The more rare a fish is, the lower the chances are that you will catch one while fishing. Using different kinds of bait can lower or increase your chances of finding more elusive fish. Black Bugs will never catch a rare or legendary fish, for example.
Making sure that you cast your line as close as possible to the center of the target can give you a rarity bonus. Using better fishing rods also increases this bonus.
If you catch 10 fish consecutively, your luck bar will fill up and you will receive another rarity bonus to your next cast. The luck rarity bonus stacks with the rarity bonus you get from your casting accuracy.