To add to rastal's guide, each type of fish has a specific tug sequence that will help you identify what you have on the line. Use this to not waste time and bait when you're looking for a specific type of fish. I'll note them by the number and frequency of small tugs, then "Pull" for the big one where you actually have to land the fish. So 2, 1, Pull would be two quick small tugs, followed by a single small tug, followed by the fish actually taking the bait (don't use A to start the battle, just push down on the LS).
Junk (buckets, cans, and clumps in the desert) will always be 1, Pull. If you don't want to waste bait, a second or so after the big Pull you can just reel it in and get "not even a nibble". I think if you try this with any fish, though, it'll take your bait (but someone else can confirm this).
FISH:
Shaman Fish: 1, 1, Pull
Sardine: 2, Pull
Blowfish: 1, 1, Pull
Rainbow Trout: 1, 1, Pull
Bream: 2, 2, Pull
Black Bass: 1, Pull
Sandfish: Pull (it will bite as the very first action on the lure)
Royal Fish: 2, 1, Pull
Blue Marlin: Pull (same as Sandfish)
Hyneria: 1, 1, Pull
Rhizodont: 2, 2, Pull
Giant Catfish: Pull (same as Sandfish)
Shark: 1, Pull
Carp: 2, Pull
Dunkleosteus: 1, 2, Pull