There are 30 caves in total. Two are "hidden" as I'll show in a minute.
It's been mentioned that this achievement may get bugged on you. If you search every cave and can't get it to unlock, you'll need to have a friend host a game, join it, then search each cave again to get it to unlock. Credit to NailedHand for the bug info.
Here is my complete map.
When you get close to a cave, your minimap will show small cave icons. These are the possible entrances/exits to the cave. The large icon appears once you enter the cave and it's counted for the achievement. (See the right side of this screen)
Make sure you go far enough in the cave for the counter to appear, like so:
There are two caves that are basically just indents in the mountains. They don't show any mini icons for entrances so you have to look closely.
Here is the first:
The second "hidden" one: