There are 4 gems for you to find: Amythest, Emerald, Ruby and Topaz. These are found randomly when digging or mining rocks.
Gems are more common in the snow biome (4% chance per rock) and volcano biome (5% chance per rock), found to the north and south respectively.
You don't actually need the gems in your inventory. If you have the gems in a vault, the achievement will still unlock.