Randomly during the story, whether you are in single player or co-op, our ex-best friend Hatty Hattington will appear on the screen holding a Golden Hat. The narrator will make a remark and then the Golden Hat flies off somewhere on the map. All that needs to be done is to bring the Hat to the exit. If you die while carrying the Hat, then it will fall in the area where you died. A word of warning: if you die by spikes or water (not sure about the electrical zappers), then the Hat will be lost, so you got a warning. If in co-op, alternate between who will get the Golden Hat.
Once you cross the exit, then you will be rewarded with 10 gems and the 1st of the ten Hats required for the achievement. Happy Hat Hunting!
EDIT: "You might also want to add that Golden Hats will appear in arena matches as well. You must be holding them as the curtain closes if you wish to receive the gems." - Darkened Night (thanks for the tidbit)