Fairly simple, as your travelling around the world, you'll see areas with a high concentration of monster around them (such as Ghouls or Harpies) and a skull mark with four little spikes sticking out of the top of its head on the map. This is a monster nest.
Firstly, to destroy the actual nest you need a bomb (Samum, Grapeshot or Dancing star do the trick), so if you haven't found the schematic for any of these yet you won't be able to do it (I'm fairly certain the schematics for all three can be found in White Orchard).
Once you've got you bombs ready you have to kill all the monsters surrounding the nest and then press A on it when the icon comes up to drop a bomb inside of it. Stand back so your not caught in the blast radius and then the nest is gone.
Once or twice once I've blown a nest up another monster has appeared that I've had to kill but that hasn't always happened to me.
Monster nests show up as question marks (?) on the map. I have found 3 nests in White orchard and at least 7 in Velen so far, but I'm sure there are a ton that I haven't managed to find yet.
Good Luck :-)