To capture a monster, you must first seriously weaken it. A monster will be ready to capture if they retreat to their home to sleep, have a pronounced limp, or you see a skull icon appearing next to their icon on the mini-map.
When they are in this state, get them in a trap (shock trap has worked well for me) and use a couple tranquilizer bombs on them. This will complete the capture.
Capturing monsters provides more rewards than killing, and cuts out the time of fighting a monster in their near-death state, where many can be even more powerful with attacks, so it's often the better way to go about finishing a hunt.