When you arrive in Rodia, the Leather Jerks will stop you and demand bullets as payment to enter the town. You can go in guns blazing at this point to try and get rid of all of them, but there will be a lot of them to fight.
An easier way to get them out of town is to pay to enter the town and head to Dengler's house on the right. On the way, the maid will offer to lock in several of the Jerks so they can't join in when the fight starts.
Once you get into the office, open fire on Dengler and his cronies will instantly appear to fight you. If you take them all out, any remaining Leather Jerks will flee the town.
You can then free the Mayor from his cage in the center of town.