These are upgrades to your basic weapons, increasing ammo capacity, reloading speed, rate of fire, damage etc. You do not have to buy them in order, you can go from level 1 to level 3 or 4, buying the previous level will not discount the next level.
Location:
On your map you will see a gun with two, three, or four stars above it in various places. (Four stars is for the DLC)
Revolver lvl 2 - Southern end of Broadway Street in Little Italy, in the circle $7,500
Revolver lvl 3 - In the park between Worth and Court in Little Italy $75,000
Pistol lvl 2 - Under Manhattan Bridge in Brooklyn $12,5000
Pistol lvl 3 - In southern Brooklyn between Willow and Furman $250,000
Magnum lvl 2 - On the west side of Galvin Street in Hell's Kitchen $25,000
Magnum lvl 3 - North of 52nd Street in the Northern side of Hell's Kitchen $250,000
Shotgun lvl 2 - North of the park near 4th Street in New Jersey - $45,000
Shotgun lvl 3 - North of the Garden Parkway Viaduct in the railroad yards in New Jersey - $450,000
Tommy gun lvl 2 - West side of E 79th Street in Midtown $50,000
Tommy gun lvl 3 - At the intersection of 25th and 3rd in Midtown $500,000