You need to take possession of these:
Fort Dry Tortuga (Easy)
Fort Eleuthera (Easy)
Fort Gibara (Easy)
Fort Castillo de Jagua (Medium)
Fort Conttoyor (Medium)
Fort Cabo de Cruz (Medium)
Fort Navassa (Medium)
Fort Charlotte (Hard)
Fort Chinchorro (Hard)
Fort Serranilla (Hard)
You should complete Sequence 04: Memory 01 to be able to attack forts. It will give you a tutorial.
The difficulty is not the fort itself. (I've seen a fort in easy waters with seven towers and a fort in hard waters with only four.) The difficulty is the ships in that area. Don't ignore the ships, they are awesome for emergency repairs during the battle. DO NOT FORGET TO GO BOARD ANY CONQUERED SHIPS LEFT IN THE AREA BEFORE YOU INFILTRATE THE FORT OR THE FORT WILL DESTROY THEM.
You want the best armor you can have as soon as you can get it. The weapons are also important, the more damage you do to the fort, the faster you can end the battle. Few ships have mortars of their own. Mortars are an excellent opening strategy, but they are best from far away. If you are getting within 200-300 meters, your cannons (and Swivel guns for weak points revealed with Heavy Shots) do wonderful damage. You'll spend a lot of time going back and forth in front of the fort, beware of your surroundings and try not to hit land too much.
Once the towers are down it's just a mad dash to kill whatever officers are marked on your map while you try to avoid getting killed by the rest of the enemies.