The fortress belongs to Paul "De Pleur" Harmon and becomes easier to take down once he is dealt with.
Like with other fortresses, sneaking in and capturing it undetected as a solo player is easier than outward appearances would suggest, even when the fort isn't weakened. The outer defenses can be bypassed by climbing the plateau to the north. This route will put you on the roof of a building next to the outer wall. You can lure the closest enemies behind the building with rocks and then drop on their heads with your Kukri, then pick off the rest of the guards.
In addition, the boat dock to the south conceals an underwater tunnel that comes up into the fort via a well which is close to plenty of cover.