The Lieutenance, at the mouth of the old harbour, was once the residence of the town’s royal governor.