The Normans originally built over the ruins of a Roman fort, then Frederick II built over the Norman castle, incorporating it into his design – the two towers of the Norman structure still stand. The bastions, with corner towers overhanging the moat, were added in the 16th century during Spanish rule, when the castle was a magnificent residence.