St Cybi, the son of a 6th-century Cornish king, became a priest and eventually washed up in North Wales, where the King of Gwynedd gave him an old Roman naval fort in which to base a religious community. The fort came to be known as Cybi's Fort (Caergybi: the Welsh name for Holyhead) and the island on which it stood became Cybi's Island (Ynys Gybi: the Welsh name for Holy Island).
You can still see the remains of the 4th-century Roman walls surrounding the present-day church yard. The Gothic church came much later, with the oldest parts built in the 13th century. Interesting medieval carvings peer out from the walls, while inside the light is softened by beautiful stained-glass windows from William Morris' workshop.