Gjirokastra's eerie hilltop castle is one of the biggest in the Balkans and easily the town's best sight, most definitely worth the steep walk up from the Old Town. Inside there's an eerie collection of armoury, two good museums, a shot down US Air Force jet and a hilariously hard to use audiotour that is included in your entry fee.
There's been a fortress here since the 12th century, although much of what can be seen today dates to the early 19th century. The castle remains somewhat infamous due to its use as a prison under the communists. One of the more quirky sights on display is that of a recovered US Air Force jet that was shot down during the communist era. The views across the valley are simply superb.