Swellendam began as a colonial settlement in 1743, when the Dutch East India Company, who wanted to consolidate their power in the area, declared the town a magisterial district. The control of the Dutch East India Company was further established with the construction of the Drostdy building which housed the residence and administrative quarters of the Landdrost or magistrate. The town is named after the first South African born Governor of the Cape, Hendrik Swellengrebel, and his wife, Helena ten Damme.

The Drostdy Museum complex gives visitors to Swellendam a comprehensive insight into the early days of Swellendam and consists of several historic buildings. These include the Drostdy building, the Old Gaol and Ambagswerf, as well as two homes, Mayville and Zanddrift. The museum offices at 18 Swellengrebel Street are also housed in a historic Georgian building, which was originally a thatched roof cottage like several other buildings in the area but modified in 1865.

The Drostdy building is a modified version of the original Drostdy building, which was built by the Dutch East India Company in 1847, to function as the residence and offices of the Landdrost or magistrate of Swellendam. The building was modified in 1813, and under British control of Swellendam the Drostdy was occupied by the Resident Magistrate until the building was sold in 1846. The Drostdy also included outbuildings that housed slaves, domestic animals and a wagon house.

The Old Gaol was built shortly after the Drostdy building, to provide accommodation for the Deputy Sheriff and Secretary. Today the Old Gaol houses a museum and a coffee shop, and there are reportedly plans to open a craft beer yard in the courtyard area of the building. The Ambagswerf, or “trades yard” shows a variety of traditional professions that were important in the development of Swellendam. These include rooms dedicated to explaining the importance of blacksmiths, coopers, tanners and wainwrights.

Mayville and Zanddrift are two houses from the early days of Swellendam that show how people lived during the early days of settlement in the area. Zanddrift in particular is an example of a traditional farmhouse and now houses the Drostdy restaurant which is famous for its collection of Coca-Cola memorabilia. Mayville, a home built between 1853 and 1855, is located adjacent to a rose garden and gives an insight into middle class Victorian living.