The Regency style main barracks building was constructed of Hawkesbury sandstone by convicts between 1841 and 1846. The first building completed was the Officers' Quarters which was completed in 1842.
The Main Barrack Block was completed in 1846 and was designed to accommodate 650 soldiers. The bell and clock were added to the building in 1856. The barracks were originally occupied by regiments of the British Army. The British troops vacated the barracks in 1870. The barracks was the premier military training site in Australia for many years, from its completion until after Federation in 1901.
The gate on Oxford Street is referred to as the Queen Victoria Gate, while the gate on Moore Park road is known as the Convict Gate.
Several important factors led to the decision to locate the Barracks where they are. Among them was the fact that Busby's Bore (the water tunnel built by convicts between 1827 and 1837) passed directly beneath the site, at a depth of around 15m. Two access shafts are located within the Barracks grounds, providing -at the time- a reliable supply of good quality drinking water.