The present brick building was erected in 1846, along with another school building located on Fourth Street, to replace an earlier school built on Gravel Hill in1829 when enrollment outgrew the space. It was used as a public school until 1857 when it was converted to a residence. The school was originally on a steep glacial gravel deposit where Subway Restaurant is now located, but was moved in 1977 to its present location. The gravel hill was later graded down to street level.