The Sandy Ground community of Rossville was settled in the 1800s by free African Americans from Maryland. By the early 1850s they organized a congregation and built a church on Crabtree Ave. (1854). The cemetery was established to the west of the church. By 1897 the congregation had outgrown its original house of worship and a new building was constructed on Bloomingdale Rd.

The earliest owner of the Coleman house, as recorded on historic maps, was Ephraim Bishop, an oysterman from Maryland. It is not known if he built the house. Mrs. Bishop sold the house in 1864 to the Reverend Isaac Coleman, pastor of the A.M.E. Zion Church, and his wife Rebecca Gray Coleman. The house remains in the possession of descendants of Mrs. Coleman today.