The original grant for Poplar Grove was from George III to Samuel Williams, and his son Thomas, who built the oldest part of the present house in 1782.
About 1792 it was bought by John Patterson, and at his death passed to his daughter, Mrs. Christopher Tompkins, the mother of “Captain” Sally Tompkins, whose home it also was. Part of a lovely old serpentine wall around the garden is still standing, and many of the old shrubs and roses; but even more interesting to the visitor is the old tide mill, which is one of the few now in existence. For many years Poplar Grove was the home of Judge Taylor Garnet. It is now owned by Mr. and Mrs. George Upton, of Cambridge, Massachusetts.