Sheen Elms was of 17th century origins and was a long rambling east-west building facing the Upper Richmond Road. Its entrance was approximately where Milton Road and Sheen Lane meet, and the grounds extended from Vernon Road along that now covered by Milton Road. Almost the whole history of this house is connected with two Mortlake families, the Taylors of Westhall and the Bankes who were the chief occupants. George Bankes, MP for Dorsetshire, was at Sheen Elms from about 1832 until 1841 when he unexpectedly had the lovely house of Kingston Lacey transferred to him by his elder brother, who went off to live in Italy. Sheen Elms was let, the family going off to Dorset until George died. His widow remarried a few years later, returned to London, and was buried in Mortlake churchyard in 1875. Her second son was left Sheen Elms in her will, her eldest having inherited Kingston Lacey, now National Trust property.
William Bankes, son of George the MP, was born here in 1836 and went to Temple Grove School. In April 1857 he was commissioned as Cornet in the 7th Queen's Own Hussars and by November of that year had arrived in India. In the following March he was involved in fighting in Lucknow where he received terrible wounds and was attacked by dysentery. He died on April 6th having been awarded the Victoria Cross.
No picture of Sheen Elms has yet been found.