After the Surrender Lee returned to Richmond and the home that his son had rented for himself and his friends at the war department. At this time the 7 Lee's lived there along with Robert's brother and his family and some of the general's staff stayed for a few days or so. The throng of people trying to see the general became burdensome to a man who wanted to rest after four years of fighting. A friend of the family offered a cottage called Derwent for them to retire too for several months. So the family packed up and left Richmond behind.

The General was offered many figurehead positions with incredible salaries but he refused to be a figurehead. So when the small school of Washington College offered him the position of president he thought it over and accepted the position. It was coincidences that his new school was literally right next to the school (VMI) that his eldest son was now teaching at. The family moved to Lexington, Virginia. The old president's house was small and so the general designed and had built a new presidents house which is still in use by the college president today. He also with help from his eldest son and another professor from the Virginia Military Institute designed the chapel for the school which is now known as Lee chapel.

The General stayed in Lexington until his death on October 12, 1870. He had returned from a church meeting and had a stroke. He was placed on a bed in the dinning room where he lingered for several days before passing.