His father made many bad investments causing the family to move from their ancestral home when Robert was only three years of age. His father spent most of Robert's life away from his family before dying on Cumberland Island along the Georgia coast. Robert helped his mother as much as he could during this time. He "was as a son and daughter" to his mother, who had to raise five children on little to no money. His mother saw to it that he was well educated and helped him to get the necessary endorsements from family friends to get his appointment to the United States Military Academy in West Point, New York. He began his studies there in June of 1825.

While at West Point Robert worked extremely hard on his studies, which is shown by his record - in his four years as a cadet he never received a single demerit and finished second in his class academically. While at West Point he tutored other students and was the Corps Adjutant, a distinction usually bestowed upon the student who ranked first in the class academically. Robert graduated 2nd in 1829 and due to his high class standing was given Brevet 2nd Lieutenant of Engineers.

Shortly after his graduation he returned to Virginia to find his mother very ill, much worse then when he last saw her, nursing her until her death on July 10, 1829. She was buried at Ravensworth Plantation and was later moved in 1913 (along with her husband Henry Lee) to Lexington, Virginia after the Lee family crypt was established at Washington & Lee University. In August 1829, Robert Lee received orders to report to Cockspur Island in Georgia to help with the building of fort Pulaski there. In April of 1831 he was transferred to Old Point for another engineering project. As this time in history, the US Corps of Engineers was overwhelmed with projects in order to quickly build up the fortifications of the fledgling nation. On June 30, 1831 Robert married Miss Mary Anna Randolf Custis, daughter of George Washington Parke Custis who was the grandson of Martha Dandridge Custis Washington. At the death of G. W. P. Custis' parents he had been adopted by George Washington. He built a grand home on a hill over looking the Potomac river, which is now known as Arlington House in the middle of Arlington National Cemetery. Miss Custis' mother was kin of the Lee family, Mrs. Mary Lee Fitzhugh Custis. After the traditional wedding and wedding visits the young couple settled into married life at Fort Monroe while the groom worked on Fort Calhoun on the water.