In Colonial times, Yorkville was an undeveloped area of forests and streams. In August 1776, George Washington stationed half of his Continental Army in Manhattan and the other half in Brooklyn. Many troops in the Yorkville area on Manhattan's upper east side were in defensive positions along the East River to protect a possible retreat off Long Island, and to inflict damage on invading land and sea British forces.^[4] Following their August 27 defeat in the Battle of Long Island, the Continentals implemented an orderly pivoting retreat in the Yorkville area, leading the enemy to entice the Continentals to fight by piping "Fly Away", about a fox running away from hounds. The Continentals' disciplined northerly retreat led to the successful Battle of Harlem Heights in September 1776.^[5]

In 1815, the Upper East Side was a farmland and market garden district.^[6] The Boston Post Road traversed the Upper East Side, locally called the Eastern Post Road; milepost 6 was near the northeast corner of Third Avenue and 81st Street. From 1833 to 1837 the New York and Harlem Railroad, one of the earliest railway systems in the United States, was extended through the Upper East Side along Fourth Avenue (later renamed Park Avenue).^[7] A hamlet grew near the 86th Street station, becoming the Yorkville neighborhood as gradual yet steady commercial development occurred. The current street grid was laid-out between 1839 and 1844 as part of the Commissioners' Plan of 1811, so the Eastern Post Road was abandoned. The community had been referred to as Yorkville before 1867.^[8]

By 1850, a significant proportion of the inhabitants of the area were the Germans and Irish that helped build the Croton Aqueduct.^[9] The area was included in the 19th administrative district, whose boundaries were 40th and 86th Street.^[10] In 1858, trams were built along Second and Third Avenues. After the American Civil War, mansions replaced slums in Yorkville.^[10] On December 30, 1878, the IRT Third Avenue Line opened, followed by the IRT Second Avenue Line in August 1879