Human settlement in the area can be traced back to approximately 400 CE, when Native Americans in the area forged three trails to better access the area's freshwater springs. During the 16th century, the Creek Muskogee tribe settled the area, where they remained until the early 1800s, when they were forced out of the area due to the discovery of gold in it.

In 1821, the federal government held a number of land lotteries in the area, resulting in the purchase of land in present-day Sandy Springs and subsequent settlement of the land. The Austin-Johnson House, the oldest unaltered house in the town, was built in 1842 on what is now Johnson Ferry Road. In 1851, Wilson Spruill donated 5 acres (2.0 ha) of land for the founding of the Sandy Springs United Methodist Church, near the natural spring for which the city is named.^{[citation needed]} In 1905, the Hammond School was built at Johnson Ferry Road and Mt. Vernon Highway, across the street from the church.^{[citation needed]}

In 1950, the state legislature blocked Atlanta from annexing the community, which remained rural until the Interstate Highway System was authorized by the Federal-Aid Highway Act of 1956. In 1959, after a fire at Hammond Elementary School, William Hartsfield, the mayor of Atlanta, urged residents to support annexation so that the area would have better firefighting protection. Community opposition killed the proposal.^{[citation needed]} In the early 1960s, Georgia 400 and Interstate 285 were constructed, connecting Sandy Springs to metro Atlanta and initiating a housing boom that brought new residents and major land development as part of the white flight from Atlanta after the Civil Rights Movement won greater racial integration within Atlanta. In 1965, Hartsfield once again proposed the annexation of the Sandy Springs area. Spokesmen for Sandy Springs promised residents to "build up a city separate from Atlanta and your Negroes and forbid any Negroes to buy, or own, or live within our limits" should they reject annexation. In 1966, annexation by Atlanta was defeated in a referendum, with two-thirds voting against.^{[citation needed]}

Efforts to incorporate Sandy Springs began in 1966 in response to attempts by the city of Atlanta to annex this unincorporated area of north Fulton County.^{[citation needed]} In the early 1970s, the city of Atlanta attempted to use a state law to force annexation of Sandy Springs, which failed after the Supreme Court of Georgia ruled that the law was unconstitutional. In response, a group of residents formed the Committee for Sandy Springs 1975 to lobby for the incorporation of Sandy Springs. During this time, proponents for an incorporated Sandy Springs argued that their taxes were disproportionately going to other, largely non-white, communities in Fulton County. In every legislative session, state legislators representing the area introduced a bill in the Georgia General Assembly to authorize a referendum on incorporation. Legislators representing Atlanta and southwestern Fulton County, who feared that tax revenue would be lost from incorporation, blocked the bills, using the procedural requirement that all local legislation be approved first by a delegation of representatives from the affected area.^{[citation needed]}

In 1989, a push was made for Sandy Springs to join neighboring Chattahoochee Plantation in Cobb County.^{[citation needed]} This move was blocked by Speaker Tom Murphy.^{[citation needed]}

In 1991, the Georgia state government determined that Sandy Springs, along with other wealthier, and predominantly white, communities in Fulton County was being taxed below statewide minimums, resulting in an increase in taxes for the area. Some Sandy Springs residents, including Mitch Skandalakis, launched a number of campaigns against the taxes, and launched an unsuccessful lawsuit against the state.

On January 16, 1997, Eric Rudolph bombed an abortion clinic in Sandy Springs.

When the Republican Party gained a majority in both houses of the Georgia General Assembly in 2005, the procedural rules previously used to prevent a vote by the full chamber were changed so that the bill was handled as a state bill and not as a local bill.^{[citation needed]} The assembly also repealed the requirement that new cities must be at least 3 miles (4.8 km) from existing cities that had stymied previous attempts to incorporate due to Sandy Springs directly bordering both Roswell and Atlanta.^{[citation needed]} The bill allowing for a referendum on incorporation was introduced and passed as HB 37. The referendum initiative was approved by the Assembly and signed by Governor Sonny Perdue.

Many residents expressed displeasure with county services, claiming, based upon financial information provided by the county, that the county was redistributing revenues to fund services in less financially stable areas of the county, ignoring local opposition to rezoning, and allowing excessive development. Many residents of unincorporated and less-developed south Fulton County strongly opposed incorporation, fearing the loss of tax revenues which fund county services. County residents outside Sandy Springs were not allowed to vote on the matter. Efforts such as requesting the Justice Department to reject the plan were unsuccessful.^{[citation needed]}

The referendum was held on June 21, 2005, and residents voted 94% in favor of incorporation. In November 2005, voters returned to the polls to elect a mayor and six city council members.^{[citation needed]} Eva Galambos, who had initiated and led the charge for incorporation, was elected mayor.^{[citation needed]} Formal incorporation occurred on December 1, making Sandy Springs the third-largest city ever to incorporate in the U.S. The city's police force and fire department began service in 2006.^{[citation needed]}

Upon incorporation, Sandy Springs initiated a nontraditional approach by operating as a public-private partnership (PPP), with all but six full-time employees being contracted.

In 2010, the city undertook a procurement process to rebid all general city services, which was won by CH2M Hill. The timing of this contract, during the Great Recession, allowed the city to leverage a cheaper contract due to the economic downturn.

In 2010, the city became the first jurisdiction in Georgia to successfully "bail out" from the preclearance requirements of Section 5 of the Voting Rights Act.

In 2019, the Sandy Springs City Council moved to scale back the PPP model, directly hiring 183 contract employees, leaving only 15 outsourced full-time workers by the end of 2019. The city will still outsource a number of services, including the city attorney's office, as well as security, street sweeping and ambulance services. The move is expected to save $2.7 million in the next year and more than $14 million over 5 years.

(Note: the 2000 census numbers are for Sandy Springs prior to incorporation, but cover the same area.)

In the official 2010 census, the population of Sandy Springs was 93,853. There were 42,334 households. The racial and ethnic composition of the population was 65.0% white, 20.0% black or African American, 0.3% Native American, 5.0% Asian, 6.9% from some other race and 2.7% from two or more races. 14.2% of the population was Hispanic or Latino of any race.

In the preceding official census of 2000, when there were 85,781 people, 39,288 households, and 19,683 families residing in the CDP, the population density was 2,274.1 people per square mile (878.1/km²). There were 42,794 housing units at an average density of 1,134.5 per square mile (438.0/km²). The racial makeup of the CDP was 77.55% White, 12.04% African American, 0.18% Native American, 3.29% Asian, 0.05% Pacific Islander, 4.94% from other races, and 1.95% from two or more races. Hispanic or Latino of any race were 9.93% of the population. According to a 2006 report by the Atlanta Jewish Federation, 15,300 Jews reside in Sandy Springs and the adjacent city of Dunwoody.

There were 48,288 households, out of which 21.1% had children under the age of 18 living with them, 40.1% were married couples living together, 7.0% had a female householder with no husband present, and 49.9% were non-families. 35.9% of all households were made up of individuals, and 6.8% had someone living alone who was 65 years of age or older. The average household size was 2.17 and the average family size was 2.87.

In the CDP, the age distribution of the population shows 17.8% under the age of 18, 10.5% from 18 to 24, 40.3% from 25 to 44, 21.6% from 45 to 64, and 9.8% who were 65 years of age or older. The median age was 33 years. For every 100 females, there were 96.6 males. For every 100 females age 18 and over, there were 95.0 males.

According to a 2008 estimate, the median income for a household in the city was $106,240, and the median income for a family was $129,810. The average income for a household was $116,406 and the average income for a family was $169,815. Males had a median income of $60,053 versus $50,030 for females. About 3.1% of families and 7.9% of the population were below the poverty line, including 8.9% of those under age 18 and 1.9% of those age 65 or over.