William Bean, traditionally recognized as Tennessee's first settler, built his cabin along Boone's Creek near Johnson City in 1769.

In the 1780s, Colonel John Tipton (1730–1813) established a farm (now the Tipton-Haynes State Historic Site) just outside what is now Johnson City. During the State of Franklin movement, Tipton was a leader of the loyalist faction, residents of the region who wanted to remain part of North Carolina rather than form a separate state. In February 1788, an armed engagement took place at Tipton's farm between Tipton and his men and the forces led by John Sevier, the leader of the Franklin faction.

Founded in 1856 by Henry Johnson as a railroad station called "Johnson's Depot", Johnson City became a major rail hub for the Southeast, as three railway lines crossed in the downtown area. In the late 19th and early 20th centuries, Johnson City served as headquarters for the narrow gauge East Tennessee and Western North Carolina Railroad (the ET&WNC, nicknamed "Tweetsie") and the standard gauge Clinchfield Railroad. Both rail systems featured excursion trips through scenic portions of the Blue Ridge Mountains and were engineering marvels of railway construction. The Southern Railway (now Norfolk Southern) also passes through the city.

During the American Civil War, before it was formally incorporated in 1869, the name of the town was briefly changed to "Haynesville" in honor of Confederate Senator Landon Carter Haynes. Henry Johnson's name was quickly restored following the war, with Johnson elected as the city's first mayor on January 3, 1870. The town grew rapidly from 1870 until 1890 as railroad and mining interests flourished. However, the national depression of 1893, which caused many railway failures (including the Charleston, Cincinnati and Chicago Railroad or "3-Cs", a predecessor of the Clinchfield) and a resulting financial panic, halted Johnson City's boom town momentum.

In 1901, the Mountain Branch of the National Home for Disabled Volunteer Soldiers (now the U.S. Veterans Affairs Medical Center and National Cemetery), Mountain Home, Tennessee was created by an act of Congress introduced by Walter P. Brownlow. Construction on this 450-acre (1.8 km²) campus, which was designed to serve disabled Civil War veterans, was completed in 1903 at a cost of $3 million. Before the completion of this facility, the assessed value of the entire town was listed at $750,000. The East Tennessee State Normal School was authorized in 1911 and the new college campus directly across from the National Soldiers Home.^{[citation needed]} Johnson City began growing rapidly and became the fifth-largest city in Tennessee by 1930.

Together with neighboring Bristol, Johnson City was a hotbed for old-time music. It hosted noteworthy Columbia Records recording sessions in 1928 known as the Johnson City Sessions. Native son "Fiddlin' Charlie" Bowman became a national recording star via these sessions. The Fountain Square area in downtown featured a host of local and traveling street entertainers including Blind Lemon Jefferson.

During the 1920s and the Prohibition era, Johnson City's ties to the bootlegging activity of the Appalachian Mountains earned the city the nickname of "Little Chicago".Stories persist that the town was one of several distribution centers for Chicago gang boss Al Capone during Prohibition. Capone had a well-organized distribution network within the southern United States for alcohol smuggling; it shipped his products from the mountain distillers to northern cities. Capone was, according to local lore, a part-time resident of Montrose Court, a luxury apartment complex now listed on the National Register of Historic Places.^{[citation needed]}

For many years, the city had a municipal "privilege tax" on carnival shows, in an attempt to dissuade traveling circuses and other transient entertainment businesses from doing business in town. The use of drums by merchants to draw attention to their goods is prohibited. Title Six, Section 106 of the city's municipal code, the so-called "Barney Fife" ordinance, empowers the city's police force to draft into involuntary service as many of the town's citizens as necessary to aid police in making arrests and in preventing or quelling any riot, unlawful assembly or breach of peace.

As of the census of 2000, there were 55,469 people, 23,720 households, and 14,018 families residing in the city. The population density was 1,412.4 per square mile. There were 25,730 housing units at an average density of 655.1 per square mile (253.0/km²). The racial makeup of the city was 90.09 percent white, 6.40 percent African American, 0.26% Native American, 1.22 percent Asian, 0.02 percent Pacific Islander, 0.69 percent from other races, and 1.32 percent from two or more races. Hispanic or Latino of any race were 1.89 percent of the population.

There were 23,720 households, out of which 25.0 percent had children under the age of 18 living with them, 44.1 percent were married couples living together, 11.6 percent had a female householder with no husband present, and 40.9 percent were non-families. 33.9 percent of all households were made up of individuals, and 11.5 percent had someone living alone who was 65 years of age or older. The average household size was 2.20, and the average family size was 2.82.

In the city, the population was spread out, with 19.8 percent under the age of 18, 13.7 percent from 18 to 24, 28.1 percent from 25 to 44, 22.5 percent from 45 to 64, and 15.9 percent who were 65 years of age or older. The median age was 37 years. For every 100 females, there were 91.1 males. For every 100 females age 18 and over, there were 88.0 males.

The median income for a household in the city was $30,835, and the median income for a family was $40,977. Males had a median income of $31,326 versus $22,150 for females. The per capita income for the city was $20,364. About 11.4 percent of families and 15.9 percent of the population were below the poverty line, including 18.9 percent of those under age 18 and 12.7 percent of those age 65 or over.