The region around the Tar River was continuously inhabited by indigenous people for 12,000 years before the first Europeans arrived, when it was home to the Tuscarora people. Europeans began settling the area after the Tuscarora War in the early 1700s. Like many other early settlements in colonial America, they settled along the fall line between the Piedmont and coastal plain, which is the point at which rivers become unnavigable sailing upstream and water flowing downstream can power a mill. The Falls of the Tar River Primitive Baptist Church was established in 1757, which still meets today, although its original building has since been replaced. Much of the community attended the church so that it served as an early form of record keeping and law enforcement with citations given for crimes.

A post office was established at the falls of the Tar River on March 22, 1816. At this point, the name "Rocky Mount" officially appears in documented history, undoubtedly referring to the rocky mound at the falls of the Tar River. The second cotton mill in North Carolina followed soon thereafter, Rocky Mount Mills, in 1818. Its proprietors were two entrepreneurs and Joel Battle, grandson of an original colonial settler to the area. Joel bought out the other proprietors before turning over the enterprise to his cousin James Smith Battle. The mill's spindles were initially operated by slaves until the 1850s and then worked exclusively by white women and girls. This female working arrangement lasted for the rest of the century.

The Battle family was also involved in the construction of the longest continuous railroad in the world up to that time, the Wilmington and Weldon Railroad, which ran about two miles (3 km) east of the mill. It connected the area to major ports in Virginia to the north and the port of Wilmington to the south. The tracks first reached Rocky Mount on Christmas Eve in 1839. In 1840, a train of cars en route to Wilmington stopped in Rocky Mount to import some "Old Nash" for special toasts at opening festivities and from there the fame of Nash County apple brandy spread. The railroad exerted a powerful influence on the development of the town so that, in 1871, the county line moved from the Tar River to its present location in the center of the tracks. The Raleigh-Tarboro stage route also passed just below Rocky Mount (roughly where I-95 and U.S. 64 run today), and for a time was the logical debarking point for railroad travelers wishing to proceed east or west.

The surrounding region was raided in 1863 during the Civil War by Union troops under the command of Brigadier General Edward E. Potter. The mill, which supplied Confederate yarn and cloth, was burned down. The mill was rebuilt after the war ended. On February 19, 1867, the village outside the mill was incorporated as a town.

The latter half of the 19th century saw the tobacco industry take shape in the state. Adjacent to the sandy coastal plain, Rocky Mount was well situated to take advantage of the rapidly rising demand for brightleaf tobacco that grew best in the sandy soil. Tobacco also shaped the city's social life. Warehouses where tobacco was stored and marketed began hosting balls for the community in the 1880s that became known as "june germans" for the time of year and style of dance. June Germans eventually transformed into all-night dance parties and attracted musicians and socialites from miles around well into the 1900s. By the end of the 19th century, tobacco had surpassed King Cotton as the town's primary agricultural product.

The turn of the 20th century saw Rocky Mount become the northern headquarters of the Atlantic Coast Line Railroad and its major repair shops and yard facilities located to the town. With it came an influx of railroad employees. In 1900, Rocky Mount's population was around 3,000. On February 28, 1907, with a population around 7,500, Rocky Mount was officially incorporated as a city. A main railroad line, a well established cotton mill, and productive farmland for brightleaf tobacco were major contributors to the area's growth and prosperity over the next decades. A vibrant central business district arose. As in the rest of the South, though, racial segregation was imposed on the community leading to white suburbs largely on the west side of town, such as Villa Place and West Haven, and black neighborhoods largely on the east side of town, like Crosstown and Around the "Y" where Jazz legend Thelonious Monk was born.

Several notable Civil Rights events occurred in Rocky Mount. In 1946, African American tobacco warehouse workers voted to organize in Rocky Mount as part of a broader nationwide movement known as Operation Dixie that lead to voter registration and political action against segregation. On November 27, 1962, Martin Luther King Jr. gave a speech at Booker T. Washington High School wherein he used his famous refrain "I have a dream" a year before his better known delivery at the March on Washington. The city also had its own sanitation workers' strike in 1978 when government sanitation workers protested their black co-worker being wrongfully arrested leading to his acquittal in court and the city later apologizing.

After WWII, the city continued to grow and the 1950s and 1960s saw the city's economy diversify into banking, manufacturing, pharmaceuticals, and the headquarters of a fast food chain known as Hardee's. New educational facilities were also built including North Carolina Wesleyan College in 1956 and Nash Community College and Edgecombe Community College in 1968. In 1970, Rocky Mount received an All-America City Award. The 1970s also saw the consolidation of the city's hospitals under Nash General Hospital and the completion of Rocky Mount–Wilson Regional Airport.

Like much of the rest of America, the 1980s onward saw urban decay of the inner city. Rocky Mount's downtown deteriorated but new neighborhoods and shopping malls were built like Golden East Crossing, and the city's boundaries expanded. In 1996, the town of Battleboro to the north of the city was annexed. In 1999, the city won its second All-America City Award.

The fall of 1999 saw two hurricanes make landfall in eastern North Carolina. Both passed over Rocky Mount: Hurricane Dennis as a tropical storm in August with 20 inches (510 mm) of rain and Hurricane Floyd in September with nearly 17 inches (430 mm) of rain. Floyd is especially memorable because most localized flooding happened quickly overnight and many residents were not aware of the flooding until the water came into their homes, requiring many to be rescued. As a result of the hurricane, the already saturated Tar River suffered the worst flooding in its recorded history, exceeding 500-year flood levels along its lower stretches, and many homes and businesses were destroyed.

The first decades of the 21st century have seen efforts to revitalize the historic downtown and projects to renovate buildings such as the train station and Douglas Block, or repurpose them like the Imperial Centre for Arts and Sciences and most recently Rocky Mount Mills. In 2007, Capitol Broadcasting Company bought the mill and is in the process of turning it into a mixed-use campus of breweries, restaurants, lofts, and event space. There have also been major new community projects such as the 143-acre (58 ha) sports complex and 165,000-square-foot (15,300 m²) downtown event center. In 2019, CSX, the successor company of the Atlantic Coast Line Railroad, broke ground on a new intermodal cargo terminal that is expected to grow the local economy in the next decade.

As of the census of 2010, 57,477 people, 23,097 households, and 14,639 families resided in the city. The population density was 1,312.6 inhabitants per square mile (606.7/km²). The city had 26,953 housing units. The racial makeup of the city was 61.3% African American, 32.4% White, 0.6% Native American, 1.0% Asian, and 1.6% from two or more races. Hispanics or Latinos of any race were 3.7% of the population.

Of the 23,097 households, 27.3% had children under the age of 18 living with them, 35.7% were married couples living together, 22.9% had a female householder with no husband present, and 36.6% were not families. About 31.4% of all households were made up of individuals living alone, and 26.6% had someone living alone who was 65 years of age or older. The average household size was 2.42 and the average family size was 3.04.

In the city, the population was distributed as 27.5% between the ages of 1 and 19, 6.4% from 20 to 24, 24% from 25 to 44, 27.9% from 45 to 64, and 14.2% who were 65 years of age or older. The median age is 38.7 years. 45.8% of the population are males compared to 54.2% for females.

The median income for a household in the city was $37,059, and for a family was $39,929. The per capita income for the city was $21,779. About 19.0% of the population is below the poverty line.

Rocky Mount's population is 40.3% religiously affiliated, below the state average of 48.9%.Christianity is the largest religion, with Baptists (13.3%) making up the largest religious group, followed by Pentecostals (4.5%) and Methodists (3.5%). Presbyterians (1.5%), Episcopalians (0.9%), and Catholics (0.8%) make up a significant amount of the Christian population as well. The remaining Christian population (15.2%) is affiliated with other churches. Islam (0.5%) is the second largest religion after Christianity.