The area which became East Harlem was rural for most of the 19th century, but residential settlements northeast of Third Avenue and East 110th Street had developed by the 1860s. The construction of the elevated transit line to Harlem in 1879 and 1880, and the building of the Lexington Avenue subway in 1919, urbanized the area, precipitating the construction of apartment buildings and brownstones. The extension of cable cars up Lexington Avenue into East Harlem was stymied by the incline created by Duffy's Hill at 103rd Street, one of the steepest grades in Manhattan. East Harlem was first populated by poor German, Irish, Scandinavian, and Eastern European Jewish immigrants, with the Jewish population standing at 90,000 around 1917. In the 1870s, Italian immigrants joined the mix after a contractor building trolley tracks on First Avenue imported Italian laborers as strikebreakers. The workers' shantytown along the East River at 106th Street was the beginning of an Italian neighborhood, with 4,000 having arrived by the mid-1880s. As more immigrants arrived, it expanded north to East 115th Street and west to Third Avenue.

East Harlem consisted of pockets of ethnically-sorted settlements – Italian, German, Irish, and Jewish – which were beginning to press up against each other, with the spaces still between them occupied by "gasworks, stockyards and tar and garbage dumps". In 1895, Union Settlement Association, one of the oldest settlement houses in New York City, began providing services in the area, offering the immigrant and low-income residents a range of community-based programs, including boys and girls clubs, a sewing school and adult education classes.

Southern Italians and Sicilians, with a moderate number of Northern Italians, soon predominated, especially in the area east of Lexington Avenue between 96th and 116th Streets and east of Madison Avenue between 116th and 125th Streets, with each street featuring people from different regions of Italy. The neighborhood became known as "Italian Harlem", the Italian American hub of Manhattan; it was the first part of Manhattan to be referred to as "Little Italy". The first Italians arrived in East Harlem in 1878, from Polla in the province of Salerno, and settled in the vicinity of 115th Street.

There were many crime syndicates in Italian Harlem from the early Black Hand to the bigger and more organized Italian gangs that formed the Italian-American Mafia. It was the founding location of the Genovese crime family, one of the Five Families that dominated organized crime in New York City. This includes the current 116th Street Crew of the Genovese family. During the 1970s, Italian East Harlem was also home to the Italian-American drug gang and murder-for-hire crew known as the East Harlem Purple Gang.

In the 1920s and early 1930s, Italian Harlem was represented in Congress by future Mayor Fiorello H. La Guardia. After becoming mayor, La Guardia helped plan a large expansion of Thomas Jefferson Park at First Avenue, between 111th and 114th Streets, in the mid-1930s. The neighborhood was represented, in the 1940s, by Italian-American civil rights lawyer, activist, and socialist Vito Marcantonio. The Italian neighborhood approached its peak in the 1930s, with over 110,000 Italian-Americans living in its crowded, run-down apartment buildings. The 1930 census showed that 81 percent of the population of Italian Harlem consisted of first- or second-generation Italian Americans, somewhat less than the concentration of Italian Americans in the Lower East Side's Little Italy with 88 percent; Italian Harlem's total population, however, was three times that of Little Italy.

The Italian community in East Harlem remained strong into the 1980s, but it has slowly diminished since then. However, Italian inhabitants and vestiges of the old Italian neighborhood remain. The annual Feast of Our Lady of Mount Carmel and the "Dancing of the Giglio", the first Italian feast in New York City, is still celebrated there every year on the second weekend of August by the Giglio Society of East Harlem. Italian retail establishments still exist, such as Rao's restaurant, started in 1896, and the original Patsy's Pizzeria which opened in 1933. In May 2011, one of the last remaining Italian retail businesses in the neighborhood, a barbershop owned by Claudio Caponigro on 116th Street, was threatened with closure by a rent increase.

Puerto Rican and Latin American migration after the First World War established an enclave at the western portion of East Harlem – around 110th Street and Lexington Avenue – which became known as "Spanish Harlem". The area slowly grew to encompass all of East Harlem, including Italian Harlem, as Italians moved out – to the Bronx, Brooklyn, upstate New York and New Jersey – and Latinos moved in during another wave of immigration in the 1940s and 1950s. Although in certain areas, particularly around Pleasant Avenue, Italian Harlem lasted through the 1970s, today most of the former Italian population is gone. Most of these predominantly older residents are clustered around Our Lady of Mount Carmel Church, mainly from 114th to 118th Streets. According to the 2000 Census, there were only 1,130 Italian-Americans still living in this area.

The newly dominant Puerto Rican population, which reached 63,000 in 1950, continued to define the neighborhood according to its needs, establishing bodegas and botánicas as it expanded; by the 1930s there was already an enclosed street market underneath the Park Avenue railroad viaduct between 111th and 116th Streets, called "La Marqueta" ("The Market").Catholic and evangelistic Protestant churches appeared in storefronts. Although "Spanish Harlem" had been in use since at least the 1930s to describe the Latino enclave – along with "Italian Harlem" and "Negro Harlem" – the name began to be used to describe the entire East Harlem neighborhood by the 1950s. Later, the name "El Barrio" ("The Neighborhood") began to be used, especially by residents of the area.

In the 1950s and 1960s, large sections of East Harlem were leveled for urban renewal projects, and the neighborhood was one of the hardest hit areas in the 1960s and 1970s as New York City struggled with deficits, race riots, urban flight, gang warfare, drug abuse, crime and poverty. Tenements were crowded, poorly maintained, and frequent targets for arson. In 1969 and 1970, a regional chapter of the Young Lords which were reorganized from a neighborhood street gang in Chicago by Jose (Cha-Cha) Jimenez, ran several programs including a Free Breakfast for Children and a Free Health Clinic to help Latino and poor families. The Young Lords came together with the Black Panthers and called for Puerto Rican independence and neighborhood empowerment. Still, as of the early 2000s, the Latin Kings gang remained prevalent in East Harlem.

By the beginning of the 21st century, East Harlem was a racially diverse neighborhood, with about a third of the population being Puerto Rican. As it has been throughout its history, it is predominantly a working-class neighborhood.

Until 2006, property values in East Harlem climbed along with those in the rest of New York City. With increased market-rate housing, including luxury condos and co-ops – most built on formerly vacant lots – there has been some decline of affordable housing in the community. A number of young professionals have settled into these recently constructed buildings. This influx of "yuppies" has caused rents to rise, more buildings in the area to get gut renovations, and changes to area demographics.

On March 12, 2014 at 9:00 EDT, a large explosion and fire at 1644–1646 Park Avenue killed eight people and injured more than 70.

The New York Post listed one part of the neighborhood – the block of Lexington Avenue between East 123rd and 124th Streets – as one of "the most dangerous blocks in the city" because police crime statistics for 2015 showed that 19 assaults had taken place there, more than for any other city block. The Post also reported that there were, according to the Harlem Neighborhood Block Association, "22 drug-treatment programs, four homeless-services providers and four transitional-living facilities" in East Harlem.

East Harlem has begun to feel the effects of gentrification. In February 2016, an article in The New York Times about "New York's Next Hot Neighborhoods" featured East Harlem as one of four such areas. A real-estate broker described it as "one of the few remaining areas in New York City where you can secure a good deal". The article mentioned new luxury developments, access to transportation, the opening of new retail stores, bars and restaurants, and national-brand stores beginning to appear on the outskirts of the neighborhood. Primarily, though, it was the cost of housing in comparison to the rest of Manhattan, which the article noted as the major factor. Beginning in 2016, the New York City government was seeking to rezone East Harlem "to facilitate new residential, commercial, community facility, and manufacturing development". The residents of the neighborhood generated a suggested zoning plan, the "East Harlem Neighborhood Plan", which was offered to the city in February 2017, but in August 2017 residents and the Manhattan Borough President, Gale Brewer, complained that the city had ignored their plan almost entirely.

In 2019, the oldest portion of the neighborhood, the blocks of East 111th through 120th Streets between Park and Pleasant Avenues, was listed on the National Register of Historic Places as the East Harlem Historic District.

The New York City Department of City Planning classifies East Harlem into two neighborhood tabulation areas: East Harlem North and East Harlem South, divided along 115th Street. The two areas had a combined population of 115,921, an increase of 1,874 (1.4%) from the combined 114,047 in the 2000 Census.

Based on data from the 2010 United States Census, the population of East Harlem North was 58,019, an increase of 871 (1.5%) from the 57,148 counted in 2000. Covering an area of 573.94 acres (232.27 ha), the neighborhood had a population density of 101.1 inhabitants per acre (64,700/sq mi; 25,000/km²). The racial makeup of the neighborhood was 6.8% (3,936) White, 35.5% (20,625) African American, 0.2% (128) Native American, 3.0% (1,766) Asian, 0.0% (9) Pacific Islander, 0.3% (185) from other races, and 1.3% (769) from two or more races. Hispanic or Latino of any race were 52.7% (30,601) of the population.

Based on data from the 2010 Census, the population of East Harlem South was 57,902, an increase of 1,003 (1.8%) from the 56,899 counted in 2000. Covering an area of 389.41 acres (157.59 ha), the neighborhood had a population density of 148.7 inhabitants per acre (95,200/sq mi; 36,700/km²). The racial makeup of the neighborhood was 17.4% (10,072) White, 24.6% (14,227) African American, 0.2% (96) Native American, 8.3% (4,802) Asian, 0.1% (55) Pacific Islander, 0.4% (218) from other races, and 1.6% (933) from two or more races. Hispanic or Latino of any race were 47.5% (27,499) of the population.

The most significant changes in the racial composition of East Harlem between 2000 and 2010 were the Asian population's increase by 109% (3,427), the White population's increase by 68% (5,689), and the Black population's decrease by 12% (4,625). Although more of the influx of Asian and White residents was in East Harlem South, the greatest percentage growth was in East Harlem North, while the Black population's decrease was evenly split. The Hispanic / Latino population also decreased by 4% (2,485), a decrease almost entirely concentrated in East Harlem South, where it fell from being the majority group to the plurality group. The small population of other races experienced a slight increase of 5% (132).

The entirety of Manhattan Community District 11, which consists of East Harlem, Randall's Island, and Ward's Island, had 124,323 inhabitants as of NYC Health's 2018 Community Health Profile, with an average life expectancy of 77.3 years.^{:2, 20} This is 3.9 years lower than the median life expectancy of 81.2 for all New York City neighborhoods.^{:53 (PDF p. 84)} Most residents are children and middle-aged adults: 21% are between the ages of 0–17, while 33% are between 25 and 44, and 23% are between 45 and 64. The ratio of college-aged and elderly residents was lower, at 10% and 13% respectively.^:2

As of 2017, the median household income in Community District 11 was $36,770. In 2018, an estimated 23% of Community District 11 residents lived in poverty, compared to 14% in all of Manhattan and 20% in all of New York City. One in nine residents (11%) were unemployed, compared to 7% in Manhattan and 9% in New York City. Rent burden, or the percentage of residents who have difficulty paying their rent, is 48% in Community District 11, compared to the boroughwide and citywide rates of 45% and 51% respectively. Based on this calculation, as of 2018, Community District 11 is considered to be gentrifying: according to the Community Health Profile, the district was low-income in 1990 and has seen above-median rent growth up to 2010.^:7

Community District 11 is a mostly low to middle income area. It is made up of first and second generation Puerto Ricans, African-Americans, Italian-Americans, Asian-Americans, West Indian Americans (especially Dominican Americans and Cuban Americans), and a growing population of Mexican Americans and Salvadoran Americans and other Central American immigrants. It has one of the highest concentrations of Puerto Ricans in all of New York City. As of 2010, the Puerto Rican population was 27.7% in zip code 10029, and 23.4% in 10035. 10035 also has a large Mexican population, at 10.7%.

As noted before, the number of Asians in East Harlem more than doubled between 2000 and 2010, largely due to Chinese people moving to East Harlem. Increasing rents in Lower Manhattan's Chinatown have driven many into public and subsidized housing developments in the neighborhood. Advocates have been calling for Chinese language services to be available in the community centers to accommodate the growing number of Chinese residents in the area. In 2000, the Chinese population in the northern portion was less than one percent, but by 2010, it has gone up to being three percent in the area. In the southern part, it rose from 4.6% to 8.4%.

Social problems, including poverty, crime, and drug addiction, have long plagued the area. Although crime rates have dropped from the historically high numbers of the past, East Harlem suffers from a relatively high violent crime rate, especially in the 25th Precinct above 115th Street. In 2019, the 25th Precinct had the third-highest rate of felony assault, the sixth-highest rates of rape and robbery, the tenth-highest rate of burglary, and the fourteenth-highest rate of murder out of the New York Police Department's 77 precincts.

East Harlem has the highest concentration of shelters and facilities in Manhattan, with eight homeless shelters, 36 drug and alcohol treatment facilities and 37 mental health treatment facilities. It also has the highest jobless rate in the entire city, as well as the city's second highest cumulative AIDS rate. The asthma rate is also 5 times larger than national levels. The neighborhood also suffers from a high poverty rate.Union Settlement Association is one of the neighborhood's largest social service agencies, reaching more than 13,000 people annually at 17 locations throughout East Harlem, through a range of programs, including early childhood education, youth development, senior services, job training, the arts, adult education, nutrition, counseling, a farmers' market, community development, and neighborhood cultural events.