Settlers first arrived in the area that would become known as the Elm community during the 1850s. Elm was located about 12 miles northwest of Dallas and, after Irving's founding in 1903, about two miles north of Irving. Nicholas Farine, one of the areas first settlers, emigrated from France as a member of the La Reunion Colony. La Reunion was located just southeast of the Irving area. The colony failed in the late 1850s, and the Farine family moved to the Elm community in 1859. Other pioneer families in the region were the Tolers, Metkers, and Storys.
Farming was the primary occupation of Elm community residents. Local farmers raised cotton, hay, and fruits and vegetables. By the early 20th century, the O'Connor family operated a large dairy in the area. In 1899, John Finley Mills, a local farmer and merchant, opened the area's first post office, which he called Finley. The post office served a population of about 75 families in the area until it was discontinued in 1902.
Residents built a one-room schoolhouse in the community in the early 1880s. The Elm School, which offered grades 1 through 7, operated until 1948, when the Irving Independent School District took it over.
The Elm community succumbed to urban growth as the city of Irving annexed the farming area during the 1950s and 1960s. Today local street names serve as reminders of the early families of Elm.