The proper term is Men of Twilight (as opposed to the Gondorians, High Men, and the Haradrim and Easterlings, Men of Darkness). The name reflects the Gondorian attitude towards them: not enemies, but also not as noble to be counted among themselves.
Most Middle Men had kinship with the Houses of Bëor or Marach. There were also Men in Middle-earth related to the Second House (the Haladin), and the men of Enedwaith and Minhiriath, later to be known as the Dunlendings. But because they spoke a language that was unknown to the Númenóreans they were not at first recognized as Middle Men.
There they were divided in two folks, who would later become House of Bëor and House of Marach, to the north-east of the Sea of Rhûn where they dwelt in the woods. The ancestors of the Hadorians dwelt in the north-east of the Sea, in the woods that there came near to the shores. The ancestors of the Beorians had reached the feet of the high hills to the south-west. Because of that distance, they both developed a craft of boat-building but met seldom.
The ancestors of the Beorians fled first and the ancestors of the Hadorians learned that only afterwards and followed on their trail through the Misty Mountains in the North (close to the dreadful Iron Mountains). Many sub-tribes of both peoples stayed behind, and when the shrinking teams (now led by Bëor and Marach) reached Beleriand, those who stayed behind occupied many parts of Eriador and northern Rhovanion. The latter appear to be akin to the House of Hador.
The Men who did not heed the summons of the Valar and Morgoth fell back in a primitive state, having no contact with the Eldar or the Edain. Instead, some of the evil Men who escaped the destruction of Beleriand came to the east and became their Kings. For this the Men were neglected by the Valar.
In the early Second Age the Middle Men were far from the coasts of the Westlands; some settled, some still wandering, or else had never passed the Misty Mountains and were scattered in eastern Rhovanion. Those learned agriculture thanks to the Entwives, and honoured them.
The Men of Eriador were mostly related to the Folk of Bëor and to a lesser degree to the Folk of Hador; some of them might be descendants of the Edain who fled the War of Wrath. They occupied mostly the shores of Lake Evendim (some Elves led by Galadriel and Celeborn for a while dwelt there side by side with the Men), the Weather Hills and the North Downs. They travelled sometimes west of the Baranduin but did not dwell there. They held the nearby Elves of Lindon in awe but were friendly to them. But they were afraid to look upon the Sea, no doubt having heard rumours of the deluge of Beleriand.
In the year S.A. 600 Vëantur first returned to Middle-earth reaching Lindon from Númenor. The news spread swiftly and filled with wonder that some of their distant kin had come, they petitioned Gil-galad to meet the mariners. Twelve messengers came and rejoiced to meet the Númenóreans on the Tower Hills. They found a people who spoke languages which were distantly related to the Númenórean tongue, Adûnaic. Númenórean scholars declared that this was because these Men were descendants of the fathers of the Edain, the Atanatári, who had not crossed the Ered Luin and entered Beleriand during the First Age.
The Númenóreans set up friendly relations with them, and declared them to be Middle Men, as opposed to the High Men (the Edain) or the Men of the Shadow (those hostile to Númenor or in the service of Sauron); this classification was modeled after that for the Elves by the Edain, being the High Elves, the Middle Elves, and the Dark Elves. In the "Middle Men" they recognised the similar stature and looks, kinship of language and friendliness to the West.
The Númenóreans began to cultivate their new friends who were into their Dark Years, grown weak and fearful, and taught them agriculture, stonecraft, smithying and their language Adûnaic. The Middle Men were comforted, populated the western shores. They revered the memory of the tall Sea-kings whom they remembered as gods hoping each time for their return
Aldarion's successors continued his works and wrecked the banks, the shorelines, great tracks and roads whom they drove into the forests northwards and southwards from the Gwathló pushing into Minhiriath and Enedwaith, establishing themselves inland as far as the river Glanduin, the southern boundary of Eregion. The Gwathuirim of Enedwaith were akin to the Folk of Haleth but were hostile to the Númenóreans, owing to the deforesting their habitat; as such they weren't considered "Middle Men" by the colonists.
Sauron recruited pre-Númenóreans and in the early second millennium he increased pressure on the West, left his stronghold in Rhûn and relocated in Mordor, drawing closer to the Númenórean sphere of influence. His regular troops attempted to conquer Eriador, hunting and killing Middle Men and the Elves and by S.A. 1700 had mastered all Eriador, up to the River Lhûn and besieged Rivendell. Eriador was already ruined when the Númenórean fleet sent by king Tar-Minastir caught Mordor's troops in the rear and utterly defeated them, but brought peace in the Westlands.
As the shadow spread over Númenor, Tar-Ciryatan sailed to Middle-earth, bringing numerous treasures back to Númenor. Later more lands were occupied, plundered or forced to pay heavy tribute in return for the lives of their inhabitants. Because of these acts they were looked upon in fear, called the "Death" itself and the Men of Middle-earth trembled at the sight of a mighty Númenórean ship on the waters of Belegaer.
In the time of Ar-Adûnakhôr, the settlers of the shores from the Gulf of Lune to Pelargir refused to join the rebellion against the Valar and were joined by exiled Faithfuls who fled the persecution. The term "Middle Men" arose among them.
The Númenóreans eventually ventured north of Pelargir and also met Men who had settled in valleys on either side of the White Mountains. So while the term "Middle Men" applied so far to the Men of Eriador they classified those people also as Middle Men, due to their friendliness to the West, and recognized as descendants of the Men who abjured the Shadow.
The Northmen of Rhovanion were counted as Middle Men, as were most people living in Eriador. During the Third Age the term Middle Men was still applied to the Men of Bree, and the remaining Northmen, such as the Men of Dale and Esgaroth the Lake-town, and of course the Rohirrim. King Valacar of Gondor became so friendly with the Middle Men of Rhovanion that he married Vidumavi, a princess of the Northmen, and his son Eldacar was of mixed blood. This led to the disastrous Kin-strife of Gondor.
However hostility developed between the Middle Men of the Haladin branch and the Númenóreans, which would endure until the end of the Third Age.
- J.R.R. Tolkien, Christopher Tolkien (ed.), The War of the Jewels, "Part Two. The Later Quenta Silmarillion: Concerning the Dwarves (Chapter 13)"
- J.R.R. Tolkien, Christopher Tolkien (ed.), The Peoples of Middle-earth, "XII. The Problem of Ros"
- J.R.R. Tolkien, Christopher Tolkien (ed.), The Peoples of Middle-earth, "Of Dwarves and Men"
- J.R.R. Tolkien, Christopher Tolkien (ed.), Unfinished Tales, "Cirion and Eorl and the Friendship of Gondor and Rohan"
- J.R.R. Tolkien, Christopher Tolkien (ed.), The Silmarillion, "Akallabêth: The Downfall of Númenor"
- J.R.R. Tolkien, The Lord of the Rings, The Two Towers, "Treebeard"
- J.R.R. Tolkien, Christopher Tolkien (ed.), Unfinished Tales, "The History of Galadriel and Celeborn"
- J.R.R. Tolkien, Christopher Tolkien (ed.), The Peoples of Middle-earth, "X. Of Dwarves and Men", "The Atani and their Languages"
- J.R.R. Tolkien, Christopher Tolkien (ed.), Unfinished Tales, "Aldarion and Erendis: The Mariner's Wife"
- J.R.R. Tolkien, Christopher Tolkien (ed.), Unfinished Tales, "A Description of the Island of Númenor"
- J.R.R. Tolkien, The Lord of the Rings, Appendix F, "The Languages and Peoples of the Third Age"