The troll-men of Far Harad are mentioned in The Return of the King and goes like this "...East rode the knights of Dol Amroth driving the enemy before them: troll-men and Variags and orcs that hated the sunlight." Also during the battle of Pelennor Fields there’s a reference, "...Easterlings with axes, and Variags of Khand, Southrons in scarlet, and out of Far Harad black men like half-trolls with white eyes and red tongues."
But no one knows if it’s a reference referring to large men who’s compared to trolls or Troll-men. In the second reference there mentioned half-trolls, so some people think that, troll-men could be a cross breed between men and trolls of some sort. Maybe it’s referring to the Olog-hai that is a sort of cross breed between troll and orc or uruk-hai. Because of that the Olog-hai has something near human intelligence.