from The Letters of J. R. R. Tolkien, Letter #144 To Naomi Mithison, 25 April 1954
Those who lingered were those who were enamoured of
Middle-earth and yet desired the unchanging beauty of the Land of the
Valar. Hence the making of the Rings; for the Three Rings were precisely endowed with the power of preservation, not of birth. Though unsullied, because they were not made by
Sauron nor touched by him,
they were nonetheless partly products of his instruction, and ultimately under the control of the One. Thus, as you will see, when the One goes, the last defenders of High-elven lore and beauty are shorn of power to hold back time, and depart.