On the first few readings of LOTR, I completely missed this duel.
It was when Frodo is on Amon Hen hill, wearing a ring, and had a duel inside:
Later, Gandalf told Aragorn, Gimli and Legolas about this duel:
So, I think, we can pretty safely assume that person talking "Take it of Fool"...