It certainly goes against the wording of their oath, where they (
Feanor and sons) expressly hold themselves to pursue
any in
Arda that hold a Silmaril- man,
elf,
maia (presumably
Luthien falls somewhere between these last two) or
vala. However, at this time
Maedhros was still able to restrain his brothers, and he was against attacking fellow
Elves to retrieve the Silmaril, in spite of what he said in the oath. When the brothers finally do attack
Doriath, he is still against the action, but it is implied that he is over-ruled by
Celegorm, Caranthir, and
Curufin. These last three are ever-ready to fulfill the oath, but do not seem to be nearly as brave or valiant as Maedhros. Perhaps Celegorm and Curufin in particular remembered what happened to them the last time they attacked Luthien, and so considered the enterprise hopeless. However, it appears it was equally hopeless for them to attack Dior, as I believe he slew at least one (if not both) of the duo.
Incidentally, I have a real problem with the later attack on the havens. Maedhros is said to have foresworn the oath (and was especially upset at the apparent death of Elured and Elurin following the oath's work in Doriath), and at first he declined to attack the havens. Yet he was tormented by the oath, and eventually led the four remaining brothers in the attack. The Oath itself was sworn to Illuvatar, and
Mandos and
Manwe were named as witnesses. Of these three, it is unclear to me which one would not have released Maedhros from the oath when he foreswore it. In the justice system in my mind (probably different than that of Manwe or
Eru), the guilt of the third kinslaying fell not to Maedhros and the
sons of Feanor (who had foresworn their oath), but to whichever of the three named who did not release the brothers from their oath. I guess such cold adherence to an
evil oath in the face of repentance does not sit well with me.