Denethor seems more a tragic character than a villain

[Ent]

what exactly does he mean by this?

As you read Gandalf's words to him, Gandalf repeats that the old heathen kings of the west who were under the domination of Sauron's sway would 'burn themselves in their pride and despair and murder their kin to ease their passing" as well. This was 'before ever a ship sailed from the West." i.e. I think before the drowning of Númenor at the time when so many had fallen under Sauron's deceits.

Clearly Denethor was aware of this part of his ancestors' history. He states it even before Gandalf makes reference back to it in their conversation.

Why he chose to make reference to it I do not know, but it is another indication of the depths of despair he had sunk to - and possibly a 'last ditch effort' if you will, to try to attribute some kind of 'valor' or 'justification' to what he's intending to do.

Gandalf's repetition and clarification of the thing is another of his attempts to pull Denethor back from his now crazed state and re-evaluate what he himself has recently said, and what it's real 'cause' was. i.e. "What are you saying, crazed Denethor -- there was no merit in being a heathen king under Sauron's domination and burning yourself and your kin alive."

that's the best I've got...

added note: I do not think this issue of 'heathen kings and burning' is referred to anywhere else in the writings. I've not found it yet at least.

 

[Ugluk]

It is possible that Tolkien was alluding to customs similar to that of the Vikings as described by Ahmad ibn Fadlan, the original work had been translated into German in the 19th century. It involved brutal killing first of some of the chieftain's horses, then the gang rape and murder of a slave girl, after which all were burned together on a ship with the chieftain's body. Note that Gandalf specifically refers to the ritual murder of those close to the one dying "And only the heathen kings, under the domination of the Dark Power, did thus, slaying themselves in pride and despair, murdering their kin to ease their own death.' This is a practice widely attested in the archaeological record, though most commonly slaves rather than relatives

You can read Ibn Fadlan's description here - but be warned, it's ugly stuff.

A Viking Burial Described by Arab Writer Ahmad ibn Fadlan

The Oseberg ship, a well-preserved Viking ship discovered in a large burial mound at the Oseberg farm near Tønsberg in Vestfold county, (Photo: norskevaapen.no) Norse funerals, or the burial custo…

thornews.com

 

[Squint-eyed Southerner]

Getting back to the OP, a character may function in the plot in a "villainous" role, but remain a tragic character. We could easily imagine the Minas Tirith Players staging "The Tragedy of Denethor, Steward of Gondor" some time into the high mimetic world of The Fourth Age.

There can be different kinds of tragedy, and tragic heroes, as several posts here have mentioned. One way of looking at them is as on a spectrum, moving from the more romantic, with stories of lost innocence or young lives cut off, like Romeo and Juliette, through what we normally think of as the typical tragic hero, who through hubris commits a provoking act against the natural order, whether deliberately like Macbeth, or inadvertently like Oedipus, and on into more ironic forms, where the term "hero" seems less and less appropriate, until we meet characters who experience only unrelieved suffering, such as those in Kafka. At either end, it's difficult to justify something like hubris or "tragic flaw".

Looked at this way, we can see Boromir in the center, with Frodo on one side, and Denethor on the other. Leaving Frodo aside, Denethor's story is clearly more ironic than Boromir's: there is a feeling that he's failed to make any real heroic point, as shown by both the manners of, and reactions to, their respective deaths. The source of at least one his grievances, the rift between the Northern and Southern kingdoms, closes up over his head, as it were. It doesn't make him less of a tragic character, but the ironic element introduces judgement by his society, much more than is the case with Boromir.