Remove duplicate words

The two options differ in more than their unit. De-duplicating words splits on whitespace and rejoins with single spaces, so line breaks and indentation do not survive it — a tidy list goes in and one long line comes out. De-duplicating lines leaves each surviving line exactly as it was, indentation included, and compares them after trimming: apple, apple and apple count as one line and only the first survives. Case is where they part company. Words are folded, lines are not, so Apple and apple are one word and two lines. That is deliberate. An identifier, a filename or the part of an email address before the @ can differ by case and mean something else, so lowercase the list first with the case converter if you do want them folded.

Punctuation travels with the word, which means cat and cat, are two different tokens and both survive; run the text through the cleaner first if that is a problem. And note what this does not tell you: it removes repeats without reporting them. If the question is how often each word appeared rather than what the unique set is, the word counter has a keyword frequency list. Order is preserved rather than sorted, so sort the result afterwards if you want both.