[patch] truncatewords filter can invalidate your HTML

HTML aware word truncation

Here's a basic HTML-aware word truncation filter.

It only counts words outside of tags and HTML comments. It only closes off tags if they were closed off (properly) in the HTML being filtered.

This time I'll upload the proper HTML-aware truncation patch

No, uh I should actually use the unit tests I created... this one works.

And speaking of unit tests... Django's unit tests don't actually work that well. When creating the unit tests, I found that:

>>> truncatewords_html('<p>one <a href="#">two - three <br>four</a> five</p>', 5)
'<p>one <a href="#">two - three <br>four</a> five...</p>'

and

>>> truncatewords_html('<p>one <a href="#">two - three <br>four</a> five</p>', 5)
'<p>one <a href="#">two - three <br>four</a> five</p>'

both validate... (changing a letter still causes a unit test fail)

So it's not actually a very good test now, is it?

That truncate_html_words() function in the patch looks wayyyyyy long and complex. Is there any way you can simplify it, perhaps by doing more of it in regular expressions rather than iterating over each character of the string?

Yep, it was. After a sleep, here's a much better version.

This is a much better (and shorter) HTML truncater

Actually, it's not that much shorter... but it is better :)

I still need to iterate through the string (to check for open tags and to get the right truncation point) but I do so with a regular expression like Adrian suggested.

Teensy change ensuring that self-closing tags are properly identified

Just a thought, the patch doesn't consider unicode at the moment - should it? Easy change if it is a consideration:

1. Compile re_words with re.UNICODE

2. In re_words, replace '[A-Za-z0-9]' with '\w' (should probably just be that anyway)

I believe this is ready for check-in, pending a core decision on SmileyChris?'s coment above re: unicode.

I'll check in what's here, since all text processing has to be unicode audited at some point. I'm not convinced this is as efficient as it could be, but we can revisit that later if it's a problem, since just having something that works is a good idea right now. Let's have people test it in practice and get some feedback; it doesn't affect any existing code, after all.

(In [4468]) Fixed #2027 -- added truncatewords_html filter that respects HTML tags whilst truncating. Patch from SmileyChris?.