Code

Opened 7 years ago

Closed 7 years ago

#4662 closed (fixed)

[unicode] truncate_html_words doesn't work for non-latin characters

Reported by: Ivan Sagalaev <Maniac@…> Owned by: jacob
Component: Uncategorized Version: unicode
Severity: Keywords:
Cc: Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

django.utils.text.truncate_html_words (and for this matter, 'truncatwords_html' filter) uses regular expression to find words. The pattern is "[A-Za-z0-9]" that doesn't match non-latin words. It should be replaced with "\w" and compiled with re.UNICODE flag.

Patch follows. There is a small backwards incompatible change of behavior since "\w" matches underscores while "[A-Za-z0-9]" doesn't and thus it now considers "_" as words. I believe this is not very bad since isolated underscores are rare in human-oriented texts.

P.S. This whole thing makes sense only for unicode branch since regular expressions in unicode mode don't work correctly for byte strings anyway.

Attachments (1)

4662.diff (589 bytes) - added by Ivan Sagalaev <Maniac@…> 7 years ago.
Patch

Download all attachments as: .zip

Change History (5)

Changed 7 years ago by Ivan Sagalaev <Maniac@…>

Patch

comment:1 Changed 7 years ago by SmileyChris

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

Comment from my original patch regarding unicode: http://code.djangoproject.com/ticket/2027#comment:6

Do we need to compile with re.UNICODE?

If you were worried about backwards compatibility, then you could use [^\W_], but I don't really think that's necessary ;P

comment:2 Changed 7 years ago by Ivan Sagalaev <Maniac@…>

Do we need to compile with re.UNICODE?

Yes. Because otherwise \w means only [A-Za-z0-9_]. re.UNICODE switches re to use unicode db to get the notion of "letter".

comment:3 Changed 7 years ago by SmileyChris

  • Triage Stage changed from Unreviewed to Ready for checkin

Oh, I actually missed the fact you were already compiling with re.U. :)

comment:4 Changed 7 years ago by mtredinnick

  • Resolution set to fixed
  • Status changed from new to closed

(In [5533]) unicode: Fixed #4662 -- Fixed a remaining ASCII assumption in
truncatewords_html(). Thanks, Ivan Sagalaev.

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.