[patch] make slugify filter to be aware of unicode

[nkeric]

Currently the slugify filter only works with english alphanumeric characters, here is a simple patch that makes it to be able to "match the characters [0-9_] plus whatever is classified as alphanumeric in the Unicode character properties database".

here is the ref:

​http://docs.python.org/lib/re-syntax.html

​http://docs.python.org/lib/node115.html

Regards,

• Eric

[nkeric]

I tested it and it handles Chinese Characters properly (should work with other language's characters too), and all Chinese punctuations are removed as expected :)

[nkeric]

new diff with re.U replaced with re.UNICODE

[nkeric]

patch the validator isSlug

[nkeric]

output string should be utf-8 encoded as input string

[nkeric]

decode utf-8 encoded string for doing re search

[Adrian Holovaty]

Closing in favor of #2489.

[(none)]

Milestone Version 1.0 deleted

[Jonas von Poser]

I found this program: Slughifi is slugify with support for international characters.

​http://amisphere.com/contrib/python-django/

​http://amisphere.com/contrib/python-django/slughifi.py

[Malcolm Tredinnick]

Somebody brought that up on the mailing list at one point. Further investigation revealed is was under the GPL license (no license is mentioned in the file, which is even worse), so we cannot use it.

We have a different solution in the works over at #4365 which I'll resolve in the near future.

[Jonas von Poser]

By now, I found a solution.

Got from ​http://www.djangosnippets.org/snippets/98/ (_string_to_slug method) that points to
​http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/251871 (Aaron Bentley)

E.g:
It translates: Á È ï ô ü ñ
to:            A E i o u n

so it's very usefull for a lot of languages.

I add a patch.

[Jonas von Poser]

Normallizes string

[Jonas von Poser]

Closing in favor of #4365