Internationalisation(charset) problems with FileField file names and core.db.backend.mysql

[little]

The function django.utils.text.get_valid_filename() is not friendly for non-latin file-names

    s = s.strip().replace(' ', '_')
    return re.sub(r'[^-A-Za-z0-9_.]', '', s)

truncates file name to underscores only: "__________.txt" for example.

Let it retun a Unicode object of string s

    return unicode(s,'utf8')

Or make it possible to overload this function to end-programmer.

[hugo]

The problem here: we can't assume anything about the filesystem of the server beside the fact that it is possible to use us-ascii in filenames. So utf-8 won't be an option - it might produce unreadable filenames. And since there are several places that function like / and ., we can't just accept any char we want, or we would open up for filesystem traversal hackery.

One way would be to just turn non-ascii chars into a uXXXX form, so that at least the filename isn't all dashes.

I move the database stuff into it's own ticket, as that isn't i18n related, but more database backend related.