charset should be ignored for the application/x-www-form-urlencoded content type.

[Mariusz Felisiak]

charset parameter is ​used in application/x-www-urlencoded content type. However, per ​the current spec (check out RFC 1866) the content type application/x-www-form-urlencoded does not have a charset and should be treated as UTF-8.

Thanks Eki Xu for the report.

[Mariusz Felisiak]

See related tickets #5076 and #14035.

[Mariusz Felisiak]

The current behavior, even if not correct, is documented and tested. Should we use a deprecation path?

[Claude Paroz]

I think it would be difficult to provide a sensible deprecation path (visible by devs, I mean), do you have a plan in mind?

[Mariusz Felisiak]

Replying to Claude Paroz:

I think it would be difficult to provide a sensible deprecation path (visible by devs, I mean), do you have a plan in mind?

We could raise a warning when self._encoding is not utf-8, that it will be ignored in Django 6.0. I'm just not sure it's worth doing.

[Shamil Abdulaev]

Greetings to you! I would like to tackle this bug and solve it!)

[Mariusz Felisiak]

Shamil, this ticket is already assign to me. We're discussing an acceptable approach.

[Claude Paroz]

Replying to Mariusz Felisiak:

We could raise a warning when self._encoding is not utf-8, that it will be ignored in Django 6.0. I'm just not sure it's worth doing.

The warning will probably only be ever raised on production servers, so hardly visible in practice.

Did you explore what will happen in practice if an incoming request is encoded in a different encoding and we try to decode it with utf-8? Server error (500)? Bad request(400)?

[Mariusz Felisiak]

Replying to Claude Paroz:

Did you explore what will happen in practice if an incoming request is encoded in a different encoding and we try to decode it with utf-8? Server error (500)? Bad request(400)?

I was only able to achieve a badly decoded string, no crash 🤔

[Claude Paroz]

Indeed, by default parse_sql is calling decode() with errors='replace', which will produce � (U+FFFD, the official REPLACEMENT CHARACTER) for invalid UTF-8 sequences. I'm still not convinced it will be a real improvement over the current situation…

[Mariusz Felisiak]

Replying to Claude Paroz:

Indeed, by default parse_sql is calling decode() with errors='replace', which will produce � (U+FFFD, the official REPLACEMENT CHARACTER) for invalid UTF-8 sequences. I'm still not convinced it will be a real improvement over the current situation…

We could raise 400 instead of silently ignoring a custom charset.

[Claude Paroz]

Replying to Mariusz Felisiak:

We could raise 400 instead of silently ignoring a custom charset.

Sure, I'd prefer that, failing loudly, instead of silently getting wrongly-encoded input.

[Adam Johnson]

A 400 response sounds sensible to me too.

[Mariusz Felisiak]

​PR

[Claude Paroz]

While starting to review the patch, and looking for more recent considerations than the old 1866 RFC, I read ​https://url.spec.whatwg.org/#application/x-www-form-urlencoded which is worth a read. Quoting a note:

A legacy server-oriented implementation might have to support encodings other than UTF-8 as well as have special logic for tuples of which the name is _charset. Such logic is not described here as only UTF-8 is conforming.

I don't necessarily re-question our previous discussions/decisions, however we might be prepared to receive some complaints as it may be that non-conforming agents start to produce BadRequest errors. Difficult to say before going to production!

[Mariusz Felisiak]

Replying to Claude Paroz:

While starting to review the patch, and looking for more recent considerations than the old 1866 RFC, I read ​https://url.spec.whatwg.org/#application/x-www-form-urlencoded which is worth a read. Quoting a note:

A legacy server-oriented implementation might have to support encodings other than UTF-8 as well as have special logic for tuples of which the name is _charset. Such logic is not described here as only UTF-8 is conforming.

I don't necessarily re-question our previous discussions/decisions, however we might be prepared to receive some complaints as it may be that non-conforming agents start to produce BadRequest errors. Difficult to say before going to production!

Unfortunately, I don't see a way to support this with a loud crash at the same time.

[Mariusz Felisiak]

I found some issues/PRs to remove charset for the application/x-www-form-urlencoded content type in other libraries. Even explicitly passing charset=utf-8 caused issues. As far as I'm aware, we can move it forward:

• ​https://github.com/requests/requests-oauthlib/issues/437
• ​https://github.com/request/request/issues/701

[Claude Paroz]

Sure, let's go ahead!

[GitHub <noreply@…>]

In 11920e7:

Fixed #34709 -- Raised BadRequest for non-UTF-8 requests with the application/x-www-form-urlencoded content type.

Thanks Eki Xu for the report.