Opened 21 months ago

Last modified 5 days ago

#27795 assigned Cleanup/optimization

Audit force_text invocations

Reported by: Claude Paroz Owned by: Claude Paroz
Component: Utilities Version: master
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

django.utils.encoding.force_text is called many times in Django code for several use cases:

  • force the evaluation of a lazy string
  • convert bytes or a bytestring (PY2) to str
  • convert an object to its string representation

A global audit is needed to clean that usage, also considering that Python 3 native strings are more wildly used.
I'm working on it currently.

Change History (38)

comment:1 Changed 21 months ago by Claude Paroz <claude@…>

In ccfd1295:

Refs #27795 -- Prevented SafeText from losing safe status on str()

This will allow to replace force_text() by str() in several places (as one of
the features of force_text is to keep the safe status).

comment:2 Changed 21 months ago by Claude Paroz <claude@…>

In 2757209c:

Refs #27795 -- Removed force_text from templatize function

comment:3 Changed 20 months ago by Claude Paroz <claude@…>

In 3a148f95:

Refs #27795 -- Removed force_text from the template layer

Thanks Tim Graham for the review.

comment:4 Changed 20 months ago by Claude Paroz <claude@…>

In 8346680e:

Refs #27795 -- Removed unneeded force_text calls

Thanks Tim Graham for the review.

comment:5 Changed 18 months ago by GitHub <noreply@…>

In e5dce7b0:

Refs #27795 -- Removed unneeded force_text calls from the Oracle backend.

comment:6 Changed 18 months ago by Claude Paroz <claude@…>

In 301de77:

Refs #27795 -- Replaced many force_text() with str()

Thanks Tim Graham for the review.

comment:7 Changed 18 months ago by Claude Paroz <claude@…>

In a8343fe7:

Refs #27795 -- Replaced force_text() with str() in feed generators

comment:8 Changed 18 months ago by Tim Graham <timograham@…>

In b39aabc:

Refs #27795 -- Reworked get_or_create test erroneously mixing bytes and str.

As CharField.to_python() now always calls str(), assigning bytes to a
CharField is no longer correct usage. Doing so results in a warning:

django/db/models/fields/init__.py:1061: BytesWarning: str() on a bytes instance

Use a unique constraint violation to trigger the database error instead.

Warning introduced in 301de774c21d055e9e5a7073e5bffdb52bc71079.

comment:9 Changed 17 months ago by Claude Paroz <claude@…>

In d842ada:

Refs #27795 -- Stopped converting integer format settings to str in JS/JSON i18n views

Thanks Tim Graham for the review.

comment:10 Changed 17 months ago by Claude Paroz <claude@…>

In f87256b1:

Refs #27795 -- Replaced force_text() with str() in contrib.messages

Thanks Tim Graham for the review.

comment:11 Changed 17 months ago by Claude Paroz <claude@…>

In 45bbb9aa:

Refs #27795 -- Removed force_text() usage in serialization framework

comment:12 Changed 9 months ago by Tim Graham <timograham@…>

In 6b2f8fb9:

Refs #27795 -- Replaced force_text() usage in django.core.management.

Use decode() since Popen.communicate() always returns bytes.

comment:13 Changed 9 months ago by Tim Graham <timograham@…>

In 16c5a334:

Refs #27795 -- Replaced force_text/bytes() with decode()/encode() in password hashers.

comment:14 Changed 9 months ago by Carlton Gibson <carlton.gibson@…>

In 6a039800:

Refs #27795 -- Replaced force_bytes() usage in django.core.cache.

comment:15 Changed 8 months ago by Tim Graham <timograham@…>

In b38532c:

Refs #27795 -- Replaced force_bytes() usage in django.core.signing.

comment:16 Changed 8 months ago by GitHub <noreply@…>

In fa75b2c:

Refs #27795 -- Removed force_bytes/text() usage in tests.

comment:17 Changed 8 months ago by Tim Graham <timograham@…>

In 7d96f0c4:

Refs #27795 -- Replaced force_bytes() usage in django.http.

comment:18 Changed 8 months ago by Jon Dufresne <jon.dufresne@…>

In 968329b:

Refs #27795 -- Replaced force_text() calls with str() in django.utils.html.

comment:19 Changed 8 months ago by Tim Graham <timograham@…>

In 1f3852b:

Refs #27795 -- Removed force_bytes() usage in django/template/loaders/cached.py.

comment:20 Changed 8 months ago by Tim Graham <timograham@…>

In 919d5981:

Refs #27795 -- Removed force_text() usage in django/test/testcases.py.

comment:21 Changed 8 months ago by Tim Graham <timograham@…>

In b8a41a28:

Refs #27795 -- Removed force_bytes() in django.test.client where possible.

comment:22 Changed 7 weeks ago by Tim Graham <timograham@…>

In 90f3ef3:

Refs #27795 -- Removed force_bytes()/smart_text() from model fields.

comment:23 Changed 7 weeks ago by Tim Graham <timograham@…>

In ee184e1:

Refs #27795 -- Removed force_bytes() usage in django/utils/cache.py.

comment:24 Changed 6 weeks ago by Tim Graham <timograham@…>

In cc79c7ee:

Refs #27795 -- Removed force_text() in Model._get_FIELD_display().

comment:25 Changed 3 weeks ago by Tim Graham <timograham@…>

In 1d65ddd9:

Refs #27795 -- Removed force_bytes() usage in db/backends/utils.py.

comment:26 Changed 3 weeks ago by Tim Graham <timograham@…>

In abeed587:

Refs #27795 -- Removed force_bytes() usage in db/backends/base/schema.py.

comment:27 Changed 3 weeks ago by Tim Graham <timograham@…>

In bb81c22:

Refs #27795 -- Removed force_bytes() usage in utils/_os.py.

comment:28 Changed 3 weeks ago by Tim Graham <timograham@…>

In 8d873503:

Refs #27795 -- Removed force_bytes() usage in contrib/auth/handlers/modwsgi.py.

comment:29 Changed 13 days ago by Tim Graham <timograham@…>

In d4fc111c:

Refs #27795 -- Removed force_bytes() usage in admindocs.

Refs #12892 is probably obsolete.

comment:30 Changed 12 days ago by Carlton Gibson <carlton.gibson@…>

In efd8a82e:

Refs #27795 -- Removed force_bytes() usage in MySQL backend.

The mysqlclient cursor attribute _last_executed is always stored as
bytes. Decode it.

TextField values are already type str. No need to decode.

comment:31 Changed 12 days ago by Carlton Gibson <carlton.gibson@…>

In a47dd99:

[2.1.x] Refs #27795 -- Removed force_bytes() usage in MySQL backend.

The mysqlclient cursor attribute _last_executed is always stored as
bytes. Decode it.

TextField values are already type str. No need to decode.

Backport of efd8a82e268a82b3ad0be77bd5b4548c30bcb4d7 from master

comment:32 Changed 12 days ago by Carlton Gibson <carlton.gibson@…>

In bdae19cf:

Refs #27795 -- Removed force_bytes() usage in sessions.

SessionBase.decode() is the inverse operation to SessionBase.encode().
As SessionBase.encode() always returns a string, SessionBase.decode()
should always be passed a string argument. Fixed the file backend, which
was the only backend still passing a bytestring.

comment:33 Changed 12 days ago by Carlton Gibson <carlton.gibson@…>

In f3f31b0:

[2.1.x] Refs #27795 -- Removed force_bytes() usage in sessions.

SessionBase.decode() is the inverse operation to SessionBase.encode().
As SessionBase.encode() always returns a string, SessionBase.decode()
should always be passed a string argument. Fixed the file backend, which
was the only backend still passing a bytestring.

Backport of bdae19cf6395d6bfee80864d9e87c4aec241eceb from master

comment:34 Changed 12 days ago by Carlton Gibson <carlton.gibson@…>

In 54eedddb:

Revert "[2.1.x] Refs #27795 -- Removed force_bytes() usage in MySQL backend."

This reverts commit a47dd99cc6883fe3e35895e8c5d4bb53996bd184.

comment:35 Changed 12 days ago by Carlton Gibson <carlton.gibson@…>

In 5045f8af:

Revert "[2.1.x] Refs #27795 -- Removed force_bytes() usage in sessions."

This reverts commit f3f31b0fc26f772b198c9683a427f2f7fe208c37.

comment:36 Changed 10 days ago by Tim Graham <timograham@…>

In 4ab071f4:

Refs #27795 -- Removed force_bytes() usage in contrib/staticfiles/storage.py.

comment:37 Changed 7 days ago by Tim Graham <timograham@…>

In 2ba588e7:

Refs #27795 -- Removed force_text() usage in db/models/sql/query.py.

comment:38 Changed 5 days ago by Tim Graham <timograham@…>

In c82893cb:

Refs #27795 -- Removed force_bytes() usage from django/utils/http.py.

django.utils.http.urlsafe_base64_encode() now returns a string, not a
bytestring. Since URLs are represented as strings,
urlsafe_base64_encode() should return a string. All uses immediately
decoded the bytestring to a string anyway.

As the inverse operation, urlsafe_base64_decode() accepts a string.

Note: See TracTickets for help on using tickets.
Back to Top