Opened 7 years ago

Closed 5 years ago

Last modified 5 years ago

#27795 closed Cleanup/optimization (fixed)

Audit force_text invocations

Reported by: Claude Paroz Owned by: Claude Paroz
Component: Utilities Version: dev
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

django.utils.encoding.force_text is called many times in Django code for several use cases:

  • force the evaluation of a lazy string
  • convert bytes or a bytestring (PY2) to str
  • convert an object to its string representation

A global audit is needed to clean that usage, also considering that Python 3 native strings are more wildly used.
I'm working on it currently.

Change History (40)

comment:1 by Claude Paroz <claude@…>, 7 years ago

In ccfd1295:

Refs #27795 -- Prevented SafeText from losing safe status on str()

This will allow to replace force_text() by str() in several places (as one of
the features of force_text is to keep the safe status).

comment:2 by Claude Paroz <claude@…>, 7 years ago

In 2757209c:

Refs #27795 -- Removed force_text from templatize function

comment:3 by Claude Paroz <claude@…>, 7 years ago

In 3a148f95:

Refs #27795 -- Removed force_text from the template layer

Thanks Tim Graham for the review.

comment:4 by Claude Paroz <claude@…>, 7 years ago

In 8346680e:

Refs #27795 -- Removed unneeded force_text calls

Thanks Tim Graham for the review.

comment:5 by GitHub <noreply@…>, 7 years ago

In e5dce7b0:

Refs #27795 -- Removed unneeded force_text calls from the Oracle backend.

comment:6 by Claude Paroz <claude@…>, 7 years ago

In 301de77:

Refs #27795 -- Replaced many force_text() with str()

Thanks Tim Graham for the review.

comment:7 by Claude Paroz <claude@…>, 7 years ago

In a8343fe7:

Refs #27795 -- Replaced force_text() with str() in feed generators

comment:8 by Tim Graham <timograham@…>, 7 years ago

In b39aabc:

Refs #27795 -- Reworked get_or_create test erroneously mixing bytes and str.

As CharField.to_python() now always calls str(), assigning bytes to a
CharField is no longer correct usage. Doing so results in a warning:

django/db/models/fields/init__.py:1061: BytesWarning: str() on a bytes instance

Use a unique constraint violation to trigger the database error instead.

Warning introduced in 301de774c21d055e9e5a7073e5bffdb52bc71079.

comment:9 by Claude Paroz <claude@…>, 7 years ago

In d842ada:

Refs #27795 -- Stopped converting integer format settings to str in JS/JSON i18n views

Thanks Tim Graham for the review.

comment:10 by Claude Paroz <claude@…>, 7 years ago

In f87256b1:

Refs #27795 -- Replaced force_text() with str() in contrib.messages

Thanks Tim Graham for the review.

comment:11 by Claude Paroz <claude@…>, 7 years ago

In 45bbb9aa:

Refs #27795 -- Removed force_text() usage in serialization framework

comment:12 by Tim Graham <timograham@…>, 6 years ago

In 6b2f8fb9:

Refs #27795 -- Replaced force_text() usage in django.core.management.

Use decode() since Popen.communicate() always returns bytes.

comment:13 by Tim Graham <timograham@…>, 6 years ago

In 16c5a334:

Refs #27795 -- Replaced force_text/bytes() with decode()/encode() in password hashers.

comment:14 by Carlton Gibson <carlton.gibson@…>, 6 years ago

In 6a039800:

Refs #27795 -- Replaced force_bytes() usage in django.core.cache.

comment:15 by Tim Graham <timograham@…>, 6 years ago

In b38532c:

Refs #27795 -- Replaced force_bytes() usage in django.core.signing.

comment:16 by GitHub <noreply@…>, 6 years ago

In fa75b2c:

Refs #27795 -- Removed force_bytes/text() usage in tests.

comment:17 by Tim Graham <timograham@…>, 6 years ago

In 7d96f0c4:

Refs #27795 -- Replaced force_bytes() usage in django.http.

comment:18 by Jon Dufresne <jon.dufresne@…>, 6 years ago

In 968329b:

Refs #27795 -- Replaced force_text() calls with str() in django.utils.html.

comment:19 by Tim Graham <timograham@…>, 6 years ago

In 1f3852b:

Refs #27795 -- Removed force_bytes() usage in django/template/loaders/cached.py.

comment:20 by Tim Graham <timograham@…>, 6 years ago

In 919d5981:

Refs #27795 -- Removed force_text() usage in django/test/testcases.py.

comment:21 by Tim Graham <timograham@…>, 6 years ago

In b8a41a28:

Refs #27795 -- Removed force_bytes() in django.test.client where possible.

comment:22 by Tim Graham <timograham@…>, 6 years ago

In 90f3ef3:

Refs #27795 -- Removed force_bytes()/smart_text() from model fields.

comment:23 by Tim Graham <timograham@…>, 6 years ago

In ee184e1:

Refs #27795 -- Removed force_bytes() usage in django/utils/cache.py.

comment:24 by Tim Graham <timograham@…>, 6 years ago

In cc79c7ee:

Refs #27795 -- Removed force_text() in Model._get_FIELD_display().

comment:25 by Tim Graham <timograham@…>, 5 years ago

In 1d65ddd9:

Refs #27795 -- Removed force_bytes() usage in db/backends/utils.py.

comment:26 by Tim Graham <timograham@…>, 5 years ago

In abeed587:

Refs #27795 -- Removed force_bytes() usage in db/backends/base/schema.py.

comment:27 by Tim Graham <timograham@…>, 5 years ago

In bb81c22:

Refs #27795 -- Removed force_bytes() usage in utils/_os.py.

comment:28 by Tim Graham <timograham@…>, 5 years ago

In 8d873503:

Refs #27795 -- Removed force_bytes() usage in contrib/auth/handlers/modwsgi.py.

comment:29 by Tim Graham <timograham@…>, 5 years ago

In d4fc111c:

Refs #27795 -- Removed force_bytes() usage in admindocs.

Refs #12892 is probably obsolete.

comment:30 by Carlton Gibson <carlton.gibson@…>, 5 years ago

In efd8a82e:

Refs #27795 -- Removed force_bytes() usage in MySQL backend.

The mysqlclient cursor attribute _last_executed is always stored as
bytes. Decode it.

TextField values are already type str. No need to decode.

comment:31 by Carlton Gibson <carlton.gibson@…>, 5 years ago

In a47dd99:

[2.1.x] Refs #27795 -- Removed force_bytes() usage in MySQL backend.

The mysqlclient cursor attribute _last_executed is always stored as
bytes. Decode it.

TextField values are already type str. No need to decode.

Backport of efd8a82e268a82b3ad0be77bd5b4548c30bcb4d7 from master

comment:32 by Carlton Gibson <carlton.gibson@…>, 5 years ago

In bdae19cf:

Refs #27795 -- Removed force_bytes() usage in sessions.

SessionBase.decode() is the inverse operation to SessionBase.encode().
As SessionBase.encode() always returns a string, SessionBase.decode()
should always be passed a string argument. Fixed the file backend, which
was the only backend still passing a bytestring.

comment:33 by Carlton Gibson <carlton.gibson@…>, 5 years ago

In f3f31b0:

[2.1.x] Refs #27795 -- Removed force_bytes() usage in sessions.

SessionBase.decode() is the inverse operation to SessionBase.encode().
As SessionBase.encode() always returns a string, SessionBase.decode()
should always be passed a string argument. Fixed the file backend, which
was the only backend still passing a bytestring.

Backport of bdae19cf6395d6bfee80864d9e87c4aec241eceb from master

comment:34 by Carlton Gibson <carlton.gibson@…>, 5 years ago

In 54eedddb:

Revert "[2.1.x] Refs #27795 -- Removed force_bytes() usage in MySQL backend."

This reverts commit a47dd99cc6883fe3e35895e8c5d4bb53996bd184.

comment:35 by Carlton Gibson <carlton.gibson@…>, 5 years ago

In 5045f8af:

Revert "[2.1.x] Refs #27795 -- Removed force_bytes() usage in sessions."

This reverts commit f3f31b0fc26f772b198c9683a427f2f7fe208c37.

comment:36 by Tim Graham <timograham@…>, 5 years ago

In 4ab071f4:

Refs #27795 -- Removed force_bytes() usage in contrib/staticfiles/storage.py.

comment:37 by Tim Graham <timograham@…>, 5 years ago

In 2ba588e7:

Refs #27795 -- Removed force_text() usage in db/models/sql/query.py.

comment:38 by Tim Graham <timograham@…>, 5 years ago

In c82893cb:

Refs #27795 -- Removed force_bytes() usage from django/utils/http.py.

django.utils.http.urlsafe_base64_encode() now returns a string, not a
bytestring. Since URLs are represented as strings,
urlsafe_base64_encode() should return a string. All uses immediately
decoded the bytestring to a string anyway.

As the inverse operation, urlsafe_base64_decode() accepts a string.

comment:39 by Claude Paroz, 5 years ago

Resolution: fixed
Status: assignedclosed

I think this is mostly done now.

comment:40 by Carlton Gibson <carlton.gibson@…>, 5 years ago

In 1e6b9e2:

Refs #27795 -- Removed an unnecessary force_bytes() call in uri_to_iri().

The value returned from urllib.parse.quote() is always a string, so can
safely call .encode().

Note: See TracTickets for help on using tickets.
Back to Top