Opened 7 months ago
Closed 7 months ago
#35368 closed Bug (duplicate)
Postgres schema test failure locally
Reported by: | bcail | Owned by: | nobody |
---|---|---|---|
Component: | Database layer (models, ORM) | Version: | dev |
Severity: | Normal | Keywords: | |
Cc: | bcail, Simon Charette | Triage Stage: | Accepted |
Has patch: | no | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description
The error is below. I bisected to commit 888b9042b3598bab6557c62de82505eec9ea62ed where the test started failing for me. I'm on Linux, using Postgres 15.
ERROR: test_add_generated_field (schema.tests.SchemaTests.test_add_generated_field) ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/local/src/django/django/db/backends/utils.py", line 103, in _execute return self.cursor.execute(sql) ^^^^^^^^^^^^^^^^^ psycopg2.errors.IndeterminateCollation: could not determine which collation to use for upper() function HINT: Use the COLLATE clause to set the collation explicitly. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/python-3.12/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/opt/python-3.12/lib/python3.12/unittest/case.py", line 636, in run self._callTestMethod(testMethod) ^^^^^^^^^^^^^^^^^ File "/opt/python-3.12/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^^^^^^^^^^ File "/opt/local/src/django/django/test/utils.py", line 446, in inner return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/opt/local/src/django/django/test/testcases.py", line 1548, in skip_wrapper return test_func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/opt/local/src/django/tests/schema/tests.py", line 923, in test_add_generated_field_contains editor.add_field(GeneratedFieldContainsModel, field) ^^^^^^^^^^^^^^^^^ File "/opt/local/src/django/django/db/backends/base/schema.py", line 754, in add_field self.execute(sql, params or None) ^^^^^^^^^^^^^^^^^ File "/opt/local/src/django/django/db/backends/postgresql/schema.py", line 48, in execute return super().execute(sql, None) ^^^^^^^^^^^^^^^^^ File "/opt/local/src/django/django/db/backends/base/schema.py", line 201, in execute cursor.execute(sql, params) ^^^^^^^^^^^^^^^^^ File "/opt/local/src/django/django/db/backends/utils.py", line 79, in execute return self._execute_with_wrappers( ^^^^^^^^^^^^^^^^^ File "/opt/local/src/django/django/db/backends/utils.py", line 92, in _execute_with_wrappers return executor(sql, params, many, context) ^^^^^^^^^^^^^^^^^ File "/opt/local/src/django/django/db/backends/utils.py", line 100, in _execute with self.db.wrap_database_errors: ^^^^^^^^^^^^^^^^^ File "/opt/local/src/django/django/db/utils.py", line 91, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value ^^^^^^^^^^^^^^^^^ File "/opt/local/src/django/django/db/backends/utils.py", line 103, in _execute return self.cursor.execute(sql) ^^^^^^^^^^^^^^^^^ django.db.utils.ProgrammingError: could not determine which collation to use for upper() function HINT: Use the COLLATE clause to set the collation explicitly.
Change History (11)
comment:1 by , 7 months ago
Component: | Uncategorized → Database layer (models, ORM) |
---|---|
Resolution: | → worksforme |
Status: | new → closed |
Type: | Uncategorized → Bug |
comment:2 by , 7 months ago
I don't know the details of what's going wrong - I've been able to run django's test suite fine, on sqlite & postgres, and then it started failing, and I found that it was that commit where the problem started. I was using psycopg2, and I just tried psycopg3, and it still failed.
Here is another detail, though: the test name that django gives as failing is "test_add_generated_field" - but in the stacktrace, it points to the test added in that commit ("test_add_generated_field_contains") - it seems that django's printing the wrong test name in the error summary. If I set parallel=1
for the tests, then the correct test name gets shown in the summary.
comment:3 by , 7 months ago
Cc: | added |
---|
Hello there!
Could you provide the exact version of psycopg2 and psycopg3 you are using to reproduce. I'm also struggling to reproduce on my side.
There might some Postgres collation shenanigans involving the distro you are using as well. Can you also reproduce with Docker?
See #35194 which seems related.
comment:4 by , 7 months ago
psycopg => 3.1.18, psycopg2 => 2.9.9
I'm on Debian 12 Bookworm - maybe there's an issue related to Debian's settings?
comment:5 by , 7 months ago
Triage Stage: | Unreviewed → Accepted |
---|
I'm able to reproduce against the Postgres 16.2 Docker image just like Mariuzs did a few weeks ago.
Not sure what the proper solution is though the failure modes seems a bit arbitrary
UPPER("test"::text) = UPPER('yes')
💥UPPER("test") = ('YES')
✅UPPER("test") = UPPER('yes')
💥
I guess we could adapt the test to avoid using UPPER
to address this immediate issue as it's not relevant to the test (it was interested in covering %
usage)
-
tests/schema/tests.py
diff --git a/tests/schema/tests.py b/tests/schema/tests.py index 86ac7382cc..ff126d446a 100644
a b class Meta: 913 913 editor.create_model(GeneratedFieldContainsModel) 914 914 915 915 field = GeneratedField( 916 expression=Q(text__ icontains="FOO"),916 expression=Q(text__contains="foo"), 917 917 db_persist=True, 918 918 output_field=BooleanField(), 919 919 )
And then re-open #35194 for further investigation into which COLLATE
incantations must be used for generated expressions. In a sense the problem is very similar to the CONCAT
vs ||
problem we ran into for GeneratedField
where CONCAT
is not immutable (#34955). Just like CONCAT
the UPPER
functions defaults to runtime information that must be explicitly declared when creating generate expressions.
comment:6 by , 7 months ago
Resolution: | worksforme |
---|---|
Status: | closed → new |
comment:7 by , 7 months ago
Thank you Simon for that ticket pointer, your plan makes sense (reopening #35194 and marking this as a dupe).
bcail, could you please double check that you are getting the error in PG 15? or is it 16.2 like the other ticket and what Simon was able to reproduce?
comment:8 by , 7 months ago
Nailed it down to a change in Postgres 15.6 released on 2024-02-08 so it makes sense that bcail ran on that only recently against PG 15.
Fix function volatility checking for GENERATED and DEFAULT expressions (Tom Lane)
These places could fail to detect insertion of a volatile function default-argument expression, or decide that a polymorphic function is volatile although it is actually immutable on the datatype of interest. This could lead to improperly rejecting or accepting a GENERATED clause, or to mistakenly applying the constant-default-value optimization in ALTER TABLE ADD COLUMN.
comment:9 by , 7 months ago
Simon, is my assumption correct that a PR is attached to this to separate the fixing of the test introduced in 888b9042b3598bab6557c62de82505eec9ea62ed vs the fix itself for the non-determinism of UPPER
?
If yes, I would suggest that, instead, the PR Refs #35194
and we close this as dupe because as I understand, the test failure is just a symptom of the issue reported there. Thanks!
comment:10 by , 7 months ago
Works for me. I'll reword the commit message so you can close this one as a dupe.
comment:11 by , 7 months ago
Resolution: | → duplicate |
---|---|
Status: | new → closed |
Thank you!
Closing as duplicate of #35194
Hello, thank you for your report!
Could you please provide more details and instructions on how to reproduce? I tested locally with PG 16.1 and also the Jenkins CI is running PG 13 and in both setups the mentioned test works OK. Could this be something specific from your environment or your models?
Closing as
worksforme
for now but do let us know if you can provide concrete steps to reproduce this.