Opened 3 years ago

Closed 23 months ago

Last modified 14 months ago

#33308 closed New feature (fixed)

Add psycopg3 backend

Reported by: Paolo Melchiorre Owned by: Florian Apolloner
Component: Database layer (models, ORM) Version: dev
Severity: Normal Keywords: database postgresql backend orm
Cc: Daniele Varrazzo, Adam Johnson, Michael P. Jung, Saeed Blanchette, Florian Apolloner, Adam Wróbel, Andreas Rammhold, Alexander Nestorov, Ben Plessinger, Thiago Bellini Ribeiro, Mikail, Anvesh Mishra, Maciej Olko, Ülgen Sarıkavak, Sage Abdullah Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

A stable version of Psycopg3 has been released for some time and sub-versions are being released which fix minor problems.
https://github.com/psycopg/psycopg

The author of Psycopg3 himself has already created some demonstration branches with the integration test of Django and Psycopg3.
https://www.psycopg.org/articles/2021/08/02/psycopg3-django-driver/

I propose to add a new database backend for PostgreSQL based on Psycopg3 in parallel with the one based on Psycopg2, we can start from the work already done.
https://github.com/dvarrazzo/django-psycopg3-backend

P.S. In the past I believe there was a similar situation when Psycpg2 support was added in parallel with version 1.

Change History (46)

comment:1 by Daniele Varrazzo, 3 years ago

Hello Paolo, thank you for looking into this.

Happy to give you my support, with explanations and eventual changes on the Psycopg side if needed.

comment:2 by Paolo Melchiorre, 3 years ago

Hi Daniele, I think the first step for you can be to rebase your Django fork [1] with the official Django [2] repository and then open a PR from your psycopg3-support branch to the Django main branch.

[1] https://github.com/dvarrazzo/django/tree/psycopg3-support
[2] https://github.com/django/django

comment:3 by Daniele Varrazzo, 3 years ago

Cc: Daniele Varrazzo added

comment:4 by Arsalan Ghassemi, 3 years ago

Triage Stage: UnreviewedAccepted

Adding support for psycopg3 seems relevant since it has a stable release, so I'm tagging it as accepted.

comment:5 by Daniele Varrazzo, 3 years ago

Hello Paolo,

I have rebased my Django branch on current main in a new psycopg3-4.1 branch and I have made the django-psycopg3-backend almost compatible with it: there are a few issues described in this commit message, together with how to run tests.

comment:6 by Paolo Melchiorre, 3 years ago

Hi Daniele

Thank, I'll take a look at both branches and let you know.

comment:7 by אורי, 3 years ago

Cc: אורי added

comment:8 by Adam Johnson, 3 years ago

Cc: Adam Johnson added

comment:9 by Michael P. Jung, 3 years ago

Cc: Michael P. Jung added

comment:10 by Saeed Blanchette, 3 years ago

Cc: Saeed Blanchette added

comment:11 by Florian Apolloner, 3 years ago

Cc: Florian Apolloner added

comment:12 by Adam Wróbel, 3 years ago

Cc: Adam Wróbel added

comment:13 by Andreas Rammhold, 3 years ago

Cc: Andreas Rammhold added

comment:14 by Alexander Nestorov, 3 years ago

Cc: Alexander Nestorov added

comment:15 by Ben Plessinger, 3 years ago

Cc: Ben Plessinger added

comment:16 by Thiago Bellini Ribeiro, 2 years ago

Cc: Thiago Bellini Ribeiro added

comment:17 by Mikail, 2 years ago

Cc: Mikail added

comment:18 by Anvesh Mishra, 2 years ago

Cc: Anvesh Mishra added

comment:19 by Maciej Olko, 2 years ago

Cc: Maciej Olko added

comment:20 by Florian Apolloner, 2 years ago

Given the CCs on this ticket I'd highly appreciate review and tests of the PR in https://github.com/django/django/pull/15687 -- remember the more reviews and tests we get the sooner this will get in :)

comment:21 by Ülgen Sarıkavak, 2 years ago

Cc: Ülgen Sarıkavak added

comment:22 by Mariusz Felisiak <felisiak.mariusz@…>, 2 years ago

In 04518e31:

Refs #33308 -- Enabled explicit GROUP BY and ORDER BY aliases.

This ensures explicit grouping from using values() before annotating an
aggregate function groups by selected aliases if supported.

The GROUP BY feature is disabled on Oracle because it doesn't support it.

comment:23 by Paolo Melchiorre, 2 years ago

Owner: changed from nobody to Florian Apolloner
Status: newassigned

Assign to @apollo13 because he's working on this issue.

comment:24 by GitHub <noreply@…>, 2 years ago

In 7990d25:

Refs #33308 -- Improved adapting DecimalField values to decimal.

comment:25 by Mariusz Felisiak <felisiak.mariusz@…>, 2 years ago

In d87a7b9:

Refs #33308 -- Stopped inheriting from FieldGetDbPrepValueMixin by PostgresOperatorLookup.

comment:26 by Mariusz Felisiak <felisiak.mariusz@…>, 2 years ago

In 5c23d9f:

Refs #33308 -- Used get_db_prep_value() to adapt JSONFields.

comment:27 by Sage Abdullah, 2 years ago

Cc: Sage Abdullah added

comment:28 by Mariusz Felisiak <felisiak.mariusz@…>, 2 years ago

In 3cafb783:

Refs #33308 -- Used psycopg's errors instead of errorcodes.

comment:29 by Mariusz Felisiak <felisiak.mariusz@…>, 2 years ago

In 149b55f:

Refs #33308 -- Ensured type handlers are registered for all PostgreSQL specific tests.

Co-authored-by: Mariusz Felisiak <felisiak.mariusz@…>

comment:30 by Mariusz Felisiak <felisiak.mariusz@…>, 2 years ago

In d3e746ac:

Refs #33308 -- Added get_type_oids() hook and simplified registering type handlers on PostgreSQL.

comment:31 by Mariusz Felisiak <felisiak.mariusz@…>, 2 years ago

In 0ff4659:

Refs #33308 -- Deprecated support for passing encoded JSON string literals to JSONField & co.

JSON should be provided as literal Python objects an not in their
encoded string literal forms.

comment:32 by Mariusz Felisiak <felisiak.mariusz@…>, 2 years ago

In 2fecf99:

Refs #33308 -- Made PostGISAdapter do not use psycopg2's Binary().

comment:33 by אורי, 2 years ago

Cc: אורי removed

comment:34 by Mariusz Felisiak <felisiak.mariusz@…>, 2 years ago

In 3e3b7f6:

Refs #33308 -- Avoided passing None to RawSQL's params.

Passing None to params causes errors in determining the data type on
psycopg3.

comment:35 by Mariusz Felisiak <felisiak.mariusz@…>, 2 years ago

In 6a216581:

Refs #33308 -- Modernized database wrapper in the PostgreSQL backend.

  • Used connection.info instead of connection.get_parameter_status() and connection.server_info which don't exist in psycopg 3.
  • Set encoding using the client_encoding connection parameter instead of connection.set_client_encoding() that doesn't exist in psycopg 3.
  • Used the dbname connection parameter instead of deprecated alias - database.

comment:36 by Mariusz Felisiak <felisiak.mariusz@…>, 2 years ago

In 2ebfbd89:

Refs #33308 -- Moved psycopg2 imports to the psycopg_any module.

comment:37 by Mariusz Felisiak <felisiak.mariusz@…>, 2 years ago

In 2f38f7b:

Refs #33308 -- Added psycopg_any.sql.quote() hook.

comment:38 by Mariusz Felisiak <felisiak.mariusz@…>, 2 years ago

In 1d90c9b:

Refs #33308 -- Added psycopg_any.IsolationLevel.

comment:39 by Mariusz Felisiak <felisiak.mariusz@…>, 2 years ago

In db7bb3b6:

Refs #33308 -- Added DatabaseOperations.compose_sql() on PostgreSQL.

comment:40 by Mariusz Felisiak, 23 months ago

Has patch: set
Triage Stage: AcceptedReady for checkin

comment:41 by Mariusz Felisiak <felisiak.mariusz@…>, 23 months ago

Resolution: fixed
Status: assignedclosed

In 09ffc5c:

Fixed #33308 -- Added support for psycopg version 3.

Thanks Simon Charette, Tim Graham, and Adam Johnson for reviews.

Co-authored-by: Florian Apolloner <florian@…>
Co-authored-by: Mariusz Felisiak <felisiak.mariusz@…>

comment:42 by Mariusz Felisiak <felisiak.mariusz@…>, 21 months ago

In a6511bc:

Refs #33308 -- Added tests for queryset ordered by annotation with nulls_first/nulls_last.

comment:43 by Mariusz Felisiak <felisiak.mariusz@…>, 21 months ago

In 312d0f88:

[4.2.x] Refs #33308 -- Added tests for queryset ordered by annotation with nulls_first/nulls_last.

Backport of a6511bc23329f1d3939571ad01d3176f8f6cb786 from main

comment:44 by GitHub <noreply@…>, 18 months ago

In 0c1518e:

Fixed #34590 -- Reverted "Refs #33308 -- Improved adapting DecimalField values to decimal."

This reverts 7990d254b0af158baf827fafbd90fe8e890f23bd.

Thanks Marc Odermatt for the report.

comment:45 by Mariusz Felisiak <felisiak.mariusz@…>, 18 months ago

In 91f8df5c:

[4.2.x] Fixed #34590 -- Reverted "Refs #33308 -- Improved adapting DecimalField values to decimal."

This reverts 7990d254b0af158baf827fafbd90fe8e890f23bd.

Thanks Marc Odermatt for the report.
Backport of 0c1518ee429b01c145cf5b34eab01b0b92f8c246 from main

comment:46 by Mariusz Felisiak <felisiak.mariusz@…>, 14 months ago

In 5e4c179:

Refs #33308 -- Removed support for passing encoded JSON string literals to JSONField & co.

Per deprecation timeline.

Note: See TracTickets for help on using tickets.
Back to Top