Opened 5 years ago

Closed 5 years ago

Last modified 22 months ago

#27996 closed New feature (fixed)

Add pgcrypto extension and GEN_RANDOM_UUID function to contrib.postgres

Reported by: Paolo Melchiorre Owned by: Paolo Melchiorre
Component: contrib.postgres Version: dev
Severity: Normal Keywords: uuid extension function random postgresql cryptography
Cc: Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Paolo Melchiorre)

After the introduction of the UUID Field in Django 1.8, I believe that django.contrib.postgres could benefit from some custom functions based on the pgcrypto extension of PostgreSQL (see https://www.postgresql.org/docs/9.6/static/pgcrypto.html). That kind of functions would be very helpful for apply a migration that adds a unique non-nullable field to a table with existing rows.

Starting from "Migrations that add unique fields" (see https://docs.djangoproject.com/en/dev/howto/writing-migrations/#migrations-that-add-unique-fields) I speed up the gen_uuid using GEN_RANDOM_UUID function changing it from:

import uuid

def gen_uuid(apps, schema_editor):
    MyModel = apps.get_model('myapp', 'MyModel')
    for row in MyModel.objects.all():
        row.uuid = uuid.uuid4()
        row.save(update_fields=['uuid'])

to

from django.contrib.postgres.functions import RandomUUID

def gen_uuid(apps, schema_editor):
    MyModel = apps.get_model('myapp', 'MyModel')
    MyModel.objects.update(uuid=RandomUUID())

Using this function on my system the time to migrate more than 10000 objects decreased from

real 0m15.988s
user 0m10.680s
sys 0m0.508s

to

real 0m2.957s
user 0m1.736s
sys 0m0.072s

I already implemented a solution for thi feature and I've created a PR

Change History (15)

comment:1 Changed 5 years ago by Paolo Melchiorre

I just submitted a full featured pull request on ghithub https://github.com/django/django/pull/8265

comment:2 Changed 5 years ago by Claude Paroz

Triage Stage: UnreviewedAccepted

comment:3 Changed 5 years ago by Paolo Melchiorre

Description: modified (diff)

comment:4 Changed 5 years ago by Paolo Melchiorre

Has patch: unset

comment:5 Changed 5 years ago by Paolo Melchiorre

Description: modified (diff)
Has patch: set

comment:6 Changed 5 years ago by Markus Holtermann

Patch needs improvement: set

comment:7 Changed 5 years ago by Paolo Melchiorre

Patch needs improvement: unset

comment:8 Changed 5 years ago by Marc Tamlyn

Triage Stage: AcceptedReady for checkin

comment:9 Changed 5 years ago by Tim Graham <timograham@…>

Resolution: fixed
Status: assignedclosed

In fcb5dbfe:

Fixed #27996 -- Added RandomUUID function and CryptoExtension to contrib.postgres.

comment:10 Changed 5 years ago by GitHub <noreply@…>

In 650bf671:

Refs #27996 -- Skipped RandomUUID test on PostgreSQL 9.3.

comment:11 Changed 22 months ago by Mariusz Felisiak <felisiak.mariusz@…>

comment:12 Changed 22 months ago by Iuri de Silvio

Hi! Today I updated my master and test migrations failed because of https://github.com/django/django/pull/13241 . You checked for is_postgresql_13 even if the connection is not postgresql. It fails with psycopg2 installed but using another db (sqlite in my case).

comment:13 Changed 22 months ago by Iuri de Silvio

For completeness, here is the stacktrace.

Traceback (most recent call last):
  File "runtests.py", line 571, in <module>
    failures = django_tests(
  File "runtests.py", line 313, in django_tests
    failures = test_runner.run_tests(
  File "django/django/test/runner.py", line 707, in run_tests
    old_config = self.setup_databases(aliases=databases)
  File "django/django/test/runner.py", line 626, in setup_databases
    return _setup_databases(
  File "django/django/test/utils.py", line 170, in setup_databases
    connection.creation.create_test_db(
  File "django/django/db/backends/base/creation.py", line 65, in create_test_db
    call_command(
  File "django/django/core/management/__init__.py", line 168, in call_command
    return command.execute(*args, **defaults)
  File "django/django/core/management/base.py", line 394, in execute
    output = self.handle(*args, **options)
  File "django/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "django/django/core/management/commands/migrate.py", line 92, in handle
    executor = MigrationExecutor(connection, self.migration_progress_callback)
  File "django/django/db/migrations/executor.py", line 18, in __init__
    self.loader = MigrationLoader(self.connection)
  File "django/django/db/migrations/loader.py", line 53, in __init__
    self.build_graph()
  File "django/django/db/migrations/loader.py", line 210, in build_graph
    self.load_disk()
  File "django/django/db/migrations/loader.py", line 112, in load_disk
    migration_module = import_module(migration_path)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "django/tests/postgres_tests/migrations/0001_setup_extensions.py", line 22, in <module>
    needs_crypto_extension = not connection.features.is_postgresql_13
AttributeError: 'DatabaseFeatures' object has no attribute 'is_postgresql_13'

comment:14 Changed 22 months ago by Mariusz Felisiak

Iuri, thanks for the report, fix is ready PR.

comment:15 Changed 22 months ago by GitHub <noreply@…>

In 88af11c:

Refs #27996 -- Fixed postgres_tests crash if not running with PostgreSQL.

Note: See TracTickets for help on using tickets.
Back to Top