Code

Opened 3 years ago

Closed 3 years ago

Last modified 18 months ago

#16250 closed Bug (fixed)

Error with new pyscopg2 2.4.2 release and tests

Reported by: anonymous Owned by: nobody
Component: Testing framework Version: 1.3
Severity: Release blocker Keywords:
Cc: ampledata, diegueus9@…, mpessas, gabor, mikeocool, shai@…, simonotron Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by jezdez)

The test command doesn't work with psycopg2==2.4.2 but does with 2.4.1:

Creating test database for alias 'default'...
Traceback (most recent call last):
  File "manage.py", line 14, in <module>
    execute_manager(settings)
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/core/management/__init__.py", line 438, in execute_manager
    utility.execute()
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/core/management/commands/test.py", line 37, in handle
    failures = test_runner.run_tests(test_labels)
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/test/simple.py", line 359, in run_tests
    old_config = self.setup_databases()
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/test/simple.py", line 296, in setup_databases
    test_db_name = connection.creation.create_test_db(self.verbosity, autoclobber=not self.interactive)
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/db/backends/creation.py", line 351, in create_test_db
    self._create_test_db(verbosity, autoclobber)
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/db/backends/creation.py", line 416, in _create_test_db
    self.set_autocommit()
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/db/backends/creation.py", line 472, in set_autocommit
    self.connection.connection.autocommit = True
psycopg2.ProgrammingError: autocommit cannot be used inside a transaction

Attachments (3)

16250-1.diff (1.4 KB) - added by ramiro 3 years ago.
Simple fix
16250-2.diff (2.5 KB) - added by ramiro 3 years ago.
More invasive fix
16250-3.diff (2.8 KB) - added by ramiro 3 years ago.
Drop .set_autocommit(), replace it with a method that explicitly states what we use it for.

Download all attachments as: .zip

Change History (46)

comment:1 Changed 3 years ago by ampledata

  • Cc ampledata added
  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

Going out on a limb and guessing this is due to the transaction control 'overhaul' in Psycopg 2.4.2:

Transaction control has been overhauled: a new connection method set_session() allows setting all the session properties affecting the transactions behaviour: the isolation level but it can also be used to have auto-commit, read-only, and deferrable transactions.

Para: http://www.initd.org/psycopg/articles/2011/06/12/psycopg-242-released/

comment:2 Changed 3 years ago by piro

The quick fix seems to reverse the test and check for the existence of set_isolation_level before autocommit.

The proper fix is to add a set_autocommit method to the specific database driver.

comment:3 Changed 3 years ago by aaugustin

  • Triage Stage changed from Unreviewed to Accepted

comment:4 Changed 3 years ago by piro

Opened a discussion about the issue in the psycopg ML.

http://archives.postgresql.org/psycopg/2011-06/msg00028.php

comment:5 Changed 3 years ago by piro

Also note that if the exception is raised it means that you have already started a transaction somehow. Asking for autocommit with a transaction already in progress is asking the drivers to be generous and give you a rollback at best. A correct behaviour for a more robust set_autocommit() could be to call connection.rollback() as well.

comment:6 Changed 3 years ago by jezdez

  • Description modified (diff)

comment:7 Changed 3 years ago by ramiro

  • Severity changed from Release blocker to Normal

comment:8 Changed 3 years ago by diegueus9

  • Cc diegueus9@… added

comment:9 Changed 3 years ago by ramiro

  • Summary changed from Error with pyscopg2 and tests to Error with new pyscopg2 2.4.2 release and app tests

I can reproduce this issue only when running application tests. It doesn't show itself when running the Django test suite.

Changed 3 years ago by ramiro

Simple fix

Changed 3 years ago by ramiro

More invasive fix

comment:10 Changed 3 years ago by ramiro

  • Has patch set

comment:11 Changed 3 years ago by mpessas

  • Cc mpessas added

comment:12 Changed 3 years ago by gabor

  • Cc gabor added

comment:13 Changed 3 years ago by russellm

  • Severity changed from Normal to Release blocker

comment:14 Changed 3 years ago by mikeocool

  • Cc mikeocool added

comment:15 Changed 3 years ago by nicolas

I would rise NotImplementedError in the base django/db/backends/creation.py instead of just 'passing'

Changed 3 years ago by ramiro

Drop .set_autocommit(), replace it with a method that explicitly states what we use it for.

comment:16 Changed 3 years ago by ramiro

  • Summary changed from Error with new pyscopg2 2.4.2 release and app tests to Error with new pyscopg2 2.4.2 release and tests

Restoring the summary -- I can see this with the django test suite, most surely I didn't use the right psycopg2 installation when tested things before.

comment:17 Changed 3 years ago by anonymous

I'm starting with django and wanted to try PostgreSQL database. After seing that ./manage.py runserver does not work, I tried with 'testserver' and found about this problem, but wonder is it just problem in the testsuite or django-1.3 does not work with psycopg2-2.4.2 in general?

comment:18 Changed 3 years ago by gour

Excuse me that my comment went through as 'anonymous' although it belongs to me. :-(

comment:19 follow-up: Changed 3 years ago by aaugustin

Django 1.3 does not work with psycopg2 2.4.2 in general; use 2.4.1.

comment:20 Changed 3 years ago by ramiro

  • Resolution set to fixed
  • Status changed from new to closed

In [16520]:

Fixed #16250 -- Made test database creation support code in the PostgreSQL DB backend compatible with psycopg2 2.4.2.

Implemented this by adding an internal hook for work that should be performed
before that point.

Also, regarding the DatabaseCreation.set_autocommit() method:

  • Stop using it for such tasks
  • Stop providing an implementation that tries to cover all the possible idioms a third party database backend DB-API 2 driver could need to activate autocommit. It is now left for third party backends to implement.

This can be backwards incompatible in the case of user applications that:

  • Had started using this method
  • Use a third a party database backend

comment:21 Changed 3 years ago by shai

  • Cc shai@… added

Just for the record, the fix was not included in Django 1.3.1; we still need 2.4.1.

comment:22 follow-up: Changed 3 years ago by anonymous

As a Django 1.3 user, how am I supposed to install psycopg2 2.4.1? It doesn't appear on PyPI, nor did I find it anywhere else. I need Windows binaries for it. Are you trying to keep it a secret?...

comment:23 in reply to: ↑ 22 Changed 3 years ago by ramiro

Replying to anonymous:

As a Django 1.3 user, how am I supposed to install psycopg2 2.4.1? It doesn't appear on PyPI, nor did I find it anywhere else. I need Windows binaries for it. Are you trying to keep it a secret?...

Why don't you ask psycopg2 questions on the psycopg2 user support channels and stop trolling (even worse, trolling as an anonymous user) here?.

comment:24 Changed 3 years ago by anonymous

Nonetheless, anonynmous's point stands that the fix should be included in the next Django 1.3

comment:25 Changed 3 years ago by anonymous

And anonymous, you can find 2.4.1 on pypi:

pip install psycopg2==2.4.1

comment:26 Changed 3 years ago by lukeplant

Bug fixes like this are no longer backported to the stable branch - it will not be in the next 1.3.X release. See https://docs.djangoproject.com/en/dev/internals/release-process/#supported-versions

This bug is not 'critical', because it can be avoided by using a different version of psycopg2.

comment:27 in reply to: ↑ 19 Changed 3 years ago by dstufft

Replying to aaugustin:

Django 1.3 does not work with psycopg2 2.4.2 in general; use 2.4.1.

I just talked to aaugustin about this statement. He said that his initial impression of the extent of Django and psycopg2 incompatibility. was incorrect. The only place in Django that is affected by the changes made in psycopg2 2.4.2 is the test runner, so psycopg2 is fine to use in cases where you do not need to run the tests (such as when deploying to production servers).

comment:28 Changed 3 years ago by aaugustin

I confirm — I can't figure out why I wrote this. Maybe my subconscious thinks that Django is unusable without the test runner :)

Thanks dstufft for correcting me.

comment:29 Changed 3 years ago by simonotron

  • Cc simonotron added

comment:30 follow-up: Changed 3 years ago by yodiaditya

When it will included in django 1.3.1 ?

comment:31 in reply to: ↑ 30 Changed 3 years ago by jakewan

Replying to yodiaditya:

When it will included in django 1.3.1 ?

See comment 26

comment:32 Changed 3 years ago by dpifke

Ubuntu users running the latest release (Oneiric) will encounter this issue, since Ubuntu ships Django 1.3.1 and Psycopg2 2.4.2.

This has been reported to Ubuntu as issue 905837:

https://bugs.launchpad.net/ubuntu/+source/python-django/+bug/905837

comment:33 Changed 3 years ago by akaihola

A quick copy-paste script for replacing the psycopg2 Debian package with 2.4.1 from PyPI on Ubuntu 11.10 and distributions based on it (like Linux Mint 12):

sudo apt-get remove python-psycopg2
sudo apt-get install -y python-dev libpq-dev
sudo apt-get install -y python-pip  # if not installed manually
sudo pip install psycopg2==2.4.1

comment:34 Changed 2 years ago by ncikic

May I ask a clarification as new djang/python developer about this issue?
Is the recommended solution to apply the fixes as in Changeset 16520 by Mr. ramiro or should i revert to psycopg2 2.4.1?
Thanks

Version 0, edited 2 years ago by ncikic (next)

comment:35 follow-up: Changed 2 years ago by aaugustin

The easiest solution is to use psycopg2 2.4.1. Note that the issue only appears when your run tests.

comment:36 in reply to: ↑ 35 ; follow-up: Changed 2 years ago by anonymous

Replying to aaugustin:

The easiest solution is to use psycopg2 2.4.1. Note that the issue only appears when your run tests.

When I use psycopg2 2.4.1, I get:

django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: cannot import name _connect

In fact I seem to get that with anything less than the latest psycopg2. I'm using pgsql-9.1 and the latest version of django.

comment:37 in reply to: ↑ 36 Changed 2 years ago by aaugustin

Replying to anonymous:

When I use psycopg2 2.4.1, I get:
django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: cannot import name _connect


I just encountered this issue after downgrading psycopg2 to 2.4.1 on Django's own CI server.

I had to wipe previous installations:

rm -r /usr/lib/python2.5/site-packages/psycopg2*
rm -r /usr/local/lib/python2.6/dist-packages/psycopg2*
rm -r /usr/local/lib/python2.7/dist-packages/psycopg2*

and then reinstall psycopg2.

comment:38 Changed 2 years ago by zap.aibulatov@…

I have installed psycopg2==2.4.1 via pip on my virtualenv, and now i have this error:

Traceback (most recent call last):
  File "./manage.py", line 9, in <module>
    execute_from_command_line(sys.argv)
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 429, in execute_from_command_line
    utility.execute()
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 261, in fetch_command
    klass = load_command_class(app_name, subcommand)
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 67, in load_command_class
    module = import_module('%s.management.commands.%s' % (app_name, name))
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/south/management/commands/__init__.py", line 10, in <module>
    import django.template.loaders.app_directories
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/template/loaders/app_directories.py", line 21, in <module>
    mod = import_module(app)
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/contrib/admin/__init__.py", line 3, in <module>
    from django.contrib.admin.helpers import ACTION_CHECKBOX_NAME
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/contrib/admin/helpers.py", line 3, in <module>
    from django.contrib.admin.util import (flatten_fieldsets, lookup_field,
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/contrib/admin/util.py", line 1, in <module>
    from django.db import models
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/db/__init__.py", line 78, in <module>
    connection = connections[DEFAULT_DB_ALIAS]
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/db/utils.py", line 93, in __getitem__
    backend = load_backend(db['ENGINE'])
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/db/utils.py", line 33, in load_backend
    return import_module('.base', backend_name)
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 24, in <module>
    raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e)
django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: can't import mx.DateTime module

comment:40 Changed 18 months ago by dave@…

I have django 1.3.1 installed as well as psycopg-2.4.1 and I still get the exact same error above.

Is there any way to work around this to get my unit tests to work?

comment:41 Changed 18 months ago by aaugustin

Upgrade to 1.3.5.

comment:42 Changed 18 months ago by dave@…

I've upgraded to 1.3.5. Same problem.

comment:43 Changed 18 months ago by aaugustin

Sorry, I confused this ticket with another one.

Either the version of psycopg you have isn't 2.4.1, or you're seeing an different problem - even if the traceback is similar.

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.