Opened 13 years ago

Closed 13 years ago

Last modified 11 years ago

#16250 closed Bug (fixed)

Error with new pyscopg2 2.4.2 release and tests

Reported by: anonymous Owned by: nobody
Component: Testing framework Version: 1.3
Severity: Release blocker Keywords:
Cc: ampledata, diegueus9@…, Apostolis Bessas, Gábor Farkas, mikeocool, shai@…, simonotron Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Jannis Leidel)

The test command doesn't work with psycopg2==2.4.2 but does with 2.4.1:

Creating test database for alias 'default'...
Traceback (most recent call last):
  File "manage.py", line 14, in <module>
    execute_manager(settings)
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/core/management/__init__.py", line 438, in execute_manager
    utility.execute()
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/core/management/commands/test.py", line 37, in handle
    failures = test_runner.run_tests(test_labels)
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/test/simple.py", line 359, in run_tests
    old_config = self.setup_databases()
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/test/simple.py", line 296, in setup_databases
    test_db_name = connection.creation.create_test_db(self.verbosity, autoclobber=not self.interactive)
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/db/backends/creation.py", line 351, in create_test_db
    self._create_test_db(verbosity, autoclobber)
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/db/backends/creation.py", line 416, in _create_test_db
    self.set_autocommit()
  File "/home/diegueus9/webapps/domicilios/lib/python2.7/site-packages/django/db/backends/creation.py", line 472, in set_autocommit
    self.connection.connection.autocommit = True
psycopg2.ProgrammingError: autocommit cannot be used inside a transaction

Attachments (3)

16250-1.diff (1.4 KB ) - added by Ramiro Morales 13 years ago.
Simple fix
16250-2.diff (2.5 KB ) - added by Ramiro Morales 13 years ago.
More invasive fix
16250-3.diff (2.8 KB ) - added by Ramiro Morales 13 years ago.
Drop .set_autocommit(), replace it with a method that explicitly states what we use it for.

Download all attachments as: .zip

Change History (46)

comment:1 by ampledata, 13 years ago

Cc: ampledata added

Going out on a limb and guessing this is due to the transaction control 'overhaul' in Psycopg 2.4.2:

Transaction control has been overhauled: a new connection method set_session() allows setting all the session properties affecting the transactions behaviour: the isolation level but it can also be used to have auto-commit, read-only, and deferrable transactions.

Para: http://www.initd.org/psycopg/articles/2011/06/12/psycopg-242-released/

comment:2 by piro, 13 years ago

The quick fix seems to reverse the test and check for the existence of set_isolation_level before autocommit.

The proper fix is to add a set_autocommit method to the specific database driver.

comment:3 by Aymeric Augustin, 13 years ago

Triage Stage: UnreviewedAccepted

comment:4 by piro, 13 years ago

Opened a discussion about the issue in the psycopg ML.

http://archives.postgresql.org/psycopg/2011-06/msg00028.php

comment:5 by piro, 13 years ago

Also note that if the exception is raised it means that you have already started a transaction somehow. Asking for autocommit with a transaction already in progress is asking the drivers to be generous and give you a rollback at best. A correct behaviour for a more robust set_autocommit() could be to call connection.rollback() as well.

comment:6 by Jannis Leidel, 13 years ago

Description: modified (diff)

comment:7 by Ramiro Morales, 13 years ago

Severity: Release blockerNormal

comment:8 by Diego Andrés Sanabria Martín, 13 years ago

Cc: diegueus9@… added

comment:9 by Ramiro Morales, 13 years ago

Summary: Error with pyscopg2 and testsError with new pyscopg2 2.4.2 release and app tests

I can reproduce this issue only when running application tests. It doesn't show itself when running the Django test suite.

by Ramiro Morales, 13 years ago

Attachment: 16250-1.diff added

Simple fix

by Ramiro Morales, 13 years ago

Attachment: 16250-2.diff added

More invasive fix

comment:10 by Ramiro Morales, 13 years ago

Has patch: set

comment:11 by Apostolis Bessas, 13 years ago

Cc: Apostolis Bessas added

comment:12 by Gábor Farkas, 13 years ago

Cc: Gábor Farkas added

comment:13 by Russell Keith-Magee, 13 years ago

Severity: NormalRelease blocker

comment:14 by mikeocool, 13 years ago

Cc: mikeocool added

comment:15 by Nicolas Lara, 13 years ago

I would rise NotImplementedError in the base django/db/backends/creation.py instead of just 'passing'

by Ramiro Morales, 13 years ago

Attachment: 16250-3.diff added

Drop .set_autocommit(), replace it with a method that explicitly states what we use it for.

comment:16 by Ramiro Morales, 13 years ago

Summary: Error with new pyscopg2 2.4.2 release and app testsError with new pyscopg2 2.4.2 release and tests

Restoring the summary -- I can see this with the django test suite, most surely I didn't use the right psycopg2 installation when tested things before.

comment:17 by anonymous, 13 years ago

I'm starting with django and wanted to try PostgreSQL database. After seing that ./manage.py runserver does not work, I tried with 'testserver' and found about this problem, but wonder is it just problem in the testsuite or django-1.3 does not work with psycopg2-2.4.2 in general?

comment:18 by gour, 13 years ago

Excuse me that my comment went through as 'anonymous' although it belongs to me. :-(

comment:19 by Aymeric Augustin, 13 years ago

Django 1.3 does not work with psycopg2 2.4.2 in general; use 2.4.1.

comment:20 by Ramiro Morales, 13 years ago

Resolution: fixed
Status: newclosed

In [16520]:

Fixed #16250 -- Made test database creation support code in the PostgreSQL DB backend compatible with psycopg2 2.4.2.

Implemented this by adding an internal hook for work that should be performed
before that point.

Also, regarding the DatabaseCreation.set_autocommit() method:

  • Stop using it for such tasks
  • Stop providing an implementation that tries to cover all the possible idioms a third party database backend DB-API 2 driver could need to activate autocommit. It is now left for third party backends to implement.

This can be backwards incompatible in the case of user applications that:

  • Had started using this method
  • Use a third a party database backend

comment:21 by Shai Berger, 13 years ago

Cc: shai@… added

Just for the record, the fix was not included in Django 1.3.1; we still need 2.4.1.

comment:22 by anonymous, 13 years ago

As a Django 1.3 user, how am I supposed to install psycopg2 2.4.1? It doesn't appear on PyPI, nor did I find it anywhere else. I need Windows binaries for it. Are you trying to keep it a secret?...

in reply to:  22 comment:23 by Ramiro Morales, 13 years ago

Replying to anonymous:

As a Django 1.3 user, how am I supposed to install psycopg2 2.4.1? It doesn't appear on PyPI, nor did I find it anywhere else. I need Windows binaries for it. Are you trying to keep it a secret?...

Why don't you ask psycopg2 questions on the psycopg2 user support channels and stop trolling (even worse, trolling as an anonymous user) here?.

comment:24 by anonymous, 12 years ago

Nonetheless, anonynmous's point stands that the fix should be included in the next Django 1.3

comment:25 by anonymous, 12 years ago

And anonymous, you can find 2.4.1 on pypi:

pip install psycopg2==2.4.1

comment:26 by Luke Plant, 12 years ago

Bug fixes like this are no longer backported to the stable branch - it will not be in the next 1.3.X release. See https://docs.djangoproject.com/en/dev/internals/release-process/#supported-versions

This bug is not 'critical', because it can be avoided by using a different version of psycopg2.

in reply to:  19 comment:27 by Donald Stufft, 12 years ago

Replying to aaugustin:

Django 1.3 does not work with psycopg2 2.4.2 in general; use 2.4.1.

I just talked to aaugustin about this statement. He said that his initial impression of the extent of Django and psycopg2 incompatibility. was incorrect. The only place in Django that is affected by the changes made in psycopg2 2.4.2 is the test runner, so psycopg2 is fine to use in cases where you do not need to run the tests (such as when deploying to production servers).

comment:28 by Aymeric Augustin, 12 years ago

I confirm — I can't figure out why I wrote this. Maybe my subconscious thinks that Django is unusable without the test runner :)

Thanks dstufft for correcting me.

comment:29 by simonotron, 12 years ago

Cc: simonotron added

comment:30 by yodiaditya, 12 years ago

When it will included in django 1.3.1 ?

in reply to:  30 comment:31 by jakewan, 12 years ago

Replying to yodiaditya:

When it will included in django 1.3.1 ?

See comment 26

comment:32 by dpifke, 12 years ago

Ubuntu users running the latest release (Oneiric) will encounter this issue, since Ubuntu ships Django 1.3.1 and Psycopg2 2.4.2.

This has been reported to Ubuntu as issue 905837:

https://bugs.launchpad.net/ubuntu/+source/python-django/+bug/905837

comment:33 by Antti Kaihola, 12 years ago

A quick copy-paste script for replacing the psycopg2 Debian package with 2.4.1 from PyPI on Ubuntu 11.10 and distributions based on it (like Linux Mint 12):

sudo apt-get remove python-psycopg2
sudo apt-get install -y python-dev libpq-dev
sudo apt-get install -y python-pip  # if not installed manually
sudo pip install psycopg2==2.4.1

comment:34 by ncikic, 12 years ago

May I ask a clarification as new djang/python developer about this issue?
Is the recommended solution to apply the fixes as in Changeset 16520 by Mr. ramiro or should i revert to psycopg2 2.4.1?
What I need is the clarification about the following comment in Changeset 16520
"Also, regarding the DatabaseCreation.set_autocommit() method:

Stop using it for such tasks
Stop providing an implementation that tries to cover all the possible idioms a third party database backend DB-API 2 driver could need to activate autocommit. It is now left for third party backends to implement. "

Does this mean the psycopg2 2.4.2 introduced the bug or it is django bug?

So the point is: Will the Changeset 16520 be part of the next django release or it is just a quick fix and for the next release it is up to psycopg2 developers to fix the problem definitely (note that at present the last psycopg2 is 2.4.4)?
Thanks

Last edited 12 years ago by ncikic (previous) (diff)

comment:35 by Aymeric Augustin, 12 years ago

The easiest solution is to use psycopg2 2.4.1. Note that the issue only appears when your run tests.

in reply to:  35 ; comment:36 by anonymous, 12 years ago

Replying to aaugustin:

The easiest solution is to use psycopg2 2.4.1. Note that the issue only appears when your run tests.

When I use psycopg2 2.4.1, I get:

django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: cannot import name _connect

In fact I seem to get that with anything less than the latest psycopg2. I'm using pgsql-9.1 and the latest version of django.

in reply to:  36 comment:37 by Aymeric Augustin, 12 years ago

Replying to anonymous:

When I use psycopg2 2.4.1, I get:
django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: cannot import name _connect


I just encountered this issue after downgrading psycopg2 to 2.4.1 on Django's own CI server.

I had to wipe previous installations:

rm -r /usr/lib/python2.5/site-packages/psycopg2*
rm -r /usr/local/lib/python2.6/dist-packages/psycopg2*
rm -r /usr/local/lib/python2.7/dist-packages/psycopg2*

and then reinstall psycopg2.

comment:38 by zap.aibulatov@…, 12 years ago

I have installed psycopg2==2.4.1 via pip on my virtualenv, and now i have this error:

Traceback (most recent call last):
  File "./manage.py", line 9, in <module>
    execute_from_command_line(sys.argv)
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 429, in execute_from_command_line
    utility.execute()
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 261, in fetch_command
    klass = load_command_class(app_name, subcommand)
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 67, in load_command_class
    module = import_module('%s.management.commands.%s' % (app_name, name))
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/south/management/commands/__init__.py", line 10, in <module>
    import django.template.loaders.app_directories
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/template/loaders/app_directories.py", line 21, in <module>
    mod = import_module(app)
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/contrib/admin/__init__.py", line 3, in <module>
    from django.contrib.admin.helpers import ACTION_CHECKBOX_NAME
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/contrib/admin/helpers.py", line 3, in <module>
    from django.contrib.admin.util import (flatten_fieldsets, lookup_field,
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/contrib/admin/util.py", line 1, in <module>
    from django.db import models
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/db/__init__.py", line 78, in <module>
    connection = connections[DEFAULT_DB_ALIAS]
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/db/utils.py", line 93, in __getitem__
    backend = load_backend(db['ENGINE'])
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/db/utils.py", line 33, in load_backend
    return import_module('.base', backend_name)
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/home/set/Envs/ruelsoft.com/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 24, in <module>
    raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e)
django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: can't import mx.DateTime module

comment:40 by dave@…, 11 years ago

I have django 1.3.1 installed as well as psycopg-2.4.1 and I still get the exact same error above.

Is there any way to work around this to get my unit tests to work?

comment:41 by Aymeric Augustin, 11 years ago

Upgrade to 1.3.5.

comment:42 by dave@…, 11 years ago

I've upgraded to 1.3.5. Same problem.

comment:43 by Aymeric Augustin, 11 years ago

Sorry, I confused this ticket with another one.

Either the version of psycopg you have isn't 2.4.1, or you're seeing an different problem - even if the traceback is similar.

Note: See TracTickets for help on using tickets.
Back to Top