Support use of pypgsql instead of psycopg

[tristanlawrence@…]

I do not see any information on the generally excellent Django website about
alternatives to psychopg as a PostGreSQL connector. I have been having some
non-Django problems getting psychopg to work so it would be nice if there
was another option such as pypgsql.

While researching I did find this page: ​http://blog.vrplumber.com/968 so it
seems that I am not the only one with this request.

[C8E]

Hi,

I do not see any information on the generally excellent Django website about alternatives to psychopg as a PostGreSQL connector. I have been having some non-Django problems getting psychopg to work so it would be nice if there was another option such as pypgsql.

Wich problems did you have? psycopg rocks :-D
If it was only a licence issue, please see below.

While researching I did find this page: ​http://blog.vrplumber.com/968 so it seems that I am not the only one with this request.

Thank you for the signalation. I've submitted it to Federico (the main psycopg author) and he reply with a comment on the page.

I've discussed this issue with him some time ago. The reason why psycopg is GPL'd is only to avoid that non free drivers will be derived from it. I can guarantee that nobody in initd will pursue no one for *using* it in non-GPL projects.

We recently rewrote the license of psycopg2 in order to explain that. See:

​http://initd.org/tracker/psycopg/file/psycopg2/trunk/LICENSE

This is the intended behavior of the license also for psycopg1, even if not explicitly stated. Or, if that make you feel better, you can use psycopg2 with Django with *one* of the two trivial patches

1. using psycopg2 -> psycopg1 compatiblity layer

   Index: django/trunk/django/core/db/backends/postgresql.py
   ===================================================================
   --- django/trunk/django/core/db/backends/postgresql.py  (revision 618)
   +++ django/trunk/django/core/db/backends/postgresql.py  (local copy)
   @@ -5,7 +5,7 @@
    """
   
    from django.core.db import base, typecasts
   -import psycopg as Database
   +import psycopg2.psycopg1 as Database
   
    DatabaseError = Database.DatabaseError
   
   

2. using either psycopg1 or psycopg2 full power, at your choice (untested)

   Index: django/trunk/django/core/db/backends/postgresql.py
   ===================================================================
   --- django/trunk/django/core/db/backends/postgresql.py  (revision 618)
   +++ django/trunk/django/core/db/backends/postgresql.py  (local copy)
   @@ -5,7 +5,12 @@
    """
   
    from django.core.db import base, typecasts
   -import psycopg as Database
   +try:
   +    import psycopg2 as Database
   +    from psycopg2.extensions import register_type
   +except ImportError:
   +    import psycopg as Database
   +    from psycopg import register_type
   
    DatabaseError = Database.DatabaseError
   
   @@ -115,13 +120,10 @@
    # Register these custom typecasts, because Django expects dates/times to be
    # in Python's native (standard-library) datetime/time format, whereas psycopg
    # use mx.DateTime by default.
   -try:
   -    Database.register_type(Database.new_type((1082,), "DATE", typecasts.typecast_date))
   -except AttributeError:
   -    raise Exception, "You appear to be using psycopg version 2, which isn't supported yet, because it's still in beta. Use psycopg version 1 instead: http://initd.org/projects/psycopg1"
   -Database.register_type(Database.new_type((1083,1266), "TIME", typecasts.typecast_time))
   -Database.register_type(Database.new_type((1114,1184), "TIMESTAMP", typecasts.typecast_timestamp))
   -Database.register_type(Database.new_type((16,), "BOOLEAN", typecasts.typecast_boolean))
   +register_type(Database.new_type((1082,), "DATE", typecasts.typecast_date))
   +register_type(Database.new_type((1083,1266), "TIME", typecasts.typecast_time))
   +register_type(Database.new_type((1114,1184), "TIMESTAMP", typecasts.typecast_timestamp))
   +register_type(Database.new_type((16,), "BOOLEAN", typecasts.typecast_boolean))
   
    OPERATOR_MAPPING = {
        'exact': '=',
   

[tristanlawrence@…]

Tried to post to "which problem" to the mailing list. That failed as I am not
subscribed, so I thought I would put it here.

My first problem was that blastwave.org does not have a psycopg package for
Solaris. But then I did a source install only to have it keep coredumping on me :(

psycopg build command:

bash-2.03# ./configure
--with-postgres-libraries=/opt/csw/postgresql/lib
--with-postgres-includes=/opt/csw/postgresql/include
--with-mxdatetime-includes=/opt/csw/lib/python2.3/site-packages/mx/DateTime/mxDateTime
--enable-devel=yes --with-python=/opt/csw/bin/python
--with-python-version=2.3

last few lines of output from:

bash-2.03# /opt/csw/lib/python2.3/site-packages/django/bin/django-admin.py
init --settings=game.settings.main

(...)
[14512] psyco_connect(): dsn = 'user=webuser dbname=gamedb', serialize = 1
[14512] psyco_connect(): minconn = 8, maxconn = 64
[14512] new_psyco_connobject(): creating new connection
[14512] new_psyco_cursobject: new cursor, keeper = 0
[14512] request_pgconn: maxconn = 64, minconn = 8, openconn = 0, availconn = 0
[14512] alloc_keeper: opening new postgresql connection
[14512] alloc_keeper: dsn = user=webuser dbname=gamedb (144778)
[14512] alloc_keeper: new posgresql connection at 297658
[14512] alloc_keeper: connection protocol version: 3
[14512] alloc_keeper: setting datestyle to iso
[14512] alloc_keeper: datestyle query executed
[14512] alloc_keeper: keeper allocated at 2eae68
[14512] request_pgconn: cursor at 2728e0 using keeper at 2eae68,
connection at 297658
[14512] new_psyco_cursobject: cursor created at 2728e0, refcnt = 1
[14512] new_psyco_connobject(): created connobject at 21f2a8, refcnt = 1
[14512] new_psyco_connobject(): stdmanager = 2728e0, stdkeeper = 2eae68
[14512] curs_switch_isolation_level: pgconn = 297658, level = 1, status = 0
[14512] psyco_conn_cursor(): conn = 21f2a8, name = Segmentation Fault
(core dumped)

It is not a django issue because I get a similar error when I try to connect to
PostGres interactively:

connection = psycopg.connect('dbname=gamedb user=webuser')

[17041] psyco_connect(): dsn = 'dbname=gamedb user=webuser', serialize = 1
[17041] psyco_connect(): minconn = 8, maxconn = 64
[17041] new_psyco_connobject(): creating new connection
[17041] new_psyco_cursobject: new cursor, keeper = 0
[17041] request_pgconn: maxconn = 64, minconn = 8, openconn = 0, availconn = 0
[17041] alloc_keeper: opening new postgresql connection
[17041] alloc_keeper: dsn = dbname=gamedb user=webuser (144228)
[17041] alloc_keeper: new posgresql connection at 1f2648
[17041] alloc_keeper: connection protocol version: 3
[17041] alloc_keeper: setting datestyle to iso
[17041] alloc_keeper: datestyle query executed
[17041] alloc_keeper: keeper allocated at 1506a8
[17041] request_pgconn: cursor at 114f20 using keeper at 1506a8,
connection at 1f2648
[17041] new_psyco_cursobject: cursor created at 114f20, refcnt = 1
[17041] new_psyco_connobject(): created connobject at 179848, refcnt = 1
[17041] new_psyco_connobject(): stdmanager = 114f20, stdkeeper = 1506a8

cursor = connection.cursor( )

[17041] psyco_conn_cursor(): conn = 179848, name = Segmentation Fault
(core dumped)
bash-2.03#

gdb on the core file didn't have anything interesting:

bash-2.03# gdb -c core
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "sparc-sun-solaris2.8".
Core was generated by `python'.
Program terminated with signal 11, Segmentation fault.
#0 0xff1331f0 in ?? ()
(gdb) bt
#0 0xff1331f0 in ?? ()
#1 0xff1862b8 in ?? ()
Previous frame identical to this frame (corrupt stack?)
(gdb) quit

And I can connect directly to Postgres using the command line client
without a problem:

bash-2.03$ bin/psql -U webuser -d gamedb
Welcome to psql 8.0.3, the PostgreSQL interactive terminal.

Type: \copyright for distribution terms

\h for help with SQL commands
\? for help with psql commands
\g or terminate with semicolon to execute query
\q to quit

gamedb=>
gamedb=> select count(*) from information_schema.tables where
table_catalog='gamedb';

count

---

92

(1 row)

gamedb=>

:/

Thanks,
Tristan

[tristanlawrence@…]

Got an email from C8E recommending that I try the mxDateTime fix from:

​http://lists.initd.org/pipermail/psycopg/2003-March/001920.html

Verified that the problem was still happening, removed mx and Psycopg
from site-packages, cleaned the two source directories, applied the
code change from the web page. I am using mx v2.0.6 rather than 2.0.4
as on that page, but I found the two define lines right as predicted.

Commented them out, recompiled mx, recompiled psycopg. Same problem:

bash-2.03# python
Python 2.3.4 (#1, Jun 29 2005, 14:00:38) [C] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> import psycopg
[21613] initpsycopg: mxDateTime module imported at fe6d734c
[21613] initpsycopg: module initialized
[21613] initpsycopg: parameters initialized
[21613] psyco_init_types: initializing NUMBER
[21613] new_psyco_typeobject: type tuple created at 17cce0
[21613] psyco_DBAPITypeObject_new: name = 1909a0, values = 17cce0, pcast = 0, ccast = 0
[21613] psyco_DBAPITypeObject_new: values refcnt = 2
[21613] psyco_DBAPITypeObject_new: object created at 10b128
[21613] psyco_add_type: typeobject = 10b128, values refcnt = 2
[21613] psyco_add_type: val = 20
[21613] psyco_add_type: val = 23
[21613] psyco_add_type: val = 21
[21613] psyco_add_type: val = 701
[21613] psyco_add_type: val = 700
[21613] psyco_add_type: val = 1700
[21613] psyco_init_types: initializing LONGINTEGER
[21613] new_psyco_typeobject: type tuple created at 17dc50
[21613] psyco_DBAPITypeObject_new: name = 18ed18, values = 17dc50, pcast = 0, ccast = 0
[21613] psyco_DBAPITypeObject_new: values refcnt = 2
[21613] psyco_DBAPITypeObject_new: object created at 10b170
[21613] psyco_add_type: typeobject = 10b170, values refcnt = 2
[21613] psyco_add_type: val = 20
[21613] psyco_init_types: initializing INTEGER
[21613] new_psyco_typeobject: type tuple created at 18e800
[21613] psyco_DBAPITypeObject_new: name = 1909c0, values = 18e800, pcast = 0, ccast = 0
[21613] psyco_DBAPITypeObject_new: values refcnt = 2
[21613] psyco_DBAPITypeObject_new: object created at 10b188
[21613] psyco_add_type: typeobject = 10b188, values refcnt = 2
[21613] psyco_add_type: val = 23
[21613] psyco_add_type: val = 21
[21613] psyco_init_types: initializing FLOAT
[21613] new_psyco_typeobject: type tuple created at 182468
[21613] psyco_DBAPITypeObject_new: name = 1909e0, values = 182468, pcast = 0, ccast = 0
[21613] psyco_DBAPITypeObject_new: values refcnt = 2
[21613] psyco_DBAPITypeObject_new: object created at 10b1a0
[21613] psyco_add_type: typeobject = 10b1a0, values refcnt = 2
[21613] psyco_add_type: val = 701
[21613] psyco_add_type: val = 700
[21613] psyco_add_type: val = 1700
[21613] psyco_init_types: initializing STRING
[21613] new_psyco_typeobject: type tuple created at 1840f0
[21613] psyco_DBAPITypeObject_new: name = 190a00, values = 1840f0, pcast = 0, ccast = 0
[21613] psyco_DBAPITypeObject_new: values refcnt = 2
[21613] psyco_DBAPITypeObject_new: object created at 10b1b8
[21613] psyco_add_type: typeobject = 10b1b8, values refcnt = 2
[21613] psyco_add_type: val = 19
[21613] psyco_add_type: val = 18
[21613] psyco_add_type: val = 25
[21613] psyco_add_type: val = 1042
[21613] psyco_add_type: val = 1043
[21613] psyco_init_types: initializing BOOLEAN
[21613] new_psyco_typeobject: type tuple created at 13e7f0
[21613] psyco_DBAPITypeObject_new: name = 190a20, values = 13e7f0, pcast = 0, ccast = 0
[21613] psyco_DBAPITypeObject_new: values refcnt = 2
[21613] psyco_DBAPITypeObject_new: object created at 10b1d0
[21613] psyco_add_type: typeobject = 10b1d0, values refcnt = 2
[21613] psyco_add_type: val = 16
[21613] psyco_init_types: initializing DATETIME
[21613] new_psyco_typeobject: type tuple created at 111e30
[21613] psyco_DBAPITypeObject_new: name = 190a40, values = 111e30, pcast = 0, ccast = 0
[21613] psyco_DBAPITypeObject_new: values refcnt = 2
[21613] psyco_DBAPITypeObject_new: object created at 10b1e8
[21613] psyco_add_type: typeobject = 10b1e8, values refcnt = 2
[21613] psyco_add_type: val = 1082
[21613] psyco_add_type: val = 1083
[21613] psyco_add_type: val = 1266
[21613] psyco_add_type: val = 1114
[21613] psyco_add_type: val = 1184
[21613] psyco_add_type: val = 704
[21613] psyco_add_type: val = 1186
[21613] psyco_init_types: initializing TIME
[21613] new_psyco_typeobject: type tuple created at 18ed50
[21613] psyco_DBAPITypeObject_new: name = 190a60, values = 18ed50, pcast = 0, ccast = 0
[21613] psyco_DBAPITypeObject_new: values refcnt = 2
[21613] psyco_DBAPITypeObject_new: object created at 10b200
[21613] psyco_add_type: typeobject = 10b200, values refcnt = 2
[21613] psyco_add_type: val = 1083
[21613] psyco_add_type: val = 1266
[21613] psyco_init_types: initializing DATE
[21613] new_psyco_typeobject: type tuple created at 1870f8
[21613] psyco_DBAPITypeObject_new: name = 190a80, values = 1870f8, pcast = 0, ccast = 0
[21613] psyco_DBAPITypeObject_new: values refcnt = 2
[21613] psyco_DBAPITypeObject_new: object created at 10b218
[21613] psyco_add_type: typeobject = 10b218, values refcnt = 2
[21613] psyco_add_type: val = 1082
[21613] psyco_add_type: val = 1114
[21613] psyco_add_type: val = 1184
[21613] psyco_init_types: initializing INTERVAL
[21613] new_psyco_typeobject: type tuple created at 18ed78
[21613] psyco_DBAPITypeObject_new: name = 190aa0, values = 18ed78, pcast = 0, ccast = 0
[21613] psyco_DBAPITypeObject_new: values refcnt = 2
[21613] psyco_DBAPITypeObject_new: object created at 10b230
[21613] psyco_add_type: typeobject = 10b230, values refcnt = 2
[21613] psyco_add_type: val = 704
[21613] psyco_add_type: val = 1186
[21613] psyco_init_types: initializing BINARY
[21613] new_psyco_typeobject: type tuple created at 183050
[21613] psyco_DBAPITypeObject_new: name = 190ac0, values = 183050, pcast = 0, ccast = 0
[21613] psyco_DBAPITypeObject_new: values refcnt = 2
[21613] psyco_DBAPITypeObject_new: object created at 10b248
[21613] psyco_add_type: typeobject = 10b248, values refcnt = 2
[21613] psyco_add_type: val = 17
[21613] psyco_init_types: initializing ROWID
[21613] new_psyco_typeobject: type tuple created at 190830
[21613] psyco_DBAPITypeObject_new: name = 190ae0, values = 190830, pcast = 0, ccast = 0
[21613] psyco_DBAPITypeObject_new: values refcnt = 2
[21613] psyco_DBAPITypeObject_new: object created at 10b260
[21613] psyco_add_type: typeobject = 10b260, values refcnt = 2
[21613] psyco_add_type: val = 26
[21613] new_psyco_typeobject: type tuple created at 102030
[21613] psyco_DBAPITypeObject_new: name = 190b00, values = 102030, pcast = 0, ccast = 0
[21613] psyco_DBAPITypeObject_new: values refcnt = 556
[21613] psyco_DBAPITypeObject_new: object created at 10b278
[21613] initpsycopg: types initialized
[21613] initpsycopg: exceptions initialized
>>> connection = psycopg.connect('dbname=gamedb user=webuser')
[21613] psyco_connect(): dsn = 'dbname=gamedb user=webuser', serialize = 1
[21613] psyco_connect(): minconn = 8, maxconn = 64
[21613] new_psyco_connobject(): creating new connection
[21613] new_psyco_cursobject: new cursor, keeper = 0
[21613] request_pgconn: maxconn = 64, minconn = 8, openconn = 0, availconn = 0
[21613] alloc_keeper: opening new postgresql connection
[21613] alloc_keeper: dsn = dbname=gamedb user=webuser (144048)
[21613] alloc_keeper: new posgresql connection at 1e13b8
[21613] alloc_keeper: connection protocol version: 3
[21613] alloc_keeper: setting datestyle to iso
[21613] alloc_keeper: datestyle query executed
[21613] alloc_keeper: keeper allocated at 1dfa50
[21613] request_pgconn: cursor at 114e30 using keeper at 1dfa50, connection at 1e13b8
[21613] new_psyco_cursobject: cursor created at 114e30, refcnt = 1
[21613] new_psyco_connobject(): created connobject at 179800, refcnt = 1
[21613] new_psyco_connobject(): stdmanager = 114e30, stdkeeper = 1dfa50
>>> cursor = connection.cursor( )
[21613] psyco_conn_cursor(): conn = 179800, name = Segmentation Fault (core dumped)

:(

Again this shows every sign of being a psycopg issue rather than a DJango issue.

PS apologies for the messy formatting in my previous update; left out the curly braces!

[tristanlawrence@…]

I have found a workaround to the psycopg issue; if psycopg is compiled without the "enable-devel" flag then it does not coredump. I have reported this issue to the psycopg list at lists.initd.org

This is with psycopg v1.1.19 compiled with gcc v3.3.2

[Adrian Holovaty]

I'm closing this, because we don't have any plans to support any other PostgreSQL driver than psycopg.

[James Bennett]

(also, note that Django can use backends which aren't part of Django itself, so if someone wants to write and maintain an alternate Postgres backend it should be fairly easy)

[Anderson Santos]

Just to put my comment, I am having a really complicated problem that is even making me wanting to give up postgresql because of a problem in psycopg. It is even creating financial problems to the company. I posted it in Django users and finally when I believed found the problem at psycopg list ​http://lists.initd.org/pipermail/psycopg/2008-October/006224.html , I can't find a workaround and the quickest solution was to be able to switch to another postgresql driver, so even if it won't be official, I believe it's really important to have more then 1 option.
Not all companies have geeks that are able to find, debug and solve problems. Another problem I had, there was a bug in Psycopg and Freebsd to install and I it was REALLY hard to find a solution.

Again, just my comment.

[James Bennett]

If you want a pypgsql-based backend, write one and use it; again, Django is not limited to only using the backend modules which ship with Django (DATABASE_ENGINE can specify either a built-in backend module, or the import path of a third-party module), so everybody who wants pypgsql should get together, write a backend module for it, and start using it.