Opened 8 years ago

Closed 7 years ago

#25454 closed Cleanup/optimization (fixed)

Module level database queries in a project prevent register_hstore from being called

Reported by: Alessandro Reichert Owned by: Claude Paroz <claude@…>
Component: contrib.postgres Version: dev
Severity: Normal Keywords: hstorefield
Cc: craig.ds@… Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no


Using hstorefield through the webserver works fine, but for applications that don't use the webserver like management commands we got a problem with models that use hstorefield. Among these problems is retrieving the hstorefield object which is retrieved as 'str' insteado of a 'dict'.

I found out a workaround that may be the cause of the bug but I've not enought knowledge to propose a fix.

Here is how to replicate the problem:

# subaccount is a hstorefield of AccountEntry model

In [8]: entry = AccountEntry.objects.get(pk=1)

In [9]: entry.subaccount
Out[9]: '"store"=>"velocity"'
In [10]: entry.subaccount.__class__
Out[10]: str

To get it working I'd to do this trick:

In [12]: from psycopg2.extras import register_hstore

In [13]: from django.db import connection

In [14]: register_hstore(connection.connection, globally=True, unicode=True)

In [15]: entry.subaccount
Out[15]: '"store"=>"velocity"'

In [16]: entry2 = AccountEntry.objects.get(pk=2)

In [17]: entry2.subaccount
Out[17]: {u'store': u'velocity'}

In [18]: entry2.subaccount.__class__
Out[18]: dict

Note that:
1) We are using django 1.8.4, python 2.7.9, psycopg2 2.6.1, postgresql 9.4.4
2) We installed the hstore extension in the template1 and we have created the django database after the hstore extension creation.
3) We have in our INSTALLED_APPS the django.contrib.postresql


Change History (8)

comment:1 Changed 8 years ago by Simon Charette

Resolution: needsinfo
Status: newclosed

Hi reichert,

It looks like django.contrib.postresql.apps.PostgresConfig.ready which is supposed to deal with register_hstore is never called.

From what I can see you reproduced using an iPython shell, was it lauched using shell and if it's not the case did you make sure to call django.setup() to make sure all apps were initialized?

Last edited 8 years ago by Simon Charette (previous) (diff)

comment:2 Changed 8 years ago by Craig de Stigter

Cc: craig.ds@… added
Resolution: needsinfo
Status: closednew

I've just had this issue. Reopening because it's marked needsinfo and I have some :)

For us, we're using shell correctly, but it looks like we're doing a database query in a silly place in our code _before_ the app cache is ready.

That is, we're forcing a database connection, which in turn sends the connection_created signal before the register_hstore signal handler is registered by PostgresConfig.ready.

This is obviously something dumb we're doing, but it'd be nice if Django was able to throw an error when the database connection is initialised too early. Then we would have found this bug ages ago. Would that be possible?

comment:3 Changed 8 years ago by Tim Graham

I'm not sure if we could do that or not (I think probably a lot of projects make queries at import time so a warning rather than exception might be needed), but could you provide a minimal project to reproduce it?

comment:4 Changed 8 years ago by Craig de Stigter

Sure thing. Here's a minimal test project I whipped up:

comment:5 Changed 8 years ago by Tim Graham

Summary: Problem using HStoreField in django shellModule level database queries in a project prevent register_hstore from being called
Triage Stage: UnreviewedAccepted
Type: BugCleanup/optimization

Thanks, I'll accept the ticket for investigation by an interested person (could be you if you are so inclined). We can always reclassify it as a documentation issue if needed.

comment:6 Changed 7 years ago by Claude Paroz

Has patch: set
Version: 1.8master

I am also experiencing this on Django's own test suite when running postgres_tests.test_hstore.

This PR should solve this.

comment:7 Changed 7 years ago by Simon Charette

Triage Stage: AcceptedReady for checkin

comment:8 Changed 7 years ago by Claude Paroz <claude@…>

Owner: set to Claude Paroz <claude@…>
Resolution: fixed
Status: newclosed

In 283b4684:

Fixed #25454 -- Ensured register_hstore_handler is called for all connections

Thanks Simon Charette for help with the patch.

Note: See TracTickets for help on using tickets.
Back to Top