Opened 2 years ago

Closed 22 months ago

#20007 closed Bug (fixed)

psycopg2 should convert SQL arrays into python unicode-arrays

Reported by: hogbait@… Owned by: EricBoersma
Component: Database layer (models, ORM) Version: master
Severity: Normal Keywords: psycopg2 unicode python2
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: yes UI/UX: no

Description

Both test cases should succeed. However, the second one fails, because the cursor returns a utf-8 encoded string array instead of a unicode array.

#!/usr/bin/env python
# coding=utf8

from django.test import TestCase
#Assuming that the default connection uses psycopg2
from django.db import connection

class UnicodeArrayTestCase(TestCase):

    def select(self, val):
        cursor = connection.cursor()
        cursor.execute("select %s", (val,))
        return cursor.fetchone()[0]

    def test_select_ascii_array(self):
        a = ["awef"]
        b = self.select(a)
        self.assertEqual(a[0], b[0])

    def test_select_unicode_array(self):
        a = [u"ᄲawef"]
        b = self.select(a)
        self.assertEqual(a[0], b[0])

The django currently configures psycopg2 to convert strings to strings, but is not configured to convert string arrays into unicode arrays. A potential fix is described here: https://github.com/django/django/pull/189

However, you should probably put the change in the cursor constructor as described here: https://github.com/django/django/commit/6595ee896a9e3bf9cc1a9bd34af0e13bd32d8a3b

Psycopg2 documentation: http://initd.org/psycopg/docs/extensions.html#psycopg2.extensions.UNICODEARRAY

Change History (6)

comment:1 Changed 2 years ago by hogbait

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Version changed from 1.5 to master

comment:2 Changed 2 years ago by aaugustin

  • Triage Stage changed from Unreviewed to Accepted

I'm not familiar with Postgres arrays but this looks reasonable.

Last edited 2 years ago by aaugustin (previous) (diff)

comment:3 Changed 2 years ago by areski

Can you add an example when you will have this issue with the Django ORM?

comment:4 Changed 2 years ago by timo

  • Patch needs improvement set

The test needs to be incorporated into the Django test suite as well.

comment:5 Changed 22 months ago by EricBoersma

  • Owner changed from nobody to EricBoersma
  • Status changed from new to assigned

comment:6 Changed 22 months ago by Tim Graham <timograham@…>

  • Resolution set to fixed
  • Status changed from assigned to closed

In ded40142a92813a8901faa4a6ca398e6679152f6:

Fixed #20007 -- Configured psycopg2 to return UnicodeArrays

Thanks hogbait for the report.

Note: See TracTickets for help on using tickets.
Back to Top