Opened 7 years ago

Closed 5 months ago

#27813 closed Bug (wontfix)

BinaryField type inconsistent between sqlite3 (bytes) and postgresql (memoryview)

Reported by: Antonio Terceiro Owned by: nobody
Component: Database layer (models, ORM) Version: 3.1
Severity: Normal Keywords:
Cc: Egor R, Ryan Cheley Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no


I have an application that has the following model:

class Attachment(models.Model):
    # ...
    data = models.BinaryField(default=None)

under sqlite3:

In [1]: Attachment.objects.last().data
Out[1]: b'Mon Feb  6 16:53:37 UTC 2017\n'

under postgresql:

In [1]: Attachment.objects.last().data
Out[1]: <memory at 0x7f1ae8e93108>

i.e. under sqlite the BinaryField values comes as bytes(), and under postgresql it comes as memoryview()

Change History (12)

comment:1 by Tim Graham, 7 years ago

I can confirm the difference, however, I can't say if a change should be made. Do you have a suggestion about what to do?

comment:2 by Antonio Terceiro, 7 years ago

I noticed this because I had a test that checked the value of the data field after a given operation, and since I was initially developing against sqlite, I compared the value against a bytes() literal b'...'. to make the test pass against postgresql as well, I had to cast the value in the database to bytes() before the comparison.

I would expect to always get the same type, either bytes or memoryview. on the other hand, everything else other than that one test just worked. the view that allows for that data to be downloaded worked just fine, because bytes and memoryview have a similar enough interface, so maybe this should be just clearly documented.

comment:3 by Tim Graham, 7 years ago

Summary: BinaryField is inconsistent between sqlite3 and postgresqlBinaryField type inconsistent between sqlite3 (bytes) and postgresql (memoryview)
Triage Stage: UnreviewedAccepted
Type: UncategorizedCleanup/optimization

Django's test suite is also casting to bytes for comparison. Accepting for further investigation.

comment:4 by Václav Řehák, 5 years ago

The same applies to Oracle backend which also returns bytes. One of the reasons we use Django in current project is that we can run it on Oracle (as needed due to historical reasons) but hopefully convert it to PostgreSQL one day. Subtle differences like this make it more complicated. What is the reason to return different data type from PostgreSQL backend?

comment:5 by Tim Graham, 5 years ago

My guess is that's how the database or database driver (psycopg2) works.

comment:6 by Stavros Korokithakis, 5 years ago

I am hit by this as well. Returning HttpResponse(mymodel.mybinaryfield) returns a literal <memory object at 0xdeadbeef> string instead of the data on Postgres.

comment:7 by Benjamin Matschke, 3 years ago

When using BinaryField as a primary key, it leads to a bug.
For example deleting such a table raises a

TypeError: '<' not supported between instances of 'memoryview' and 'memoryview'

Besides, the resulting type should be consistent, independent on the underlying database.

comment:8 by Benjamin Matschke, 3 years ago

Type: Cleanup/optimizationBug
Version: 1.103.1

comment:9 by Egor R, 2 years ago

Cc: Egor R added

comment:10 by Simon Charette, 8 months ago

David Sanders noticed that psycopg>=3.0.17 has started to return bytes instead of memoryview so I wonder if we should resolve as ''wont-fix'' here given an available and even preferred solution is available given psycopg2 is on the way out,

comment:11 by Ryan Cheley, 7 months ago

Cc: Ryan Cheley added

comment:12 by Antonio Terceiro, 5 months ago

Resolution: wontfix
Status: newclosed

yeah, it makes sense to close this since the fix is actually elsewhere. I'm closing.

Note: See TracTickets for help on using tickets.
Back to Top