values() and values_list() don't work when datetime fields are specified and a GeoManager is used

[brett_energysavvy]

I've attached an app which can be used to reproduce this as well.

from django.db import models
from django.contrib.gis.db.models.manager import GeoManager

class MyModel(models.Model):
    some_time = models.DateTimeField()
    
    objects = GeoManager()

def some_function():
    # Assuming there are instances of MyModel, this code will
    # blow up when evaluated
    list(MyModel.objects.values_list('id', 'some_time'))

This results in the following stack:

Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\django\core\handlers\base.py", line 114, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "C:\Users\Brett\Desktop\mysite\mysite\views.py", line 10, in repro_view
    list(MyModel.objects.values_list('id', 'some_time'))
  File "C:\Python27\lib\site-packages\django\db\models\query.py", line 96, in __iter__
    self._fetch_all()
  File "C:\Python27\lib\site-packages\django\db\models\query.py", line 854, in _fetch_all
    self._result_cache = list(self.iterator())
  File "C:\Python27\lib\site-packages\django\db\models\query.py", line 1068, in iterator
    for row in self.query.get_compiler(self.db).results_iter():
  File "C:\Python27\lib\site-packages\django\db\models\sql\compiler.py", line 746, in results_iter
    row = self.resolve_columns(row, fields)
  File "C:\Python27\lib\site-packages\django\contrib\gis\db\models\sql\compiler.py", line 182, in resolve_columns
    values.append(self.query.convert_values(value, field, self.connection))
  File "C:\Python27\lib\site-packages\django\contrib\gis\db\models\sql\query.py", line 76, in convert_values
    return super(GeoQuery, self).convert_values(value, field, connection)
  File "C:\Python27\lib\site-packages\django\db\models\sql\query.py", line 289, in convert_values
    return connection.ops.convert_values(value, field)
  File "C:\Python27\lib\site-packages\django\db\backends\sqlite3\base.py", line 267, in convert_values
    return parse_datetime_with_timezone_support(value)
  File "C:\Python27\lib\site-packages\django\db\backends\sqlite3\base.py", line 46, in parse_datetime_with_timezone_support
    dt = parse_datetime(value)
  File "C:\Python27\lib\site-packages\django\utils\dateparse.py", line 67, in parse_datetime
    match = datetime_re.match(value)
TypeError: expected string or buffer

This appears to also happen in Django 1.6. Any help on a workaround would be useful as well.

[Aymeric Augustin]

I haven't reproduced the problem yet but the use case is absolutely valid and the traceback proves the problem.

It's probably my fault :(

[brett_energysavvy]

Have you had a chance to look at this? I can't seem to find a workaround for this and it's blocking our development team from upgrading to Django 1.5/1.6.

[Aymeric Augustin]

I've been distracted by my work on app-loading, sorry.

[Aymeric Augustin]

This bug only happens on Spatialite. Unfortunately I never managed to install it. I tried again tonight without success.

I wrote a test case, which I'm attaching. It passes under PostGIS.

[Akis Kesoglou]

Python 3.3, Django@master, I believe the following is related, here is the traceback:

Traceback (most recent call last):
  ...
  File "<virtualenv>/src/django/django/db/models/query.py", line 140, in __iter__
    self._fetch_all()
  File "<virtualenv>/src/django/django/db/models/query.py", line 962, in _fetch_all
    self._result_cache = list(self.iterator())
  File "<virtualenv>/src/django/django/db/models/query.py", line 267, in iterator
    offset=len(aggregate_select))
  File "<virtualenv>/src/django/django/db/models/query.py", line 1441, in get_cached_row
    cached_row = get_cached_row(row, index_end, using, klass_info)
  File "<virtualenv>/src/django/django/db/models/query.py", line 1429, in get_cached_row
    obj = klass(*fields)
  File "<virtualenv>/src/django/django/db/models/base.py", line 387, in __init__
    setattr(self, field.attname, val)
  File "<virtualenv>/src/django/django/contrib/gis/db/models/proxy.py", line 65, in __set__
    obj.__class__.__name__, gtype, type(value)))
TypeError: Cannot set Location GeometryProxy (POINT) with value of type: <class 'bytes'>

Fix is easy for python 3; just add bytes to the list of types at django.contrib.gis.db.models.proxy line 60.

The weird thing is it only happens when testing, not on runserver.

[Akis Kesoglou]

Forget my comment above -- it is totally unrelated. I tried hard to find the cause of the error, but in the end I figured I'd forgotten to add the GeoManager to a custom intermediary ('through') model for the m2m relation, hence the exception. On the other hand, maybe I could create a ticket for the ugliness of the error above, for a user error that I'm sure is not that uncommon.

Excuse the noise from my part.

[Claude Paroz]

The problem is caused by resolve_columns in django/contrib/gis/db/models/sql/compiler.py, which calls self.query.convert_values if self.query.geo_values is True. However, the date value is already converted in a proper Python Datetime at the start of the method. It happens only in Spatialite because the basic convert_values don't touch date/time values, only the sqlite3 backend does. I see two ways forward:

1. Just test in parse_datetime that the value is not already a Datetime object.
2. Try to be more clever at the resolve_columns level to only call convert_values on geometry values (ORM knowledge required).

[Marc Tamlyn <marc.tamlyn@…>]

In e9103402c0fa873aea58a6a11dba510cd308cb84:

Fixed #18757, #14462, #21565 -- Reworked database-python type conversions

Complete rework of translating data values from database

Deprecation of SubfieldBase, removal of resolve_columns and
convert_values in favour of a more general converter based approach and
public API Field.from_db_value(). Now works seamlessly with aggregation,
.values() and raw queries.

Thanks to akaariai in particular for extensive advice and inspiration,
also to shaib, manfre and timograham for their reviews.