Test failure with spatialite

[Julien Phalip]

I'm seeing a test failure with spatialite 4.0.0:

vagrant@djangocore:~$ runtests2.6-spatialite gis
Creating test database for alias 'default'...
Creating test database for alias 'other'...
.........................F.....................................................................................................................................................
======================================================================
FAIL: test02_osr (django.contrib.gis.tests.test_spatialrefsys.SpatialRefSysTest)
Testing getting OSR objects from SpatialRefSys model objects.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/django/django/contrib/gis/tests/test_spatialrefsys.py", line 76, in test02_osr
    self.assertTrue(srs.proj4 in sd['proj4'])
AssertionError: False is not True

----------------------------------------------------------------------
Ran 175 tests in 0.697s

FAILED (failures=1)
Destroying test database for alias 'default'...
Destroying test database for alias 'other'...

All the version numbers can be found in the latest djangocore-box specs: ​https://github.com/jphalip/djangocore-box/blob/master/provisioning/shell/install-gis.sh

[Julien Phalip]

Marking as blocker to be sure 1.5 doesn't ship with a test failure.

[Claude Paroz]

Julien, could you replace the assertTrue by an assertIn so as we can plainly see the strings involved in the comparison?

[Julien Phalip]

Hi Claude, this is the updated error message:

======================================================================
FAIL: test02_osr (django.contrib.gis.tests.test_spatialrefsys.SpatialRefSysTest)
Testing getting OSR objects from SpatialRefSys model objects.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/django/django/contrib/gis/tests/test_spatialrefsys.py", line 76, in test02_osr
    self.assertIn(srs.proj4, sd['proj4'])
AssertionError: u'+proj=longlat +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +no_defs ' not found in ['+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs ', '+proj=longlat +datum=WGS84 +no_defs ']

[Claude Paroz]

That's strange. When reading libspatialite 4.0.0 sources, I can see the proper definition of the 4326 epsg definition (src/srsinit/epsg_inlined_wgs84_00.c):

p = add_epsg_def (filter, first, last, 4326, "epsg", 4326,
        "WGS 84");
add_proj4text (p, 0,
        "+proj=longlat +datum=WGS84 +no_defs");

What's the result of:
SELECT * FROM spatial_ref_sys WHERE srid=4326;

[Julien Phalip]

This is what I get when running that query:

[(4326, u'epsg', 4326, u'WGS 84', u'+proj=longlat +datum=WGS84 +no_defs', u'GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]]')]

[Claude Paroz]

I've just compiled libspatialite 4.0.0 and ran the gis tests without errors. I have really no clue what your failure comes from.

[Claude Paroz]

Quoting Anssi:
+1 for not holding 1.5 for this one. Spatialite isn't exactly the most used DB for gis, and the failure doesn't look like data corruption or crashing type.

[Claude Paroz]

I've finally been able to reproduce the issue after upgrading my main system to Debian wheezy (using Spatialite 3). This has nothing to do with spatial_ref_sys data in Spatialite, but rather a consequence of the way the gdal.SpatialReference is initialized with a pseudo srs WKT.

Julien, would you be able to test the attached patch?

[Tim Graham]

I can reproduce this using djangocore-box:

vagrant@djangocore:~$ runtests2.7-spatialite django.contrib.gis
Creating test database for alias 'default'...
Creating test database for alias 'other'...
..s..................sssssssssss................ssssss......................................................................................F................................Esssss.......................................................................................
======================================================================
ERROR: django.contrib.gis.tests.geoapp.tests (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: django.contrib.gis.tests.geoapp.tests
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 252, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 230, in _get_module_from_name
    __import__(name)
  File "/django/django/contrib/gis/tests/geoapp/tests.py", line 207, in <module>
    class GeoLookupTest(TestCase):
  File "/django/django/contrib/gis/tests/geoapp/tests.py", line 307, in GeoLookupTest
    if postgis_bug_version():
  File "/django/django/contrib/gis/tests/geoapp/tests.py", line 27, in postgis_bug_version
    spatial_version = getattr(connection.ops, "spatial_version", (0,0,0))
  File "/django/django/utils/functional.py", line 49, in __get__
    res = instance.__dict__[self.func.__name__] = self.func(instance)
  File "/django/django/contrib/gis/db/backends/spatialite/operations.py", line 133, in spatial_version
    six.reraise(ImproperlyConfigured, ImproperlyConfigured(new_msg), sys.exc_info()[2])
  File "/django/django/contrib/gis/db/backends/spatialite/operations.py", line 127, in spatial_version
    version = self.spatialite_version_tuple()[1:]
  File "/django/django/contrib/gis/db/backends/spatialite/operations.py", line 273, in spatialite_version_tuple
    version = self.spatialite_version()
  File "/django/django/contrib/gis/db/backends/spatialite/operations.py", line 264, in spatialite_version
    return self._get_spatialite_func('spatialite_version()')
  File "/django/django/contrib/gis/db/backends/spatialite/operations.py", line 242, in _get_spatialite_func
    cursor = self.connection._cursor()
  File "/django/django/db/backends/__init__.py", line 131, in _cursor
    self.ensure_connection()
  File "/django/django/db/backends/__init__.py", line 126, in ensure_connection
    self.connect()
  File "/django/django/db/backends/__init__.py", line 113, in connect
    conn_params = self.get_connection_params()
  File "/django/django/db/backends/sqlite3/base.py", line 322, in get_connection_params
    "settings.DATABASES is improperly configured. "
ImproperlyConfigured: Cannot determine the SpatiaLite version for the "" database (error was "settings.DATABASES is improperly configured. Please supply the NAME value.").  Was the SpatiaLite initialization SQL loaded on this database?


======================================================================
FAIL: test02_osr (django.contrib.gis.tests.test_spatialrefsys.SpatialRefSysTest)
Testing getting OSR objects from SpatialRefSys model objects.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/django/django/contrib/gis/tests/test_spatialrefsys.py", line 77, in test02_osr
    six.assertRegex(self, srs.proj4, sd['proj4_re'])
  File "/django/django/utils/six.py", line 430, in assertRegex
    return getattr(self, _assertRegex)(*args, **kwargs)
AssertionError: Regexp didn't match: '\\+proj=longlat (\\+ellps=WGS84 )?\\+datum=WGS84 \\+no_defs ' not found in u'+proj=longlat +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +no_defs '

----------------------------------------------------------------------
Ran 266 tests in 2.289s

FAILED (failures=1, errors=1, skipped=23)

After the patch is applied, none of the GIS tests run:

vagrant@djangocore:~$ runtests2.7-spatialite django.contrib.gis
Traceback (most recent call last):
  File "/django/tests/runtests.py", line 364, in <module>
    options.failfast, args)
  File "/django/tests/runtests.py", line 193, in django_tests
    state = setup(verbosity, test_labels)
  File "/django/tests/runtests.py", line 140, in setup
    test_modules = get_test_modules()
  File "/django/tests/runtests.py", line 67, in get_test_modules
    from django.contrib.gis.tests.utils import HAS_SPATIAL_DB
  File "/django/django/contrib/gis/tests/utils.py", line 34, in <module>
    from django.contrib.gis.db.backends.spatialite.models import SpatialRefSys
  File "/django/django/contrib/gis/db/backends/spatialite/models.py", line 45, in <module>
    class SpatialRefSys(models.Model, SpatialRefSysMixin):
  File "/django/django/contrib/gis/db/backends/spatialite/models.py", line 54, in SpatialRefSys
    if connection.ops.spatial_version[0] >= 3:
  File "/django/django/utils/functional.py", line 49, in __get__
    res = instance.__dict__[self.func.__name__] = self.func(instance)
  File "/django/django/contrib/gis/db/backends/spatialite/operations.py", line 133, in spatial_version
    six.reraise(ImproperlyConfigured, ImproperlyConfigured(new_msg), sys.exc_info()[2])
  File "/django/django/contrib/gis/db/backends/spatialite/operations.py", line 127, in spatial_version
    version = self.spatialite_version_tuple()[1:]
  File "/django/django/contrib/gis/db/backends/spatialite/operations.py", line 273, in spatialite_version_tuple
    version = self.spatialite_version()
  File "/django/django/contrib/gis/db/backends/spatialite/operations.py", line 264, in spatialite_version
    return self._get_spatialite_func('spatialite_version()')
  File "/django/django/contrib/gis/db/backends/spatialite/operations.py", line 242, in _get_spatialite_func
    cursor = self.connection._cursor()
  File "/django/django/db/backends/__init__.py", line 131, in _cursor
    self.ensure_connection()
  File "/django/django/db/backends/__init__.py", line 126, in ensure_connection
    self.connect()
  File "/django/django/db/backends/__init__.py", line 113, in connect
    conn_params = self.get_connection_params()
  File "/django/django/db/backends/sqlite3/base.py", line 322, in get_connection_params
    "settings.DATABASES is improperly configured. "
django.core.exceptions.ImproperlyConfigured: Cannot determine the SpatiaLite version for the "" database (error was "settings.DATABASES is improperly configured. Please supply the NAME value.").  Was the SpatiaLite initialization SQL loaded on this database?

[Ramiro Morales]

I've uploaded a modified version of Claude's patch that works for me (spatialite 3.x here, note how the version conditional was changed to only in version 4.x otherwise most of the GIS tests fail becuase AFAICS libspatialite 3 doesn't create the new srtex column).

Perhaps this can be divided in two different commits?

[Claude Paroz]

Ramiro, please go ahead. I can confirm that the srtext column is 4.x only (was previously named srs_wkt, but not used in GeoDjango).

[Ramiro Morales <cramm0@…>]

In 9a73e7f40c851cef262dd36da444c81492632ee5:

Fixed #19678 -- GeoDjango test failure with spatialite >= 3.0.

Thanks Julien for the report and Claude for the fix.

[Ramiro Morales <cramm0@…>]

In 362dd68fb20be195462af22448416c9895ce7df7:

Added new 'srtext' spatialite 4.x SpatialRefSys column to its model.

This is for general consistency in the GeoDjango DB backends.
Thanks Claude for the fix.

Refs #19678.