GeoDjango missing pyspatialite support

[Daniel Sokolowski]

Pyspatialite is a modified version of pysqlite that builds sqlite with spatiliate support already in it. This module is easier to install compared to pysqlite especially when utilizing a package installer such as PIP; making GeoDjango support this package is relatively trivial.

Further Explanation:
It appears that the emergent way to run django sites is now Virtualenv+PIP. Adding support for pyspatiallite allows one to setup GeoDjango sqlite3 support utilizing PIP and hence allows for the creation of requiremnts.txt which in turn makes the project that much more portable. One can not use PIP to install pysqlite as by default it does not enable loading of sqlite extensions.

[Daniel Sokolowski]

Code + Doc patch

[anonymous]

the code is clear and is backwards compatible

[juanpex]

Replying to anonymous:

the code is clear and is backwards compatible

this is my comment.

[Daniel Sokolowski]

Can someone please mark it "Ready for checkin" ? I just applied the patch on "Django-1.5.dev17930-py2.6-dev_r17930":

$  patch -p0 < patch-feb2312\@1646.diff
(Stripping trailing CRs from patch.)
patching file docs/ref/contrib/gis/install.txt
Hunk #1 succeeded at 66 (offset 2 lines).
Hunk #2 succeeded at 461 (offset 2 lines).
(Stripping trailing CRs from patch.)
patching file django/db/backends/sqlite3/base.py
Hunk #1 succeeded at 22 with fuzz 1.
Hunk #2 succeeded at 353 (offset 13 lines).
(Stripping trailing CRs from patch.)
patching file django/contrib/gis/db/backends/spatialite/base.py

[Claude Paroz]

pyspatialite seems to suffer from build/install issues. See for example those logs: ​http://code.activestate.com/pypm/pyspatialite/
I was able to reproduce those errors on my system.

[Claude Paroz]

Note also that the following issue did not receive any feedback for two months:
​http://code.google.com/p/pyspatialite/issues/detail?id=6

[Daniel Sokolowski]

Thanks for taking the time to review it; I've looked at the error logs and the build fails as it is missing header files - unfortunately one has to use the system package management to install those as pip won't work. Also the branch is labeled experimental. I did an upgrade from 2.6 to this experimental branch here and it build properly:

gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions build/temp.linux-i686-2.6/src/module.o build/temp.linux-i686-2.6/src/connection.o build/temp.linux-i686-2.6/src/cursor.o build/temp.linux-i686-2.6/src/cache.o build/temp.linux-i686-2.6/src/microprotocols.o build/temp.linux-i686-2.6/src/prepare_protocol.o build/temp.linux-i686-2.6/src/statement.o build/temp.linux-i686-2.6/src/util.o build/temp.linux-i686-2.6/src/row.o build/temp.linux-i686-2.6/amalgamation/sqlite3.o build/temp.linux-i686-2.6/amalgamation/spatialite.o -L/Library/Frameworks/GEOS.framework/unix/lib -L/Library/Frameworks/PROJ.framework/unix/lib -lgeos -lgeos_c -lproj -o build/lib.linux-i686-2.6/pyspatialite/_spatialite.so
Successfully installed pyspatialite

Your second concern again results in header files missing; in this case it's the geos package - see line amalgamation\spatialite.c:102:20: error: geos_c.h: No such file or directory.

Both results are due to the fact PIP only does python packages and it won't even complain if 'outside' requirements are missing, these supporting header files would need to be installed either way to compile sqlite3 with 'extension' support.

The patch ought to include a NOTE advising that one must install supporting GEOS, GDAL, PROJ.4 files? Since these issues are not package related after that it should be good for RFC. thoughts?

[Claude Paroz]

Yes, please, provide any complementary instructions for system libraries needed. What's strange with my system is that such libraries are installed, as I'm running Geodjango/Postgis without problems. It might be an issue with library paths.

[Daniel Sokolowski]

Will do; on a side note is your system Debian, Ubuntu? If so source files and binary files are split up, i.e. libgeos libgeos-dev.

[Claude Paroz]

Yes, Debian. And you are right, the -dev packages are not installed, so that should be part of the complementary instructions.

[Daniel Sokolowski]

Documentation updated and a pull requested created: ​https://github.com/django/django/pull/198

[Claude Paroz]

Thanks, we are close now. But I think that the docs are still not clear enough about what steps are needed to install SpatiaLite. The subsections are currently:

• SQLite
• SpatiaLite library (libspatialite) and tools (spatialite)
• pyspatialite
• pysqlite2

AFAIR, the pyspatialite method does not need any other step, right? In that case, wouldn't it be better to structure the section this way:

• pyspatialite
• Installing from sources
  • SQLite
  • SpatiaLite library (libspatialite) and tools (spatialite)
  • pysqlite2

[Daniel Sokolowski]

In all honestly when I was updating the docs I was thinking that the install docs need to be overhauled. The install from source should be removed and replaced with concise installs for specific database + linux combo using PIP and system package installers.

GeoDjango installation is overly confusing but does not need to be, I was think along the lines:

• Requirements Overview
• Django + PostgreSQL + GeoDjango Stack
  • Debian/Ubuntu
  • CentOS
  • ...
• Django + MySQL + GeoDjango Stack
  • Debian/Ubuntu
  • !CentOS
  • ...
• !Django + !SQLite + GeoDjango Stack
  • Debian/Ubuntu
  • !CentOS
  • ...

Thoughts ?

[Claude Paroz]

Sure, feel free to propose a reorganisation of the install docs. However, I'd prefer not mixing patches, and this could happen in a separate ticket.

[Anssi Kääriäinen]

I tested the pull request 198 with pyspatialite on Ubuntu 12.04. What I did:

sudo apt-get install libgeos-dev
sudo pip install pyspatialite
./runtests --settings=test_spatialite.py gis

Result was a core dump. Output (but not the actual dump) attached.

Of course, I can't apply the PR as the above issue needs some resolution first.

Additionally, please check the install.txt changes for whitespace errors, lines longer than 80 chars and use spaces instead of tabs.

[Anssi Kääriäinen]

Core dump on Ubuntu 12.04

[Daniel Sokolowski]

I no longer believe there is a need for this patch - installing through Debian apt system all the requirements as stated by the docs bypasses all of the PIP issues I have encountered and makes this not needed fix at least for me; however feel free to change it.

[Paul Garner]

Just dropping in to say anything that can ease the pain of setting up GeoDjango is much appreciated. I'm trying to do it primarily via Homebrew and Pip... it'd be nice if it worked. The current instructions all lead to various dead ends

I'm currently stuck here:
​http://stackoverflow.com/questions/11093593/python-the-pysqlite-library-does-not-support-c-extension-loading

If using pip install pyspatialite makes things more 'just work' then PLEASE do it. It's not helpful to say oh yeah I got my install working through Debian apt... it has zero value to users not on Debian.

Can a core dev please set this back to 'Accepted' as it had previously been marked?

[Paul Garner]

Well after actually trying to apply the patch and go this route I think there are more problems. Basically pyspatialite is broken.

​https://code.google.com/p/pyspatialite/issues/detail?id=5
​https://code.google.com/p/pyspatialite/issues/detail?id=9

[anonymous]

As a note, it's being/is fixed (at the moment, there is a new pre-release package on pypi. after a bit of testing, I'll upgrade it to stable)

[Ramiro Morales]

Replying to anentropic:

Just dropping in to say anything that can ease the pain of setting up GeoDjango is much appreciated. I'm trying to do it primarily via Homebrew and Pip... it'd be nice if it worked. The current instructions all lead to various dead ends

I'm currently stuck here:
​http://stackoverflow.com/questions/11093593/python-the-pysqlite-library-does-not-support-c-extension-loading

The answer to that SO question is the same as the one the Geodjango docs have mentioned for some time: ​https://docs.djangoproject.com/en/1.6/ref/contrib/gis/install/spatialite/#pysqlite2

[...]

Can a core dev please set this back to 'Accepted' as it had previously been marked?

Having tried the pyspatialite path myself recently, even with the latest development code, and getting the segmentation faults Anssi mentioned in comment:18 back in Jul '12 when trying to execute syncdb seems to indicate that it hasn't yet reached a point where we can:

• Support pypatialite by replacing the module used by the main SQLite Django DB backend.
• Recommend it as the easiest path to get GIS support with SQLite.

and that the documented process of using platform-packages plus possibly building pysqlite2 with a modified configuration still is the safest strategy to reach success.

Because of this, I think the 'someday/maybe' tag is appropiate at this point.

[Paul Garner]

I am still unable to install Spatialite. The solution in SO question and GeoDjango docs mentioned above don't work for me. Full details here:
​http://stackoverflow.com/questions/20505665/install-spatialite-for-python-geodjango-on-os-x

[Tim Graham]

pysqlite 2.7.0+ can be installed using pip as C extension loading is now ​enabled by default. I believe that removes the motivation for this ticket.