Opened 12 years ago

Closed 8 years ago

#17756 closed New feature (wontfix)

GeoDjango missing pyspatialite support

Reported by: Daniel Sokolowski Owned by:
Component: GIS Version: dev
Severity: Normal Keywords:
Cc: juanpex, taylor.mitchell@…, reinout@…, sfllaw@…, Paul Garner Triage Stage: Someday/Maybe
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: no UI/UX: no

Description

Pyspatialite is a modified version of pysqlite that builds sqlite with spatiliate support already in it. This module is easier to install compared to pysqlite especially when utilizing a package installer such as PIP; making GeoDjango support this package is relatively trivial.

Further Explanation:
It appears that the emergent way to run django sites is now Virtualenv+PIP. Adding support for pyspatiallite allows one to setup GeoDjango sqlite3 support utilizing PIP and hence allows for the creation of requiremnts.txt which in turn makes the project that much more portable. One can not use PIP to install pysqlite as by default it does not enable loading of sqlite extensions.

Attachments (2)

patch-feb2312@1646.diff (4.4 KB ) - added by Daniel Sokolowski 12 years ago.
Code + Doc patch
17756_error.txt (13.8 KB ) - added by Anssi Kääriäinen 12 years ago.
Core dump on Ubuntu 12.04

Download all attachments as: .zip

Change History (30)

comment:1 by Daniel Sokolowski, 12 years ago

Owner: changed from nobody to Daniel Sokolowski
Status: newassigned

by Daniel Sokolowski, 12 years ago

Attachment: patch-feb2312@1646.diff added

Code + Doc patch

comment:2 by Daniel Sokolowski, 12 years ago

Has patch: set

comment:3 by anonymous, 12 years ago

the code is clear and is backwards compatible

in reply to:  3 comment:4 by juanpex, 12 years ago

Cc: juanpex added

Replying to anonymous:

the code is clear and is backwards compatible

this is my comment.

comment:5 by Aymeric Augustin, 12 years ago

Triage Stage: UnreviewedAccepted

comment:6 by Taylor Mitchell, 12 years ago

Cc: taylor.mitchell@… added

comment:7 by Daniel Sokolowski, 12 years ago

Can someone please mark it "Ready for checkin" ? I just applied the patch on "Django-1.5.dev17930-py2.6-dev_r17930":

$  patch -p0 < patch-feb2312\@1646.diff
(Stripping trailing CRs from patch.)
patching file docs/ref/contrib/gis/install.txt
Hunk #1 succeeded at 66 (offset 2 lines).
Hunk #2 succeeded at 461 (offset 2 lines).
(Stripping trailing CRs from patch.)
patching file django/db/backends/sqlite3/base.py
Hunk #1 succeeded at 22 with fuzz 1.
Hunk #2 succeeded at 353 (offset 13 lines).
(Stripping trailing CRs from patch.)
patching file django/contrib/gis/db/backends/spatialite/base.py

comment:8 by Claude Paroz, 12 years ago

pyspatialite seems to suffer from build/install issues. See for example those logs: http://code.activestate.com/pypm/pyspatialite/
I was able to reproduce those errors on my system.

comment:9 by Claude Paroz, 12 years ago

Note also that the following issue did not receive any feedback for two months:
http://code.google.com/p/pyspatialite/issues/detail?id=6

comment:10 by Daniel Sokolowski, 12 years ago

Thanks for taking the time to review it; I've looked at the error logs and the build fails as it is missing header files - unfortunately one has to use the system package management to install those as pip won't work. Also the branch is labeled experimental. I did an upgrade from 2.6 to this experimental branch here and it build properly:

gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions build/temp.linux-i686-2.6/src/module.o build/temp.linux-i686-2.6/src/connection.o build/temp.linux-i686-2.6/src/cursor.o build/temp.linux-i686-2.6/src/cache.o build/temp.linux-i686-2.6/src/microprotocols.o build/temp.linux-i686-2.6/src/prepare_protocol.o build/temp.linux-i686-2.6/src/statement.o build/temp.linux-i686-2.6/src/util.o build/temp.linux-i686-2.6/src/row.o build/temp.linux-i686-2.6/amalgamation/sqlite3.o build/temp.linux-i686-2.6/amalgamation/spatialite.o -L/Library/Frameworks/GEOS.framework/unix/lib -L/Library/Frameworks/PROJ.framework/unix/lib -lgeos -lgeos_c -lproj -o build/lib.linux-i686-2.6/pyspatialite/_spatialite.so
Successfully installed pyspatialite

Your second concern again results in header files missing; in this case it's the geos package - see line amalgamation\spatialite.c:102:20: error: geos_c.h: No such file or directory.

Both results are due to the fact PIP only does python packages and it won't even complain if 'outside' requirements are missing, these supporting header files would need to be installed either way to compile sqlite3 with 'extension' support.

The patch ought to include a NOTE advising that one must install supporting GEOS, GDAL, PROJ.4 files? Since these issues are not package related after that it should be good for RFC. thoughts?

comment:11 by Claude Paroz, 12 years ago

Patch needs improvement: set

Yes, please, provide any complementary instructions for system libraries needed. What's strange with my system is that such libraries are installed, as I'm running Geodjango/Postgis without problems. It might be an issue with library paths.

comment:12 by Daniel Sokolowski, 12 years ago

Will do; on a side note is your system Debian, Ubuntu? If so source files and binary files are split up, i.e. libgeos libgeos-dev.

comment:13 by Claude Paroz, 12 years ago

Yes, Debian. And you are right, the -dev packages are not installed, so that should be part of the complementary instructions.

comment:14 by Daniel Sokolowski, 12 years ago

Documentation updated and a pull requested created: https://github.com/django/django/pull/198

comment:15 by Claude Paroz, 12 years ago

Thanks, we are close now. But I think that the docs are still not clear enough about what steps are needed to install SpatiaLite. The subsections are currently:

  • SQLite
  • SpatiaLite library (libspatialite) and tools (spatialite)
  • pyspatialite
  • pysqlite2

AFAIR, the pyspatialite method does not need any other step, right? In that case, wouldn't it be better to structure the section this way:

  • pyspatialite
  • Installing from sources
    • SQLite
    • SpatiaLite library (libspatialite) and tools (spatialite)
    • pysqlite2

comment:16 by Daniel Sokolowski, 12 years ago

In all honestly when I was updating the docs I was thinking that the install docs need to be overhauled. The install from source should be removed and replaced with concise installs for specific database + linux combo using PIP and system package installers.

GeoDjango installation is overly confusing but does not need to be, I was think along the lines:

  • Requirements Overview
  • Django + PostgreSQL + GeoDjango Stack
    • Debian/Ubuntu
    • CentOS/RedHat
    • ...
  • Django + MySQL + GeoDjango Stack
    • Debian/Ubuntu
    • CentOS/RedHat
    • ...
  • Django + !SQLite + GeoDjango Stack
    • Debian/Ubuntu
    • CentOS/RedHat
    • ...

Thoughts ?

Last edited 12 years ago by Daniel Sokolowski (previous) (diff)

comment:17 by Claude Paroz, 12 years ago

Sure, feel free to propose a reorganisation of the install docs. However, I'd prefer not mixing patches, and this could happen in a separate ticket.

comment:18 by Anssi Kääriäinen, 12 years ago

I tested the pull request 198 with pyspatialite on Ubuntu 12.04. What I did:

sudo apt-get install libgeos-dev
sudo pip install pyspatialite
./runtests --settings=test_spatialite.py gis

Result was a core dump. Output (but not the actual dump) attached.

Of course, I can't apply the PR as the above issue needs some resolution first.

Additionally, please check the install.txt changes for whitespace errors, lines longer than 80 chars and use spaces instead of tabs.

by Anssi Kääriäinen, 12 years ago

Attachment: 17756_error.txt added

Core dump on Ubuntu 12.04

comment:19 by Reinout van Rees, 11 years ago

Cc: reinout@… added

comment:20 by Simon Law, 11 years ago

Cc: sfllaw@… added

comment:21 by Tim Graham, 11 years ago

Easy pickings: unset

comment:22 by Daniel Sokolowski, 11 years ago

Owner: Daniel Sokolowski removed
Status: assignednew
Triage Stage: AcceptedSomeday/Maybe

I no longer believe there is a need for this patch - installing through Debian apt system all the requirements as stated by the docs bypasses all of the PIP issues I have encountered and makes this not needed fix at least for me; however feel free to change it.

comment:23 by Paul Garner, 10 years ago

Cc: Paul Garner added

Just dropping in to say anything that can ease the pain of setting up GeoDjango is much appreciated. I'm trying to do it primarily via Homebrew and Pip... it'd be nice if it worked. The current instructions all lead to various dead ends

I'm currently stuck here:
http://stackoverflow.com/questions/11093593/python-the-pysqlite-library-does-not-support-c-extension-loading

If using pip install pyspatialite makes things more 'just work' then PLEASE do it. It's not helpful to say oh yeah I got my install working through Debian apt... it has zero value to users not on Debian.

Can a core dev please set this back to 'Accepted' as it had previously been marked?

comment:24 by Paul Garner, 10 years ago

Well after actually trying to apply the patch and go this route I think there are more problems. Basically pyspatialite is broken.

https://code.google.com/p/pyspatialite/issues/detail?id=5
https://code.google.com/p/pyspatialite/issues/detail?id=9

comment:25 by anonymous, 10 years ago

As a note, it's being/is fixed (at the moment, there is a new pre-release package on pypi. after a bit of testing, I'll upgrade it to stable)

in reply to:  23 comment:26 by Ramiro Morales, 10 years ago

Replying to anentropic:

Just dropping in to say anything that can ease the pain of setting up GeoDjango is much appreciated. I'm trying to do it primarily via Homebrew and Pip... it'd be nice if it worked. The current instructions all lead to various dead ends

I'm currently stuck here:
http://stackoverflow.com/questions/11093593/python-the-pysqlite-library-does-not-support-c-extension-loading

The answer to that SO question is the same as the one the Geodjango docs have mentioned for some time: https://docs.djangoproject.com/en/1.6/ref/contrib/gis/install/spatialite/#pysqlite2

[...]

Can a core dev please set this back to 'Accepted' as it had previously been marked?

Having tried the pyspatialite path myself recently, even with the latest development code, and getting the segmentation faults Anssi mentioned in comment:18 back in Jul '12 when trying to execute syncdb seems to indicate that it hasn't yet reached a point where we can:

  • Support pypatialite by replacing the module used by the main SQLite Django DB backend.
  • Recommend it as the easiest path to get GIS support with SQLite.

and that the documented process of using platform-packages plus possibly building pysqlite2 with a modified configuration still is the safest strategy to reach success.

Because of this, I think the 'someday/maybe' tag is appropiate at this point.

comment:27 by Paul Garner, 10 years ago

I am still unable to install Spatialite. The solution in SO question and GeoDjango docs mentioned above don't work for me. Full details here:
http://stackoverflow.com/questions/20505665/install-spatialite-for-python-geodjango-on-os-x

comment:28 by Tim Graham, 8 years ago

Resolution: wontfix
Status: newclosed

pysqlite 2.7.0+ can be installed using pip as C extension loading is now enabled by default. I believe that removes the motivation for this ticket.

Note: See TracTickets for help on using tickets.
Back to Top