Opened 3 years ago

Last modified 17 months ago

#17756 new New feature

GeoDjango missing pyspatialite support

Reported by: danols Owned by:
Component: GIS Version: master
Severity: Normal Keywords:
Cc: juanpex, taylor.mitchell@…, reinout@…, sfllaw@…, anentropic Triage Stage: Someday/Maybe
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: no UI/UX: no

Description

Pyspatialite is a modified version of pysqlite that builds sqlite with spatiliate support already in it. This module is easier to install compared to pysqlite especially when utilizing a package installer such as PIP; making GeoDjango support this package is relatively trivial.

Further Explanation:
It appears that the emergent way to run django sites is now Virtualenv+PIP. Adding support for pyspatiallite allows one to setup GeoDjango sqlite3 support utilizing PIP and hence allows for the creation of requiremnts.txt which in turn makes the project that much more portable. One can not use PIP to install pysqlite as by default it does not enable loading of sqlite extensions.

Attachments (2)

patch-feb2312@1646.diff (4.4 KB) - added by danols 3 years ago.
Code + Doc patch
17756_error.txt (13.8 KB) - added by akaariai 3 years ago.
Core dump on Ubuntu 12.04

Download all attachments as: .zip

Change History (29)

comment:1 Changed 3 years ago by danols

  • Needs documentation unset
  • Needs tests unset
  • Owner changed from nobody to danols
  • Patch needs improvement unset
  • Status changed from new to assigned

Changed 3 years ago by danols

Code + Doc patch

comment:2 Changed 3 years ago by danols

  • Has patch set

comment:3 follow-up: Changed 3 years ago by anonymous

the code is clear and is backwards compatible

comment:4 in reply to: ↑ 3 Changed 3 years ago by juanpex

  • Cc juanpex added

Replying to anonymous:

the code is clear and is backwards compatible

this is my comment.

comment:5 Changed 3 years ago by aaugustin

  • Triage Stage changed from Unreviewed to Accepted

comment:6 Changed 3 years ago by tmitchell

  • Cc taylor.mitchell@… added

comment:7 Changed 3 years ago by danols

Can someone please mark it "Ready for checkin" ? I just applied the patch on "Django-1.5.dev17930-py2.6-dev_r17930":

$  patch -p0 < patch-feb2312\@1646.diff
(Stripping trailing CRs from patch.)
patching file docs/ref/contrib/gis/install.txt
Hunk #1 succeeded at 66 (offset 2 lines).
Hunk #2 succeeded at 461 (offset 2 lines).
(Stripping trailing CRs from patch.)
patching file django/db/backends/sqlite3/base.py
Hunk #1 succeeded at 22 with fuzz 1.
Hunk #2 succeeded at 353 (offset 13 lines).
(Stripping trailing CRs from patch.)
patching file django/contrib/gis/db/backends/spatialite/base.py

comment:8 Changed 3 years ago by claudep

pyspatialite seems to suffer from build/install issues. See for example those logs: http://code.activestate.com/pypm/pyspatialite/
I was able to reproduce those errors on my system.

comment:9 Changed 3 years ago by claudep

Note also that the following issue did not receive any feedback for two months:
http://code.google.com/p/pyspatialite/issues/detail?id=6

comment:10 Changed 3 years ago by danols

Thanks for taking the time to review it; I've looked at the error logs and the build fails as it is missing header files - unfortunately one has to use the system package management to install those as pip won't work. Also the branch is labeled experimental. I did an upgrade from 2.6 to this experimental branch here and it build properly:

gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions build/temp.linux-i686-2.6/src/module.o build/temp.linux-i686-2.6/src/connection.o build/temp.linux-i686-2.6/src/cursor.o build/temp.linux-i686-2.6/src/cache.o build/temp.linux-i686-2.6/src/microprotocols.o build/temp.linux-i686-2.6/src/prepare_protocol.o build/temp.linux-i686-2.6/src/statement.o build/temp.linux-i686-2.6/src/util.o build/temp.linux-i686-2.6/src/row.o build/temp.linux-i686-2.6/amalgamation/sqlite3.o build/temp.linux-i686-2.6/amalgamation/spatialite.o -L/Library/Frameworks/GEOS.framework/unix/lib -L/Library/Frameworks/PROJ.framework/unix/lib -lgeos -lgeos_c -lproj -o build/lib.linux-i686-2.6/pyspatialite/_spatialite.so
Successfully installed pyspatialite

Your second concern again results in header files missing; in this case it's the geos package - see line amalgamation\spatialite.c:102:20: error: geos_c.h: No such file or directory.

Both results are due to the fact PIP only does python packages and it won't even complain if 'outside' requirements are missing, these supporting header files would need to be installed either way to compile sqlite3 with 'extension' support.

The patch ought to include a NOTE advising that one must install supporting GEOS, GDAL, PROJ.4 files? Since these issues are not package related after that it should be good for RFC. thoughts?

comment:11 Changed 3 years ago by claudep

  • Patch needs improvement set

Yes, please, provide any complementary instructions for system libraries needed. What's strange with my system is that such libraries are installed, as I'm running Geodjango/Postgis without problems. It might be an issue with library paths.

comment:12 Changed 3 years ago by danols

Will do; on a side note is your system Debian, Ubuntu? If so source files and binary files are split up, i.e. libgeos libgeos-dev.

comment:13 Changed 3 years ago by claudep

Yes, Debian. And you are right, the -dev packages are not installed, so that should be part of the complementary instructions.

comment:14 Changed 3 years ago by danols

Documentation updated and a pull requested created: https://github.com/django/django/pull/198

comment:15 Changed 3 years ago by claudep

Thanks, we are close now. But I think that the docs are still not clear enough about what steps are needed to install SpatiaLite. The subsections are currently:

  • SQLite
  • SpatiaLite library (libspatialite) and tools (spatialite)
  • pyspatialite
  • pysqlite2

AFAIR, the pyspatialite method does not need any other step, right? In that case, wouldn't it be better to structure the section this way:

  • pyspatialite
  • Installing from sources
    • SQLite
    • SpatiaLite library (libspatialite) and tools (spatialite)
    • pysqlite2

comment:16 Changed 3 years ago by danols

In all honestly when I was updating the docs I was thinking that the install docs need to be overhauled. The install from source should be removed and replaced with concise installs for specific database + linux combo using PIP and system package installers.

GeoDjango installation is overly confusing but does not need to be, I was think along the lines:

  • Requirements Overview
  • Django + PostgreSQL + GeoDjango Stack
    • Debian/Ubuntu
    • CentOS/RedHat
    • ...
  • Django + MySQL + GeoDjango Stack
    • Debian/Ubuntu
    • CentOS/RedHat
    • ...
  • Django + !SQLite + GeoDjango Stack
    • Debian/Ubuntu
    • CentOS/RedHat
    • ...

Thoughts ?

Last edited 3 years ago by danols (previous) (diff)

comment:17 Changed 3 years ago by claudep

Sure, feel free to propose a reorganisation of the install docs. However, I'd prefer not mixing patches, and this could happen in a separate ticket.

comment:18 Changed 3 years ago by akaariai

I tested the pull request 198 with pyspatialite on Ubuntu 12.04. What I did:

sudo apt-get install libgeos-dev
sudo pip install pyspatialite
./runtests --settings=test_spatialite.py gis

Result was a core dump. Output (but not the actual dump) attached.

Of course, I can't apply the PR as the above issue needs some resolution first.

Additionally, please check the install.txt changes for whitespace errors, lines longer than 80 chars and use spaces instead of tabs.

Changed 3 years ago by akaariai

Core dump on Ubuntu 12.04

comment:19 Changed 2 years ago by reinout

  • Cc reinout@… added

comment:20 Changed 2 years ago by sfllaw

  • Cc sfllaw@… added

comment:21 Changed 20 months ago by timo

  • Easy pickings unset

comment:22 Changed 20 months ago by danols

  • Owner danols deleted
  • Status changed from assigned to new
  • Triage Stage changed from Accepted to Someday/Maybe

I no longer believe there is a need for this patch - installing through Debian apt system all the requirements as stated by the docs bypasses all of the PIP issues I have encountered and makes this not needed fix at least for me; however feel free to change it.

comment:23 follow-up: Changed 19 months ago by anentropic

  • Cc anentropic added

Just dropping in to say anything that can ease the pain of setting up GeoDjango is much appreciated. I'm trying to do it primarily via Homebrew and Pip... it'd be nice if it worked. The current instructions all lead to various dead ends

I'm currently stuck here:
http://stackoverflow.com/questions/11093593/python-the-pysqlite-library-does-not-support-c-extension-loading

If using pip install pyspatialite makes things more 'just work' then PLEASE do it. It's not helpful to say oh yeah I got my install working through Debian apt... it has zero value to users not on Debian.

Can a core dev please set this back to 'Accepted' as it had previously been marked?

comment:24 Changed 19 months ago by anentropic

Well after actually trying to apply the patch and go this route I think there are more problems. Basically pyspatialite is broken.

https://code.google.com/p/pyspatialite/issues/detail?id=5
https://code.google.com/p/pyspatialite/issues/detail?id=9

comment:25 Changed 18 months ago by anonymous

As a note, it's being/is fixed (at the moment, there is a new pre-release package on pypi. after a bit of testing, I'll upgrade it to stable)

comment:26 in reply to: ↑ 23 Changed 17 months ago by ramiro

Replying to anentropic:

Just dropping in to say anything that can ease the pain of setting up GeoDjango is much appreciated. I'm trying to do it primarily via Homebrew and Pip... it'd be nice if it worked. The current instructions all lead to various dead ends

I'm currently stuck here:
http://stackoverflow.com/questions/11093593/python-the-pysqlite-library-does-not-support-c-extension-loading

The answer to that SO question is the same as the one the Geodjango docs have mentioned for some time: https://docs.djangoproject.com/en/1.6/ref/contrib/gis/install/spatialite/#pysqlite2

[...]

Can a core dev please set this back to 'Accepted' as it had previously been marked?

Having tried the pyspatialite path myself recently, even with the latest development code, and getting the segmentation faults Anssi mentioned in comment:18 back in Jul '12 when trying to execute syncdb seems to indicate that it hasn't yet reached a point where we can:

  • Support pypatialite by replacing the module used by the main SQLite Django DB backend.
  • Recommend it as the easiest path to get GIS support with SQLite.

and that the documented process of using platform-packages plus possibly building pysqlite2 with a modified configuration still is the safest strategy to reach success.

Because of this, I think the 'someday/maybe' tag is appropiate at this point.

comment:27 Changed 17 months ago by anentropic

I am still unable to install Spatialite. The solution in SO question and GeoDjango docs mentioned above don't work for me. Full details here:
http://stackoverflow.com/questions/20505665/install-spatialite-for-python-geodjango-on-os-x

Note: See TracTickets for help on using tickets.
Back to Top