#23804 closed New feature (fixed)
Add a RasterField to GeoDjango
Reported by: | Daniel Wiesmann | Owned by: | Daniel Wiesmann |
---|---|---|---|
Component: | GIS | Version: | dev |
Severity: | Normal | Keywords: | raster gdal |
Cc: | Triage Stage: | Ready for checkin | |
Has patch: | yes | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description
Raster integrations is becoming stronger for database back-ends such as PostGIS, so an integration of raster data into Django would open new possibilities for online mapping. Handling raster data in online environments is currently hard to accomplish and integration into a web framework would make this easier in many cases.
Adding a RasterField makes sense as a first step to handling raster data input and output. Spatial querying and other operations could then be built on top of that.
Change History (33)
comment:1 by , 10 years ago
Triage Stage: | Unreviewed → Accepted |
---|
comment:2 by , 10 years ago
Owner: | changed from | to
---|---|
Status: | new → assigned |
comment:3 by , 10 years ago
I have made good process in creating some basic raster functionality, including gdal bindings, a GDALRaster object, a RasterField and a Tiler engine to create tiles from GDALRasters, and tests for most of it.
For now I have kept all new code in a submodule of contrib/gis/gdal, see
https://github.com/geodesign/django/tree/gdalraster/django/contrib/gis/gdal/raster
Now I was thinking of moving the module to master and starting to write the documentation for it. But I am not sure if it would be better to integrate it directly into the contrib.gis.gdal module. In the gdal.raster module, I have mimicked the structure of that module (for instance by creating a prototypes/ds.py file with the ctypes gdal bindings).
Does anyone have time to give me some advice or discuss how to proceed from here?
Note: I branched off the stable/1.7.x branch because like this I am able to use the code in one of my projects and play with it in a full project development environment.
comment:4 by , 10 years ago
Nice to see quick progress!
I think that we could aim for more integration with main gdal module. For example, when I compare driver.py and raster/driver.py, I see much code duplication. The RasterField
model should live somewhere in gis/db/models
(maybe by creating a fields module). I also would like to see all prototypes in gdal/prototypes and tests in gdal/tests. Maybe the other files could be left in a raster
module. These are just quick thoughts, without deep looking in the code.
comment:5 by , 10 years ago
Daniel, look at this pull request: https://github.com/django/django/pull/3648
Could you review/comment on it, and tell if you think it's going in a good direction?
comment:8 by , 10 years ago
The base is now in. I'll continue to work on integrating Daniel's branch.
comment:9 by , 10 years ago
I branched off master and started integrating the raster files into the structure with the new driver. I followed your suggestions above. The new branch is this one:
https://github.com/geodesign/django/tree/raster/django/contrib/gis/gdal
- Moved all tests into gdal.tests folder
- Separated constants from utils functions
- Moved RasterField into
django.contrib.gis.db.models.fields
module
Most of the changes are file-rearangements.
The raster field can leverage some of the features around the GeometryFields, but here some of the code that can be re-used is named Geometry* for instance the GeometryProxy and the geom_type
property. Both are related to lazy-instanciation of field target instances (Rasters or Geometries).
See:
https://github.com/geodesign/django/blob/raster/django/contrib/gis/db/models/fields.py#L356
https://github.com/geodesign/django/blob/raster/django/contrib/gis/db/models/fields.py#L390
comment:10 by , 10 years ago
The RasterField is still very minimalistic, here are some thoughts of what is missing/could be improved:
- Integration with different db backends. Currently the raster IO is tailored towrards PostGIS. The value passed to the db is the output of the
wkb
property of the GDALRaster, which converts between PostGIS WKB and GDALRaster. This seems to work in the test-databases. The db_type is fixed to'raster'
, which is also PostGIS specific.
- A deeper integration with the GeometryField would be beneficial. For instance the SRID property could be shared in a master class. But this is currently integrated into GeometryField directly. So integrating the two would probably require a GeoField master class or so.
comment:11 by , 10 years ago
I've prepared a new (small) step, with the following patch. I've deliberately chosen some different paths from your branch. It's not to contradict you by pleasure :-), I may totally be wrong or naive with certain choices, but I'm trying to create the most Pythonesque API as possible. Feel free to criticize and contradict me in return!
comment:12 by , 10 years ago
Just had a quick look, this is awsome, its very clean like this. I can't resist to add a quote of a movie I saw the other day:
"You've got to understand your limitations. It's your limitations that make you the wonderful disaster that you most probably are. For me that is where collaboration comes in. To take an idea that is blind and unformed and that has been hatched largely in solitude and allow these strange collaborator creatures to morph it into something else, something that is better, that's really something to see."
-- Nick Cave on collaboration from 20,000 Days on Earth
My main concern for now on this is that it limits the data source to be only from a file. You can't instantiate a source from memory, which is what I had intended to use in the raster field. And in-memory rasters work just the same way as the file based ones.
Were you thinking of having a separate GDALRaster object that can be created from memory and used in a field? If you have time, maybe you could briefly summarize how you would see the rest of the code to look like.
Will have a closer look over the next few days.
comment:13 by , 10 years ago
Thanks for the quote :-)
My main concern for now on this is that it limits the data source to be only from a file.
Don't fear, it's just a technique, going step after step. Of course, the current functionality is almost unusable for anything serious. And like you I'm looking forward to new possibilities. Don't stop to write code, I wouldn't be able to write anything without taking inspiration from your work, seriously.
Now I may at some point question the extent of the code needed to be in the framework and the code which could be better in an external project. For me the RasterField
code is clearly an objective at short term. But the tiling functionality, for example, I'm still not completely convinced it belongs to Django core. We'll see.
comment:14 by , 10 years ago
I am aware that some of the code in the raster branch might be too high level for Django core and am happy to discuss this. Here is a start:
For the img property
I kept the img
property as part of it as this might be needed for showing the raster in the admin interface.
For the Tiler egine/utility
I envisioned the Tiler engine to be the raster version of the LayerMapping utility (see link below). For larger rasters, it is unpractical to load them as one single raster, so in many use cases some sort of tiling will be necessary. The tiler I wrote is very focused on creating tiles for TMS, which is probably not general enough as a tiling utitlity. Still I think something along those lines is necessary for loading large rasters from files into a db.
https://github.com/django/django/blob/master/django/contrib/gis/utils/layermapping.py
comment:17 by , 10 years ago
Note that to add the RasterField, we'll probably be facing the same issue as #23879 (skipping model creation for non PostGIS).
comment:18 by , 10 years ago
I was working on integrating pixel value read and write support. When you have time, please have a look at the following pull request:
comment:19 by , 10 years ago
Thanks for the great work, I will definitely put that in my review list.
Unfortunately, this will not be included in Django 1.8 (currently feature frozen), but let's aim for a bright raster support in Django 1.9 (including the raster db field, hopefully)!
comment:20 by , 10 years ago
Has patch: | set |
---|
comment:21 by , 10 years ago
I added a HEX representation for the GDAL Rasters to the above pull request. Rasters can be instantiated from HEX data and converted into that format too.
I was trying to find a standard for Raster data in binary/hex but unfortunately such a standard does not seem to exist.
http://gis.stackexchange.com/questions/130209/is-there-an-open-standard-for-raster-information
http://lists.osgeo.org/pipermail/gdal-dev/2015-January/040807.html
So I guess to create a RasterField, the conversion to the different databases will be quite different (I think for geometries, the OpenGIS specification is used in most spatial databases). I would propose to focusing on PostGIS support for a RasterField first and add sqlite and oracle afterwards.
comment:22 by , 10 years ago
I added a first take at a RasterField for PostGIS backends to the pull request.
The testing is very basic and I have not yet written any documentation for it and it lacks proper Index creation. However, I wanted to get feedback on the basic implementation before workin on it more.
comment:23 by , 10 years ago
Sorry, I was busy with #24214 which is no1 priority for GIS in 1.9. I should have time to review part of your work soon.
comment:24 by , 10 years ago
Patch needs improvement: | set |
---|
comment:27 by , 10 years ago
Here is a first draft of a Raster field for PostGIS >= 2.0 backends. I am not sure if I covered all the important aspects of creating a field, but it should serve as a starting point at least.
comment:29 by , 10 years ago
I updated the pull request for the first RasterField implementation. The raster field should now be fairly complete, but it became quite large due to that. Its hard to split things at this point though, as the different elements are quite interrelated. I hope its still an acceptable size.
Here is a description of the current state:
- The field is implemented for PostGIS backends only.
- Similar to
GeometryFields
, the field srid can be specified and the raster will automatically be transformed to the field srid upon saving.
- A spatial index for the raster is built by default, and can be disabled with
spatial_index=False
.
- The
null
andblank
flags can be specified.
- Lazy instantiation works for the raster field as well.
- Tests are in place for the RasterField and corresponding migration commands and operations
For this, I made some changes on the existing codebase:
- Generalized the
GeometryProxy
and renamed toSpatialProxy
. This is used for lazy-instantiation of objects related to spatial fields.
- Override the
get_indexes
introspection function for postgis, as the spatial indices are based on expressions and are not straight indices (usingST_ConvexHull
). So the sql code retrieving the list of indices has to be adopted to that.
- Update the
get_placeholder
function in thePostGISOperator
to implicitly reproject rasters that are passed to a field with a different srid than the field srid.
- Change in
PostGISSchemaEditor
to create spatial indices on raster (wrapping the index creation sql with theST_ConvexHull
function).
- Split parts of the
GeometryField
into a generalizedBaseSpatialField
class. TheGeometryField
and theRasterField
subclass theBaseSpatialField
.
- Updated the
gis_migrations
test module to also include theRasterField
into the tests.
comment:30 by , 9 years ago
Patch needs improvement: | unset |
---|---|
Triage Stage: | Accepted → Ready for checkin |
(Daniel is author of https://github.com/geodesign/django-raster/)