Opened 3 years ago

Closed 2 years ago

Last modified 2 years ago

#30489 closed Bug (fixed)

Django RasterField deserialization bug with pixeltype flags

Reported by: Ivor Bosloper Owned by: Ivor Bosloper
Component: GIS Version: dev
Severity: Normal Keywords: RasterField
Cc: Hasan Ramezani Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Ivor Bosloper)

After inserting some raster data with raster2pgsql into a Django model table with a RasterField column, I get a list index out of range when querying the table with a Django Queryset.

...
File "django/contrib/gis/db/models/fields.py" in from_db_value
  360.         return connection.ops.parse_raster(value)
File "django/contrib/gis/db/backends/postgis/operations.py" in parse_raster
  369.         return from_pgraster(value)
File "django/contrib/gis/db/backends/postgis/pgraster.py" in from_pgraster
  57.         pixeltype = POSTGIS_TO_GDAL[pixeltype]

It turns out the pixeltype value used is 39 while the POSTGIS_TO_GDAL list is only 16 elements long. The database field contains valid data but can not be deserialized with Django.

Steps for reproduction:

# Django model
class RasterModel(models.Model):
    rast = models.RasterField(srid=4326)

# raw sql, single pixel raster with nodata bit set
insert into app_rastermodel values(1, REPLACE('01 0000 0100 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 E6100000 0100 0100 6 2 03 03', ' ', '')::raster);

# query generating Exception
RasterModel.objects.get(pk=1)

Analysis: if we look at the Raster specification, the pixeltype is a byte of which the 4 highest bits are flags and the lowest 4 bits are the real pixeltype. Quoting the specification:

 Pixel type and storage flag
 ---------------------------

 Pixel type specifies type of pixel values in a band.
 Storage flag specifies whether the band data is stored
 as part of the datum or is to be found on the server's
 filesytem.

 There are currently 11 supported pixel value types, so 4
 bits are enough to account for all. We'll reserve
 the upper 4 bits for generic flags and define upmost as
 storage flag:
 
 #define BANDTYPE_FLAGS_MASK 0xF0
 #define BANDTYPE_PIXTYPE_MASK 0x0F

 #define BANDTYPE_FLAG_OFFDB     (1<<7)
 #define BANDTYPE_FLAG_HASNODATA (1<<6)
 #define BANDTYPE_FLAG_ISNODATA  (1<<5)
 #define BANDTYPE_FLAG_RESERVED3 (1<<4)

However, Django deserialization code only considers a single flag (BANDTYPE_FLAG_HASNODATA, bit 6, value 64):

# django/contrib/gis/db/backends/postgis/pgraster.py
def from_pgraster(data):
        ...
        # Subtract nodata byte from band nodata value if it exists
        has_nodata = pixeltype >= 64
        if has_nodata:
            pixeltype -= 64
       ...

The erroneous pixeltype 39 in my example actually had the BANDTYPE_FLAG_ISNODATA (bit 5, value 32) bit set which indicates all rastervalues are nodata.

I have created (my first django) patch and hope somebody can assist me in getting it correct and merged.

Change History (12)

comment:1 Changed 3 years ago by Ivor Bosloper

Description: modified (diff)

comment:3 Changed 3 years ago by Claude Paroz

Triage Stage: UnreviewedAccepted

comment:4 Changed 3 years ago by Ivor Bosloper

Description: modified (diff)

comment:5 Changed 3 years ago by Daniel Wiesmann

This is indeed a bug in the raster parser. I guess it has not shown so far because the flags are not an issue as long as you create rasters using Django. GDAL (and thuse GDALRaster) does not know about a "all nodata" flag as far as I know, and it does not allow things like different datatypes in bands in the same raster.

I had a look at the PR, but I am not familiar with bitwise operators in python, so I need a little more time. I'll do a more detailed review tomorrow and share my thoughts on GitHub.

comment:6 Changed 3 years ago by Mariusz Felisiak

Owner: changed from nobody to Ivor Bosloper
Patch needs improvement: set
Status: newassigned

comment:7 Changed 2 years ago by Hasan Ramezani

Patch needs improvement: unset
Last edited 2 years ago by Mariusz Felisiak (previous) (diff)

comment:8 Changed 2 years ago by Hasan Ramezani

Cc: Hasan Ramezani added

comment:9 Changed 2 years ago by Mariusz Felisiak <felisiak.mariusz@…>

In 828e3b13:

Refs #30489 -- Made from_pgraster()/to_pgraster() use BANDTYPE_FLAG_HASNODATA and bitwise operators for nodata flag.

comment:10 Changed 2 years ago by Mariusz Felisiak

Triage Stage: AcceptedReady for checkin
Version: 2.2master

comment:11 Changed 2 years ago by Mariusz Felisiak <felisiak.mariusz@…>

Resolution: fixed
Status: assignedclosed

In 7e15795:

Fixed #30489 -- Fixed RasterField deserialization with pixeltype flags.

Thanks Ivor Bosloper for the original patch.

comment:12 Changed 2 years ago by Mariusz Felisiak <felisiak.mariusz@…>

In 9358da7:

Refs #30489 -- Fixed RasterFieldTest.test_deserialize_with_pixeltype_flags() when run without numpy.

Note: See TracTickets for help on using tickets.
Back to Top