Django RasterField deserialization bug with pixeltype flags

[Ivor Bosloper]

After inserting some raster data with raster2pgsql into a Django model table with a RasterField column, I get a list index out of range when querying the table with a Django Queryset.

...
File "django/contrib/gis/db/models/fields.py" in from_db_value
  360.         return connection.ops.parse_raster(value)
File "django/contrib/gis/db/backends/postgis/operations.py" in parse_raster
  369.         return from_pgraster(value)
File "django/contrib/gis/db/backends/postgis/pgraster.py" in from_pgraster
  57.         pixeltype = POSTGIS_TO_GDAL[pixeltype]

It turns out the pixeltype value used is 39 while the POSTGIS_TO_GDAL list is only 16 elements long. The database field contains valid data but can not be deserialized with Django.

Steps for reproduction:

# Django model
class RasterModel(models.Model):
    rast = models.RasterField(srid=4326)

# raw sql, single pixel raster with nodata bit set
insert into app_rastermodel values(1, REPLACE('01 0000 0100 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 E6100000 0100 0100 6 2 03 03', ' ', '')::raster);

# query generating Exception
RasterModel.objects.get(pk=1)

Analysis: if we look at the ​Raster specification, the pixeltype is a byte of which the 4 highest bits are flags and the lowest 4 bits are the real pixeltype. Quoting the specification:

 Pixel type and storage flag
 ---------------------------

 Pixel type specifies type of pixel values in a band.
 Storage flag specifies whether the band data is stored
 as part of the datum or is to be found on the server's
 filesytem.

 There are currently 11 supported pixel value types, so 4
 bits are enough to account for all. We'll reserve
 the upper 4 bits for generic flags and define upmost as
 storage flag:
 
 #define BANDTYPE_FLAGS_MASK 0xF0
 #define BANDTYPE_PIXTYPE_MASK 0x0F

 #define BANDTYPE_FLAG_OFFDB     (1<<7)
 #define BANDTYPE_FLAG_HASNODATA (1<<6)
 #define BANDTYPE_FLAG_ISNODATA  (1<<5)
 #define BANDTYPE_FLAG_RESERVED3 (1<<4)

However, Django deserialization code only considers a single flag (BANDTYPE_FLAG_HASNODATA, bit 6, value 64):

# django/contrib/gis/db/backends/postgis/pgraster.py
def from_pgraster(data):
        ...
        # Subtract nodata byte from band nodata value if it exists
        has_nodata = pixeltype >= 64
        if has_nodata:
            pixeltype -= 64
       ...

The erroneous pixeltype 39 in my example actually had the BANDTYPE_FLAG_ISNODATA (bit 5, value 32) bit set which indicates all rastervalues are nodata.

I have created (my first django) patch and hope somebody can assist me in getting it correct and merged.

[Ivor Bosloper]

See ​https://github.com/django/django/pull/11381

[Daniel Wiesmann]

This is indeed a bug in the raster parser. I guess it has not shown so far because the flags are not an issue as long as you create rasters using Django. GDAL (and thuse GDALRaster) does not know about a "all nodata" flag as far as I know, and it does not allow things like different datatypes in bands in the same raster.

I had a look at the PR, but I am not familiar with bitwise operators in python, so I need a little more time. I'll do a more detailed review tomorrow and share my thoughts on GitHub.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 828e3b13:

Refs #30489 -- Made from_pgraster()/to_pgraster() use BANDTYPE_FLAG_HASNODATA and bitwise operators for nodata flag.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 7e15795:

Fixed #30489 -- Fixed RasterField deserialization with pixeltype flags.

Thanks Ivor Bosloper for the original patch.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 9358da7:

Refs #30489 -- Fixed RasterFieldTest.test_deserialize_with_pixeltype_flags() when run without numpy.