Geodjango running through memory

[Taylor]

I have been trying to upload a raster to my postgis backend and keep running into

django.db.utils.OperationalError: cannot allocate memory for output buffer

I try to upload the file in the django shell using the following:

raster = GDALRaster('rasters/data/my_image.tif', write=True) 
r = RastModel(name="Foo", image=raster)
r.save()

The image is only about a gb and I have plenty of ram to handle it. When I save the raster it runs through almost all 32gb of ram and then throws the error. From all the docs and posts I have seen it appears that I am uploading the raster correctly so I am assuming there is a bug somewhere.

>>> rast = GDALRaster('rasters/data/raw_tempe.tif', write=True)
>>> r = Rast(name='Test', image=rast)
>>> r.save()
Traceback (most recent call last):
  File "/mnt/f/Projects/gis322/env/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
psycopg2.OperationalError: cannot allocate memory for output buffer


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/mnt/f/Projects/gis322/env/lib/python3.6/site-packages/django/db/models/base.py", line 718, in save
    force_update=force_update, update_fields=update_fields)
  File "/mnt/f/Projects/gis322/env/lib/python3.6/site-packages/django/db/models/base.py", line 748, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/mnt/f/Projects/gis322/env/lib/python3.6/site-packages/django/db/models/base.py", line 831, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/mnt/f/Projects/gis322/env/lib/python3.6/site-packages/django/db/models/base.py", line 869, in _do_insert
    using=using, raw=raw)
  File "/mnt/f/Projects/gis322/env/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/mnt/f/Projects/gis322/env/lib/python3.6/site-packages/django/db/models/query.py", line 1136, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/mnt/f/Projects/gis322/env/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1289, in execute_sql
    cursor.execute(sql, params)
  File "/mnt/f/Projects/gis322/env/lib/python3.6/site-packages/django/db/backends/utils.py", line 100, in execute
    return super().execute(sql, params)
  File "/mnt/f/Projects/gis322/env/lib/python3.6/site-packages/django/db/backends/utils.py", line 68, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/mnt/f/Projects/gis322/env/lib/python3.6/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/mnt/f/Projects/gis322/env/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
  File "/mnt/f/Projects/gis322/env/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/mnt/f/Projects/gis322/env/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.OperationalError: cannot allocate memory for output buffer

[Simon Charette]

Please provide a full traceback so we can determine in which phase the process runs out of memory. I assume it's happening at on the save() call?

Also while I'm not that familiar with GDAL rasters I know that TIFF images can be heavily compressed so it's possible that even if the compressed file is only 1GB it could be way larger uncompressed.

[Taylor]

Replying to Simon Charette:

Please provide a full traceback so we can determine in which phase the process runs out of memory. I assume it's happening at on the save() call?

Also while I'm not that familiar with GDAL rasters I know that TIFF images can be heavily compressed so it's possible that even if the compressed file is only 1GB it could be way larger uncompressed.

I added the traceback above let me know if I can supply anything else to help resolve the issue. If wanted I could supply the image as well since it is freely available landsat imagery.

[Tim Graham]

Yes, when opening a ticket you should include enough details to reproduce the problem.

If you can't debug the issue yourself to explain why Django is at fault, generally, you should ask for help on our support channels before opening a ticket. I see you already opened ​a question on Stackoverflow so that's fine. Please don't use this ticket tracker as a second level support channel.

I see that the exception comes from PostgreSQL, not Django. If you find that Django is at fault, feel free to reopen.

[Simon Charette]

Taylor, you might want to try getting help from the ​geodjango mailing list as well or the ​GIS dedicated stack exchange instead or the generic StackOverflow.

If you've noticed the process chew through all of your RAM then the uncompression of the image looks like a smoking gun to me. You could try using an image manipulation tool to uncompress the image to disk and compare its file size.

As Tim said nothing proves this is an actual bug with Django so we cannot dedicate time to investigate further. If you can come up with a more detailed report of what is happening please report it back here even if Django is not at fault as it might help others hitting a similar issue. Finally I'd suggest you include your exact model definition in your reports as well as it's not completely clear whether or not RastModel.image is an ImageField or RasterField and mention your are using PostGIS with a RASTER column on the GIS stack exchange in the latter case. Best of luck.