Opened 5 years ago

Closed 4 years ago

Last modified 21 months ago

#13488 closed Uncategorized (fixed)

Exceptions in GEOS I/O object destructor at process exit

Reported by: mroach@… Owned by: jbronn
Component: GIS Version: 1.2
Severity: Normal Keywords: gis
Cc: brandon.konkle@…, proppy@…, craig.destigter@…, sheats@… Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: no UI/UX: no

Description (last modified by jbronn)

I see this traceback in GeoDjango when the application exits. Not exactly
sure but there may be some memory leakage here.

Exception TypeError: "'NoneType' object is not callable" in <bound method_WKTReader.__del__ ofdjango.contrib.gis.geos.prototypes.io._WKTReader object at 0x6b1a790>> ignored

Exception TypeError: "'NoneType' object is not callable" in <bound method _WKBReader.__del__ of <django.contrib.gis.geos.prototypes.io._WKBReader object at 0x6b43690>> ignored

Exception TypeError: "'NoneType' object is not callable" in <bound method WKBWriter.__del__ of <django.contrib.gis.geos.prototypes.io.WKBWriter object at 0x6b1ad50>> ignored

To reproduce, just run a distance function (for example:

airport.objects.distance("POINT (-74.3166750000000036 40.9000249999999994)")

Whent the application exits (or exit from a python prompt), you'll get the
exceptions.

Attachments (2)

13488.diff (607 bytes) - added by Rozza 5 years ago.
Handle dying threads
13488.1.diff (1.9 KB) - added by jbronn 4 years ago.
Create reference to thread_context in GEOSFunc instances.

Download all attachments as: .zip

Change History (28)

comment:1 Changed 5 years ago by anonymous

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

comment:2 Changed 5 years ago by anonymous

  • Component changed from django.contrib.comments to GIS

comment:3 Changed 5 years ago by jbronn

  • Description modified (diff)
  • Owner changed from nobody to jbronn
  • Status changed from new to assigned

comment:4 Changed 5 years ago by jbronn

I can't reproduce. I need details on the platform you're using. What version of Python/Django/GEOS/PostGIS? What OS are you running on (instinct tells me RHEL)?

I don't think memory leakage is occurring. GeoDjango instantiates up to 5 GEOS I/O objects per-thread. These references are cleaned up by the __del__ method on the I/O objects, however in your particular install of Python the GC order has the I/O object destructor wrapper destroyed before the Python object -- hence why it doesn't exist at the time of interpreter exit. The GEOS I/O references aren't being cleaned upon exit of the process, so even though the references aren't released the process memory space should be reclaimed by the OS regardless because the process is terminating.

In other words, it may look bad but it's relatively benign. But again, I need more details so I can confirm what's going on and possibly tweak variable names so it won't occur on platforms of your type.

comment:5 Changed 5 years ago by jbronn

  • milestone 1.2 deleted

This isn't a 1.2 release blocker.

comment:6 Changed 5 years ago by jbronn

  • Summary changed from GeoDjango to Exceptions in GEOS I/O object destructor at process exit

comment:7 Changed 5 years ago by mroach@…

We are running:
python: 2.6.5
django: trunk
GEOS: 3.2.1
PostGIS: 3.2.1 (i think)
OS: Arch linux

comment:8 Changed 5 years ago by bkonkle

I'm also getting this message upon exit. I'm running Django on an OS X Snow Leopard laptop, using Postgres 8.3, GEOS 3.1.1, Proj 4.7.0, PostGIS 1.3.6, and GDAL 1.6.3. We haven't started running Django 1.2 on our Ubuntu 8.04 dev server yet, but when we do I'll check to see if the message shows up there as well.

comment:9 Changed 5 years ago by bkonkle

  • Cc brandon.konkle@… added

comment:10 Changed 5 years ago by proppy

  • Cc proppy@… added

Changed 5 years ago by Rozza

Handle dying threads

comment:11 Changed 5 years ago by Rozza

I've added an attachment that handles / catches the dying thread scenario and quietly lets the rest in peace.

I have no idea how I'd write a test to prove it works - but I tested on my machine it stops the warnings.

I based it on how _threading_local.py from in the python library handles destruction.

comment:12 Changed 5 years ago by anonymous

  • Has patch set

comment:13 Changed 5 years ago by Rozza

  • Resolution set to wontfix
  • Status changed from assigned to closed

Updated after conversation on the geo_django google group.

The root cause for these problems is the same: the module holding the 
destructor is garbage collected in the dying thread before the geometry 
or I/O object is itself garbage collected (and tries to call the now 
non-existent function). 

To compound the issue, it appears that GC order changes depending on the 
version of Python used, as well as the platform it is running on.  In 
other words, Python 2.7 on Snow Leopard won't show these errors while 
Python 2.5 on Arch Linux will. 

However, for the ticket you opened (#13843), the developer may prevent 
these errors by _not_ creating module-level globals of GEOS geometry 
objects.  Here's an example of a module-level global: 


from django.contrib.gis.geos import GEOSGeometry 
geom = GEOSGeometry('POINT(5 23)') 
def my_view(): 
    do_something_with(geom) 


Instead of having `geom` defined in the global scope, it would be better 
to have it instantiated in the local scope of the view function instead. 

As it stands I'm -1 on the #13843 patch.  While it reduces log noise 
in your case, it masks that there could be an underlying problem with 
the developer's code.  Regardless, I'm keeping the ticket open because 
absent a technological fix, there should be at least note in the 
documentation on how to avoid it. 

-Justin 

comment:14 Changed 5 years ago by jbronn

  • Resolution wontfix deleted
  • Status changed from closed to reopened

comment:15 Changed 5 years ago by jbronn

I use thread-local module-level global I/O objects that developer has no control over, thus re-opening.

comment:16 Changed 5 years ago by jbronn

  • milestone set to 1.3
  • Patch needs improvement set
  • Triage Stage changed from Unreviewed to Ready for checkin
  • Version changed from 1.2-beta to 1.2

I got an Arch Linux VM set up and got the problem reproduced using standard packages and with the code from the tutorial. This is what I get in the logs:

[Wed Jul 07 08:26:48 2010] [error] Exception KeyError: KeyError(140158140106560,) in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
[Wed Jul 07 08:26:48 2010] [error] Exception KeyError: KeyError(140158140106560,) in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
[Wed Jul 07 08:26:48 2010] [error] Exception KeyError: KeyError(140158140106560,) in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
[Wed Jul 07 08:26:48 2010] [error] Exception KeyError: KeyError(140158140106560,) in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
[Wed Jul 07 08:26:48 2010] [error] Exception KeyError: KeyError(140158140106560,) in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
[Wed Jul 07 08:26:48 2010] [error] Exception KeyError: KeyError(140158140106560,) in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
[Wed Jul 07 08:26:48 2010] [error] Exception KeyError: KeyError(140158140106560,) in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
[Wed Jul 07 08:26:48 2010] [error] Exception KeyError: KeyError(140158140106560,) in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
[Wed Jul 07 08:26:48 2010] [error] Exception KeyError: KeyError(140158140106560,) in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
[Wed Jul 07 08:26:48 2010] [error] Exception KeyError: KeyError(140158140106560,) in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
[Wed Jul 07 08:26:48 2010] [error] Exception KeyError: KeyError(140158140106560,) in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
[Wed Jul 07 08:26:48 2010] [notice] caught SIGTERM, shutting down
[Wed Jul 07 08:26:48 2010] [error] Exception AttributeError: "'NoneType' object has no attribute 'destroy_geom'" in 
[Wed Jul 07 08:26:48 2010] [error] Exception AttributeError: "'NoneType' object has no attribute 'destroy_geom'" in 

comment:17 Changed 5 years ago by jbronn

  • Triage Stage changed from Ready for checkin to Accepted

comment:18 Changed 5 years ago by cdestigter

  • Cc craig.destigter@… added

Changed 4 years ago by jbronn

Create reference to thread_context in GEOSFunc instances.

comment:19 Changed 4 years ago by jbronn

Need feedback from users to test this out on their platforms. This patch tries to address root cause of these error messages, rather than masking them in a try/except statement.

comment:20 Changed 4 years ago by jbronn

There are a few of you CC'ed on this ticket, anyone tried out the patch? I'd like to get this into 1.3.

comment:21 Changed 4 years ago by sheats

  • Cc sheats@… added

I just tried the patch and it works fine -- error message goes away completely. How refreshing!

comment:22 Changed 4 years ago by jbronn

  • Resolution set to fixed
  • Status changed from reopened to closed

Fixed in r15378.

comment:23 Changed 4 years ago by jbronn

In [15379]:

[1.2.X] Fixed #13488 -- No longer generate unhandled exceptions that may occur when destructors of global GEOS I/O objects are called on process termination.

Backport of r15378 from trunk.

comment:24 Changed 4 years ago by jacob

  • milestone 1.3 deleted

Milestone 1.3 deleted

comment:25 Changed 21 months ago by matt.rigal@…

  • Easy pickings unset
  • Severity set to Normal
  • Type set to Uncategorized
  • UI/UX unset

I actually didn't want to reopen the ticket, but I do experience the same issues on Django 1.5.1 with Gdal 1.10 and Geos 3.3.5 on MacOSX 10.8. And I do not have module-level globals.

comment:26 Changed 21 months ago by timo

@matt.rigal -- please open a new ticket with details.

Note: See TracTickets for help on using tickets.
Back to Top