Opened 6 years ago

Last modified 6 years ago

#29878 new Bug

GEOSContextHandle leaks probably due to thread local object destructing order — at Initial Version

Reported by: Yong Li Owned by: nobody
Component: GIS Version: 1.11
Severity: Normal Keywords: GEOS leak GIS
Cc: Sergey Fedoseev Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

I have observed consistent GEOSContextHandle leaks when using Django Geometry features in temporary threads. And I can avoid the leaks by manually clearing all attributes of django.contrib.gis.geos.prototypes.io.thread_context. My theory is that destructors of attributes in io.thread_context call some GEOSFunc objects, and that can create new GEOSContextHandle while Python is clearing thread local storage.

  1. threadsafe.thread_context.handle is cleared
  2. io.thread_context attributes are cleared
  3. io.thread_context attributes are destructed, and then created new threadsafe.thread_context.handle.

When I am trying a minimized sample, I also found that it got double free or corruption error very often if I do GC in main thread right after the thread using GEOS is joined. So this may be another big issue

BTW, I am using Python 2.7.12 and django 1.11. But after checking the latest Django code, I think the issue is still there.

My sample code is:

Code highlighting:

#!/usr/bin/env python
import gc
import threading
from django.contrib.gis.geos import GEOSGeometry

_old_objs = None
_new_objs = None
_first_time = True


def gc_objects():
    gc.collect()
    objs_counts = {}
    gc_objs = {}
    for obj in gc.get_objects():
        key = str(type(obj))
        if key in objs_counts:
            objs_counts[key] += 1
        else:
            gc_objs[key] = obj
            objs_counts[key] = 1
    return objs_counts, gc_objs


def dump_memory_leaks():
    global _old_objs
    global _new_objs
    global _first_time
    if _first_time:
        _old_objs, _ = gc_objects()
        _first_time = False
    else:
        _new_objs, gc_objs = gc_objects()
        leaked = {}
        for k, v in _new_objs.iteritems():
            old_v = _old_objs.get(k)
            if old_v:
                diff = _new_objs[k] - old_v
                if diff > 0:
                    leaked[str(k)] = diff
            else:
                leaked[str(k)] = v

        print "Leaks: {}".format(leaked)

        _new_objs = None


def use_geos():
    GEOSGeometry('POINT(5 23)')
    # These lines can get rid of the GEOSContextHandle leak
    # from django.contrib.gis.geos.prototypes.io import thread_context as io_thread_context
    # io_thread_context.__dict__.clear()
    dump_memory_leaks()


if __name__ == '__main__':
    for i in xrange(10):
        t = threading.Thread(target=use_geos)
        t.start()
        t.join()
        # If I do GC here, it will crash with "double free or corruption" at random `i`
        # dump_memory_leaks()


Change History (0)

Note: See TracTickets for help on using tickets.
Back to Top