GEOS Polygons and Collections (across versions) cause segmentation faults on macOS arm64 (M1)

[Aapo Rista]

I got new Macbook Air (M1, 2020) and when I'm trying to use a project transferred from previous Intel Macbook Pro, I get Segmentation fault 11 in functions that use MultiPoint, MultiLinestring or MultiPolygon.

How to reproduce:

Create a virtualenv, a Django project and start the shell:

cd /tmp/ && python3 -m venv venv multisegfault && source multisegfault/bin/activate && pip install django && django-admin startproject segfaulttest && cd segfaulttest && python manage.py shell

Copy paste this line to the shell:

from django.contrib.gis.geos import MultiLineString; MultiLineString()

You should get Segmentation fault: 11.

It may be possible that GEOSGeom_createCollection_r causes crash, but I don't know how to try it outside Django.

Libraries are installed using HomeBrew and they have versions:

Django 3.2rc1 and 3.1.7

$ gdal-config --version
3.2.2

$ geos-config --version
3.9.1

$ python -V
Python 3.9.2

# SELECT PostGIS_full_version();
POSTGIS="3.1.1 aaf4c79" [EXTENSION] PGSQL="130" GEOS="3.9.1-CAPI-1.14.2" PROJ="7.2.1" LIBXML="2.9.10" LIBJSON="0.15" LIBPROTOBUF="1.3.3" WAGYU="0.5.0 (Internal)"

Everything is good on Intel Macbook Pro (same library versions) and on Ubuntu 20.04 (default versions).

[Mariusz Felisiak]

Duplicate of #32544.

[Mariusz Felisiak]

Can you enable faulthandler and check the origin of a segmentation fault?

catchsegv python -X faulthandler script.py

[Aapo Rista]

I'm still figuring out how to run catchsegv on Mac Os, but here are the rest:

$ python -X faulthandler manage.py shell
Python 3.9.2 (default, Mar 24 2021, 20:21:37) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from django.contrib.gis.geos import MultiLineString; MultiLineString()
Fatal Python error: Segmentation fault

Current thread 0x00000001009b7d40 (most recent call first):
  File "/path/to/Project/venv/lib/python3.9/site-packages/django/contrib/gis/geos/prototypes/threadsafe.py", line 49 in __call__
  File "/path/to/Project/venv/lib/python3.9/site-packages/django/contrib/gis/geos/libgeos.py", line 152 in __call__
  File "/path/to/Project/venv/lib/python3.9/site-packages/django/contrib/gis/geos/collections.py", line 56 in _create_collection
  File "/path/to/Project/venv/lib/python3.9/site-packages/django/contrib/gis/geos/collections.py", line 36 in __init__
  File "<console>", line 1 in <module>
  File "/opt/homebrew/Cellar/python@3.9/3.9.2_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/code.py", line 90 in runcode
  File "/opt/homebrew/Cellar/python@3.9/3.9.2_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/code.py", line 74 in runsource
  File "/opt/homebrew/Cellar/python@3.9/3.9.2_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/code.py", line 258 in push
  File "/opt/homebrew/Cellar/python@3.9/3.9.2_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/code.py", line 232 in interact
  File "/opt/homebrew/Cellar/python@3.9/3.9.2_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/code.py", line 301 in interact
  File "/path/to/Project/venv/lib/python3.9/site-packages/django/core/management/commands/shell.py", line 82 in python
  File "/path/to/Project/venv/lib/python3.9/site-packages/django/core/management/commands/shell.py", line 100 in handle
  File "/path/to/Project/venv/lib/python3.9/site-packages/django/core/management/base.py", line 398 in execute
  File "/path/to/Project/venv/lib/python3.9/site-packages/django/core/management/base.py", line 354 in run_from_argv
  File "/path/to/Project/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 413 in execute
  File "/path/to/Project/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 419 in execute_from_command_line
  File "/path/to/Project/django_project/mydata/manage.py", line 17 in main
  File "/path/to/Project/django_project/mydata/manage.py", line 21 in <module>
Segmentation fault: 11

[Aapo Rista]

Ok, I found something in Console's Crash Reports (I left out a few hundred of lines). See GEOSGeom_createCollection_r there.

Again, I think this is not GEOS/GDAL/Django version related issue (as assumed in #32544), but a bug in GEOS/Python/libgeos or something.

Process:               Python [86891]
Path:                  /opt/homebrew/*/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python
Identifier:            Python
Version:               3.9.2 (3.9.2)
Code Type:             ARM-64 (Native)
Parent Process:        bash [86834]
Responsible:           Terminal [1531]
User ID:               502

Date/Time:             2021-03-27 16:48:34.731 +0200
OS Version:            macOS 11.2.3 (20D91)
Report Version:        12
Anonymous UUID:        1F81D35A-CC9F-662C-B696-4BC56365499D

Sleep/Wake UUID:       24C66F23-05FD-4B16-BF9A-9BEF835E38FA

Time Awake Since Boot: 450000 seconds
Time Since Wake:       6200 seconds

System Integrity Protection: enabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x000000016dda4000
Exception Note:        EXC_CORPSE_NOTIFY

Termination Signal:    Segmentation fault: 11
Termination Reason:    Namespace SIGNAL, Code 0xb
Terminating Process:   exc handler [86891]

VM Regions Near 0x16dda4000:
    Stack                       16cda4000-16dda4000    [ 16.0M] rw-/rwx SM=ALI  thread 0
--> 
    Submap                      180000000-190000000    [256.0M] r--/r-- SM=SHM  machine-wide VM submap

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libgeos_c.1.dylib             	0x000000010528873c GEOSGeom_createCollection_r + 88
1   libgeos_c.1.dylib             	0x000000010528872c GEOSGeom_createCollection_r + 72
2   libffi.dylib                  	0x0000000194e64050 ffi_call_SYSV + 80
3   libffi.dylib                  	0x0000000194e6c9d8 ffi_call_int + 944
4   _ctypes.cpython-39-darwin.so  	0x0000000103f38684 _ctypes_callproc + 856
5   _ctypes.cpython-39-darwin.so  	0x0000000103f330f4 PyCFuncPtr_call + 220
6   org.python.python             	0x00000001024cbbe0 _PyObject_Call + 128
7   org.python.python             	0x00000001025c2030 _PyEval_EvalFrameDefault + 40440
8   org.python.python             	0x00000001025b7290 _PyEval_EvalCode + 436
9   org.python.python             	0x00000001024cbe7c _PyFunction_Vectorcall + 364

[Brenton Partridge]

I can confirm that this is indeed a Django bug, and I was able to replicate segfaults on Django's main branch, so it appears the GEOS 3.9 update in https://code.djangoproject.com/ticket/32544 did not fix this. It's also been replicated in e.g. ​https://github.com/libgeos/geos/issues/528.

It appears that in the current GEOS integration, argument signatures were omitted in favor of automatically casting arguments for a few polygon-related API calls. While this appears to have worked fine on x86 and x64, on macOS arm64 (for native M1 processors), it causes seemingly random data to be provided in place of arguments after the first, reliably causing segmentation faults whenever a Polygon, MultiPolygon, or MultiLineString is constructed. Underlying CFFI internals may be less lenient to unspecified arguments in Apple's implementation than on other platforms. Moving to explicit signatures works fine, and I see no reason having explicit signatures should cause problems on other platforms (though we should ensure CI passes).

I've submitted a PR at ​https://github.com/django/django/pull/15214.

All GIS tests now pass, and the changes are fully covered by test_geos. The deeper issue in testing is that CI does not run on Apple hardware, so it will be difficult to prevent regressions. However, the code changes are small and well-documented, and I'd suggest that Apple/arm64 CI will be important but not urgent.

For organizations using older versions of Django, I've also made a monkey-patch available at ​https://gist.github.com/bpartridge/26a11b28415d706bfb9993fc28767d68 which can be added in any initialization code (or even called from settings.py) before Polygon objects are created. When applied in test settings against the main branch, all GIS tests pass as well; functionality is identical to the PR above.

PR should be ready for review; happy to make changes as requested! Happy holidays - and hopefully this brings some holiday cheer to developers who might be seeing segfaults on their newly gifted machines!

[Mariusz Felisiak]

Thanks for the investigation! It ends with Segmentation fault so IMO it qualifies for a backport to Django 4.0 as a crashing bug.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 19fb8388:

Fixed #32600 -- Fixed Geometry collections and Polygon segmentation fault on macOS ARM64.

[Mariusz Felisiak <felisiak.mariusz@…>]

In b85ceaab:

[4.0.x] Fixed #32600 -- Fixed Geometry collections and Polygon segmentation fault on macOS ARM64.

Backport of 19fb838803f63eef0726a370050443b693f109be from main