ORM error on apache + wsgi

[Christian]

When I deployed the application on the serve, a Ubuntu 16.04.01, python3.5 in virtualenv, sqlite, apache2 + wsgi I experienced this "strange" error:
Environment:

Request Method: GET
Request URL: http://www.6245.it/command/send/3

Django Version: 1.9.9
Python Version: 3.5.2
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django_extensions',
 'django.contrib.sites',
 'rest_framework',
 'engine',
 'formtools',
 'allauth',
 'allauth.account',
 'allauth.socialaccount']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware']



Traceback:

File "/var/www/virtualenvs/mailinglist/lib/python3.5/site-packages/django/core/handlers/base.py" in get_response
  149.                     response = self.process_exception_by_middleware(e, request)

File "/var/www/virtualenvs/mailinglist/lib/python3.5/site-packages/django/core/handlers/base.py" in get_response
  147.                     response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/var/www/virtualenvs/mailinglist/lib/python3.5/site-packages/django/views/generic/base.py" in view
  68.             return self.dispatch(request, *args, **kwargs)

File "/var/www/virtualenvs/mailinglist/lib/python3.5/site-packages/django/views/generic/base.py" in dispatch
  88.         return handler(request, *args, **kwargs)

File "/var/www/virtualenvs/mailinglist/lib/python3.5/site-packages/django/views/generic/base.py" in get
  157.         context = self.get_context_data(**kwargs)

File "/var/www/mailinglist/engine/views/command.py" in get_context_data
  52. 		su.delivery(kwargs["shipment_id"])

File "/var/www/mailinglist/engine/common/shipment.py" in delivery
  115. 			tu.parse()

File "/var/www/mailinglist/engine/common/template.py" in parse
  108. 			parser.feed(s)

File "/var/www/mailinglist/engine/common/template.py" in start
  64. 				p.save()

File "/var/www/virtualenvs/mailinglist/lib/python3.5/site-packages/django/db/models/base.py" in save
  708.                        force_update=force_update, update_fields=update_fields)

File "/var/www/virtualenvs/mailinglist/lib/python3.5/site-packages/django/db/models/base.py" in save_base
  736.             updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)

File "/var/www/virtualenvs/mailinglist/lib/python3.5/site-packages/django/db/models/base.py" in _save_table
  801.                                       forced_update)

File "/var/www/virtualenvs/mailinglist/lib/python3.5/site-packages/django/db/models/base.py" in _do_update
  831.         filtered = base_qs.filter(pk=pk_val)

File "/var/www/virtualenvs/mailinglist/lib/python3.5/site-packages/django/db/models/query.py" in filter
  790.         return self._filter_or_exclude(False, *args, **kwargs)

File "/var/www/virtualenvs/mailinglist/lib/python3.5/site-packages/django/db/models/query.py" in _filter_or_exclude
  808.             clone.query.add_q(Q(*args, **kwargs))

File "/var/www/virtualenvs/mailinglist/lib/python3.5/site-packages/django/db/models/sql/query.py" in add_q
  1243.         clause, _ = self._add_q(q_object, self.used_aliases)

File "/var/www/virtualenvs/mailinglist/lib/python3.5/site-packages/django/db/models/sql/query.py" in _add_q
  1269.                     allow_joins=allow_joins, split_subq=split_subq,

File "/var/www/virtualenvs/mailinglist/lib/python3.5/site-packages/django/db/models/sql/query.py" in build_filter
  1203.             condition = self.build_lookup(lookups, col, value)

File "/var/www/virtualenvs/mailinglist/lib/python3.5/site-packages/django/db/models/sql/query.py" in build_lookup
  1097.                     lhs = self.try_transform(lhs, name, lookups)

File "/var/www/virtualenvs/mailinglist/lib/python3.5/site-packages/django/db/models/sql/query.py" in try_transform
  1115.                 (name, lhs.output_field.__class__.__name__))

Exception Type: FieldError at /command/send/3
Exception Value: Unsupported lookup 'exact' for AutoField or join on the field not permitted.

When I run the same application with the same data with the builtin development server the error doesn't raise. I've attached the requirements (obtained via pip freeze) and the file with the full code.

The strange think is that after one or two "F5" to reload the page the code works without errors.

[Tim Graham]

Please see TicketClosingReasons/UseSupportChannels for way to get help debugging your application. If it turns out to be a bug in Django then you can reopen the ticket explaining why Django is at fault. Thanks!

[Christian]

This is a BUG in interaction between lxml parser and django orm, I guess.

I cannot investigate where and why the problem raises: I don't know django and lxml internals so well.
I've moved the ORM saving mechanism outside the parser and eveything works. But this is not the solution.

I've posted the link to thi ticket also on the lxml trouble ticketing system.

[Shai Berger]

Hi,

I've looked at the data you've supplied, and I must agree with Tim: The report, in its current state, cannot be used to investigate the supposed bug. Please try to reduce it to a test-case that demonstrates the problem and is easy to set up, so we can at least reproduce it.

I suspect that there may be an actual problem -- in your code or in Django -- related to the fact that you try to manipulate a Foreign Key by changing only the underlying id attribute. But we need you to find a reliable way to demonstrate it.

[Christian]

HI,

thanks. Now I modified the code simply cutting the db update code outside the start method of the lxml parser and everything works fine.
The strange thing is that the error happened only when I copied the code to the production system with apache + wsgi. It has never happened during development using development server. And second thing: why after a couple of page refresh the code works?

I'll try to reduce the code, but I suspect that there is a problem related to multi-process/threading running when deployed on apache and the error is due to interaction between "processes"

Christian