Django

Code

Ticket #5632 (new)

Opened 8 months ago

Last modified 1 month ago

Problems with DB objects on a multithreaded scenario

Reported by: claudio.marinozzi@thenetplanet.com (alias PyMan) Assigned to: nobody
Component: Database wrapper Version: SVN
Keywords: multi thread db Cc:
Triage Stage: Accepted Has patch: 1
Needs documentation: 0 Needs tests: 0
Patch needs improvement: 0

Description

Discussion topic that started here in Django Developers

There are some critical functions like get_or_create, create, delete, add, remove, etc. that if used in a multithreaded scenario could be crash or "damage data" (some samples in the discussion above) when working on same DB-data, or even not, at the same time.

These operations should be serialized (mutexed) or something like that.

I attach a diff file containing the quick, but working, solution. It's a decorated function used on those critical functions that does the mutex thing. Maybe the decoration should be added to some other functions too. Still don't know Django that good.

Attachments

dbmultithreaded.diff (5.8 kB) - added by claudio.marinozzi@thenetplanet.com on 09/28/07 05:16:25.
diff created with Django on Trunk at h12pm Italy time
dbmultithreaded2.diff (5.9 kB) - added by teepark on 03/26/08 12:24:50.
dbmultithreaded3.diff (6.3 kB) - added by teepark on 03/26/08 12:35:27.
based on SVN trunk r7364

Change History

09/28/07 05:16:25 changed by claudio.marinozzi@thenetplanet.com

  • attachment dbmultithreaded.diff added.

diff created with Django on Trunk at h12pm Italy time

09/28/07 10:52:11 changed by ubernostrum

  • needs_better_patch set to 1.
  • needs_tests changed.
  • needs_docs changed.

09/28/07 14:12:32 changed by ikelly

Please use the syntax "get_or_create = mutex(get_or_create)" for your decorators rather than the "@mutex" syntax. The reason for this is that the @ syntax is not supported in Python 2.3.

Also, please don't put your name in the code. If you want to be credited for your contributions, that's fine, but the proper place for it is in the AUTHORS file. See http://www.djangoproject.com/documentation/contributing/#coding-style for more information.

On a more stylistic note, this code: 

        try:
            res = func(*args, **kwargs)
        except:
            __serializer.release()
            raise
        __serializer.release()

is exactly equivalent to this: 

        try:
            res = func(*args, **kwargs)
        finally:
            __serializer.release()

12/02/07 18:54:07 changed by Simon G <dev@simon.net.nz>

  • has_patch set to 1.
  • stage changed from Unreviewed to Accepted.

03/26/08 12:24:50 changed by teepark

  • attachment dbmultithreaded2.diff added.

03/26/08 12:35:27 changed by teepark

  • attachment dbmultithreaded3.diff added.

based on SVN trunk r7364

03/26/08 12:36:35 changed by anonymous

  • needs_better_patch deleted.

sorry for the two tickets, I meant to include Claudio and myself in AUTHORS

04/15/08 21:20:36 changed by teepark

I don't have the necessary permission to modify/remove tickets otherwise I would remove mine.

This doesn't fix the multi-process situation - each process would just acquire it's own process-specific lock.

Also, get_or_create shouldn't be used with non-unique fields anyway (you get an AssertionError? if more than one row comes up), and get_or_create in the development version catches IntegrityErrors?, which will result from the unique constraint being violated by simultaneous get_or_creates.

So claudio's and my patches don't fix the whole problem, and the development version already does for sane cases. Sorry for all the unnecessary posts & patches.

Add/Change #5632 (Problems with DB objects on a multithreaded scenario)




Change Properties
Action