Opened 4 years ago

Closed 4 years ago

Last modified 3 years ago

#26152 closed Cleanup/optimization (fixed)

django.setup() hangs django if ever used in a module that's imported by an app

Reported by: Harry Percival Owned by: nobody
Component: Documentation Version: 1.9
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

The docs suggest using django.setup() for any script that needs to be run standalone:

https://docs.djangoproject.com/en/1.9/topics/settings/#calling-django-setup-is-required-for-standalone-django-usage

But this causes problems if a standalone script also contains some code that is imported by a regular django app.

Here's a minimal repro:

https://github.com/hjwp/recursive_django_setup_bug_minimal_repro/commit/15208c126796980cbe238ffe168d8b1bafaace26

if foo.py contains django.setup(), and myapp.models.py tries to import from foo.py, that now causes a hang such that no manage.py call will ever complete. ouch!

The solution/workaround is to put any calls to django.setup() inside a

if __name__ == '__main__':
    import django
    django.setup()

I gather that fixing django.setup() / apps.populate() to make it reentrant is hard. So this could be a quick documentation fix instead?

cf #18251, a more complex version of this bug I think, which is in fact quite simple to run into...

Change History (16)

comment:1 Changed 4 years ago by Harry Percival

I want to submit a documentation fix, but I'm not sure where. The new docs don't really have a section that specifically recommends using django.setup() for standalone scripts?

comment:2 Changed 4 years ago by Harry Percival

aha, there it is, in docs/ref/applications.txt.

comment:3 Changed 4 years ago by Tim Graham

Component: UncategorizedDocumentation
Type: UncategorizedCleanup/optimization

How about in the topics/settings section that you linked to in the description? Something like, "When writing standlone scripts, keep them as minimal as possible and put application logic in other modules of your application. You can't import scripts that call django.setup() in your application as this will lead to a deadlock."

comment:4 Changed 4 years ago by Harry Percival

hi tim, yes, sorry, i was looking at an out-of-date code tree...

comment:5 Changed 4 years ago by Harry Percival

comment:6 Changed 4 years ago by Tim Graham

Triage Stage: UnreviewedAccepted

I don't think it's a good practice to write "standalone scripts" that are imported Django itself.

comment:7 Changed 4 years ago by Aymeric Augustin

We may be able to detect that case and raise an exception instead of hanging. (I didn't look at the code too closely.)

comment:8 Changed 4 years ago by Harry Percival

Hi Aymeric, there's a couple of ways I can think of that would detect cycles and then a couple of things we could do with it:

detect options:

  • import the traceback module and examine the call stack for a previous call to django.setup()
  • set some global variable in the module, _setup_called = True, or whatever it may be. (there is already a class variable called .ready, which is set to true at the end of the apps.populate() call. So maybe we could just add a second class variable that's set at the beginning)

what to do:

  • raise an exception
  • or, just an early return. Maybe setup() should just be idempotent

comment:9 Changed 4 years ago by Harry Percival

Just been digging into this. And I am not an experienced coder of multithreaded code, so I probably shouldn't be allowed.

Here's the current code:

        # populate() might be called by two threads in parallel on servers
        # that create threads before initializing the WSGI callable.
        with self._lock:
            if self.ready:
                return

            # app_config should be pristine, otherwise the code below won't
            # guarantee that the order matches the order in INSTALLED_APPS.
            if self.app_configs:
                raise RuntimeError("populate() isn't reentrant")

            # [...] rest of populate code
            self.ready = True

So switching _lock to being a threading.RLock will have the desired effect of raising the "populate isn't reentrant" exception that's already been defined, if populate is ever called recursively by the same thread.

I'm not sure I understand the code though. Under what circumstances, with the current threading.Lock(), would we expect the RuntimeError to be encountered?

comment:10 Changed 4 years ago by Aymeric Augustin

Yes, I was thinking of adding a boolean flag when the process starts.

I /think/ we discussed using a RLock before and rejected it, but I'm not sure. I have to investigate. I'll try to find time.

comment:11 Changed 4 years ago by Harry Percival

I might be wrong but, given the use of threading.Lock(), isn't the RunTimeError if self.app_configs theoretically impossible?

comment:12 Changed 4 years ago by Tim Graham <timograham@…>

Resolution: fixed
Status: newclosed

In 0fb1185:

Fixed #26152 -- Documented how to avoid django.setup() deadlock in standalone scripts.

comment:13 Changed 4 years ago by Harry Percival

Hi all, can I propose a re-open? We just got bitten by this thing today again, where calling django.setup() was messing with the logging setup in weird and unpredictable ways in our unit tests.

It seems like the ideal version of django.setup() should be idempotent, or only able to be called once?

Or should I open a different ticket?

comment:14 Changed 4 years ago by Tim Graham <timograham@…>

In fe41a134:

[1.9.x] Fixed #26152 -- Documented how to avoid django.setup() deadlock in standalone scripts.

Backport of 0fb1185538aeec9004fb9c84d96b81dc2778f66a from master

comment:15 Changed 4 years ago by Tim Graham

I guess a new ticket is appropriate since the "Documentation" bit has been completed.

comment:16 Changed 3 years ago by Aymeric Augustin

Harry: eventually I found the time to investigate this issue and filed #27176. Feel free to comment on that issue if you have anything to add.

Note: See TracTickets for help on using tickets.
Back to Top