FORCE_SCRIPT_NAME is not respected for static URLs

[Brian Helba]

​The documentation for the STATIC_URL setting states:

If STATIC_URL is a relative path, then it will be prefixed by the server-provided value of SCRIPT_NAME (or / if not set). This makes it easier to serve a Django application in a subpath without adding an extra configuration to the settings.

However, when using FORCE_SCRIPT_NAME, the value of STATIC_URL when serving requests is never updated appropriately. This breaks URL construction via django.templatetags.static.static (used in templates as {% static ... %}).

For example, this causes the Django Admin pages to break when using FORCE_SCRIPT_NAME to serve Django under a subpath. Generally, using FORCE_SCRIPT_NAME causes Django to behave incorrectly: view URLs are constructed to respect it, static URLs are constructed to not respect it.

I believe that this bug likely also affects static URLs when using the SCRIPT_NAME WSGI environment variable too, but I haven't verified that yet.

---

There is definitely some history here, but I believe previous bug reports have struggled to articulate this problem. I'm including all of the below detail in the hope that this issue will be understood well enough to acknowledge it as a legitimate bug.

Long ago, the following reports were made, which I believe are not relevant to this problem (but I'm summarizing them because they've been claimed to be duplicates of this problem):

• #7930 only discussed view URLs (and using reverse()); this now works as expected, but static URLs are still broken
• #30634 is probably irrelevant; it claimed vague problems with SCRIPT_NAME and runserver; the problem here is general to all servers, including both runserver and WSGI
• #31724 is probably a true duplicate of #7930

More recently, we've seen:

• #34892 is probably the same as this problem, but the reporter struggled to articulate the behavior and I believe it was mistakenly closed as a duplicate of the aforementioned old issues
• #35985 claimed that this is problem is limited to using threading within management commands, then got derailed by the niche use case and suggestions around the low-level set_script_prefix API

---

I believe that I understand the exact cause of the bug. Note, my use of some lifecycle events and specific thread names may be limited to the runserver case, but the exact same behaviors manifest with WSGI (and I believe that equivalent things are occurring with a multiprocess lifecycle).

1. django.setup() is ​called very early by runserver
2. django.setup() ​calls set_script_prefix
3. set_script_prefix ​correctly sets django.urls.base._prefixes.value, but only for the current thread (since _prefixes is a thread-local object)
4. a new thread, django-main-thread, ​is spawned; again, I believe (and can locate if necessary) that an equivalent event also happens in a WSGI lifecycle
5. check_url_settings ​runs and accesses settings.STATIC_URL; importantly, this is the first time in the startup lifecycle that settings.STATIC_URL has ever been accessed
6. django.conf.__getattr__ ​has special logic for STATIC_URL, so it calls the staticmethod LazySettings._add_script_prefix
7. LazySettings._add_script_prefix ​calls get_script_prefix
8. get_script_prefix ​looks at django.urls.base._prefixes.value, but it's running in a new thread (step 4), so it doesn't contain the FORCE_SCRIPT_NAME value (step 3) and returns "/"
9. django.conf.__getattr__ (step 6) permanently caches the incorrect value of settings.STATIC_URL in LazySettings.__dict__ (which is not thread-local); all future requests for settings.STATIC_URL will receive the incorrect value ("/", instead of settings.FORCE_SCRIPT_NAME)
10. The first HTTP request comes in
11. WSGIHandler.__call__ ​calls get_script_name
12. get_script_name ​correctly returns settings.FORCE_SCRIPT_NAME
13. WSGIHandler.__call__ ​calls set_script_prefix with the correct argument (the value settings.FORCE_SCRIPT_NAME)
14. set_script_prefix (just as in step 3) ​finally sets django.urls.base._prefixes.value (which, remember, is a thread-local variable, but will persist for at least the remainder of this HTTP request) with the correct value
15. While rendering the HTTP response, a template or some code calls ​the function django.templatetags.static; assume that the app "django.contrib.staticfiles" is installed and configured to use some subclass of StaticFilesStorage (which is Django's typical configuration)
16. django.contrib.staticfiles.storage.staticfiles_storage.url() ​is called
17. staticfiles_storage (an instance of StaticFilesStorage) ​is lazily constructed
18. StaticFilesStorage.__init__ is called; assume it has no arguments from settings.STORAGES["staticfiles"]["OPTIONS"] (​this is Django's default)
19. StaticFilesStorage.__init__ ​defaults to set its self._base_url to settings.STATIC_URL, but settings.STATIC_URL returns an incorrect value (step 9)
20. Continuing the call in step 16, FilesystemStorage.url ​forms the actual URL from self.base_url
21. self.base_url, ​a cached property, relies on the incorrect self._base_url
22. A static file URL is returned with the incorrect base URL, not respecting settings.FORCE_SCRIPT_NAME
23. Subsequent HTTP requests skip steps 17-19, but otherwise reply the work from step 10 onwards (and also result in incorrect static URLs)

In summary, the problem is that although there's code to attempt to set (via set_script_prefix) the correct script name both on Django's startup and again on every individual HTTP request, settings.STATIC_URL slips between a lifecycle "crack" and ends up with the wrong value (which doesn't incorporate the script name, contrary to its documentation), which persists over the entire request-response lifecycle.

[Brian Helba]

In terms of policies for a solution, although settings.FORCE_SCRIPT_NAME might practically be constant over Django's lifecycle, it's completely allowed by WSGI for the value of the SCRIPT_NAME environment variable to change on each request (and there's code in step 13 to support this). So, I think it follows that that settings.STATIC_URL should also be allowed to change on each new HTTP request. I'd encourage this path.

If we require that settings.STATIC_URL shouldn't change, it allows for some simpler fixes that only require STATIC_URL to eventually respect FORCE_SCRIPT_NAME and maybe the first-presented value of a SCRIPT_NAME environment variable. I think this would lead to confusing behavior for anyone using SCRIPT_NAME.

Alternatively, we could change ​the documentation for STATIC_URL to state that it doesn't respect SCRIPT_NAME, but that would be a larger policy change (not a simple bugfix) and it'd lead to a slipperly slope of just dropping all support for SCRIPT_NAME / FORCE_SCRIPT_NAME, etc. (which is out of scope for this issue). I think this change is too ambitious and would remove useful functionality for serving Django within subpaths behind reverse proxies, so I don't support it.

---

I see two reasonable changes:

1. Change the behavior (in step 6-9) of settings.STATIC_URL, so it's not permanently cached. Each access of settings.STATIC_URL would recomputed the value, using LazySettings._add_script_prefix.
2. Change set_script_prefix to reach into settings and invalidate the cache of STATIC_URL, causing step 13 to have its full intended effect. This is technically more efficient than having no cache of STATIC_URL, but introduces additional circularity in the relationship between all these methods and further breaks the abstraction of LazySettings (as LazySettings caches itself internally, but has its cache invalidated externally).

[Brian Helba]

Finally, I see that LazySettings ​also caches MEDIA_URL in a similar way. I haven't fully thought about the impact on the default (a.k.a. media) storage, but there may be similar issues there too. I'm trying to keep this issue more focused on static files, as they definitely cause breakages with the default Django Admin.

[Jacob Walls]

Thanks for the thorough report. Reproduced 👍

Regression in 7dbbe65647a54283fadf2c51f11a750a74425d7b given the side effects left by the system check in step 4 and django.contrib.staticfiles.utils.check_settings, which does something similar.

Reverting that commit fixes the FORCE_SCRIPT_NAME case; I would expect it to fix an invariant SCRIPT_NAME as well. I think triage on #34028 missed this, since we can't just document that prefixes shouldn't be set outside the request/response cycle and expect this to work, since these two system checks are *also* outside the request/response cycle.

See this comment ​anticipating a revert when someone posted on the closed PR, also encouraging us not to focus on varying values for SCRIPT_NAME:

There is an open ticket to that extend somewhere and I feel we might need to have to revert this. That said, settings are considered to be static, so if this changes per request Django will not be happy either way since the value was already passed to default_storage and used to instantiate the backend :/

Would you like to submit a PR?

[Jacob Walls]

/cc some folks from #34028

[Florian Apolloner]

I fear I don't have anything useful to add, personally I think we should burn it down and rebuild (the whole SCRIPT_NAME/FORCE_SCRIPT_NAME interaction with static files etc). For context I can offer https://code.djangoproject.com/ticket/35985#comment:5 which provides some "workarounds" -- "workarounds" in quotes since I don't think this is a workaround but how it should actually behave :D