Path.resolve(strict=True) raises PermissionError.

[leonyxz]

Hello,

Python 3.8
FreeBSD 11.3-RELEASE-p12 with custom settings mentioned later.

On vanilla configuration of FreeBSD there is no issue.
Installation of FreeBSD in question has mac_bsdextended turned on (​https://www.freebsd.org/cgi/man.cgi?query=mac_bsdextended&sektion=4&apropos=0&manpath=FreeBSD+12.1-RELEASE+and+Ports)
Using ugidfw (​https://www.freebsd.org/cgi/man.cgi?query=ugidfw&sektion=8&manpath=freebsd-release-ports), the following rule has been set:

---

0 subject not uid root gid nobody object gid wheel type d mode sx

---

Attempts to run Django 3.1 result in

---

$ python manage.py runserver 0.0.0.0:1337
Traceback (most recent call last):
  File "manage.py", line 22, in <module>
    main()
  File "manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/home/userlogin/.virtualenvs/django31/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/home/userlogin/.virtualenvs/django31/lib/python3.8/site-packages/django/core/management/__init__.py", line 345, in execute
    settings.INSTALLED_APPS
  File "/home/userlogin/.virtualenvs/django31/lib/python3.8/site-packages/django/conf/__init__.py", line 83, in __getattr__
    self._setup(name)
  File "/home/userlogin/.virtualenvs/django31/lib/python3.8/site-packages/django/conf/__init__.py", line 70, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/userlogin/.virtualenvs/django31/lib/python3.8/site-packages/django/conf/__init__.py", line 177, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/home/userlogin/domains/django31.userlogin.com/public_python/project/settings.py", line 16, in <module>
    BASE_DIR = Path(__file__).resolve(strict=True).parent.parent
  File "/usr/local/lib/python3.8/pathlib.py", line 1159, in resolve
    s = self._flavour.resolve(self, strict=strict)
  File "/usr/local/lib/python3.8/pathlib.py", line 355, in resolve
    return _resolve(base, str(path)) or sep
  File "/usr/local/lib/python3.8/pathlib.py", line 339, in _resolve
    target = accessor.readlink(newpath)
  File "/usr/local/lib/python3.8/pathlib.py", line 439, in readlink
    return os.readlink(path)
PermissionError: [Errno 13] Permission denied: '/usr'

---

The issue seems to be comnected with following piece code
​https://github.com/python/cpython/blob/master/Lib/pathlib.py#L342-L346

---

    try:
                    target = accessor.readlink(newpath)
                except OSError as e:
                    if e.errno != EINVAL and strict:
                        raise
                    # Not a symlink, or non-strict mode. We just leave the path
                    # untouched.
                    path = newpath

---

With ugidfw enabled, os.readlink raises a PermissionError, which is unhandled by pathlib.

[Carlton Gibson]

Hi. Interesting. I'm struggling to see what if anything we can or should do here.

So this is the call we're making:

BASE_DIR = Path(__file__).resolve(strict=True).parent.parent

from settings.py.

If that raises PermissionDenied then (surely, I want to say) your permissions are just set wrong (too strict).

I can't see how it's meant to be an issue in Django?

(Sorry if I'm missing it...)

[Mariusz Felisiak]

#31945 was marked as a duplicate.

[tytusd]

Hi Carlton, thanks for looking into this. I have just been hit by this problem in all of my Django-based projects (as I host all of them in similarly set up environments). It looks like its clearly a regression - everything works fine with Django==3.0.8. I created a ticket #31945 (https://code.djangoproject.com/ticket/31945) with my stack trace clearly showing the regression (sorry I did not find this ticket before submitting mine..) - please take a look. There is no way to give the user running the website full access to entire '/usr' directory (server admins won't agree, it is a managed environment). And I do not see any reason why would this be needed? The project is set up inside of user's home dir, similarly the virtual environment is inside of the user's home directory.

What has changed between 3.0.8 and 3.1.0 that this problem started occuring?

This issue prevents me from updating Django version in all of my projects. I would really appreciate if some fix could be considered.

Replying to Carlton Gibson:

Hi. Interesting. I'm struggling to see what if anything we can or should do here.

So this is the call we're making:

BASE_DIR = Path(__file__).resolve(strict=True).parent.parent

from settings.py.

If that raises PermissionDenied then (surely, I want to say) your permissions are just set wrong (too strict).

I can't see how it's meant to be an issue in Django?

(Sorry if I'm missing it...)

[tytusd]

After updating Django from 3.0.8 to 3.1.0 it became unusable - I cannot call any manage.py management command in the production environment, where the user does not have root access and has very limited access to the /usr directory (it is a managed environment). Normally everything works fine, as Django projects run within virtual environments. Unfortunately, after the update, the Django code clearly attempts to access /usr directory (even though it is running inside of a virtual environment located elsewhere, within the user home directory). After downgrading back to 3.0.8 everything works fine again.

Stack trace below:

(web) [XYZ@s39]:<~/domains/XXX/public_python>$ python manage.py migrate
Traceback (most recent call last):
  File "manage.py", line 21, in <module>
    main()
  File "manage.py", line 17, in main
    execute_from_command_line(sys.argv)
  File "/usr/home/XYZ/.virtualenvs/web/lib/python3.6/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/usr/home/XYZ/.virtualenvs/web/lib/python3.6/site-packages/django/core/management/__init__.py", line 377, in execute
    django.setup()
  File "/usr/home/XYZ/.virtualenvs/web/lib/python3.6/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/usr/home/XYZ/.virtualenvs/web/lib/python3.6/site-packages/django/apps/registry.py", line 114, in populate
    app_config.import_models()
  File "/usr/home/XYZ/.virtualenvs/web/lib/python3.6/site-packages/django/apps/config.py", line 211, in import_models
    self.models_module = import_module(models_module_name)
  File "/usr/home/XYZ/.virtualenvs/web/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/home/XYZ/.virtualenvs/web/lib/python3.6/site-packages/django/contrib/auth/models.py", line 2, in <module>
    from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
  File "/usr/home/XYZ/.virtualenvs/web/lib/python3.6/site-packages/django/contrib/auth/base_user.py", line 8, in <module>
    from django.contrib.auth import password_validation
  File "/usr/home/XYZ/.virtualenvs/web/lib/python3.6/site-packages/django/contrib/auth/password_validation.py", line 160, in <module>
    class CommonPasswordValidator:
  File "/usr/home/XYZ/.virtualenvs/web/lib/python3.6/site-packages/django/contrib/auth/password_validation.py", line 170, in CommonPasswordValidator
    DEFAULT_PASSWORD_LIST_PATH = Path(__file__).resolve(strict=True).parent / 'common-passwords.txt.gz'
  File "/usr/local/lib/python3.6/pathlib.py", line 1141, in resolve
    s = self._flavour.resolve(self, strict=strict)
  File "/usr/local/lib/python3.6/pathlib.py", line 346, in resolve
    return _resolve(base, str(path)) or sep
  File "/usr/local/lib/python3.6/pathlib.py", line 330, in _resolve
    target = accessor.readlink(newpath)
  File "/usr/local/lib/python3.6/pathlib.py", line 440, in readlink
    return os.readlink(path)
PermissionError: [Errno 13] Permission denied: '/usr'
(web) [XYZ@s39]:<~/domains/XXX/public_python>$ pip install Django==3.0.8
Collecting Django==3.0.8
  Downloading Django-3.0.8-py3-none-any.whl (7.5 MB)
     |████████████████████████████████| 7.5 MB 4.5 MB/s 
Requirement already satisfied: sqlparse>=0.2.2 in /usr/home/XYZ/.virtualenvs/web/lib/python3.6/site-packages (from Django==3.0.8) (0.3.1)
Requirement already satisfied: asgiref~=3.2 in /usr/home/XYZ/.virtualenvs/web/lib/python3.6/site-packages (from Django==3.0.8) (3.2.10)
Requirement already satisfied: pytz in /usr/home/XYZ/.virtualenvs/web/lib/python3.6/site-packages (from Django==3.0.8) (2020.1)
Installing collected packages: Django
  Attempting uninstall: Django
    Found existing installation: Django 3.1
    Uninstalling Django-3.1:
      Successfully uninstalled Django-3.1
Successfully installed Django-3.0.8
(web) [XYZ@s39]:<~/domains/XXX/public_python>$ python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, database, pages, sessions
Running migrations:
  No migrations to apply.
(web) [XYZ@s39]:<~/domains/XXX/public_python>$

[Claude Paroz]

It looks strange that your user has not at least traversal directory permissions on /usr (the last x in dir permissions), as your home dir is inside /usr (which looks like a pecularity by itself).

[Carlton Gibson]

OK, let's re-open just to take a look. Still seems like a permission issue to me, but it's come up twice, so let's double-check.

edeec1247e52de6fc32cee93e96d4ce36003ea4b added the strict parameter (for the password validation case at least). Does removing that solve the issue?
(I can't see that the original report can be valid though... — surely you need permissions to your project folder? 🤔)

I'll CC Jon, who made most of the pathlib related updates here.

[tytusd]

I do confirm that removing "strict=True" parameter from the call to .resolve() solves the problem - everything works fine again.

I do have full access to the project folder (the project is located within subfolders in my home directory - '/usr/home/XYZ/domains/XXX/public_python - I have full access to all the folders starting from /usr/home/XYZ/ - not before it).

Also, I also have read and execute access to, for example, '/usr/local/bin' (where original Python interpreters reside and are copied from to virtual envs created and located inside of my home directory). But I do not have any access to '/usr'

(web) [qcgeo@s39]:<~>$ ls /usr
ls: /usr: Permission denied
(web) [XYZ@s39]:<~>$ ls /usr/local/bin/python*
/usr/local/bin/python            /usr/local/bin/python3           /usr/local/bin/python3.6         /usr/local/bin/python3.7m
/usr/local/bin/python-config     /usr/local/bin/python3-config    /usr/local/bin/python3.6-config  /usr/local/bin/python3.7m-config
/usr/local/bin/python2           /usr/local/bin/python3.5         /usr/local/bin/python3.6m        /usr/local/bin/python3.8
/usr/local/bin/python2-config    /usr/local/bin/python3.5-config  /usr/local/bin/python3.6m-config /usr/local/bin/python3.8-config
/usr/local/bin/python2.7         /usr/local/bin/python3.5m        /usr/local/bin/python3.7         /usr/local/bin/pythontex
/usr/local/bin/python2.7-config  /usr/local/bin/python3.5m-config /usr/local/bin/python3.7-config
(web) [XYZ@s39]:<~>$ 

Replying to Carlton Gibson:

OK, let's re-open just to take a look. Still seems like a permission issue to me, but it's come up twice, so let's double-check.

edeec1247e52de6fc32cee93e96d4ce36003ea4b added the strict parameter (for the password validation case at least). Does removing that solve the issue?
(I can't see that the original report can be valid though... — surely you need permissions to your project folder? 🤔)

I'll CC Jon, who made most of the pathlib related updates here.

[Mariusz Felisiak]

Thanks for extra info, I think we should remove strict=True in CommonPasswordValidator. I will reopen #31945 because that's not a duplicate but a separate issue.

[Mariusz Felisiak]

OK let's fix both and keep them in the first ticket, sorry for the noise.

[Mariusz Felisiak]

​PR

[GitHub <noreply@…>]

In e39e727:

Fixed #31912 -- Removed strict=True in Path.resolve() in project template and CommonPasswordValidator.

This caused permission errors when user didn't have permissions to
all intermediate directories in a Django installation path.

Thanks tytusd and leonyxz for reports.

Regression in edeec1247e52de6fc32cee93e96d4ce36003ea4b and
26554cf5d1e96db10d0d5f4b69683a22fb82fdf8.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 14a19700:

[3.1.x] Fixed #31912 -- Removed strict=True in Path.resolve() in project template and CommonPasswordValidator.

This caused permission errors when user didn't have permissions to
all intermediate directories in a Django installation path.

Thanks tytusd and leonyxz for reports.

Regression in edeec1247e52de6fc32cee93e96d4ce36003ea4b and
26554cf5d1e96db10d0d5f4b69683a22fb82fdf8.
Backport of e39e727ded673e74016b5d3658d23cbe20234d11 from master

[Jakub Szafrański]

This issue is still present in utils/autoreload.py and can cause Django to fail with the following traceback:

Traceback (most recent call last):
  File "manage.py", line 21, in <module>
    main()
  File "manage.py", line 17, in main
    execute_from_command_line(sys.argv)
  File "/usr/home/XXX/.virtualenvs/go/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/usr/home/XXX/.virtualenvs/go/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/home/XXX/.virtualenvs/go/lib/python3.8/site-packages/django/core/management/base.py", line 328, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/home/XXX/.virtualenvs/go/lib/python3.8/site-packages/django/core/management/commands/runserver.py", line 60, in execute
    super().execute(*args, **options)
  File "/usr/home/XXX/.virtualenvs/go/lib/python3.8/site-packages/django/core/management/base.py", line 369, in execute
    output = self.handle(*args, **options)
  File "/usr/home/XXX/.virtualenvs/go/lib/python3.8/site-packages/django/core/management/commands/runserver.py", line 95, in handle
    self.run(**options)
  File "/usr/home/XXX/.virtualenvs/go/lib/python3.8/site-packages/django/core/management/commands/runserver.py", line 102, in run
    autoreload.run_with_reloader(self.inner_run, **options)
  File "/usr/home/XXX/.virtualenvs/go/lib/python3.8/site-packages/django/utils/autoreload.py", line 599, in run_with_reloader
    start_django(reloader, main_func, *args, **kwargs)
  File "/usr/home/XXX/.virtualenvs/go/lib/python3.8/site-packages/django/utils/autoreload.py", line 584, in start_django
    reloader.run(django_main_thread)
  File "/usr/home/XXX/.virtualenvs/go/lib/python3.8/site-packages/django/utils/autoreload.py", line 299, in run
    self.run_loop()
  File "/usr/home/XXX/.virtualenvs/go/lib/python3.8/site-packages/django/utils/autoreload.py", line 305, in run_loop
    next(ticker)
  File "/usr/home/XXX/.virtualenvs/go/lib/python3.8/site-packages/django/utils/autoreload.py", line 345, in tick
    for filepath, mtime in self.snapshot_files():
  File "/usr/home/XXX/.virtualenvs/go/lib/python3.8/site-packages/django/utils/autoreload.py", line 361, in snapshot_files
    for file in self.watched_files():
  File "/usr/home/XXX/.virtualenvs/go/lib/python3.8/site-packages/django/utils/autoreload.py", line 260, in watched_files
    yield from iter_all_python_module_files()
  File "/usr/home/XXX/.virtualenvs/go/lib/python3.8/site-packages/django/utils/autoreload.py", line 105, in iter_all_python_module_files
    return iter_modules_and_files(modules, frozenset(_error_files))
  File "/usr/home/XXX/.virtualenvs/go/lib/python3.8/site-packages/django/utils/autoreload.py", line 141, in iter_modules_and_files
    resolved_path = path.resolve(strict=True).absolute()
  File "/usr/local/lib/python3.8/pathlib.py", line 1180, in resolve
    s = self._flavour.resolve(self, strict=strict)
  File "/usr/local/lib/python3.8/pathlib.py", line 362, in resolve
    return _resolve(base, str(path)) or sep
  File "/usr/local/lib/python3.8/pathlib.py", line 346, in _resolve
    target = accessor.readlink(newpath)
  File "/usr/local/lib/python3.8/pathlib.py", line 451, in readlink
    return os.readlink(path)
PermissionError: [Errno 13] Permission denied: '/usr'

Just to explain a little, this configuration is pretty common in shared-system environments - it is intentional and prevents users from enumerating local users on the system and accessing their home directories in case of miss-configured permissions.

[Mariusz Felisiak]

This is a regression in e28671187903e6aca2428374fdd504fca3032aee (Django 3.0) so it's not a release blocker and it doesn't qualify for a backport. Please open a new ticket.

[Mariusz Felisiak]

I've created a new ticket, see #32223.