Failing test for black formatter missing install simulation

[Roelzkie]

The test on test.user_commands.tests.UtilsTests.test_run_formatters_handles_oserror_for_black_path attempts to assert a ​`FileNotFoundError` when simulating a missing black formatter installation.

./runtests.py user_commands.tests.UtilsTests.test_run_formatters_handles_oserror_for_black_path

However, it fails on the darwin platform, at least on MacOS v14.7.2 (Sonoma) M1 machine. I tested on a Linux platform, and the test passed.

The root issue is due to the subprocess.run command, which yields different OSError results on different OS's when the test reaches the ​`subprocess.run(["nonexistent", "--fast", "--")`].

For darwin (at least on MacOS v14.7.2 - M1), it yields a NotADirectoryError / OSError(20, "Not a directory") which fails the test.
For linux, it yields a FileNotFoundError / OSError(2, "File not found") which passes the test.

Ideally, this test should be able to handle it regardless of the machine platform.

See failed test full stacktrace: ​https://dpaste.org/pY48h (link will expire, see result below instead).

FAIL: test_run_formatters_handles_oserror_for_black_path (user_commands.tests.UtilsTests.test_run_formatters_handles_oserror_for_black_path) [FileNotFoundError]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/user1/.pyenv/versions/3.13.2/lib/python3.13/unittest/case.py", line 58, in testPartExecutor
    yield
  File "/Users/user1/.pyenv/versions/3.13.2/lib/python3.13/unittest/case.py", line 556, in subTest
    yield
  File "/Users/user1/projects/django/tests/user_commands/tests.py", line 584, in test_run_formatters_handles_oserror_for_black_path
    self.assertIn(exception.__qualname__, parsed_error)
    ^^^^^^^
  File "/Users/user1/.pyenv/versions/3.13.2/lib/python3.13/unittest/case.py", line 1174, in assertIn
    self.fail(self._formatMessage(msg, standardMsg))
    ^^^^^^^^^^^
  File "/Users/user1/.pyenv/versions/3.13.2/lib/python3.13/unittest/case.py", line 732, in fail
    raise self.failureException(msg)
    ^^^^^^^^^^^^^^^
AssertionError: 'FileNotFoundError' not found in 'Formatters failed to launch:Traceback (most recent call last):
  File "/Users/user1/projects/django/django/core/management/utils.py", line 175, in run_formatters
    subprocess.run(
    ~~~~~~~~~~~~~~^
        [black_path, "--fast", "--", *written_files],
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        capture_output=True,
        ^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/user1/.pyenv/versions/3.13.2/lib/python3.13/subprocess.py", line 556, in run
    with Popen(*popenargs, **kwargs) as process:
         ~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user1/.pyenv/versions/3.13.2/lib/python3.13/subprocess.py", line 1038, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                        pass_fds, cwd, env,
                        ^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
                        gid, gids, uid, umask,
                        ^^^^^^^^^^^^^^^^^^^^^^
                        start_new_session, process_group)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user1/.pyenv/versions/3.13.2/lib/python3.13/subprocess.py", line 1974, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
NotADirectoryError: [Errno 20] Not a directory: \'nonexistent\''
​
----------------------------------------------------------------------

Update: Please see ​https://github.com/django/django/pull/19591

[Sarah Boyce]

Mike would you be able to confirm this and that the PR fixes the issue? (See from #36162 you might have a similar setup)

[Mike Edmunds]

Replying to Sarah Boyce:

Mike would you be able to confirm this and that the PR fixes the issue? (See from #36162 you might have a similar setup)

I'm not able to reproduce the issue as reported. (But I'm on macOS 15.5 not 14.7. I'm not sure why that would make a difference. Looks like we're both on Python 3.13.2.)

% python ./runtests.py user_commands.tests.UtilsTests.test_run_formatters_handles_oserror_for_black_path

Testing against Django installed in '~/dev/django/django' with up to 8 processes
Found 1 test(s).
System check identified no issues (0 silenced).
.
----------------------------------------------------------------------
Ran 1 test in 0.009s

OK

% sw_vers
ProductName:            macOS
ProductVersion:         15.5
BuildVersion:           24F74

% uname -mrsv 
Darwin 24.5.0 Darwin Kernel Version 24.5.0: Tue Apr 22 19:48:46 PDT 2025; root:xnu-11417.121.6~2/RELEASE_ARM64_T8103 arm64

% python --version
Python 3.13.2

% git rev-parse --short=7 HEAD
23529b6

[Roelzkie]

Replying to Mike Edmunds:

Replying to Sarah Boyce:

Mike would you be able to confirm this and that the PR fixes the issue? (See from #36162 you might have a similar setup)

I'm not able to reproduce the issue as reported. (But I'm on macOS 15.5 not 14.7. I'm not sure why that would make a difference. Looks like we're both on Python 3.13.2.)

% python ./runtests.py user_commands.tests.UtilsTests.test_run_formatters_handles_oserror_for_black_path

Testing against Django installed in '~/dev/django/django' with up to 8 processes
Found 1 test(s).
System check identified no issues (0 silenced).
.
----------------------------------------------------------------------
Ran 1 test in 0.009s

OK

% sw_vers
ProductName:            macOS
ProductVersion:         15.5
BuildVersion:           24F74

% uname -mrsv 
Darwin 24.5.0 Darwin Kernel Version 24.5.0: Tue Apr 22 19:48:46 PDT 2025; root:xnu-11417.121.6~2/RELEASE_ARM64_T8103 arm64

% python --version
Python 3.13.2

% git rev-parse --short=7 HEAD
23529b6

Thank you for checking. I'm not sure why it would break. I pulled again from the latest main branch just to be sure, but the test still failed. I also followed your commands to show more information.

$ python ./runtests.py user_commands.tests.UtilsTests.test_run_formatters_handles_oserror_for_black_path
Testing against Django installed in '/Users/user1/django/django' with up to 8 processes
Found 1 test(s).
System check identified no issues (0 silenced).
F
======================================================================
FAIL: test_run_formatters_handles_oserror_for_black_path (user_commands.tests.UtilsTests.test_run_formatters_handles_oserror_for_black_path) [FileNotFoundError]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/user1/django/tests/user_commands/tests.py", line 584, in test_run_formatters_handles_oserror_for_black_path
    self.assertIn(exception.__qualname__, parsed_error)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'FileNotFoundError' not found in 'Formatters failed to launch:Traceback (most recent call last):\n  File "/Users/user1/django/django/core/management/utils.py", line 175, in run_formatters\n    subprocess.run(\n    ~~~~~~~~~~~~~~^\n        [black_path, "--fast", "--", *written_files],\n        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n        capture_output=True,\n        ^^^^^^^^^^^^^^^^^^^^\n    )\n    ^\n  File "/Users/user1/.pyenv/versions/3.13.2/lib/python3.13/subprocess.py", line 569, in run\n    with Popen(*popenargs, **kwargs) as process:\n         ~~~~~^^^^^^^^^^^^^^^^^^^^^^\n  File "/Users/user1/.pyenv/versions/3.13.2/lib/python3.13/subprocess.py", line 1051, in __init__\n    self._execute_child(args, executable, preexec_fn, close_fds,\n    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n                        pass_fds, cwd, env,\n                        ^^^^^^^^^^^^^^^^^^^\n    ...<5 lines>...\n                        gid, gids, uid, umask,\n                        ^^^^^^^^^^^^^^^^^^^^^^\n                        start_new_session, process_group)\n                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/Users/user1/.pyenv/versions/3.13.2/lib/python3.13/subprocess.py", line 1987, in _execute_child\n    raise child_exception_type(errno_num, err_msg, err_filename)\nNotADirectoryError: [Errno 20] Not a directory: \'nonexistent\'\n'

----------------------------------------------------------------------
Ran 1 test in 0.008s

FAILED (failures=1)

$ sw_vers
ProductName:            macOS
ProductVersion:         14.7.2
BuildVersion:           23H311

$ uname -mrsv
Darwin 23.6.0 Darwin Kernel Version 23.6.0: Fri Nov 15 15:13:56 PST 2024; root:xnu-10063.141.1.702.7~1/RELEASE_ARM64_T8103 arm64

$ python --version
Python 3.13.2

$ git rev-parse --short=7 HEAD
23529b6

[Natalia Bidart]

Accepting based on the conversations in the ticket and the PR.

Roelzkie, Mike, Jacob: are either of you using a VM to run the tests? Or all you all running "native" macos? (if that makes sense)

[Roelzkie]

Replying to Natalia Bidart:

Accepting based on the conversations in the ticket and the PR.

Roelzkie, Mike, Jacob: are either of you using a VM to run the tests? Or all you all running "native" macos? (if that makes sense)

I'm running the test on a native macOS in my case.

[Jacob Walls]

I was also testing on a bare metal macOS.

[nessita <124304+nessita@…>]

In 58fc404:

Fixed #36479 -- Improved how FileNotFoundError is triggered in code formatter tests.

Ensured the test for formatter subprocess FileNotFoundError doesn't rely
on platform-specific behavior, improving reliability on macOS and other
systems by consistently using pathlib to build test paths.