compilemessages should ignore common environment and dependency directories by default

[Ayoub Bouaik]

The compilemessages management command recursively traverses the entire project tree looking for locale directories. However, it currently lacks default ignore patterns for common python virtual environments (venv, .venv, env, .env) and modern frontend package management directories (node_modules).

Because of this, if a project contains a third-party package inside node_modules or a local virtual environment that includes localized files with formatting quirks (such as a Byte Order Mark), compilemessages attempts to process them, resulting in unexpected execution overhead or critical crashes on files the developer did not write.

Steps to Reproduce
Inside a clean project workspace, run the following commands to simulate a third-party dependency with a BOM-encoded translation file:

1. Create a dummy package locale path: mkdir -p node_modules/mock-package/locale/en/LC_MESSAGES/

2. Add a dummy translation file containing a UTF-8 BOM flag: printf '\xEF\xBB\xBF# Dummy translation file\nmsgid ""\nmsgstr ""\n' > node_modules/mock-package/locale/en/LC_MESSAGES/django.po

3. Attempt to compile messages: python manage.py compilemessages

Expected Behavior
Django should automatically skip top-level dependency and virtual environment folders like node_modules, venv, and .venv unless explicitly instructed otherwise via the -i / --ignore CLI flags.

Actual Output / Traceback
The /home/ayoub/django_test/node_modules/mock-package/locale/en/LC_MESSAGES/django.po file has a BOM (Byte Order Mark). Django only supports .po files encoded in UTF-8 and without any BOM.
CommandError: compilemessages generated one or more errors.

[Jacob Walls]

Thanks for the ticket. Not sure yet about adjusting the default, and if we did, I assume we'd change makemessages as well?

Eager to hear from others.

I do think it's a good idea to show this usage in the docs, as it's more "real-world" than the hypothetical we have, e.g.

docs/ref/django-admin.txt

@@ -193 +193 @@
 
 .. console::
 
-    django-admin compilemessages --ignore=cache --ignore=outdated/*/locale
+    django-admin compilemessages --ignore=node_modules --ignore=outdated/*/locale
 
 ``createcachetable``
 --------------------

[Mike Edmunds]

I was about to "needsnewfeatureprocess" this. I think it makes a lot of sense, but there's an open-ended discussion about exactly which directories should be ignored. (.bower? .gradle? .gems? build? dist? etc.)

I'd probably suggest using .gitignore, as that should be an accurate list of things to ignore for any given project. (For projects using git. But what about other VCS ignore files? And even git's ignore mechanism is more complicated than a single .gitignore file.)

With all those questions, a new features discussion seemed appropriate. (But I agree about updating the docs example.)

[Jacob Walls]

Thanks Mike, that matches my gut feeling. Let's neeedsnewfeatureprocess this one, and then I can put up a PR for the minor docs tweak since you agree.

Ayoub, if you'd like to advance this one, you can open an issue on github.com/django/new-features to kick off a discussion about how best to derive the default value. Thanks.

[Natalia Bidart]

Django can't possibly maintain a hardcoded list of directories to ignore: venv names, editor artifacts, build tool outputs, and frontend tooling directories vary too widely across projects and teams. Any default list would be incomplete from day one and would require ongoing maintenance as the ecosystem evolves. For me, this is an unnecessary maintenance burden that we could and should avoid.

The right answer is already in place: compilemessages (and makemessages) both accept --ignore for exactly this purpose. Users should configure their own ignore patterns to match their project layout. For anything global, git's config handles per-user preferences without touching Django at all.

I would say this is a wontfix, unless I'm missing something?

[Mike Edmunds]

Replying to Natalia Bidart:

Django can't possibly maintain a hardcoded list of directories to ignore […]

Agreed.

The right answer is already in place: compilemessages (and makemessages) both accept --ignore for exactly this purpose. Users should configure their own ignore patterns to match their project layout. …

Yes, but…

… For anything global, git's config handles per-user preferences without touching Django at all.

I don't think compilemessages pays any attention to git's ignore mechanisms or global config, so users would need to replicate every ignore pattern as --ignore options.

A new compilemessages option like --ignore-file=.gitignore or --respect-gitignore or --use-vcs-ignore might be a reasonable feature request, but that would need discussion. Thus the newfeatureprocess suggestion. (Note that fully respecting git's ignore configuration is its own can of worms: ​https://github.com/biomejs/biome/discussions/6628.)

[Jacob Walls <jacobtylerwalls@…>]

In d5797ae5:

Refs #37188 -- Used a more realistic --ignore in compilemessages example.

[Jacob Walls <jacobtylerwalls@…>]

In 1b3c749:

[6.1.x] Refs #37188 -- Used a more realistic --ignore in compilemessages example.

Backport of d5797ae57d4822ee77091c8e8e5d35e5f4efca53 from main.

[Natalia Bidart]

Replying to Mike Edmunds:

I don't think compilemessages pays any attention to git's ignore mechanisms or global config, so users would need to replicate every ignore pattern as --ignore options.

Fair point, though I think that's a reasonable ask. Medium to large projects are very likely already orchestrating commands via a Makefile, justfile, or similar. Folding in the right --ignore incantation there is low friction IMO. And for projects with more specific needs, subclassing the management command is always an option (I forget often enough that this is a perfectly valid option).

A new compilemessages option like --ignore-file=.gitignore or --respect-gitignore or --use-vcs-ignore might be a reasonable feature request, but that would need discussion. Thus the newfeatureprocess suggestion. (Note that fully respecting git's ignore configuration is its own can of worms: ​https://github.com/biomejs/biome/discussions/6628.)

Thank you for the newfeatureprocess suggestion, I agree it's a good next step from the POV of a community driven project.

Personally I'm skeptical the feature is worth it. Django's goal is to solve the 80% well and give users hooks to handle the rest, and I think --ignore already covers the common case adequately. The Biome thread is a good illustration of why: even with a dedicated option for this, they ended up only reading .gitignore (not .git/info/exclude, not the global config) because git's ignore mechanism is a layered stack, not a single file. Users relying on those other layers would still be surprised. Also TIL about all those layers! :scream:

[Ayoub Bouaik]

Replying to Jacob Walls:

Thanks Mike, that matches my gut feeling. Let's neeedsnewfeatureprocess this one, and then I can put up a PR for the minor docs tweak since you agree.

Ayoub, if you'd like to advance this one, you can open an issue on github.com/django/new-features to kick off a discussion about how best to derive the default value. Thanks.

Thanks Jacob

I've opened a discussion in the django/new-features repository to continue exploring possible approaches:

​https://github.com/django/new-features/issues/183

Looking forward to hearing everyone's thoughts.