Make Django release artifacts PEP 625 compliant

[Sarah Boyce]

After the recent security release, I received some email notifications from PyPI:

This email is notifying you of an upcoming deprecation that we have determined may affect you as a result of your recent upload to 'Django'.

In the future, PyPI will require all newly uploaded source distribution filenames to comply with PEP 625. Any source distributions already uploaded will remain in place as-is and do not need to be updated.

Specifically, your recent upload of 'Django-5.1.4.tar.gz' is incompatible with PEP 625 because it does not contain the normalized project name 'django'.

In most cases, this can be resolved by upgrading the version of your build tooling to a later version that supports PEP 625 and produces compliant filenames.

If you have questions, you can email admin@… to communicate with the PyPI admin@… to communicate with the PyPI administrators.

I believe this relates to: ​https://github.com/pypi/warehouse/issues/12245
There's a chance that they might stop supporting non-normalized names by "end of year": ​https://github.com/pypi/warehouse/issues/12245#issuecomment-2272378958

I believe the name normalization was also discussed here: ​https://github.com/django/django/pull/17806#issuecomment-2176465382 hence cc-ing some people involved from that PR

I'm not 100% certain the path forward, it feels to me we might need to make a change and backport to all supported versions and perhaps announce this on the blog. I have no preference, but I need help to know what we should do here.

[Claude Paroz]

AFAIR, we refrained from using a recent setuptools with requires = ["setuptools>=61.0.0,<69.3.0"], to avoid that change (Django to django). Maybe you could try a setuptools version > 69.3 and see what's happening?

[Nick Pope]

Certainly using v69.3.0 as a minimum version for setuptools should make us compliant with PEP 625.

But, honestly, I remain as baffled as last time I looked into this. I found ​this comment which basically says that the source distribution filename should follow the same normalisation rules as the binary distribution filename and that states that uppercase should be replaced with lowercase. So why, oh why, do I get the following?

$ git clean -fdX
$ python -m build
...
Successfully built django-5.2.dev20241206115354.tar.gz and Django-5.2-py3-none-any.whl

I even tried with the latest setuptools, at the time of writing v75.6.0.

While we could go to the PyPA and say, "What the heck?", it's probably just easier to change name = Django to name = django in pyproject.toml and move on:

$ git clean -fdX
$ python -m build
...
Successfully built django-5.2.dev20241206115354.tar.gz and django-5.2-py3-none-any.whl

It also seems concerning that the version in the wheel filename is incomplete...

Here is a draft ​PR for consideration.

I'm not sure what we'll have to do regarding backports, but it might mean a more significant backport of the setup.cfg to pyproject.toml changes.

[Nick Pope]

Also looking at ​https://packaging.python.org/en/latest/specifications/binary-distribution-format/#file-name-convention...

Oh the irony that "django" is shown as an example of the distribution name... :')

[Baptiste Mispelon]

AFAIR, we refrained from using a recent setuptools with requires = setuptools>=61.0.0,<69.3.0, to avoid that change (Django to django).

Yes, that was done in this merge request: ​https://github.com/django/django/pull/17806#issuecomment-2176465382

I've just tried to build with the latest setuptools (python -m pip install build twine && python -m build from a clean checkout of the git repo) and it creates two files:

• django-5.2.dev20241206115354.tar.gz
• Django-5.2-py3-none-any.whl

The inconsistent naming seems to be a bug in setuptools [1] (mentionned in the comment on the patch I linked above).

My opinion is that we should be using the latest version of setuptools and let it generate the filenames for the builds. Those filenames are internal plumbing and are not normally exposed to users. They also don't change how Django is installed when using tools like pip, since those tools will normalize names internally as well.
One hitch with this plan is that the current release documentation (both the public one and I assume the ad-hoc scripts used by the fellows) makes assumptions about those filenames that would not hold true anymore.

[1] ​https://github.com/pypa/setuptools/issues/3777

[Baptiste Mispelon]

I took a look at the website's code to see what changes might be needed if/when the build files' naming structure changes.

It's not too bad, but it's still quite substantial: ​https://github.com/django/djangoproject.com/pull/1801.
With the changes proposed in this merge request there would be a slight change in the release process, since the build files should now be uploaded via the release creation form, and not via scp as currently documented.

[Baptiste Mispelon]

Hi, me again 😁

I took a look at the Jenkins confirm-release job mentionned in the release documentation and the only assumption it makes is that the checksum file is named Django-${VERSION}.checksum.txt, then it fetches the names of the build files (tarball and wheel as of now) from the checksum file itself.

So we can either keep the capitalized name for the checksum, or change one line in the Jenkins config.

[Natalia Bidart]

Hello everyone, I'm assigning this ticket to myself to "own" the transition process. I have read comments, reviewed PRs and pondered about this. I have some thoughts:

• For the changes to djangoproject.com, I added a newer comment suggesting to make the tarball and checksum a cached property (we could use something not fancy but simple like "If the lowercase django-... file exists, return that, otherwise return the uppercase one".
  • Uploading the tarballs via the Django admin feels a step in the wrong direction when thinking about automating the Django release process.
• For the changes to Django itself from Nick, those look good, we should think what to do for 5.0 and 4.2. We may need to backport the pyproject.toml changes to 5.0 and 4.2, I'm unclear about the implications of this, but it certainly has advantages from the POV of maintaining the project.
  • It would certainly help the release process to use the same release procedure for all versions.

[Natalia Bidart]

Added ​comment on the setuptools ​issue about the casing inconsistency for binaries asking for further details if available.

[Natalia Bidart]

After lots of reading, I think we are dealing with a mix of issues. The PyPI deprecation warning is only about the tarball, not about the wheel name. Specifically, there are two relevant PEPs:

• PEP 625, which addresses naming conventions for source distributions, and
• PEP 491, which addresses naming conventions for wheels.

As Baptiste said in comment:4, current setuptools (75.6.0) produces lowercase django tarball and capitalized Django wheel. If we proceed with this as-is, we would resolve the "PyPI warning" for the source distribution, addressing that part of the issue. However, we'd still need smart tools and a flexible download page to handle both naming formats. I think this is doable and would allow us to "detach" ourselves from the pressure of solving the wheel naming inconsistency from external sources.

[Natalia Bidart]

The fix in setuptools has landed, we "just" need a release from them: ​https://github.com/pypa/setuptools/issues/3777. And then we need to progress the ticket for djangoproject.com so downloads for lowercase Django woks.

[Nick Pope]

Patch updated to use setuptools v75.8.1 which was released containing the necessary fix.

[Natalia Bidart]

Commented on how after lots of work from Baptiste to adequate djangoproject.com to handle artifacts with any casing, we have a case-agnostic URL that we can use to download the tarball and wheel. This is now in a "test new version" script located in https://code.djangoproject.com/wiki/ReleaseTestNewVersion. With that in mind, and with PR #19178 which is about to be merged, the fixing branch would really "just" be the setuptools upgrade.

[nessita <124304+nessita@…>]

In 08dae5bd:

Refs #35980 -- Updated internal docs for artifact upload and release via admin site.

Recent updates to djangoproject.com extended the Release model to
support uploading and storing artifacts and checksum files. This branch
updates the "How to release Django" docs to match the new release flow.

[Natalia <124304+nessita@…>]

In 2344787:

[5.2.x] Refs #35980 -- Updated internal docs for artifact upload and release via admin site.

Recent updates to djangoproject.com extended the Release model to
support uploading and storing artifacts and checksum files. This branch
updates the "How to release Django" docs to match the new release flow.

Backport of 08dae5bd46a02a2e2798028c83d97b1894beffb8 from main.

[Natalia Bidart]

As an update, we are going to land Nick's changes after this Wednesday, when Sarah will make security releases and the 5.2 final release.

[nessita <124304+nessita@…>]

In 3ae049b:

Fixed #35980 -- Updated setuptools to normalize package names in built artifacts.

[Natalia <124304+nessita@…>]

In cd458ea:

[5.2.x] Fixed #35980 -- Updated setuptools to normalize package names in built artifacts.

Backport of 3ae049b26b995c650c41ef918d5f60beed52b4ba from main.

[Natalia <124304+nessita@…>]

In bbf376b:

[5.1.x] Fixed #35980 -- Updated setuptools to normalize package names in built artifacts.

Backport of 3ae049b26b995c650c41ef918d5f60beed52b4ba from main.

[nessita <124304+nessita@…>]

In 3456eee4:

[4.2.x] Fixed #35980 -- Updated setuptools to normalize package names in built artifacts.

Backport of 3ae049b26b995c650c41ef918d5f60beed52b4ba from main.

[nessita <124304+nessita@…>]

In 42ab993:

Refs #35980 -- Added release note about changes in release artifacts filenames.

[Natalia <124304+nessita@…>]

In ab5c0664:

[5.2.x] Refs #35980 -- Added release note about changes in release artifacts filenames.

Backport of 42ab99309d347f617d60751c2e8d627fb2963049 from main.

[Natalia <124304+nessita@…>]

In 85bdeb3:

[5.1.x] Refs #35980 -- Added release note about changes in release artifacts filenames.

Backport of 42ab99309d347f617d60751c2e8d627fb2963049 from main.

[Natalia <124304+nessita@…>]

In dc365ca:

[4.2.x] Refs #35980 -- Added release note about changes in release artifacts filenames.

Backport of 42ab99309d347f617d60751c2e8d627fb2963049 from main.

[GitHub <noreply@…>]

In 1307b8a:

Fixed #36402, Refs #35980 -- Updated built package name in reusable apps tutorial for PEP 625.

[Natalia <124304+nessita@…>]

In 81320d0a:

[5.2.x] Fixed #36402, Refs #35980 -- Updated built package name in reusable apps tutorial for PEP 625.

Backport of 1307b8a1cb05762147736d0f347792b33f645390 from main.

[Natalia <124304+nessita@…>]

In 129750a8:

[5.1.x] Fixed #36402, Refs #35980 -- Updated built package name in reusable apps tutorial for PEP 625.

Backport of 1307b8a1cb05762147736d0f347792b33f645390 from main.

[Natalia <124304+nessita@…>]

In c5b42632:

[4.2.x] Fixed #36402, Refs #35980 -- Updated built package name in reusable apps tutorial for PEP 625.

Backport of 1307b8a1cb05762147736d0f347792b33f645390 from main.