Opened 6 years ago

Last modified 7 weeks ago

#28905 assigned Cleanup/optimization

Overhaul extra_requires to include more optional dependencies

Reported by: Jaap Roes Owned by: Nick Pope
Component: Packaging Version: dev
Severity: Normal Keywords: setup optional dependencies packages requirements
Cc: Herbert Fortes, Nick Pope Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: no UI/UX: no

Description

Django has a number of optional features that require the installation of additional Python packages.

Right now it's possible to pip install Django[bcrypt,argon2] to install Django and the optional packages (at the correct minimum version) required for bcrypt/argon2 support.

However this is not possible for other optional features, like database/cache backend, gis, certain image related features etc.

I've created a pull request adding the optional dependencies found in the test requirements (initially only psycopg2 for postgresql, but after a comment from pope1ni I followed their suggestion to include more). It changes the set of extra_requires to:

{
    'admindocs': ['docutils'],
    'argon2': ['argon2_cffi>=16.1.0'],
    'bcrypt': ['bcrypt'],
    'gis': [
        'geoip2',
        'numpy'
    ],
    'geoip': ['geoip2'],
    'jinja2': ['Jinja2>=2.9.2'],
    'memcached': ['python3-memcached'],
    'numpy': ['numpy'],
    'mysql': ['mysqlclient>=1.3.7'],
    'oracle': ['Oracle'],
    'images': ['Pillow'],
    'postgresql': ['psycopg2>=2.5.4'],
    'pylibmc': ['pylibmc; sys.platform != "win32"'],
    'sqlparse': ['sqlparse'],
    'selenium': ['selenium'],
    'test-parallel': ['tblib'],
    'yaml': ['PyYAML'],
}

So end users can do things like:

$ pip install Django[mysql,sqlparse,images,memcached]
$ pip install Django[postgresql,gis]
$ pip install Django[selenium,test-parallel]

Not only would this make it much easier to install a particular combination of Django and optional dependencies correctly, it also makes some of the optional features/dependencies a little bit more discoverable.

This might not the full set of optional dependencies, or the right grouping/naming, so a review is more than welcome.

Besides adding this to setup.py, additional changes to the documentation, test instructions and scripts, code warnings etc. will probably be necessary.

Change History (24)

comment:1 by Jaap Roes, 6 years ago

Component: UncategorizedPackaging

comment:2 by Sergey Fedoseev, 6 years ago

Has patch: set
Triage Stage: UnreviewedAccepted

comment:3 by Jaap Roes, 6 years ago

I have no idea where to go from here. I assume posting this to the developer mailinglist is the best way to gather feedback on this and come to a consensus of which packages and groupings.

edit: https://groups.google.com/forum/#!topic/django-developers/6IQ5VeY6jY8

Last edited 6 years ago by Jaap Roes (previous) (diff)

comment:4 by Carlton Gibson, 6 years ago

Related #27682

comment:5 by Carlton Gibson, 6 years ago

Needs documentation: set

comment:6 by Jaap Roes, 6 years ago

Easy pickings: set

I don't have the time/energy to get this done, but I feel that this would be an excellent, and fairly easy, ticket for someone to work on during the Djangocon sprints

comment:7 by oliver, 6 years ago

Owner: changed from nobody to oliver
Status: newassigned

comment:8 by oliver, 6 years ago

Owner: oliver removed
Status: assignednew

comment:9 by Israel Fermín Montilla, 6 years ago

Can I claim it?, I have some spare time I could invest on this

comment:10 by Herbert Fortes, 6 years ago

Cc: Herbert Fortes added

comment:11 by Jaap Roes, 6 years ago

Yes please claim it, it would be really helpful to, for example, identify pieces of documentation that should be updated when this lands.

comment:12 by Niranj Rajasekaran, 6 years ago

Owner: set to Niranj Rajasekaran
Status: newassigned

comment:13 by Jaap Roes, 5 years ago

I just discovered another optional dependency, on Linux one can install pyinotify for better reloading behaviour (https://docs.djangoproject.com/en/dev/ref/django-admin/#runserver).

So maybe adding {'dev': ['pyinotify ; sys_platform == "linux"']} would be helpful as well.

comment:14 by Nick Pope, 5 years ago

Needs documentation: unset
Owner: changed from Niranj Rajasekaran to Nick Pope

New PR.

comment:15 by Jaap Roes, 5 years ago

Thanks Nick for taking this ticket and trying to move it forwards.

The additional documentation about optional dependencies makes so much sense, it alone would satisfy some of the discoverability issues with optional dependencies. It would be nice if the overview of optional dependencies could link to the relevant sections in the docs. E.g. caching, databases, GIS, password hashers, template engines, serializers etc.

The grouping/inclusion of dependencies seems to be the main stumbling block. Adding extra_requires groups adds a maintenance burden on these groups, as in, they'll need to work for future versions of Django. For this reason I think it's probably not the right choice to name groups directly after the package they are installing (pylibmc, pyinotify, selenium etc.).

Maybe it would be easier to just start out with just adding the database extra's? Those seem non-controversial and the package names and version specs are non-obvious, so having those in Django's setup.py has a clear advantage.

It might also be interesting to figure out how to "deprecate" extra requirement groups, so at some point it would be possible to phase out the bcrypt or argon2 extra requirements.

I was also thinking, extra requirements would also be a nice place to include other "official" Django projects. E.g. pip install django[mysql,localflavor,comments] or pip install django[postgresql,channels].

Anyway, I don't think this ticket will go anywhere until someone with a more authoritative voice makes a decision what's wanted and unwanted.

in reply to:  15 comment:16 by Nick Pope, 5 years ago

Thanks for the feedback, please see responses below.

... It would be nice if the overview of optional dependencies could link to the relevant sections in the docs. E.g. caching, databases, GIS, password hashers, template engines, serializers etc.

Agreed. Will add in links to useful sections in the documentation.

... For this reason I think it's probably not the right choice to name groups directly after the package they are installing (pylibmc, pyinotify, selenium etc.).

Maybe it would be easier to just start out with just adding the database extra's? ...

I still feel that adding support for some of the packages other than database backends would be very useful. There should at the very least be an easy option to install all non-database backend packages, particularly useful for removing tests/requirements/*.txt.

It might also be interesting to figure out how to "deprecate" extra requirement groups, so at some point it would be possible to phase out the bcrypt or argon2 extra requirements.

I'm not sure it is possible to "deprecate" - in fact it doesn't really make sense - a package must be available during a deprecation period. However, when the package is removed, the group must either remain empty in extra_requires or be removed entirely if documented in the release notes.

I was also thinking, extra requirements would also be a nice place to include other "official" Django projects. E.g. pip install django[mysql,localflavor,comments] or pip install django[postgresql,channels].

I think it best to leave these out of the extra_requires option, but perhaps document them on the optional dependencies page along with a link to other django-related projects.

comment:17 by Carlton Gibson, 5 years ago

Patch needs improvement: set

comment:18 by Rishabh Arya, 4 years ago

Owner: changed from Nick Pope to Rishabh Arya

comment:19 by Manav Agarwal, 3 years ago

Can I claim this? I think I can move it forward.

comment:20 by Nick Pope, 3 years ago

Owner: changed from Rishabh Arya to Nick Pope

comment:21 by Carlton Gibson, 18 months ago

Easy pickings: unset

comment:22 by Natalia Bidart, 7 weeks ago

Cc: Nick Pope added

Nick, do you see any value in trying to resurface/push this? I have been recently asking about dependency management following a crusade to pin black to 23.12.1 for stable/4.2.x (PR #17781 and PR #17793), so I wondered if there would be a way to define deps (and versions) in a single place to be useful for all GitHub Actions, docs building, pre-commit hooks and tests/tox definitions.

comment:23 by Claude Paroz, 7 weeks ago

FWIW, I just pushed a commit I had prepare some times ago as a PR, which moves content of setup.* to pyproject.toml. I don't know if that changes anything in the game…

comment:24 by Nick Pope, 7 weeks ago

Hi Natalia. Sorry for the delay getting back to you. I'll dust off what I had and see where things stand - certainly things could be simplified and dependency stuff is scattered, in particular from a documentation PoV. Some of my ongoing work toward refreshing django-docker-box has exposed some areas that are particularly non obvious.

Thanks, Claude. I'll review that PR. It feels like it is time to reconsider the more complete migration to pyproject.toml in light of the distutils removal, deprecation of support for calling setup.py directly, and stabilisation of support for pyproject.toml in most tooling including setuptools.

Note: See TracTickets for help on using tickets.
Back to Top