Cache python dependencies on GitHub Actions

[Jacob Walls]

Good task for new-ish contributor:
Make a small survey of how other major python projects cache their dependencies in their Github Actions workflows, and implement something for Django.

Note there are several workflows that would make use of this.

Adam Johnson's blog:
​https://adamj.eu/tech/2023/11/02/github-actions-faster-python-virtual-environments/

This would shave about a minute off the test job.

[Jacob Walls]

Nilesh, there's a contributor who is already submitting a pull request for this, so just be aware, you may wish to restate your intentions or collaborate with them. Pravin, please accommodate Nilesh if they would prefer to keep working. Also, you have several other PRs open, not all of which follow the contribution guidelines for commit & PR titles. It would be better to focus on bringing those to completion first.

​PR (draft)

[Nilesh Pahari]

Replying to Jacob Walls:

Nilesh, there's a contributor who is already submitting a pull request for this, so just be aware, you may wish to restate your intentions or collaborate with them. Pravin, please accommodate Nilesh if they would prefer to keep working. Also, you have several other PRs open, not all of which follow the contribution guidelines for commit & PR titles. It would be better to focus on bringing those to completion first.

​PR (draft)

Hi!
I wasn't able to find any other major project using the pattern mentioned in the Adam Johnson's blog, so I just replicated the pattern (mentioned in the blog) on all the workflows in order to cache the whole virtual environment rather than just the python wheels. I have already made the changes and will be raising a PR shortly. Also just to confirm, the note about having several open PRs was directed at me?

[Jacob Walls]

Nope, it was not. Sorry for the confusion!

[Jacob Walls]

​PR

[Nilesh Pahari]

Workflows are now passing. I also made them more robust by adding a check to only create the virtual environment when it doesn’t already exist.

[Jacob Walls]

This turned out to be more subtle than I expected.

We were using cache: 'pip', so we were availing ourselves of cached wheels. As detailed on Adam's blog post, the potential optimization here is about not moving the wheels into place and not checking for updated dependencies, which an application with exact pins wouldn't need to do. We don't pin. I didn't have this internalized when writing the ticket.

Nilesh did a great job applying my feedback, but after the latest round I realized we had two different patterns to maintain -- one for PRs, and another on pushes to main to get "canary" testing of breakages in latest dependencies -- but on top of that, I realized we'd want to evict the caches at some point. After a month? Write a new cache-eviction job, or use the month in the cache key?

I ran this by Natalia and we agreed that the complexity/maintenance tradeoff was starting to get worse here. So our recommendation is to do nothing for now.

Thanks Nilesh for the investigation.