Add metadata API to Tasks

[Jake Howard]

A TaskResult should have metadata - a writable persistent dict (JSON-compatible) containing additional data about the task, but outside of its arguments.

Proposed API:

• my_task.using(metadata={"trace_id": 123}).enqueue(foo=1) would enqueue the my_task task with metadata {"trace_id": 123} and arguments foo=1.
• Metadata could be retrieved using task context context.metadata
• If the context.metadata is modified, it is saved when a task finishes (successfully or otherwise)
• Metadata can be saved within a task using context.save_metadata() (or .asave_metadata)
• metadata is always a dict, and must be strictly JSON-supported types (as task args and kwargs are currently)
• Metadata support is opt-in - backends must set the supports_metadata feature and implement save_metadata and/or asave_metadata
• By convention, metadata keys prefixed with _ should not be used by end users. Keys prefixed with _django should be considered reserved by Django and its first-party backends. Backend or library developers are free to use their own narrow-scoped reservd prefixes, so long as they start with _. They may be stored, but are stripped out before being shown by the user, and are ignored during save.

See also ​https://github.com/django/new-features/issues/141

[David Smith]

The Steering Council gave their support to move this forward on the new-features repo.

[mudassar]

​https://github.com/django/django/pull/22123