Make makemigrations scriptable / script-friendly

[Chris Jerdonek]

Currently, the makemigrations management command doesn't lend itself well to scripting. For example, it writes its progress output to stdout rather than stderr. Also, there doesn't appear to be a structured / programmatic way to figure out what files it has created.

My use case is that in my development environment, I'd like to be able to run makemigrations in a Docker container, find out what files were added (e.g. from makemigrations's output), and then copy those files from the Docker container to my development machine so they can be added to source control.

Currently, there doesn't seem to be an easy way to do this. One way, for example, is to manually read makemigrations's output to find out what apps were affected, and then inspect the directories yourself for the new files.

Better, for example, would be if makemigrations could write the paths to the created files to stdout.

[astandley]

The current makemigrations command would seem to allow for everything you have asked.

# Define People and Poll models
>> python -m manage makemigrations
Migrations for 'newspaper':
  newspaper\migrations\0001_initial.py
    - Create model People
Migrations for 'polls':
  polls\migrations\0001_initial.py
    - Create model Poll

# Add new_field to People Model
>> python -m manage makemigrations
Migrations for 'newspaper':
  newspaper\migrations\0002_people_new_field.py
    - Add field new_field to people

Addressing each point:

Output written to stdout rather than stderr.

You can redirect stdout to stderr for the execution of the command if you want the output in stderr. (Using sys in python, or pipes for a shell script)

A structured / programmatic way to figure out what files it has created.

Run a regex on the output of the migration command. Example pattern: r'Migrations for '(?P<app_name>[^']*)':\n (?P<migration_file>[^\n]*)'

Is there a reason this would not meet your needs?

[Chris Jerdonek]

I don't consider parsing log output with regexes to be structured output or a programmatic API. That seems brittle and unnecessarily complicated, and also would be hard for Django to make backwards compatibility guarantees about.

What I had in mind was something like log output going to stderr, and the paths of the created files going to stdout -- one per line. If something fancier was needed, json could be outputted. With something like that, there wouldn't be any need for regex parsing and the API would be well-defined.

[Tim Graham]

It seems complicated. For example, what if makemigrations requires interactive input from the questioner?

[Chris Jerdonek]

My original use case was a non-interactive one (inside a container). But again, you raise a good point. Obviously, piping to stdout won't work if interactivity is required (because you'd want user prompts to go to stdout). This is true of any script, not just Django management commands. Other than that, I don't think the changes I've described would hurt things in that case, aside from possibly the "reliability" issue you mentioned here. That though could be addressed by my follow-up comment to yours. If we wanted a fancier solution, the "structured" stdout could be outputted only in non-interactive mode.

[Tim Graham]

I'm skeptical, but I guess if you have a patch to propose, we can evaluate it.

[Tim Graham]

I closed #29470 as a duplicate.

[Amir Hadi]

If you are developing with Docker, why are you not just mounting your development machines source directory into the container, execute makemigrations and then you have the migrations directly on your machine. This way you can save yourself from parsing anything.

[Jacob Walls]

Until I remedy the test failure on Windows.

[Jacob Walls]

I'm going to take a look at implementing Chris's suggested log() method. (See PR)

[Jacob Walls]

Up to date w/r/t feedback but for commit reorganization and suggestion to put the new log() method in a standalone PR, which I am willing to do!

[Jacob Walls]

Got around to reorganizing the commits and opening a first-step ​refactoring PR to add log() per Chris's suggestion.

If for whatever reason this is not desired, by all means, close the smaller PR and we can just work on one branch. Cheers.

[Mariusz Felisiak <felisiak.mariusz@…>]

In f153e92:

Refs #29026 -- Added log() to makemigrations.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 0ab58c12:

Refs #29026 -- Allowed customizing InteractiveMigrationQuestioner's prompt destination.

Previously, the questioner did not obey the value of stdout provided
to the command.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 6f78cb6b:

Fixed #29026 -- Added --scriptable option to makemigrations.