Migrations broken with django 1.7.1 when changing null from true to false.

[Andrey Antukh]

Hi!

I start having strange exceptions when I have migrated a project from django 1.7 to 1.7.1

(taiga)[3/5.0.7]{1}niwi@niwi:~/taiga/taiga-back> python manage.py sqlmigrate projects 0006
Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/niwi/.virtualenvs/taiga/lib/python3.4/site-packages/Django-1.7.2.dev20141030224451-py3.4.egg/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/home/niwi/.virtualenvs/taiga/lib/python3.4/site-packages/Django-1.7.2.dev20141030224451-py3.4.egg/django/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/niwi/.virtualenvs/taiga/lib/python3.4/site-packages/Django-1.7.2.dev20141030224451-py3.4.egg/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/niwi/.virtualenvs/taiga/lib/python3.4/site-packages/Django-1.7.2.dev20141030224451-py3.4.egg/django/core/management/commands/sqlmigrate.py", line 30, in execute
    return super(Command, self).execute(*args, **options)
  File "/home/niwi/.virtualenvs/taiga/lib/python3.4/site-packages/Django-1.7.2.dev20141030224451-py3.4.egg/django/core/management/base.py", line 338, in execute
    output = self.handle(*args, **options)
  File "/home/niwi/.virtualenvs/taiga/lib/python3.4/site-packages/Django-1.7.2.dev20141030224451-py3.4.egg/django/core/management/commands/sqlmigrate.py", line 61, in handle
    sql_statements = executor.collect_sql(plan)
  File "/home/niwi/.virtualenvs/taiga/lib/python3.4/site-packages/Django-1.7.2.dev20141030224451-py3.4.egg/django/db/migrations/executor.py", line 77, in collect_sql
    migration.apply(project_state, schema_editor, collect_sql=True)
  File "/home/niwi/.virtualenvs/taiga/lib/python3.4/site-packages/Django-1.7.2.dev20141030224451-py3.4.egg/django/db/migrations/migration.py", line 108, in apply
    operation.database_forwards(self.app_label, schema_editor, project_state, new_state)
  File "/home/niwi/.virtualenvs/taiga/lib/python3.4/site-packages/Django-1.7.2.dev20141030224451-py3.4.egg/django/db/migrations/operations/fields.py", line 139, in database_forwards
    schema_editor.alter_field(from_model, from_field, to_field)
  File "/home/niwi/.virtualenvs/taiga/lib/python3.4/site-packages/Django-1.7.2.dev20141030224451-py3.4.egg/django/db/backends/schema.py", line 473, in alter_field
    self._alter_field(model, old_field, new_field, old_type, new_type, old_db_params, new_db_params, strict)
  File "/home/niwi/.virtualenvs/taiga/lib/python3.4/site-packages/Django-1.7.2.dev20141030224451-py3.4.egg/django/db/backends/schema.py", line 617, in _alter_field
    sql, params = tuple(zip(*actions))
ValueError: need more than 0 values to unpack

I start research.

Firstly I have surprised that the last created migration does not include null=False change: ​https://github.com/taigaio/taiga-back/blob/improvements-migrations/taiga/projects/migrations/0006_auto_20141029_1040.py but this, more surprisingly with django 1.7 emits correct sql migration without having null=False in a migration file:

(taiga)[3/5.0.7]niwi@niwi:~/taiga/taiga-back> python manage.py sqlmigrate projects 0006
BEGIN;
--
-- MIGRATION NOW PERFORMS OPERATION THAT CANNOT BE WRITTEN AS SQL:
-- Raw Python operation
--
ALTER TABLE "projects_project" ALTER COLUMN "total_milestones" SET NOT NULL;
ALTER TABLE "projects_project" ALTER COLUMN "total_milestones" DROP DEFAULT;

COMMIT;

This is a proper behavior?

But with django 1.7.1 it seems not detects correctly the null attribute change, that causes that actions variable on django/db/backends/schema.py:617 to be empty, which in turn, causes that exception.

I have tried set manually the null=False parameter on the migrations file, and it no such effect (works with django 1.7 and raises exception with django 1.7.1)

[Tim Graham]

Did you generate the migration with Django 1.7 or 1.7.1? If the former, could you try regenerating it with 1.7.1? I think #23609 (fixed in 1.7.1) could be related.

[Andrey Antukh]

The migration is generated by django 1.7 but I have tried generate it with django 1.7.1, but it no such effect. The same exception is raised and the same behavior on creating the migration file (missing null=True and blank=True on alterfield action).

Also, I have tried with 1.7.2dev (stable/1.7.x branch) and also I have found the same exception.

[Tim Graham]

I've having trouble reproducing the issue. Could you outline the minimal steps (ideally one that doesn't involve a big project like the example you linked)? Not sure it matters, but which database backend are you using?

[Andrey Antukh]

We are using the psycopg2 backend.

Tomorrow I'll try to make a small project for make easy to reproduce it.

But the process is very simple:

• havind django 1.7, I have created a migration to one model that already have one integer field with null=True, blank=True and default=0 on first migration, changing null and blank parameters both to False.
• update to django 1.7.1
• try apply the created migration

[Markus Holtermann]

I can confirm the problem

[Markus Holtermann]

I opened a pull-request to fix this ticket: ​https://github.com/django/django/pull/3452

[Markus Holtermann]

As long as you don't have a high traffic site, you should get it working with the following workaround:

class Migration(migrations.Migration):
    dependencies = [
        # your dependencies here
    ]

    operations = [
        migrations.AlterField(
            model_name='mymodel',
            name='myfield',
            field=models.IntegerField(null=True, blank=True, default=None),
        ),
        migrations.AlterField(
            model_name='mymodel',
            name='myfield',
            field=models.IntegerField(default=0),
        ),
    ]

What happened: Unfortunately I didn't took into account in the other patch that the default sticks between a NULL and NOT NULL change :-/. Thus changing the default from e.g. 0 to NULL while sticking to the NULL constraint and changing it back to 0 together with adding the NOT NULL constraint should work.

[Andrey Antukh]

Thanks for this workaround. In our case that is not very problematic and we are reverted temporally to django 1.7.

Thanks for the quick fix.

[Tim Graham <timograham@…>]

In 715ccfde24f9f2b7f6710429370a1eff3c78fc2a:

Fixed #23738 -- Allowed migrating from NULL to NOT NULL with the same default value

Thanks to Andrey Antukh for the report.

[Tim Graham <timograham@…>]

In 40ad022d5e366f56424dcdb885607d13c00a550e:

[1.7.x] Fixed #23738 -- Allowed migrating from NULL to NOT NULL with the same default value

Thanks to Andrey Antukh for the report.

Backport of 715ccfde24 from master