FK value not inserted in database in data migration

[Rakan Alhneiti]

Hello,

I am using "django_dynamic_scraper" to scrape data off the internet. I started my project by creating an empty migration that basically adds a default scraper.

Models as defined in dynamic_scraper app:

class ScrapedObjClass(models.Model):
    name = models.CharField(max_length=200)
    scraper_scheduler_conf = models.TextField(default='\
"MIN_TIME": 15,\n\
"MAX_TIME": 10080,\n\
"INITIAL_NEXT_ACTION_FACTOR": 10,\n\
"ZERO_ACTIONS_FACTOR_CHANGE": 20,\n\
"FACTOR_CHANGE_FACTOR": 1.3,\n')
    checker_scheduler_conf = models.TextField(default='\
"MIN_TIME": 1440,\n\
"MAX_TIME": 10080,\n\
"INITIAL_NEXT_ACTION_FACTOR": 1,\n\
"ZERO_ACTIONS_FACTOR_CHANGE": 5,\n\
"FACTOR_CHANGE_FACTOR": 1.3,\n')
    comments = models.TextField(blank=True)
    
    def __unicode__(self):
        return self.name

    class Meta:
        ordering = ['name',]


class ScrapedObjAttr(models.Model):
    ATTR_TYPE_CHOICES = (
        ('S', 'STANDARD'),
        ('T', 'STANDARD (UPDATE)'),
        ('B', 'BASE'),
        ('U', 'DETAIL_PAGE_URL'),
        ('I', 'IMAGE'),
    )
    name = models.CharField(max_length=200)
    obj_class = models.ForeignKey(ScrapedObjClass)
    attr_type = models.CharField(max_length=1, choices=ATTR_TYPE_CHOICES)
    
    def __unicode__(self):
        return self.name + " (" + self.obj_class.__unicode__() + ")"

Here's my migration:

# -*- coding: utf-8 -*-
from __future__ import unicode_literals

from django.db import models, migrations


def add_youtube_scraper(apps, schema_editor):
    Scraper = apps.get_model('dynamic_scraper', 'Scraper')
    ScrapedObjClass = apps.get_model('dynamic_scraper', 'ScrapedObjClass')
    ScrapedObjAttr = apps.get_model('dynamic_scraper', 'ScrapedObjAttr')
    ScraperElem = apps.get_model('dynamic_scraper', 'ScraperElem')

    scraped_obj_class = ScrapedObjClass()
    scraped_obj_class.name = 'Youtube Video'
    scraped_obj_class.save()

    scraped_attrs_map = {}
    scraped_attrs_list = [
        {'name': 'base', 'type': 'B'},
        {'name': 'url', 'type': 'U'},
        {'name': 'title', 'type': 'S'},
        {'name': 'body', 'type': 'S'},
        {'name': 'images', 'type': 'I'},
        {'name': 'videos', 'type': 'S'},
    ]

    for scraped_attr in scraped_attrs_list:
        scraped_obj_attr = ScrapedObjAttr()
        scraped_obj_attr.name = scraped_attr['name']
        scraped_obj_attr.attr_type = scraped_attr['type']
        scraped_obj_attr.obj_class_id = scraped_obj_class.id
        scraped_obj_attr.save()
        scraped_attrs_map[scraped_attr['name']] = scraped_obj_attr


class Migration(migrations.Migration):

    dependencies = [
    ]

    operations = [
        migrations.RunPython(add_youtube_scraper)
    ]

When i run python manage.py migrate i get the following error:

Running migrations:
  Applying content_scraper.0001_initial...
Traceback (most recent call last):
  File "./manage.py", line 11, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 338, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 161, in handle
    executor.migrate(targets, plan, fake=options.get("fake", False))
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 68, in migrate
    self.apply_migration(migration, fake=fake)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 102, in apply_migration
    migration.apply(project_state, schema_editor)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/migration.py", line 108, in apply
    operation.database_forwards(self.app_label, schema_editor, project_state, new_state)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/operations/special.py", line 117, in database_forwards
    self.code(from_state.render(), schema_editor)
  File "/app/content_scraper/migrations/0001_initial.py", line 32, in add_youtube_scraper
    scraped_obj_attr.save()
  File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 589, in save
    force_update=force_update, update_fields=update_fields)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 617, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 698, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 731, in _do_insert
    using=using, raw=raw)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/manager.py", line 92, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 921, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 921, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 82, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: null value in column "obj_class_id" violates not-null constraint
DETAIL:  Failing row contains (19, base, null, B).

Basically what i am trying to do is assign a value to a ForeignKey. I tried printing the value of the created scraper object class and it's 19 as in the last line of the traceback. I also checked the SQL statement generated:

INSERT INTO "dynamic_scraper_scrapedobjattr" ("name", "attr_type") VALUES (%s, %s) RETURNING "dynamic_scraper_scrapedobjattr"."id"

As you can see the obj_class attribute of the ScrapedObjAttr is omitted, which is a weird behaviour.

[Shai Berger]

Hello,

Thanks for taking the trouble to submit this problem report. Some points arise:

1) Per Django's maintenance policy, only major bugs (security issues or data-loss problems) will be fixed in 1.7. Can you please try to reproduce the issue with the current 1.8 release?

2) While you're at it, if the problem still exists, and to help pinpoint it, please try to replace the line

scraped_obj_attr.obj_class_id = scraped_obj_class.id

with

scraped_obj_attr.obj_class = scraped_obj_class

In principle, it should be equivalent.

3) Could you please clarify how you dealt with django-dynamic-scraper's migrations? The current source has a migrations folder, but those are South migrations, incompatible with Django>=1.7.

Thanks,
Shai.

[Tim Graham]

Is the migration with RunPython the initial migration, i.e. 0001_initial.py? If so, this is incorrect -- you need an initial migration which creates all the models in the app.

[Shai Berger]

Replying to timgraham:

Is the migration with RunPython the initial migration, i.e. 0001_initial.py? If so, this is incorrect -- you need an initial migration which creates all the models in the app.

If I understand correctly, the migration belongs to content_scraper (an app with no models of its own?) but it uses the models from dynamic_scraper, which appears to be an unmigrated app (its public repository has only South migrations). This should be valid, as far as I understand, but only if my assumptions indeed hold.

[Rakan Alhneiti]

Thanks for the follow up on this ticket.

I started off the project using Django 1.6 which is how i managed to run the south migrations in the first place.

I upgraded to django 1.7 and then upon your request to 1.8, removed the ".id" part and i was able to regenerate the issue on 1.8.2 as well.

As for Tim's comment, Shaib's assumptions are correct. My "content_scraper" app has no models, it just provides a "default" data migration for the models provided by "dynamic_scraper".

Let me know how i can be of any further help.

Thanks,
Rakan

[Tim Graham]

You could try adding an initial migration to the dynamic_scraper app (use ​settings.MIGRATION_MODULES). Then update your data migration to have a dependency on that migration.

[Rakan Alhneiti]

That worked.

I ran:

python manage.py makemigrations dynamic_scraper

Then

python manage.py migrate dynamic_scraper --fake

Because the tables already exist.

And ran migrate again on my app and everything ran as expected.

Could you please provide more context of why i needed to create a migration for this third party app?

[Tim Graham]

I think the problem is roughly described in ​the dependencies section of the migrations docs. Basically, migrations and models without migrations don't interact very well.

I don't think we are going to invest any time in trying to remedy this as migrations will be compulsory for all apps in 1.9. We could add a sentence to the 1.8 docs in the dependencies section, "In addition, any models that are used in RunPython operations must have migrations."

[Tim Graham <timograham@…>]

In df6a4cac:

[1.8.x] Refs #24853 -- Documented a limitation of RunPython and unmigrated apps.