Code


Version 7 (modified by gcc, 11 months ago) (diff)

--

Continuous Integration

Ticket #19891 describes the use case for getting Travis CI to run Django's test suite automatically on each pull request, to help core devs to know whether it's ready for merging.

That ticket describes a number of "outstanding issues" holding back the completion of this task. The status of those issues (i.e. which remain outstanding) is not clear from reading the ticket, so I'll attempt to document and clarify them here.

The Story So Far

  • Florian's unmerged code to configure Travis is here.
  • In particular, his Travis configuration file is here.
  • This config file downloads and runs a shell script which does most of the setup work.
  • In particular, it downloads $CONFIG_SERVER/test_$DB\_$GIS.py and saves it as tests/django_settings.py, which runtests uses to load its settings.
  • Travis is configured to run on this fork, so it only tests pull requests made to this fork.
  • Travis builds every branch that you push to. So every branch has to have a .travis.yml file, preferably with the same contents. This is annoying!

Notifications

  • "But really, we don't want to bother you guys with tons of emails for broken/fixed pull requests. Hence, I propose adding to the .travis.yml:" comment 12 DONE
  • "However, having notifications on IRC might be nice. I'm not an avid user myself so I'm open to any suggestions. Refer here" TODO low priority

Other

  • apollo13 prefers his master branch over the travisci branch.
  • Travis is currently building all branches. Need to build only master and ensure that every other branch has at least a minimal .travis.yml file to disable building it. TODO
  • Build of the travisci branch is currently failing - ouch! TODO urgent
  • The Travis config files in travisci and master branches differ TODO

Postgres Crashing

Postgres on Travis crashes when we run all the tests due to running out of disk space:

======================================================================
ERROR: test_cache_read_for_model_instance (cache.tests.DBCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/aptivate/django/django/test/testcases.py", line 259, in __call__
    self._post_teardown()
  File "/home/travis/build/aptivate/django/django/test/testcases.py", line 523, in _post_teardown
    self._fixture_teardown()
  File "/home/travis/build/aptivate/django/django/test/testcases.py", line 538, in _fixture_teardown
    skip_validation=True, reset_sequences=False)
  File "/home/travis/build/aptivate/django/django/core/management/__init__.py", line 161, in call_command
    return klass.execute(*args, **defaults)
  File "/home/travis/build/aptivate/django/django/core/management/base.py", line 283, in execute
    output = self.handle(*args, **options)
  File "/home/travis/build/aptivate/django/django/core/management/base.py", line 413, in handle
    return self.handle_noargs(**options)
  File "/home/travis/build/aptivate/django/django/core/management/commands/flush.py", line 84, in handle_noargs
    emit_post_sync_signal(set(all_models), verbosity, interactive, db)
  File "/home/travis/build/aptivate/django/django/core/management/sql.py", line 202, in emit_post_sync_signal
    interactive=interactive, db=db)
  File "/home/travis/build/aptivate/django/django/dispatch/dispatcher.py", line 182, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/home/travis/build/aptivate/django/django/contrib/auth/management/__init__.py", line 97, in create_permissions
    auth_app.Permission.objects.using(db).bulk_create(perms)
  File "/home/travis/build/aptivate/django/django/db/models/query.py", line 443, in bulk_create
    self._batched_insert(objs_without_pk, fields, batch_size)
  File "/home/travis/build/aptivate/django/django/db/transaction.py", line 313, in __exit__
    connection.set_autocommit(True)
  File "/home/travis/build/aptivate/django/django/db/backends/__init__.py", line 330, in set_autocommit
    self._set_autocommit(autocommit)
  File "/home/travis/build/aptivate/django/django/db/backends/postgresql_psycopg2/base.py", line 175, in _set_autocommit
    self.connection.autocommit = autocommit
InterfaceError: connection already closed

======================================================================
ERROR: test_cache_read_for_model_instance_with_deferred (cache.tests.DBCacheTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/aptivate/django/tests/cache/tests.py", line 807, in setUp
    management.call_command('createcachetable', self._table_name, verbosity=0, interactive=False)
  File "/home/travis/build/aptivate/django/django/core/management/__init__.py", line 161, in call_command
    return klass.execute(*args, **defaults)
  File "/home/travis/build/aptivate/django/django/core/management/base.py", line 283, in execute
    output = self.handle(*args, **options)
  File "/home/travis/build/aptivate/django/django/core/management/base.py", line 383, in handle
    label_output = self.handle_label(label, **options)
  File "/home/travis/build/aptivate/django/django/core/management/commands/createcachetable.py", line 57, in handle_label
    curs = connection.cursor()
  File "/home/travis/build/aptivate/django/django/db/backends/__init__.py", line 159, in cursor
    cursor = util.CursorWrapper(self._cursor(), self)
  File "/home/travis/build/aptivate/django/django/db/backends/__init__.py", line 131, in _cursor
    return self.create_cursor()
  File "/home/travis/build/aptivate/django/django/db/utils.py", line 99, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/travis/build/aptivate/django/django/db/backends/__init__.py", line 131, in _cursor
    return self.create_cursor()
  File "/home/travis/build/aptivate/django/django/db/backends/postgresql_psycopg2/base.py", line 141, in create_cursor
    cursor = self.connection.cursor()
InterfaceError: connection already closed

... postgres logs:

2013-05-19 08:29:13 UTC STATEMENT:  INSERT INTO "test cache table" (cache_key, value, expires) VALUES (':1:addkey1', 'gAJYCAAAAG5ld3ZhbHVlcQEu', '2013-05-19 03:34:13')
2013-05-19 08:29:21 UTC PANIC:  could not write to file "pg_xlog/xlogtemp.11884": No space left on device
2013-05-19 08:29:21 UTC STATEMENT:  COMMIT
2013-05-19 08:29:21 UTC LOG:  server process (PID 11884) was terminated by signal 6: Aborted
2013-05-19 08:29:21 UTC LOG:  terminating any other active server processes
2013-05-19 08:29:21 UTC WARNING:  terminating connection because of crash of another server process
2013-05-19 08:29:21 UTC DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2013-05-19 08:29:21 UTC HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2013-05-19 08:29:21 UTC LOG:  all server processes terminated; reinitializing
2013-05-19 08:29:21 UTC LOG:  database system was interrupted; last known up at 2013-05-19 08:27:16 UTC
2013-05-19 08:29:21 UTC LOG:  database system was not properly shut down; automatic recovery in progress
2013-05-19 08:29:21 UTC LOG:  redo starts at 0/5045358
2013-05-19 08:29:32 UTC FATAL:  the database system is in recovery mode
2013-05-19 08:29:32 UTC FATAL:  the database system is in recovery mode
2013-05-19 08:29:43 UTC FATAL:  the database system is in recovery mode
2013-05-19 08:29:43 UTC FATAL:  the database system is in recovery mode
travis_fold:end:after_failure
Done. Your build exited with 1.

It seems that this might happen when disk writes are slow, which I can imagine happening in a virtual machine.

Can any Postgres expert help debug and fix this?