Opened 14 years ago

Last modified 13 years ago

#12007 new Cleanup/optimization

djangoadmin loaddata gives odd error if it runs out of memory

Reported by: chrisw Owned by: nobody
Component: Core (Serialization) Version: 1.1
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Alex Gaynor)

Hi All,

I tried to import a 3.2GB xml file, partly to see if Django loads the whole file into memory when it's importing and partly to see how large pg client transactions work. Whichever caused the problem, I have a fair idea that even thought the machine has 4GB of memory, the process ran out of memory.

I'd expect a MemoryError if that were the case, what I actually got was:

  File "django/core/management/__init__.py", line 362, in execute_manager
    utility.execute()
  File "/django/core/management/__init__.py", line 303, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "django/core/management/base.py", line 195, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "django/core/management/base.py", line 222, in execute
    output = self.handle(*args, **options)
  File "django/core/management/commands/loaddata.py", line 200, in handle
    transaction.leave_transaction_management()
  File "django/db/transaction.py", line 74, in leave_transaction_management
    raise TransactionManagementError("This code isn't under transaction management")
django.db.transaction.TransactionManagementError: This code isn't under transaction management

cheers,

Chris

Change History (5)

comment:1 by Russell Keith-Magee, 14 years ago

Component: UncategorizedSerialization
Triage Stage: UnreviewedAccepted

For those looking to dig into this, it's important to identify whether this is a database issue (i.e., the database runs out of memory if you try to do a 3GB transaction) or a Django issue (i.e., a problem with our wrapper loading the data file into memory).

Its also important to note the analog with #5423 - that ticket deals with making dumpdata behave as an iterator. I'm not marking this as a dupe because the solution won't necessarily be the same, but it's worth noting if someone wants to tackle both problems.

comment:2 by Alex Gaynor, 14 years ago

Description: modified (diff)

Please use preview

comment:3 by Peter Baumgartner, 13 years ago

Severity: Normal
Type: Cleanup/optimization

comment:4 by Aymeric Augustin, 12 years ago

UI/UX: unset

Change UI/UX from NULL to False.

comment:5 by Aymeric Augustin, 12 years ago

Easy pickings: unset

Change Easy pickings from NULL to False.

Note: See TracTickets for help on using tickets.
Back to Top