Opened 7 years ago

Last modified 5 years ago

#12007 new Cleanup/optimization

djangoadmin loaddata gives odd error if it runs out of memory

Reported by: chrisw Owned by: nobody
Component: Core (Serialization) Version: 1.1
Severity: Normal Keywords:
Cc: Triage Stage: Accepted
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by Alex Gaynor)

Hi All,

I tried to import a 3.2GB xml file, partly to see if Django loads the whole file into memory when it's importing and partly to see how large pg client transactions work. Whichever caused the problem, I have a fair idea that even thought the machine has 4GB of memory, the process ran out of memory.

I'd expect a MemoryError if that were the case, what I actually got was:

  File "django/core/management/__init__.py", line 362, in execute_manager
    utility.execute()
  File "/django/core/management/__init__.py", line 303, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "django/core/management/base.py", line 195, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "django/core/management/base.py", line 222, in execute
    output = self.handle(*args, **options)
  File "django/core/management/commands/loaddata.py", line 200, in handle
    transaction.leave_transaction_management()
  File "django/db/transaction.py", line 74, in leave_transaction_management
    raise TransactionManagementError("This code isn't under transaction management")
django.db.transaction.TransactionManagementError: This code isn't under transaction management

cheers,

Chris

Change History (5)

comment:1 Changed 7 years ago by Russell Keith-Magee

Component: UncategorizedSerialization
Needs documentation: unset
Needs tests: unset
Patch needs improvement: unset
Triage Stage: UnreviewedAccepted

For those looking to dig into this, it's important to identify whether this is a database issue (i.e., the database runs out of memory if you try to do a 3GB transaction) or a Django issue (i.e., a problem with our wrapper loading the data file into memory).

Its also important to note the analog with #5423 - that ticket deals with making dumpdata behave as an iterator. I'm not marking this as a dupe because the solution won't necessarily be the same, but it's worth noting if someone wants to tackle both problems.

comment:2 Changed 7 years ago by Alex Gaynor

Description: modified (diff)

Please use preview

comment:3 Changed 5 years ago by Peter Baumgartner

Severity: Normal
Type: Cleanup/optimization

comment:4 Changed 5 years ago by Aymeric Augustin

UI/UX: unset

Change UI/UX from NULL to False.

comment:5 Changed 5 years ago by Aymeric Augustin

Easy pickings: unset

Change Easy pickings from NULL to False.

Note: See TracTickets for help on using tickets.
Back to Top