Code

Opened 6 months ago

Closed 6 months ago

#21245 closed Cleanup/optimization (duplicate)

Long-running process with many SQL operations crashes when DEBUG is True

Reported by: pztrick Owned by: nobody
Component: Database layer (models, ORM) Version: 1.5
Severity: Normal Keywords:
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

When DEBUG=True, SQL queries are buffered in memory and for long-running processes this can cause the server to kill the process.

In my particular use case, I was crashing when executing a South data migration which ingested large datasets into my PostGIS database.

There is a monkey patch available on SO: http://stackoverflow.com/a/7769117/1545769

After including this monkey patch in each of my data migration files, the migrations completed successfully.

Here is why I am creating an issue:
1) There isn't a very helpful message when the process is terminated; you have to deduce it from looking at PostgreSQL logs or /var/log/syslog to see that low memory was the perpetrator.
2) There is no way to configure in settings.py to disable the CursorDebugWrapper (e.g. settings.DISABLE_DEBUG_CURSOR boolean)
3) I wonder if we can't make Django smart enough to trap Out-of-memory errors to present an informative exception via Python and/or...
4) ...smartly manage the connection.queries buffer to prevent it from using too much RAM in the first place?

N2 would be an easy pickings solution (here: https://github.com/django/django/blob/master/django/db/backends/__init__.py), but it wouldn't solve N1 and developers might not know that RAM is the issue. So I think N3/N4 should be considered.

Attachments (0)

Change History (1)

comment:1 Changed 6 months ago by claudep

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Resolution set to duplicate
  • Status changed from new to closed

Duplicate of #12581

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.