Django

Code

Ticket #2638 (closed: duplicate)

Opened 2 years ago

Last modified 2 years ago

[patch] default ForeignKey field results in terrible scaling perfomance

Reported by: heckj@mac.com Assigned to: mtredinnick
Milestone: Component: Validators
Version: SVN Keywords:
Cc: Triage Stage: Design decision needed
Has patch: 1 Needs documentation: 0
Needs tests: 0 Patch needs improvement: 0

Description

Normally I'm not sure I'd call this a defect, but the default values for models.ForiegnKey? (specifically raw_admin_id) have such a negative performance impact that I couldn't avoid it.

The discussion on the django-user's mailing list at http://groups.google.com/group/django-users/browse_thread/thread/ca0640e1cd0149a4/484695e3983c1858?lnk=gst&q=AddManipulator+performance&rnum=1#484695e3983c1858 caught my attention.

While I really like the behavior that's enabled without raw_admin_id, I think the default value should favor the scaling/performance side of Django, with the additional functionality of returning potential values from the validator being the opt in.

I've posted the test case code I used at http://www.rhonabwy.com/django_perf_test.zip

Attachments

joetest.zip (12.4 kB) - added by heckj@mac.com on 08/31/06 23:37:55.
test project illustrating the performance degredation
2638.diff (1.8 kB) - added by heckj@mac.com on 09/02/06 21:10:57.
attaching patch (from trunk) for a suggested resolution. Includes update to documentation

Change History

08/31/06 23:37:55 changed by heckj@mac.com

  • attachment joetest.zip added.

test project illustrating the performance degredation

09/01/06 09:17:57 changed by Karen Tracey <graybark@bellsouth.net>

Yes, I think I've run into this also. I actually thought it was this problem (list_display containing ForeignKey create a dead loop):

http://groups.google.com/group/django-users/browse_thread/thread/1d73cec767f42c38/84538ca609dceee2?lnk=gst&q=infinite+loop&rnum=1

but in my case I didn't have a loop in my ForeignKey relations, just over 100,000 potential values for the ForeignKey field. Trying to turn on edit_inline in the admin for a model that pointed to one of these things via ForeignKey resulted in what appeared to be an infinite loop. I only realized later that the problem was some code in admin was trying to populate a list of choices for these things, which isn't feasible for my DB. Malcolm alerted me that he thought I would also hit the problem outside of admin if I just tried to create a manipulator for form processing, but since I haven't gotten to needing to do that yet and I don't need the edit_inline functionality, I hadn't gotten around to worrying about/investigating that yet. I'm glad to hear there is already a way to turn this behavior off; for my DB it would be better if "off" was the default.

Just figured I'd mention that I did run afoul of this, in case it influences what you decide to do with it.

09/02/06 21:10:57 changed by heckj@mac.com

  • attachment 2638.diff added.

attaching patch (from trunk) for a suggested resolution. Includes update to documentation

09/02/06 21:12:03 changed by heckj@mac.com

  • version changed from 0.95 to SVN.
  • summary changed from default ForeignKey field results in terrible scaling perfomance to [patch] default ForeignKey field results in terrible scaling perfomance.

Added patch for a suggested resolution. I also included an update to the documentation associated with the code change.

09/21/06 09:49:59 changed by mtredinnick

This isn't the right solution, although your diagnosis of the problem is more or less accurate. A better solution, I suspect, is to make the get_choices() and/or get_default_choices() functions in the manipulators sufficiently lazy so that they don't load all the values unless they are needed (and they usually aren't needed in things like AddManipulator? and ChangeManipulator? settings).

09/24/06 07:09:46 changed by mtredinnick

  • owner changed from adrian to mtredinnick.

02/25/07 19:06:01 changed by SmileyChris

  • stage changed from Unreviewed to Design decision needed.

03/12/07 07:49:50 changed by mtredinnick

  • status changed from new to closed.
  • resolution set to duplicate.

Marking this as a dupe of #3436, since that ticket has a patch that I believe should fix the root cause.

Add/Change #2638 ([patch] default ForeignKey field results in terrible scaling perfomance)




Change Properties
Action