Code

Opened 4 years ago

Closed 4 years ago

#13710 closed (invalid)

raw() sql bug when using a model with many fields with long names

Reported by: Renskers Owned by: tobias
Component: Database layer (models, ORM) Version: 1.2
Severity: Keywords: raw sql joins deffered class
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: UI/UX:

Description

I love the new raw() function to write custom sql and get back ORM objects. However, when using joins, things get weird.

As long as you use "select *", everything works fine:

for r in Model.objects.raw('SELECT * FROM model INNER JOIN othermodel ON othermodel.model_id = model.id'): 
    print r 

However, this can result in a LOT of columns that you don't need, and possible duplicate column names.

Solution would seem to be this:

for r in Model.objects.raw('SELECT model.id, model.name FROM model INNER JOIN othermodel ON othermodel.model_id = model.id'): 
    print r 

Sadly, this doesn't work. I get an "type() argument 1 must be string, not unicode" error from django/db/models/query_utils.py in deferred_class_factory, line 274.

Weirdly enough, this does work:

for r in Model.objects.raw('SELECT model.* FROM model INNER JOIN othermodel ON othermodel.model_id = model.id'): 
    print r 

Attachments (2)

13712.diff (811 bytes) - added by tobias 4 years ago.
test showing the specified query works (and this is not a bug)
13710.diff (811 bytes) - added by tobias 4 years ago.
same patch, named for this bug (not the duplicate)

Download all attachments as: .zip

Change History (13)

comment:1 Changed 4 years ago by Renskers

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

I found the source of the problem. My model has a lot of fields, and most of them have long, verbose names.

The error is caused in django/db/models/query_utils.py, on line 267:
name = "%s_Deferred_%s" % (model.name, '_'.join(sorted(list(attrs))))

This results in a *very* long string, which cannot be handled, or so it seems.

When I change this line to something like this, it does work:
name = "%s_Deferred_%s" % (model.name, '_lala')

It's weird that this problem only presents itself when you don't use model.*. When using model.* in the raw query, everything works fine.

This also fixes bug #13712.

comment:2 Changed 4 years ago by Renskers

  • Summary changed from raw() sql bug when using joins to raw() sql bug when using a model with many fields with long names

comment:3 Changed 4 years ago by anonymous

This problem also presents itself in the admin. When I try to load the admin page for my model, I get this error:

Exception Type: TypeError at /admin/biobench/measurement/
Exception Value: __init__() keywords must be strings

Traceback:

File "/opt/APPS/biobench/acc/Biobench/packages/django/core/handlers/base.py" in get_response
  100.                     response = callback(request, *callback_args, **callback_kwargs)
File "/opt/APPS/biobench/acc/Biobench/packages/django/contrib/admin/options.py" in wrapper
  239.                 return self.admin_site.admin_view(view)(*args, **kwargs)
File "/opt/APPS/biobench/acc/Biobench/packages/django/utils/decorators.py" in _wrapped_view
  76.                     response = view_func(request, *args, **kwargs)
File "/opt/APPS/biobench/acc/Biobench/packages/django/views/decorators/cache.py" in _wrapped_view_func
  69.         response = view_func(request, *args, **kwargs)
File "/opt/APPS/biobench/acc/Biobench/packages/django/contrib/admin/sites.py" in inner
  190.             return view(request, *args, **kwargs)
File "/opt/APPS/biobench/acc/Biobench/packages/django/utils/decorators.py" in _wrapper
  21.             return decorator(bound_func)(*args, **kwargs)
File "/opt/APPS/biobench/acc/Biobench/packages/django/utils/decorators.py" in _wrapped_view
  76.                     response = view_func(request, *args, **kwargs)
File "/opt/APPS/biobench/acc/Biobench/packages/django/utils/decorators.py" in bound_func
  17.                 return func(self, *args2, **kwargs2)
File "/opt/APPS/biobench/acc/Biobench/packages/django/contrib/admin/options.py" in changelist_view
  1,071.             'selection_note': _('0 of %(cnt)s selected') % {'cnt': len(cl.result_list)},
File "/opt/APPS/biobench/acc/Biobench/packages/django/db/models/query.py" in __len__
  81.                 self._result_cache = list(self.iterator())
File "/opt/APPS/biobench/acc/Biobench/packages/django/db/models/query.py" in iterator
  274.                             only_load=only_load)
File "/opt/APPS/biobench/acc/Biobench/packages/django/db/models/query.py" in get_cached_row
  1,208.             obj = klass(**dict(zip(field_names, fields)))

My model has 175 fields (yeah...), which on average are 40 characters long.

comment:4 Changed 4 years ago by ondrej.kohout@…

  • Keywords deffered class added

We have the same problem even with a small number of fields.

Using some_queryset.only('somefield') results in new model/contenttype creation with
diferred fields. We've used this qs method in some south migration. South
sends post_syncdb after model creations. Then django contrib.auth wants to add
permission after receiving this signal and there is a problem with
Permission.name (i.e. 'Can add page_ deferred_layout_migrated_site_id_skin_template')
field size limited to max length=50 (or Permission.codename limited to 100)

This is maybe more a problem of South application (sending signals of undeclared model
creations), but this is also a cause of uncontrolled size of generated model name in deferred_class_factory()

comment:5 Changed 4 years ago by tobias

  • Owner changed from nobody to tobias
  • Status changed from new to assigned

attempting to reproduce...

Changed 4 years ago by tobias

test showing the specified query works (and this is not a bug)

Changed 4 years ago by tobias

same patch, named for this bug (not the duplicate)

comment:6 Changed 4 years ago by tobias

  • Resolution set to worksforme
  • Status changed from assigned to closed

I was NOT able to reproduce this bug. I attached a test showing that the query you've given works in the test environment. If you're still having issues please try to update this test to generate the error you're experiencing, or provide more information about your environment such that we can reproduce the bug.

comment:7 Changed 4 years ago by Renskers

  • Resolution worksforme deleted
  • Status changed from closed to reopened

It only generates an error when your model has a lot of fields, which have long names. The code in django/db/models/query_utils.py, line 267, generates a string that is too long to handle. With "normal" models (i.e. not a ton of fields with long names), the query does indeed work fine.

This is explained in my first comment on this ticket. I could create a small Django project to prove this error, if that would help.

comment:8 Changed 4 years ago by tobias

Yes, some code (as little as possible) demonstrating the issue would be great, thanks.

comment:9 Changed 4 years ago by Renskers

OK, did some extensive testing. Turns out, my problem has nothing to do with the number of fields on a model. Rather, I had some fields with unicode names, done like this:

MyModel.add_to_class(u'field_name', models.CharField(max_length=30))

I used this add_to_class way of adding fields to a model because the fields were already defined somewhere else. But, the field names are unicode strings (because they are also used in other languages with weird characters). This causes the "type() argument 1 must be string, not unicode" error in deferred_class_factory.

Now that I know this, I can cast my field names to the proper string type before adding them to my model. But maybe it's also a good idea to cast the name variable to a string in the deferred_class_factory function, just to be safe (see django/db/models/query_utils.py, line 267)

Still, the size of the name variable can really get out of hand when you have a lot of fields. In my case, it was a string with over 8000 characters...

comment:10 Changed 4 years ago by anonymous

this is a probleblem with mentioned uncontrolled size of generated model name. Collecting field names to make a model name.. nono

comment:11 Changed 4 years ago by tobias

  • Resolution set to invalid
  • Status changed from reopened to closed

As far as I can tell, add_to_class is an undocumented method and hence unsupported. Probably just best if you ensure that you pass in non-unicode strings.

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.