Opened 14 years ago
Closed 14 years ago
#13710 closed (invalid)
raw() sql bug when using a model with many fields with long names
Reported by: | Kevin Renskers | Owned by: | Tobias McNulty |
---|---|---|---|
Component: | Database layer (models, ORM) | Version: | 1.2 |
Severity: | Keywords: | raw sql joins deffered class | |
Cc: | Triage Stage: | Unreviewed | |
Has patch: | no | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description
I love the new raw() function to write custom sql and get back ORM objects. However, when using joins, things get weird.
As long as you use "select *", everything works fine:
for r in Model.objects.raw('SELECT * FROM model INNER JOIN othermodel ON othermodel.model_id = model.id'): print r
However, this can result in a LOT of columns that you don't need, and possible duplicate column names.
Solution would seem to be this:
for r in Model.objects.raw('SELECT model.id, model.name FROM model INNER JOIN othermodel ON othermodel.model_id = model.id'): print r
Sadly, this doesn't work. I get an "type() argument 1 must be string, not unicode" error from django/db/models/query_utils.py in deferred_class_factory, line 274.
Weirdly enough, this does work:
for r in Model.objects.raw('SELECT model.* FROM model INNER JOIN othermodel ON othermodel.model_id = model.id'): print r
Attachments (2)
Change History (13)
comment:1 by , 14 years ago
comment:2 by , 14 years ago
Summary: | raw() sql bug when using joins → raw() sql bug when using a model with many fields with long names |
---|
comment:3 by , 14 years ago
This problem also presents itself in the admin. When I try to load the admin page for my model, I get this error:
Exception Type: TypeError at /admin/biobench/measurement/ Exception Value: __init__() keywords must be strings
Traceback:
File "/opt/APPS/biobench/acc/Biobench/packages/django/core/handlers/base.py" in get_response 100. response = callback(request, *callback_args, **callback_kwargs) File "/opt/APPS/biobench/acc/Biobench/packages/django/contrib/admin/options.py" in wrapper 239. return self.admin_site.admin_view(view)(*args, **kwargs) File "/opt/APPS/biobench/acc/Biobench/packages/django/utils/decorators.py" in _wrapped_view 76. response = view_func(request, *args, **kwargs) File "/opt/APPS/biobench/acc/Biobench/packages/django/views/decorators/cache.py" in _wrapped_view_func 69. response = view_func(request, *args, **kwargs) File "/opt/APPS/biobench/acc/Biobench/packages/django/contrib/admin/sites.py" in inner 190. return view(request, *args, **kwargs) File "/opt/APPS/biobench/acc/Biobench/packages/django/utils/decorators.py" in _wrapper 21. return decorator(bound_func)(*args, **kwargs) File "/opt/APPS/biobench/acc/Biobench/packages/django/utils/decorators.py" in _wrapped_view 76. response = view_func(request, *args, **kwargs) File "/opt/APPS/biobench/acc/Biobench/packages/django/utils/decorators.py" in bound_func 17. return func(self, *args2, **kwargs2) File "/opt/APPS/biobench/acc/Biobench/packages/django/contrib/admin/options.py" in changelist_view 1,071. 'selection_note': _('0 of %(cnt)s selected') % {'cnt': len(cl.result_list)}, File "/opt/APPS/biobench/acc/Biobench/packages/django/db/models/query.py" in __len__ 81. self._result_cache = list(self.iterator()) File "/opt/APPS/biobench/acc/Biobench/packages/django/db/models/query.py" in iterator 274. only_load=only_load) File "/opt/APPS/biobench/acc/Biobench/packages/django/db/models/query.py" in get_cached_row 1,208. obj = klass(**dict(zip(field_names, fields)))
My model has 175 fields (yeah...), which on average are 40 characters long.
comment:4 by , 14 years ago
Keywords: | deffered class added |
---|
We have the same problem even with a small number of fields.
Using some_queryset.only('somefield') results in new model/contenttype creation with
diferred fields. We've used this qs method in some south migration. South
sends post_syncdb after model creations. Then django contrib.auth wants to add
permission after receiving this signal and there is a problem with
Permission.name (i.e. 'Can add page_ deferred_layout_migrated_site_id_skin_template')
field size limited to max length=50 (or Permission.codename limited to 100)
This is maybe more a problem of South application (sending signals of undeclared model
creations), but this is also a cause of uncontrolled size of generated model name in deferred_class_factory()
comment:5 by , 14 years ago
Owner: | changed from | to
---|---|
Status: | new → assigned |
attempting to reproduce...
by , 14 years ago
Attachment: | 13712.diff added |
---|
test showing the specified query works (and this is not a bug)
comment:6 by , 14 years ago
Resolution: | → worksforme |
---|---|
Status: | assigned → closed |
I was NOT able to reproduce this bug. I attached a test showing that the query you've given works in the test environment. If you're still having issues please try to update this test to generate the error you're experiencing, or provide more information about your environment such that we can reproduce the bug.
comment:7 by , 14 years ago
Resolution: | worksforme |
---|---|
Status: | closed → reopened |
It only generates an error when your model has a lot of fields, which have long names. The code in django/db/models/query_utils.py, line 267, generates a string that is too long to handle. With "normal" models (i.e. not a ton of fields with long names), the query does indeed work fine.
This is explained in my first comment on this ticket. I could create a small Django project to prove this error, if that would help.
comment:8 by , 14 years ago
Yes, some code (as little as possible) demonstrating the issue would be great, thanks.
comment:9 by , 14 years ago
OK, did some extensive testing. Turns out, my problem has nothing to do with the number of fields on a model. Rather, I had some fields with unicode names, done like this:
MyModel.add_to_class(u'field_name', models.CharField(max_length=30))
I used this add_to_class way of adding fields to a model because the fields were already defined somewhere else. But, the field names are unicode strings (because they are also used in other languages with weird characters). This causes the "type() argument 1 must be string, not unicode" error in deferred_class_factory.
Now that I know this, I can cast my field names to the proper string type before adding them to my model. But maybe it's also a good idea to cast the name variable to a string in the deferred_class_factory function, just to be safe (see django/db/models/query_utils.py, line 267)
Still, the size of the name variable can really get out of hand when you have a lot of fields. In my case, it was a string with over 8000 characters...
comment:10 by , 14 years ago
this is a probleblem with mentioned uncontrolled size of generated model name. Collecting field names to make a model name.. nono
comment:11 by , 14 years ago
Resolution: | → invalid |
---|---|
Status: | reopened → closed |
As far as I can tell, add_to_class is an undocumented method and hence unsupported. Probably just best if you ensure that you pass in non-unicode strings.
I found the source of the problem. My model has a lot of fields, and most of them have long, verbose names.
The error is caused in django/db/models/query_utils.py, on line 267:
name = "%s_Deferred_%s" % (model.name, '_'.join(sorted(list(attrs))))
This results in a *very* long string, which cannot be handled, or so it seems.
When I change this line to something like this, it does work:
name = "%s_Deferred_%s" % (model.name, '_lala')
It's weird that this problem only presents itself when you don't use model.*. When using model.* in the raw query, everything works fine.
This also fixes bug #13712.