Opened 2 years ago

Closed 2 years ago

Last modified 2 years ago

#28075 closed Bug (fixed)

USE_THOUSAND_SEPARATOR = True, causes foreignkey id to become localized

Reported by: George Tantiras Owned by: Jon Dufresne
Component: Forms Version: 1.11
Severity: Release blocker Keywords:
Cc: Jon Dufresne Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by George Tantiras)

After upgrading to Django 1.11, when a foreignkey points to an id with more than 3 digits, it gets localized.

In django admin, when I visit an already existing instance and hit "Save and Continue Editing", I receive the following error:

Exception raised while rendering {% include %} for template 'admin/change_form.html'. Empty string rendered instead.
Traceback (most recent call last):
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/template/loader_tags.py", line 216, in render
    return template.render(context)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/template/base.py", line 209, in render
    return self._render(context)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/template/base.py", line 199, in _render
    return self.nodelist.render(context)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/template/base.py", line 990, in render
    bit = node.render_annotated(context)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/template/base.py", line 957, in render_annotated
    return self.render(context)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/template/defaulttags.py", line 216, in render
    nodelist.append(node.render_annotated(context))
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/template/base.py", line 957, in render_annotated
    return self.render(context)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/template/defaulttags.py", line 529, in render
    return self.nodelist.render(context)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/template/base.py", line 990, in render
    bit = node.render_annotated(context)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/template/base.py", line 957, in render_annotated
    return self.render(context)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/template/defaulttags.py", line 216, in render
    nodelist.append(node.render_annotated(context))
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/template/base.py", line 957, in render_annotated
    return self.render(context)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/template/defaulttags.py", line 411, in render
    return strip_spaces_between_tags(self.nodelist.render(context).strip())
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/template/base.py", line 990, in render
    bit = node.render_annotated(context)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/template/base.py", line 957, in render_annotated
    return self.render(context)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/template/defaulttags.py", line 322, in render
    return nodelist.render(context)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/template/base.py", line 990, in render
    bit = node.render_annotated(context)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/template/base.py", line 957, in render_annotated
    return self.render(context)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/template/base.py", line 1046, in render
    return render_value_in_context(output, context)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/template/base.py", line 1024, in render_value_in_context
    value = force_text(value)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/utils/encoding.py", line 76, in force_text
    s = six.text_type(s)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/utils/html.py", line 385, in <lambda>
    klass.__str__ = lambda self: mark_safe(klass_str(self))
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/forms/boundfield.py", line 41, in __str__
    return self.as_widget()
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/forms/boundfield.py", line 120, in as_widget
    **kwargs
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/forms/widgets.py", line 220, in render
    context = self.get_context(name, value, attrs)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/contrib/admin/widgets.py", line 281, in get_context
    'rendered_widget': self.widget.render(name, value, attrs),
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/dal/widgets.py", line 150, in render
    widget = super(WidgetMixin, self).render(name, value, attrs)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/forms/widgets.py", line 220, in render
    context = self.get_context(name, value, attrs)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/forms/widgets.py", line 665, in get_context
    context = super(Select, self).get_context(name, value, attrs)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/forms/widgets.py", line 627, in get_context
    context['widget']['optgroups'] = self.optgroups(name, context['widget']['value'], attrs)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/dal/widgets.py", line 140, in optgroups
    self.filter_choices_to_render(selected_choices)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/dal/widgets.py", line 183, in filter_choices_to_render
    pk__in=[c for c in selected_choices if c]
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/db/models/query.py", line 781, in filter
    return self._filter_or_exclude(False, *args, **kwargs)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/db/models/query.py", line 799, in _filter_or_exclude
    clone.query.add_q(Q(*args, **kwargs))
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/db/models/sql/query.py", line 1260, in add_q
    clause, _ = self._add_q(q_object, self.used_aliases)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/db/models/sql/query.py", line 1286, in _add_q
    allow_joins=allow_joins, split_subq=split_subq,
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/db/models/sql/query.py", line 1220, in build_filter
    condition = self.build_lookup(lookups, col, value)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/db/models/sql/query.py", line 1114, in build_lookup
    return final_lookup(lhs, rhs)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/db/models/lookups.py", line 24, in __init__
    self.rhs = self.get_prep_lookup()
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/db/models/lookups.py", line 219, in get_prep_lookup
    rhs_value = self.lhs.output_field.get_prep_value(rhs_value)
  File "/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/db/models/fields/__init__.py", line 962, in get_prep_value
    return int(value)
ValueError: invalid literal for int() with base 10: '1.211'

Changing the setting:

USE_THOUSAND_SEPARATOR = False

renders everything back to normal.

Change History (18)

comment:1 Changed 2 years ago by George Tantiras

Description: modified (diff)

comment:2 Changed 2 years ago by George Tantiras

Description: modified (diff)

comment:3 Changed 2 years ago by Tim Graham

In the traceback, I see you're using django-autocomplete-light (dal). Can you reproduce this issue without that package to rule out an issue there? Could you provide a sample project to more easily reproduce the issue. I tried to briefly reproduce it using but putting a foreign key in ModelAdmin.raw_id_fields but couldn't reproduce. Ideally, you could also bisect to find the commit where the behavior changed.

comment:4 Changed 2 years ago by Tim Graham

Resolution: needsinfo
Status: newclosed

comment:5 Changed 2 years ago by George Tantiras

To reproduce the issue, I followed the below steps:

In a fresh Django-1.11 installation:

  • Create the app separ
  • separ/models.py:
    from django.db import models
    
    
    class Parent(models.Model):
        a_field = models.BooleanField()
    
    
    class Child(models.Model):
        parent_key = models.ForeignKey(Parent)
    
  • separ/admin.py:
    from django.contrib import admin
    from separ.models import Parent, Child
    
    
    class ParentAdmin(admin.ModelAdmin):
        pass
    
    admin.site.register(Parent, ParentAdmin)
    
    
    class ChildAdmin(admin.ModelAdmin):
        pass
    
    admin.site.register(Child, ChildAdmin)
    
  • in the settings.py file add the separ module in the INSTALLED_APPS and the line:
        USE_THOUSAND_SEPARATOR = True
    
  • Load the Parent model with data:
    >>> from separ.models import Parent, Child
    >>> for i in range(1,1500): 
    ...        a = Parent(a_field = True)
    ...        a.save()
    >>> a = Parent.objects.get(id=1235)
    >>> b = Child(parent_key=a)
    >>> b.save()
    
  • Visit the relevant url and hit 'Save and Add Another":
    http://127.0.0.1:8000/admin/separ/child/1/change/
    

An error message appears: Select a valid choice. That choice is not one of the available choices.

Trying to debug the error I have found that:

/home/flyer/.virtualenvs/erad/lib/python3.4/site-packages/django/contrib/admin/options.py(1438)_changeform_view()
   1437         ModelForm = self.get_form(request, obj)
-> 1438         if request.method == 'POST':
   1439             form = ModelForm(request.POST, request.FILES, instance=obj)

ipdb> request.POST
<QueryDict: {'parent_key': ['1.235'], ... }

The result of the request.POST command contains the id of the related model localised: ['1.235'].

I have not yet performed a bisect because I have not done it before, but I will give it a try.

Last edited 2 years ago by George Tantiras (previous) (diff)

comment:6 Changed 2 years ago by George Tantiras

Summary: USE_THOUSANDS_SEPARATOR = True, causes foreignkey id to become localizedUSE_THOUSAND_SEPARATOR = True, causes foreignkey id to become localized

comment:7 Changed 2 years ago by George Tantiras

Description: modified (diff)

comment:8 Changed 2 years ago by George Tantiras

Resolution: needsinfo
Status: closednew

comment:9 Changed 2 years ago by Tim Graham

Which locale are you using?

comment:10 Changed 2 years ago by George Tantiras

In the example, am using the settings.py as it comes with the default django installation.
However, if it might make sense, I have the following files in my project tree, for the locales en, el:

  • <project name>formats/en/formats.py, <project name>formats/el/formats.py
    DATE_FORMAT = 'd-m-Y'
    TIME_FORMAT = 'H:i'
    DATETIME_FORMAT = 'd-m-Y H:i'
    YEAR_MONTH_FORMAT = 'F Y'
    MONTH_DAY_FORMAT = 'F j'
    SHORT_DATE_FORMAT = 'm/d/Y'
    SHORT_DATETIME_FORMAT = 'm/d/Y P'
    FIRST_DAY_OF_WEEK = 1
    
    # BUT here use the Python strftime format syntax,
    # LIST: http://docs.python.org/library/datetime.html#strftime-strptime-behavior
    
    DATE_INPUT_FORMATS = (
        '%d-%m-%Y',     # '21-03-2014'
    )
    TIME_INPUT_FORMATS = (
        '%H:%M:%S',     # '17:59:59'
        '%H:%M',        # '17:59'
    )
    DATETIME_INPUT_FORMATS = (
        '%d-%m-%Y %H:%M',     # '21-03-2014 17:59'
    )
    
    DECIMAL_SEPARATOR = ','
    THOUSAND_SEPARATOR = '.'
    NUMBER_GROUPING = 3
    

comment:11 Changed 2 years ago by Tim Graham

Unfortunately, I still can't reproduce. Can you provide a sample project I could download and run?

comment:12 Changed 2 years ago by George Tantiras

I have uploaded a slightly modified default project which reflects the issue under Python 3.4.2 here:

https://github.com/raratiru/28075

comment:13 Changed 2 years ago by Tim Graham

Component: UncategorizedForms
Severity: NormalRelease blocker
Triage Stage: UnreviewedAccepted
Type: UncategorizedBug

Thanks, I can reproduce using that project. I bisected the regression to 6d8979f4c2fbfb9fd5db92acd72489cbbcbdd5d1.

comment:14 Changed 2 years ago by Tim Graham

Cc: Jon Dufresne added

I marked #28090 as a duplicate -- it's another case where the <select> value is inadvertently localized.

comment:15 Changed 2 years ago by Jon Dufresne

Owner: changed from nobody to Jon Dufresne
Status: newassigned

I can reproduce and have written a test. I'll further investigate a fix.

comment:16 Changed 2 years ago by Jon Dufresne

Has patch: set

comment:17 Changed 2 years ago by Tim Graham <timograham@…>

Resolution: fixed
Status: assignedclosed

In 581879a:

Fixed #28075 -- Prevented ChoiceWidget from localizing option values.

comment:18 Changed 2 years ago by Tim Graham <timograham@…>

In 1442e299:

[1.11.x] Fixed #28075 -- Prevented ChoiceWidget from localizing option values.

Backport of 581879a510e58841e5780a5d1fdb4995733d2036 from master

Note: See TracTickets for help on using tickets.
Back to Top