Code

Opened 5 years ago

Last modified 4 weeks ago

#10933 new Bug

Avoid " TypeError: Cannot convert Decimal("0.0000") to Decimal " when the decimal module has been reloaded

Reported by: gagravarr Owned by: nobody
Component: Database layer (models, ORM) Version: 1.3
Severity: Normal Keywords: dceu2011
Cc: me@…, jonaskoelker, MaDeuce Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: yes Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

If for some reason the decimal module gets reloaded (seems fairly easy to trigger when using runserver, but we've seen it once or twice wiht apache + mod_python too), then calling db_obj.save() on a model with a decimal field will blow up (see http://groups.google.com/group/django-users/browse_thread/thread/7da92d7f5d6e2a53 for example)

One workaround is to have extra code in the decimal field logic in django, to detect when the value is no longer being recognised as a Decimal but is one, and port it over to the new decimal object.

--- django/db/models/fields/__init__.py (revision 9643)
+++ django/db/models/fields/__init__.py (working copy)
@@ -579,6 +579,11 @@
     def to_python(self, value):
         if value is None:
             return value
+        # Work around reload(decimal) problems
+        if not isinstance(value, decimal.Decimal) and \
+               len(str(value.__class__).split("'")) == 3 and \
+               str(value.__class__).split("'")[1] == 'decimal.Decimal':
+            return decimal.Decimal( value.to_eng_string() )
         try:
             return decimal.Decimal(value)
         except decimal.InvalidOperation:

I'm not sure if this is an ideal fix or not, but it'd be great to have this (or something like it) in django to help avoid the issue

Attachments (1)

ticket10933.patch (1.8 KB) - added by willhardy 3 years ago.
Tests and cleaner fix

Download all attachments as: .zip

Change History (30)

comment:1 Changed 5 years ago by orzel

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset

hello. I'm hit by this same problem. I'm using django trunk (mainly because I hoped this bug was fixed), and python 2.6.2.

comment:2 Changed 5 years ago by lukeplant

If you could reproduce the error using a reload() statement in a test (probably put it in regressiontests/model_fields/tests.py), that would be great. After that we could decide on a fix. The fix above is probably OK. I'd really like to know why 'decimal' is being reloaded, but given that this is causing genuine problems for people, it's better to fix it than leave it until we know the root cause.

comment:3 Changed 4 years ago by SmileyChris

  • Needs tests set
  • Triage Stage changed from Unreviewed to Accepted

comment:4 Changed 4 years ago by orzel

  • Version changed from 1.0 to SVN

Hello. Just to confirm that the bug (really annoying if you have 'send error mail' on) in django trunk as of today, although it's really, really worse. If i disable the above patch (that i'm using for very long), i got such a mail for almost every save(). I think it happenned less often in may.
I'm using apache + mod_python.
The updated patch is :

--- a/django/db/models/fields/__init__.py       Tue Jan 12 16:33:40 2010 +0100
+++ b/django/db/models/fields/__init__.py       Tue Jan 12 16:57:48 2010 +0100
@@ -761,6 +761,11 @@
     def to_python(self, value):
         if value is None:
             return value
+        # Work around reload(decimal) problems
+        if not isinstance(value, decimal.Decimal) and \
+               len(str(value.__class__).split("'")) == 3 and \
+               str(value.__class__).split("'")[1] == 'decimal.Decimal':
+            return decimal.Decimal( value.to_eng_string() )
         try:
             return decimal.Decimal(value)
         except decimal.InvalidOperation:

comment:5 Changed 4 years ago by hotani

I'm having this problem with latest SVN (13053). The occasional errors are e-mailed to me and I am unable to reproduce. I'm going to try applying the patch above and seeing if that helps.

Using apache/mod_wsgi

comment:6 Changed 3 years ago by rikwade

I believe I am encountering this problem with my Django application. Last couple of lines of Django debug:

[...]
 File "/usr/lib/python2.5/decimal.py", line 656, in __new__
   raise TypeError("Cannot convert %r to Decimal" % value)

TypeError: Cannot convert Decimal("10.5") to Decimal

I am running Django 1.2.5 with Apache 2 and WSGI. A code fix or documentation note with supported workaround would be appreciated.

comment:7 Changed 3 years ago by lukeplant

  • Severity set to Normal
  • Type set to Bug

comment:8 Changed 3 years ago by mindthief

  • Easy pickings unset

Hi,
I added the above patch and re-ran python setup.py install to reinstall django. Now, the problem is still there but occurs in the patch I just added!

Exception Type: TypeError
Exception Value:
Cannot convert Decimal('1') to Decimal
Exception Location: /usr/lib/python2.6/decimal.py in new, line 651
Python Executable: /usr/bin/python
Python Version: 2.6.5
Python Path:
['/usr/lib/python2.6',

'/usr/lib/python2.6/plat-linux2',
'/usr/lib/python2.6/lib-tk',
'/usr/lib/python2.6/lib-old',
'/usr/lib/python2.6/lib-dynload',
'/usr/lib/python2.6/dist-packages',

The stacktrace where the error specifically occurs is:

File "/usr/local/lib/python2.6/dist-packages/django/db/models/fields/subclassing.py" in inner

  1. return func(*args, kwargs)

File "/usr/local/lib/python2.6/dist-packages/django/db/models/fields/init.py" in get_db_prep_save

  1. # this method.

File "/usr/local/lib/python2.6/dist-packages/django/db/models/fields/init.py" in to_python

  1. if not isinstance(value, decimal.Decimal) and \

File "/usr/lib/python2.6/decimal.py" in new

  1. raise TypeError("Cannot convert %r to Decimal" % value)

Exception Type: TypeError at /vote
Exception Value: Cannot convert Decimal('1') to Decimal

comment:9 Changed 3 years ago by anonymous

  • Patch needs improvement set

Changed 3 years ago by willhardy

Tests and cleaner fix

comment:10 Changed 3 years ago by willhardy

  • Keywords dceu2011 added
  • Patch needs improvement unset
  • UI/UX unset

comment:11 Changed 3 years ago by willhardy

This is a problem when you combine isinstance with reload. Because we can't avoid using isinstance in the decimal model, I've created a new decimal object when TypeError is raised, using the as_tuple().

comment:12 follow-up: Changed 3 years ago by adrian.boczkowski@…

This problem also occurs with Django 1.3 on Apache + mod_wsgi. I fix it by changing (sorry, I can't make .diff file):

try:
             return decimal.Decimal(value)

to:

try:
             return decimal.Decimal(str(value))

I know it's rather ugly workaround than beautiful fix ;-), but it seems to work fine.

comment:13 follow-up: Changed 3 years ago by ejohnstone@…

  • Version changed from SVN to 1.3

I saw a similar problem today, and it was isolated to Internet Explorer 7 and 8.
This error did not show up on Firefox 6.01, nor Opera 11.51

In my case, I had mistakenly written

return Decimal(self.bid_price - self.amt_paid)

And self.bid_price and self.amt_paid were already of type 'Decimal'.
IE raised the error on this, yet Firefox and Opera didn't.
This seems strange to me.
I am using django 1.3.0

comment:14 in reply to: ↑ 13 Changed 3 years ago by adrian.boczkowski@…

Check something like this before return statement:

if type(self.bid_price) != type(self.amt_paid):
   raise Exception

I guess you will see an error. It raises an exception, because decimal module gets reloaded (as I read comments before), and the first decimal.Decimal and the second decimal.Decimal aren't the same. If you wan't to workaround it now, cast Decimals to str, and then to Decimal (decimal.Decimal(str(self.bid.price))) - I used it, but you'll probably have to change many lines of your code...

Has anyone seen this error on other platforms (not apache+mod_wsgi based servers)? It seems it's rather mod_wsgi bug.

Replying to ejohnstone@…:

I saw a similar problem today, and it was isolated to Internet Explorer 7 and 8.
This error did not show up on Firefox 6.01, nor Opera 11.51

In my case, I had mistakenly written

return Decimal(self.bid_price - self.amt_paid)

And self.bid_price and self.amt_paid were already of type 'Decimal'.
IE raised the error on this, yet Firefox and Opera didn't.
This seems strange to me.
I am using django 1.3.0

comment:15 Changed 3 years ago by ejohnstone@…

Here's the approach I am taking, now, that seems to be working. First thanks for the str() suggestion. It seemed to work sometimes, meaning I didn't catch it in all places or there was something else going on.

However I found that configurations #2 and #3 below worked, so far, without depending on the str() fix.

I will post 3 configurations from my /usr/local/etc/apache22/extra/httpd-vhosts.conf (FreeBSD path). The first one is the configuration that does not work, i.e that gives the error. The second one works, and the third one works. I've commented them with the relevant information, and I'm currently using the third one.

      ServerName my_site.com
      ......
      ......
      ## Configuration #1
      ## WSGIScriptAlias / /path_to_my_app/apache/django.wsgi

      ## Configuration #2
      ## Solved decimal loading problem
      ## http://www.defitek.com/blog/2010/06/29/cant-adapt-type-decimal-error-with-django-apache-postgresql-psycopg2/
      ## http://groups.google.com/group/django-users/browse_thread/thread/44c2670de1509ac5
      ## WSGIDaemonProcess my_site
      ## WSGIScriptAlias / /path_to_my_app/apache/django.wsgi process-group=my_site application-group=%{GLOBAL}

      ## Configuration #3
      ## Also solved decimal loading problem
      ## From "Django 1.1 Testing and Debugging" by Karen M. Tracey
      ## Great! book
      WSGIScriptAlias / /path_to_my_app/apache/django.wsgi
      WSGIDaemonProcess my_site
      WSGIProcessGroup my_site

Information about version, OS, hardware:

pkg_info -XI 'wsgi|python|django|apache' |egrep -v registration
ap22-mod_wsgi-3.3_1 Python WSGI adapter module for Apache
apache-2.2.19       Version 2.2.x of Apache web server with prefork MPM.
py27-django-1.3     High-level Python Web framework
python27-2.7.2_1    An interpreted object-oriented programming language

uname -rpmi
8.0-RELEASE-p4-jc2 amd64 amd64 jail8

comment:16 in reply to: ↑ 12 Changed 2 years ago by anonymous

Replying to adrian.boczkowski@…:

This problem also occurs with Django 1.3 on Apache + mod_wsgi. I fix it by changing (sorry, I can't make .diff file):

try:
             return decimal.Decimal(value)

to:

try:
             return decimal.Decimal(str(value))

I know it's rather ugly workaround than beautiful fix ;-), but it seems to work fine.

Hi Adrian,
Just asking on where you applied this fix?

comment:17 Changed 2 years ago by pauldwaite

I’ve also seen this bug (specifically when rendering Decimal values via a custom template tag).

I’m utterly mystified as to how I’m seeing it though, because it’s on a site that I run on a virtual private server, the disk image for which is a copy of a VMWare virtual machine that I run on my laptop to test the site. This bug only happens on the live environment, and not on my laptop, even though both are running from the same original disk image.

I presume I must have changed *something* on the live disk image, but I don’t recall doing so (I’m reasonably careful about that).

comment:18 Changed 2 years ago by anonymous

I can confirm this bug on Django 1.3 on Apache + mod_wsgi. I have tried applying adrian's patch. As I can't reproduce the error, I just get them emailed to me occasionally, I cannot yet confirm that it works.

comment:19 Changed 2 years ago by maersu

Replying to adrian.boczkowski@…:

I confirm this bug with django 1.3, ngnix, uwsgi, Python 2.6. Reloading uwsgi solves the problem (which is of course not a proper solution)
The bug is not reproducible and occurs rarely. Its really hard to debug ...

(System: Ubuntu 10.04.2 LTS i686)

Interesting Discussion on Google Groups

Last edited 2 years ago by maersu (previous) (diff)

comment:20 Changed 2 years ago by maersu

  • Cc me@… added

comment:21 Changed 18 months ago by jonaskoelker

  • Cc jonaskoelker added

comment:22 follow-up: Changed 17 months ago by jonaskoelker

http://code.google.com/p/modwsgi/wiki/ApplicationIssues#Multiple_Python_Sub_Interpreters explicitly mentions decimal.Decimal in the context of psycopg2. I'm experiencing this problem using sqlite3 as my database, so it's obviously not limited to PostgreSQL. I think this explains why the workaround/solution in comment:15 works. HTH :-)

comment:23 in reply to: ↑ 22 Changed 16 months ago by MaDeuce

  • Cc MaDeuce added

Replying to jonaskoelker:

http://code.google.com/p/modwsgi/wiki/ApplicationIssues#Multiple_Python_Sub_Interpreters explicitly mentions decimal.Decimal in the context of psycopg2. I'm experiencing this problem using sqlite3 as my database, so it's obviously not limited to PostgreSQL. I think this explains why the workaround/solution in comment:15 works. HTH :-)

Like jonaskoelker, I'm using sqlite3 with Django 1.3, mod_wsgi 3.3, and have the problem. Unfortunately, modifying my httpd.conf to align with that of comment:15 does not make the problem go away. I will try the patch offered in comment:18, as I'm out of other ideas. However, since I don't understand what is causing mod_wsgi to reload Decimal in the first place, I don't have much confidence that it will resolve my specific problem (even though it may indeed resolve the psycopg2 problem). If someone could offer pointers as to how to track down the root cause of the reload, I'd be highly appreciative. I can only reproduce after about an hour of intense load on a production system, so it's difficult to conduct experiments. I'm also a little concerned that the fix of comment:18 is 1.5 years old and (AFAICT) has not been incorporated into the Django code base; are there any negative side-effects?

comment:24 Changed 16 months ago by anonymous

One more data point: I'm also getting this using apache and django 1.4.3

comment:25 Changed 12 months ago by anonymous

Was getting following error twice a day running django 1.4.5, libapache_mod_wsgi 3.3, apache 2.2.22, python 2.7.3 on Debian 7.0:

[..]
  File "/usr/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 847, in to_python
    return decimal.Decimal(value)

  File "/usr/lib/python2.7/decimal.py", line 658, in __new__
    raise TypeError("Cannot convert %r to Decimal" % value)

TypeError: Cannot convert Decimal('18') to Decimal

with the following WSGI request parameters:

 'mod_wsgi.application_group': 'hostname|',
 'mod_wsgi.callable_object': 'application',
 'mod_wsgi.handler_script': '',
 'mod_wsgi.input_chunked': '0',
 'mod_wsgi.listener_host': '',
 'mod_wsgi.listener_port': '80',
 'mod_wsgi.process_group': '',
 'mod_wsgi.request_handler': 'wsgi-script',
 'mod_wsgi.script_reloading': '1',
 'mod_wsgi.version': (3, 3),
 'wsgi.errors': <mod_wsgi.Log object at 0xb8788ae8>,
 'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter object at 0xb90561d0>,
 'wsgi.input': <mod_wsgi.Input object at 0xb87886d8>,
 'wsgi.multiprocess': True,
 'wsgi.multithread': False,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'http',
 'wsgi.version': (1, 1)}>

First change to virtual host config:

<VirtualHost *:80>
...
     ServerName servername
     WSGIScriptAlias / /path/to/wsgi.py
+    WGSIDaemonProcess servername
+    WSGIProcessGroup servername
</VirtualHost>
...

Next day same decimal reloading error. WSGI request parameters:

...
 'mod_wsgi.application_group': 'hostname|',
 'mod_wsgi.callable_object': 'application',
 'mod_wsgi.handler_script': '',
 'mod_wsgi.input_chunked': '0',
 'mod_wsgi.listener_host': '',
 'mod_wsgi.listener_port': '80',
 'mod_wsgi.process_group': 'servername',
 'mod_wsgi.request_handler': 'wsgi-script',
 'mod_wsgi.script_reloading': '1',
 'mod_wsgi.version': (3, 3),
 'wsgi.errors': <mod_wsgi.Log object at 0xba83ced0>,
 'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter object at 0xba8ea4a0>,
 'wsgi.input': <mod_wsgi.Input object at 0xba82f228>,
 'wsgi.multiprocess': False,
 'wsgi.multithread': True,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'http',
 'wsgi.version': (1, 1)}>

Second change to apache config:

<VirtualHost *:80>
...
     ServerName servername
-    WSGIScriptAlias / /path/to/wsgi.py
+    WSGIScriptAlias / /path/to/wsgi.py process-group=servername application-group=%{GLOBAL}
     WSGIDaemonProcess servername
-    WSGIProcessGroup servername
+    WSGIScriptReloading Off
...
</VirtualHost>

which seems to be in a similar vein to the docs https://docs.djangoproject.com/en/1.4/howto/deployment/wsgi/modwsgi/#using-mod-wsgi-daemon-mode

No errors for a month now, current WSGI request parameters.

...
 'mod_wsgi.application_group': '',
 'mod_wsgi.callable_object': 'application',
 'mod_wsgi.handler_script': '',
 'mod_wsgi.input_chunked': '0',
 'mod_wsgi.listener_host': '',
 'mod_wsgi.listener_port': '80',
 'mod_wsgi.process_group': 'servername',
 'mod_wsgi.request_handler': 'wsgi-script',
 'mod_wsgi.script_reloading': '0',
 'mod_wsgi.version': (3, 3),
 'wsgi.errors': <mod_wsgi.Log object at 0xb97a5278>,
 'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter object at 0xb9814de8>,
 'wsgi.input': <mod_wsgi.Input object at 0xb97a5098>,
 'wsgi.multiprocess': False,
 'wsgi.multithread': True,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'http',
 'wsgi.version': (1, 1)}>

comment:26 Changed 12 months ago by anonymous

Appendum to above:

  1. Running small office server
  2. Using sqlite3 database
  3. Exception raised by saving models (as above) and model methods as so:
    class Item(models.Model):
        def amount(self):
            return foo # A decimal
    
    class Purchase(models.Models):
        def tax(self):
            return item.amount*Decimal('0.1')
    

comment:27 Changed 2 months ago by anonymous

I found this bug in Django 1.6.1, using runserver and sqlite3.

The model is:

class Car(models.Model):
  price = models.DecimalField(max_digits=7, decimal_places=2)
  maker = models.ForeignKey(CarMaker, null=True, blank=True, related_name="%(app_label)s_%(class)s_result")

And the piece of code that raised the exception:

try:
  car1.maker = maker1
  car1.save()
except:
  # manage error

It raised:

Cannot convert Decimal('3.03') to Decimal

Can provide the whole traceback if needed.

Can someone change the bug version to 1.6?

comment:28 follow-up: Changed 4 weeks ago by anonymous

I have also been having problems with this bug.

Apache/2.2.22 (Ubuntu)
Apache wsgi module 2.7
Python 2.7.3
Django 1.6

My solution is to 'monkey-patch' the DecimalField class in the models.py file where I use it:

import decimal
from django.db import models

############
## Patch problem with Decimal field
############

def to_python(self, value):
    if value is None:
        return value
    try:
        return decimal.Decimal(str(value))
    except decimal.InvalidOperation:
        raise exceptions.ValidationError(
            self.error_messages['invalid'],
            code='invalid',
            params={'value': value},
        )

models.DecimalField.to_python = to_python


comment:29 in reply to: ↑ 28 Changed 4 weeks ago by warnes

Replying to anonymous:

Wasn't logged-in when I posted.

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as new
The owner will be changed from nobody to anonymous. Next status will be 'assigned'
as The resolution will be set. Next status will be 'closed'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.