Avoid " TypeError: Cannot convert Decimal("0.0000") to Decimal " when the decimal module has been reloaded

[gagravarr]

If for some reason the decimal module gets reloaded (seems fairly easy to trigger when using runserver, but we've seen it once or twice wiht apache + mod_python too), then calling db_obj.save() on a model with a decimal field will blow up (see ​http://groups.google.com/group/django-users/browse_thread/thread/7da92d7f5d6e2a53 for example)

One workaround is to have extra code in the decimal field logic in django, to detect when the value is no longer being recognised as a Decimal but is one, and port it over to the new decimal object.

--- django/db/models/fields/__init__.py (revision 9643)
+++ django/db/models/fields/__init__.py (working copy)
@@ -579,6 +579,11 @@
     def to_python(self, value):
         if value is None:
             return value
+        # Work around reload(decimal) problems
+        if not isinstance(value, decimal.Decimal) and \
+               len(str(value.__class__).split("'")) == 3 and \
+               str(value.__class__).split("'")[1] == 'decimal.Decimal':
+            return decimal.Decimal( value.to_eng_string() )
         try:
             return decimal.Decimal(value)
         except decimal.InvalidOperation:

I'm not sure if this is an ideal fix or not, but it'd be great to have this (or something like it) in django to help avoid the issue

[Thomas Capricelli]

hello. I'm hit by this same problem. I'm using django trunk (mainly because I hoped this bug was fixed), and python 2.6.2.

[Luke Plant]

If you could reproduce the error using a reload() statement in a test (probably put it in regressiontests/model_fields/tests.py), that would be great. After that we could decide on a fix. The fix above is probably OK. I'd really like to know why 'decimal' is being reloaded, but given that this is causing genuine problems for people, it's better to fix it than leave it until we know the root cause.

[Thomas Capricelli]

Hello. Just to confirm that the bug (really annoying if you have 'send error mail' on) in django trunk as of today, although it's really, really worse. If i disable the above patch (that i'm using for very long), i got such a mail for almost every save(). I think it happenned less often in may.
I'm using apache + mod_python.
The updated patch is :

--- a/django/db/models/fields/__init__.py       Tue Jan 12 16:33:40 2010 +0100
+++ b/django/db/models/fields/__init__.py       Tue Jan 12 16:57:48 2010 +0100
@@ -761,6 +761,11 @@
     def to_python(self, value):
         if value is None:
             return value
+        # Work around reload(decimal) problems
+        if not isinstance(value, decimal.Decimal) and \
+               len(str(value.__class__).split("'")) == 3 and \
+               str(value.__class__).split("'")[1] == 'decimal.Decimal':
+            return decimal.Decimal( value.to_eng_string() )
         try:
             return decimal.Decimal(value)
         except decimal.InvalidOperation:

[hotani]

I'm having this problem with latest SVN (13053). The occasional errors are e-mailed to me and I am unable to reproduce. I'm going to try applying the patch above and seeing if that helps.

Using apache/mod_wsgi

[rikwade]

I believe I am encountering this problem with my Django application. Last couple of lines of Django debug:

[...]
 File "/usr/lib/python2.5/decimal.py", line 656, in __new__
   raise TypeError("Cannot convert %r to Decimal" % value)

TypeError: Cannot convert Decimal("10.5") to Decimal

I am running Django 1.2.5 with Apache 2 and WSGI. A code fix or documentation note with supported workaround would be appreciated.

[mindthief]

Hi,
I added the above patch and re-ran python setup.py install to reinstall django. Now, the problem is still there but occurs in the patch I just added!

Exception Type: TypeError
Exception Value:
Cannot convert Decimal('1') to Decimal
Exception Location: /usr/lib/python2.6/decimal.py in new, line 651
Python Executable: /usr/bin/python
Python Version: 2.6.5
Python Path:
['/usr/lib/python2.6',

'/usr/lib/python2.6/plat-linux2',
'/usr/lib/python2.6/lib-tk',
'/usr/lib/python2.6/lib-old',
'/usr/lib/python2.6/lib-dynload',
'/usr/lib/python2.6/dist-packages',

The stacktrace where the error specifically occurs is:

File "/usr/local/lib/python2.6/dist-packages/django/db/models/fields/subclassing.py" in inner

28. return func(*args, kwargs)

File "/usr/local/lib/python2.6/dist-packages/django/db/models/fields/init.py" in get_db_prep_save

786. # this method.

File "/usr/local/lib/python2.6/dist-packages/django/db/models/fields/init.py" in to_python

761. if not isinstance(value, decimal.Decimal) and \

File "/usr/lib/python2.6/decimal.py" in new

651. raise TypeError("Cannot convert %r to Decimal" % value)

Exception Type: TypeError at /vote
Exception Value: Cannot convert Decimal('1') to Decimal

[Will Hardy]

Tests and cleaner fix

[Will Hardy]

This is a problem when you combine isinstance with reload. Because we can't avoid using isinstance in the decimal model, I've created a new decimal object when TypeError is raised, using the as_tuple().

[adrian.boczkowski@…]

This problem also occurs with Django 1.3 on Apache + mod_wsgi. I fix it by changing (sorry, I can't make .diff file):

try:
             return decimal.Decimal(value)

to:

try:
             return decimal.Decimal(str(value))

I know it's rather ugly workaround than beautiful fix ;-), but it seems to work fine.

[ejohnstone@…]

I saw a similar problem today, and it was isolated to Internet Explorer 7 and 8.
This error did not show up on Firefox 6.01, nor Opera 11.51

In my case, I had mistakenly written

return Decimal(self.bid_price - self.amt_paid)

And self.bid_price and self.amt_paid were already of type 'Decimal'.
IE raised the error on this, yet Firefox and Opera didn't.
This seems strange to me.
I am using django 1.3.0

[adrian.boczkowski@…]

Check something like this before return statement:

if type(self.bid_price) != type(self.amt_paid):
   raise Exception

I guess you will see an error. It raises an exception, because decimal module gets reloaded (as I read comments before), and the first decimal.Decimal and the second decimal.Decimal aren't the same. If you wan't to workaround it now, cast Decimals to str, and then to Decimal (decimal.Decimal(str(self.bid.price))) - I used it, but you'll probably have to change many lines of your code...

Has anyone seen this error on other platforms (not apache+mod_wsgi based servers)? It seems it's rather mod_wsgi bug.

Replying to ejohnstone@…:

I saw a similar problem today, and it was isolated to Internet Explorer 7 and 8.
This error did not show up on Firefox 6.01, nor Opera 11.51

In my case, I had mistakenly written

return Decimal(self.bid_price - self.amt_paid)

And self.bid_price and self.amt_paid were already of type 'Decimal'.
IE raised the error on this, yet Firefox and Opera didn't.
This seems strange to me.
I am using django 1.3.0

[ejohnstone@…]

Here's the approach I am taking, now, that seems to be working. First thanks for the str() suggestion. It seemed to work sometimes, meaning I didn't catch it in all places or there was something else going on.

However I found that configurations #2 and #3 below worked, so far, without depending on the str() fix.

I will post 3 configurations from my /usr/local/etc/apache22/extra/httpd-vhosts.conf (FreeBSD path). The first one is the configuration that does not work, i.e that gives the error. The second one works, and the third one works. I've commented them with the relevant information, and I'm currently using the third one.

      ServerName my_site.com
      ......
      ......
      ## Configuration #1
      ## WSGIScriptAlias / /path_to_my_app/apache/django.wsgi

      ## Configuration #2
      ## Solved decimal loading problem
      ## http://www.defitek.com/blog/2010/06/29/cant-adapt-type-decimal-error-with-django-apache-postgresql-psycopg2/
      ## http://groups.google.com/group/django-users/browse_thread/thread/44c2670de1509ac5
      ## WSGIDaemonProcess my_site
      ## WSGIScriptAlias / /path_to_my_app/apache/django.wsgi process-group=my_site application-group=%{GLOBAL}

      ## Configuration #3
      ## Also solved decimal loading problem
      ## From "Django 1.1 Testing and Debugging" by Karen M. Tracey
      ## Great! book
      WSGIScriptAlias / /path_to_my_app/apache/django.wsgi
      WSGIDaemonProcess my_site
      WSGIProcessGroup my_site

Information about version, OS, hardware:

pkg_info -XI 'wsgi|python|django|apache' |egrep -v registration
ap22-mod_wsgi-3.3_1 Python WSGI adapter module for Apache
apache-2.2.19       Version 2.2.x of Apache web server with prefork MPM.
py27-django-1.3     High-level Python Web framework
python27-2.7.2_1    An interpreted object-oriented programming language

uname -rpmi
8.0-RELEASE-p4-jc2 amd64 amd64 jail8

[anonymous]

Replying to adrian.boczkowski@…:

This problem also occurs with Django 1.3 on Apache + mod_wsgi. I fix it by changing (sorry, I can't make .diff file):

try:
             return decimal.Decimal(value)

to:

try:
             return decimal.Decimal(str(value))

I know it's rather ugly workaround than beautiful fix ;-), but it seems to work fine.

Hi Adrian,
Just asking on where you applied this fix?

[pauldwaite]

I’ve also seen this bug (specifically when rendering Decimal values via a custom template tag).

I’m utterly mystified as to how I’m seeing it though, because it’s on a site that I run on a virtual private server, the disk image for which is a copy of a VMWare virtual machine that I run on my laptop to test the site. This bug only happens on the live environment, and not on my laptop, even though both are running from the same original disk image.

I presume I must have changed *something* on the live disk image, but I don’t recall doing so (I’m reasonably careful about that).

[anonymous]

I can confirm this bug on Django 1.3 on Apache + mod_wsgi. I have tried applying adrian's patch. As I can't reproduce the error, I just get them emailed to me occasionally, I cannot yet confirm that it works.

[Marcel Eyer]

Replying to adrian.boczkowski@…:

I confirm this bug with django 1.3, ngnix, uwsgi, Python 2.6. Reloading uwsgi solves the problem (which is of course not a proper solution)
The bug is not reproducible and occurs rarely. Its really hard to debug ...

(System: Ubuntu 10.04.2 LTS i686)

[Jonas Kölker]

​http://code.google.com/p/modwsgi/wiki/ApplicationIssues#Multiple_Python_Sub_Interpreters explicitly mentions decimal.Decimal in the context of psycopg2. I'm experiencing this problem using sqlite3 as my database, so it's obviously not limited to PostgreSQL. I think this explains why the workaround/solution in comment:15 works. HTH :-)

[MaDeuce]

Replying to jonaskoelker:

​http://code.google.com/p/modwsgi/wiki/ApplicationIssues#Multiple_Python_Sub_Interpreters explicitly mentions decimal.Decimal in the context of psycopg2. I'm experiencing this problem using sqlite3 as my database, so it's obviously not limited to PostgreSQL. I think this explains why the workaround/solution in comment:15 works. HTH :-)

Like jonaskoelker, I'm using sqlite3 with Django 1.3, mod_wsgi 3.3, and have the problem. Unfortunately, modifying my httpd.conf to align with that of comment:15 does not make the problem go away. I will try the patch offered in comment:18, as I'm out of other ideas. However, since I don't understand what is causing mod_wsgi to reload Decimal in the first place, I don't have much confidence that it will resolve my specific problem (even though it may indeed resolve the psycopg2 problem). If someone could offer pointers as to how to track down the root cause of the reload, I'd be highly appreciative. I can only reproduce after about an hour of intense load on a production system, so it's difficult to conduct experiments. I'm also a little concerned that the fix of comment:18 is 1.5 years old and (AFAICT) has not been incorporated into the Django code base; are there any negative side-effects?

[anonymous]

One more data point: I'm also getting this using apache and django 1.4.3

[anonymous]

Was getting following error twice a day running django 1.4.5, libapache_mod_wsgi 3.3, apache 2.2.22, python 2.7.3 on Debian 7.0:

[..]
  File "/usr/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 847, in to_python
    return decimal.Decimal(value)

  File "/usr/lib/python2.7/decimal.py", line 658, in __new__
    raise TypeError("Cannot convert %r to Decimal" % value)

TypeError: Cannot convert Decimal('18') to Decimal

with the following WSGI request parameters:

 'mod_wsgi.application_group': 'hostname|',
 'mod_wsgi.callable_object': 'application',
 'mod_wsgi.handler_script': '',
 'mod_wsgi.input_chunked': '0',
 'mod_wsgi.listener_host': '',
 'mod_wsgi.listener_port': '80',
 'mod_wsgi.process_group': '',
 'mod_wsgi.request_handler': 'wsgi-script',
 'mod_wsgi.script_reloading': '1',
 'mod_wsgi.version': (3, 3),
 'wsgi.errors': <mod_wsgi.Log object at 0xb8788ae8>,
 'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter object at 0xb90561d0>,
 'wsgi.input': <mod_wsgi.Input object at 0xb87886d8>,
 'wsgi.multiprocess': True,
 'wsgi.multithread': False,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'http',
 'wsgi.version': (1, 1)}>

First change to virtual host config:

<VirtualHost *:80>
...
     ServerName servername
     WSGIScriptAlias / /path/to/wsgi.py
+    WGSIDaemonProcess servername
+    WSGIProcessGroup servername
</VirtualHost>
...

Next day same decimal reloading error. WSGI request parameters:

...
 'mod_wsgi.application_group': 'hostname|',
 'mod_wsgi.callable_object': 'application',
 'mod_wsgi.handler_script': '',
 'mod_wsgi.input_chunked': '0',
 'mod_wsgi.listener_host': '',
 'mod_wsgi.listener_port': '80',
 'mod_wsgi.process_group': 'servername',
 'mod_wsgi.request_handler': 'wsgi-script',
 'mod_wsgi.script_reloading': '1',
 'mod_wsgi.version': (3, 3),
 'wsgi.errors': <mod_wsgi.Log object at 0xba83ced0>,
 'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter object at 0xba8ea4a0>,
 'wsgi.input': <mod_wsgi.Input object at 0xba82f228>,
 'wsgi.multiprocess': False,
 'wsgi.multithread': True,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'http',
 'wsgi.version': (1, 1)}>

Second change to apache config:

<VirtualHost *:80>
...
     ServerName servername
-    WSGIScriptAlias / /path/to/wsgi.py
+    WSGIScriptAlias / /path/to/wsgi.py process-group=servername application-group=%{GLOBAL}
     WSGIDaemonProcess servername
-    WSGIProcessGroup servername
+    WSGIScriptReloading Off
...
</VirtualHost>

which seems to be in a similar vein to the docs ​https://docs.djangoproject.com/en/1.4/howto/deployment/wsgi/modwsgi/#using-mod-wsgi-daemon-mode

No errors for a month now, current WSGI request parameters.

...
 'mod_wsgi.application_group': '',
 'mod_wsgi.callable_object': 'application',
 'mod_wsgi.handler_script': '',
 'mod_wsgi.input_chunked': '0',
 'mod_wsgi.listener_host': '',
 'mod_wsgi.listener_port': '80',
 'mod_wsgi.process_group': 'servername',
 'mod_wsgi.request_handler': 'wsgi-script',
 'mod_wsgi.script_reloading': '0',
 'mod_wsgi.version': (3, 3),
 'wsgi.errors': <mod_wsgi.Log object at 0xb97a5278>,
 'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter object at 0xb9814de8>,
 'wsgi.input': <mod_wsgi.Input object at 0xb97a5098>,
 'wsgi.multiprocess': False,
 'wsgi.multithread': True,
 'wsgi.run_once': False,
 'wsgi.url_scheme': 'http',
 'wsgi.version': (1, 1)}>

[anonymous]

Appendum to above:

1. Running small office server
2. Using sqlite3 database
3. Exception raised by saving models (as above) and model methods as so:

   class Item(models.Model):
       def amount(self):
           return foo # A decimal
   
   class Purchase(models.Models):
       def tax(self):
           return item.amount*Decimal('0.1')
   

[anonymous]

I found this bug in Django 1.6.1, using runserver and sqlite3.

The model is:

class Car(models.Model):
  price = models.DecimalField(max_digits=7, decimal_places=2)
  maker = models.ForeignKey(CarMaker, null=True, blank=True, related_name="%(app_label)s_%(class)s_result")

And the piece of code that raised the exception:

try:
  car1.maker = maker1
  car1.save()
except:
  # manage error

It raised:

Cannot convert Decimal('3.03') to Decimal

Can provide the whole traceback if needed.

Can someone change the bug version to 1.6?

[anonymous]

I have also been having problems with this bug.

Apache/2.2.22 (Ubuntu)
Apache wsgi module 2.7
Python 2.7.3
Django 1.6

My solution is to 'monkey-patch' the DecimalField class in the models.py file where I use it:

import decimal
from django.db import models

############
## Patch problem with Decimal field
############

def to_python(self, value):
    if value is None:
        return value
    try:
        return decimal.Decimal(str(value))
    except decimal.InvalidOperation:
        raise exceptions.ValidationError(
            self.error_messages['invalid'],
            code='invalid',
            params={'value': value},
        )

models.DecimalField.to_python = to_python

[warnes]

Replying to anonymous:

Wasn't logged-in when I posted.

[Tim Graham]

As noted in comment 25, I think this is probably related to incorrect server configuration. If it's still an issue, we need much more specific information about how to reproduce it.