Opened 5 years ago

Closed 5 years ago

Last modified 4 years ago

#20161 closed Uncategorized (invalid)

save errors with multiproccesing

Reported by: oddghost@… Owned by: nobody
Component: Database layer (models, ORM) Version: 1.5
Severity: Normal Keywords:
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no


i have an exception then trying to process big data multiproccesed

    return self.get_query_set().get(*args, **kwargs)
  File "/opt/Envs/akkord/local/lib/python2.7/site-packages/django/db/models/", line 546, in save
  File "/opt/Envs/akkord/local/lib/python2.7/site-packages/django/db/models/", line 382, in get
    force_update=force_update, update_fields=update_fields)
  File "/opt/Envs/akkord/local/lib/python2.7/site-packages/django/db/models/", line 654, in save_base
  File "/opt/Envs/akkord/local/lib/python2.7/site-packages/django/db/", line 134, in commit_unless_managed
    num = len(clone)
  File "/opt/Envs/akkord/local/lib/python2.7/site-packages/django/db/models/", line 90, in __len__
  File "/opt/Envs/akkord/local/lib/python2.7/site-packages/django/db/backends/", line 221, in commit_unless_managed
    self._result_cache = list(self.iterator())
  File "/opt/Envs/akkord/local/lib/python2.7/site-packages/django/db/models/", line 301, in iterator
    for row in compiler.results_iter():
  File "/opt/Envs/akkord/local/lib/python2.7/site-packages/django/db/models/sql/", line 775, in results_iter
  File "/opt/Envs/akkord/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/", line 240, in _commit
    return self.connection.commit()
DatabaseError: error with no message from the libpq
    for rows in self.execute_sql(MULTI):
  File "/opt/Envs/akkord/local/lib/python2.7/site-packages/django/db/models/sql/", line 854, in <lambda>
    result = iter((lambda: cursor.fetchmany(GET_ITERATOR_CHUNK_SIZE)),
ProgrammingError: no results to fetch

#coding: utf8
__author__ = 'eri'


import urllib2 as urllib
from BeautifulSoup import BeautifulSoup
import re
from import BaseCommand, CommandError
from akkordi.models import *
from django.db import transaction
from multiprocessing import Process, Queue, current_process,Lock
import time

class Command(BaseCommand):
    help = "parses"

    def worker(self,input, output,lock):

        def artist(a):
            #some logic
            for line in lines:
                #logic and some .save()

        def make_query(p,lock):
            listhtml = urllib.urlopen(_tableurl%p)
            listsoup = BeautifulSoup(listhtml)
            table    = listsoup.find('table',width="600")
            artists  = table.findAll('a',href=_match)

            for a in artists:

        for page in iter(input.get, 'STOP'):
            result = make_query(page,lock)

    def handle(self, *args, **options):

        """Runs everything"""

        pages = range(1,26)

        # Create queues
        task_queue = Queue()
        done_queue = Queue()
        lock = Lock()

        for page in pages:
            #Start worker processes
        for i in range(NUMBER_OF_PROCESSES):
            Process(target=self.worker, args=(task_queue, done_queue,lock)).start()

            # Get and print results
        for i in range(len(pages)):
            self.stdout.write(u'Ready №%s' % done_queue.get())

        # Tell child processes to stop
        for i in range(NUMBER_OF_PROCESSES):
            print "Stopping Process #%s" % i

if i spawn one task - works fine, but slow

Change History (5)

comment:1 Changed 5 years ago by Aymeric Augustin

Resolution: invalid
Status: newclosed

Database connections don't survive forking. This isn't specific to Django.

comment:2 Changed 5 years ago by anonymous

fails on postgre
with mysql works

comment:3 in reply to:  1 Changed 5 years ago by anonymous

Replying to aaugustin:

Database connections don't survive forking. This isn't specific to Django.

Even if i set new django.db.connection.cursor for each fork?

comment:4 Changed 5 years ago by anonymous

You need to close the connection before forking.

comment:5 Changed 4 years ago by anonymous

Was there a valid work around for this? I can get my code working with 2 to 3 workers but with 3 I see the django.db.utils.DatabaseError: error with no message from the libpq error. How do you close the connection before forking?

I am doing something like this. I am assuming that the update is the culprit because when removed it works:

locations = Location.objects.all()

from multiprocessing import Pool
p = Pool(3), locations)

def process_location(location):

... some processing
... other processing

Note: See TracTickets for help on using tickets.
Back to Top