Code

Opened 6 years ago

Closed 19 months ago

Last modified 19 months ago

#6648 closed Cleanup/optimization (worksforme)

random seed identical among preforked FastCGI instances

Reported by: nezroy@… Owned by: nobody
Component: Core (Other) Version: master
Severity: Normal Keywords: random seed fastcgi fcgi prefork
Cc: flori@…, mmitar@… Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

While writing a basic captcha module, I was encountering a lot of duplicate strings of "random" text that I could not explain. Then I remembered seeing this comment on a separate session issue: http://code.djangoproject.com/ticket/1180#comment:20

Based on that info, I setup some extra logging, and it did appear that each of my FastCGI threads were producing identical random values. I was specifically using a call like this: random.choice('ABCDEFGHIJKLMNOPQRSTUVWXYZ'). It's possible other random methods are not affected, I was only interested in that one. It seems the random seed is set prior to the fork, and each forked process thereafter produces identical random output as a result.

In my setup, I run manage.py with method=prefork, runfcgi, and a static number of processes. This is on an Ubuntu Gutsy AMD 64-bit (on Xen) with Python 2.5.1. I'm currently running against SVN Django, revision 7049.

My workaround was to add a middleware class with an init function that sets the random seed. This gets called at "startup" in each of the new FCGI processes, giving them unique random output. Obviously not ideal, but it gets the job done in my case.

import logging, logging.config
import os, random, time

class ReseedRandom(object):
  def __init__(self):
    logging.debug("setting new random seed for this process")
    random.seed("%d%s" % (os.getpid(), time.ctime()))

It would be much cleaner if the manage.py FCGI launcher could seed its random number generator AFTER the fork occurs, rather than before (or not at all, as the case may be).

Attachments (2)

6648_fix_random_number_generator_fcgi_fork.diff (785 bytes) - added by peritus <peritus@…> 6 years ago.
Resets the random number generator's seed for each fcgi-child
6648_fix_random_number_generator_fcgi_fork-2.diff (712 bytes) - added by mitar 3 years ago.
For Django 1.3+

Download all attachments as: .zip

Change History (16)

comment:1 Changed 6 years ago by Simon Greenhill

  • Needs documentation unset
  • Needs tests unset
  • Patch needs improvement unset
  • Triage Stage changed from Unreviewed to Design decision needed

Changed 6 years ago by peritus <peritus@…>

Resets the random number generator's seed for each fcgi-child

comment:2 Changed 6 years ago by peritus <peritus@…>

  • Has patch set
  • Triage Stage changed from Design decision needed to Unreviewed

I can reproduce this - my users could :(

Given the following view and lighty+fastcgi+prefork:

def do_random(request):
    return HttpResponse(random.sample('ABCDEFGHIJKLMNOPQRSTUVWXYZ', 5))

Produces in subsequent requests:

XDQIU AJMNL JIQXN XDQIU IROXL AJMNL YAPJL CUYOK ODKLA JIQXN WACZR IROXL CYUJE SNQLX YAPJL CUYOK UMOWL ODKLA PIXQG WACZR CYUJE

which is *NOT* random.

Applying the patch, the same view produces

QBLDN THPXF VSJQB RIPTF EZQRL OSNZY HQSOB VJSED HQBTO UTOJI VSTFY NHPBY QMITZ WJGZB QVDRN QCIOT GWYVQ RGZHE TXHKI

which seems fairly random.

comment:3 Changed 6 years ago by anonymous

  • Cc flosch@… added

comment:4 Changed 6 years ago by anonymous

  • Cc flori@… added; flosch@… removed

comment:5 Changed 6 years ago by peritus <peritus@…>

  • Component changed from Uncategorized to Core framework

Reported this upstream:

http://trac.saddi.com/flup/ticket/35

I think, django should to add the workaround though.

comment:6 Changed 6 years ago by peritus <peritus@…>

This has been fixed in flup >= 1.0.2 (to be released):

http://trac.saddi.com/flup/changeset/137726cdcd67

comment:7 Changed 6 years ago by anonymous

  • Triage Stage changed from Unreviewed to Accepted

comment:8 Changed 5 years ago by buffi

The current patch is broken.
ctime() will only produce different output each second (an example output is 'Mon Apr 27 14:18:38 2009'), so the same process will get the same random seed if two requests are handed to it during the same second.

It would be better to use something like random.seed("%d%f" % (getpid(), time())) , but I'm unsure if this is the correct way to fix this issue, so I won't submit a patch.

comment:9 Changed 4 years ago by anonymous

Actually by using '%d%f' % (getpid(), time()) you're usually decreasing the randomness of the generator since if you don't pass anything then os.urandom() will be used on most supporting operating systems (windows and linux for example), which is more random than hash() which will be used in the case of passing a string to seed().

comment:10 Changed 3 years ago by julien

  • Type set to Cleanup/optimization

comment:11 Changed 3 years ago by patchhammer

  • Easy pickings unset
  • Patch needs improvement set
  • Severity set to Normal

6648_fix_random_number_generator_fcgi_fork.diff fails to apply cleanly on to trunk

Changed 3 years ago by mitar

For Django 1.3+

comment:12 Changed 3 years ago by mitar

  • Cc mmitar@… added
  • Patch needs improvement unset
  • UI/UX unset

I have attached a new patch version. It simply calls random.seed() as this itself tries to initialize from os.urandom if available or current time.

I came upon this problem with current Django version and python-flup package from Debian (1.0.2).

comment:13 Changed 19 months ago by lrekucki

  • Resolution set to worksforme
  • Status changed from new to closed

nginx + flup from pypi (both 1.0.2 and 1.0.3.dev-20110405) seem to work for me.

comment:14 Changed 19 months ago by mitar

Have you tried preforking?

Could there be a test case made which would somehow test this?

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.