Opened 7 months ago

Closed 7 months ago

Last modified 7 months ago

#32740 closed Bug (fixed)

Running colorama.init() at import time causes deployment error

Reported by: Leon Matthews Owned by: Carlton Gibson
Component: Core (Other) Version: 3.2
Severity: Release blocker Keywords: apache mod_wsgi
Cc: MinchinWeb Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Ticket 31216 added terminal colour output on Windows. Unfortunately, there is a small problem with it that causes deployments to fail under Linux/Apache/mod_wsgi, leaving me stuck on Django 3.1.

I've tracked it down to line 16 of django/code/management/color.py. There, the function colorama.init() is being called, and being called at import time (with caveats, see below). My deployment setup uses Apache mod_wsgi with the setting WSGIRestrictStdout on (its recommended default). During import of the logging system colorama.init() accesses sys.stdout which is prohibited by this setting causing a dreaded 500 error.

Running code during import is best avoided if possible. I understand that colorama.init() must be called at some point, but could it please be moved into a function instead? Would the function colour_style() in the same module be suitable? Doing so would make it possible for me use my logging configuration to avoid the function call.

Some notes:

  • My (production) logging configuration doesn't use colour, but the import chain forces a run of colorama.init() anyway.
  • A work-around would be simply not install colorama, as init() is only called it's import succeeds. Unfortunately, colorama is being installed into my project's virtualenv by a 3rd-party dependency, and I don't want to drop that package. Also, I use colorama in the custom management commands of a couple of my projects.
  • I apologise for not testing my deployments during the 3.2 beta. I'm kicking myself, because I did test upgrades of my main projects for the beta and release candidates. I fixed all the deprecation warnings and made sure my test suite passed - but I didn't try a test deployment. I fell prey to the classic it works on my computer blunder!

Attachments (1)

django-ticket-32740.txt (2.2 KB) - added by Leon Matthews 7 months ago.
Traceback from server

Download all attachments as: .zip

Change History (13)

Changed 7 months ago by Leon Matthews

Attachment: django-ticket-32740.txt added

Traceback from server

comment:1 Changed 7 months ago by Carlton Gibson

Cc: MinchinWeb added

Hello again. I'm supposing we should fix this, but I'm a little intrigued by the last line of that traceback:

  File "/srv/websites/digitaladvisor.nz/env/lib/python3.8/site-packages/colorama/ansitowin32.py", line 59, in closed
26	    return stream.closed

The docs you link have this for justification:

A well behaved Python WSGI application should never attempt to write any data directly to sys.stdout or use the print statement without directing it to an alternate file object.

But `closed` is just checking the status of the stream. It's not writing data, so shouldn't trigger an error. To that extent it kind of looks like a bug in mod_wsgi. 🤔

could it please be moved into a function instead?

An earlier iteration of the PR did have this call nested inside a function.

comment:2 Changed 7 months ago by Carlton Gibson

I've opened an issue on `mod_wsgi` to see what Graham's thoughts are there.

Even if we were to nest the call to colorama.init(), the same path would be hit, just later, and there'd still be a compatibility issue with mod_wsgi.

(I don't think that requiring the logging config to avoid the call is particularly robust...)

comment:3 Changed 7 months ago by Carlton Gibson

Mariusz suggested in conversation that we might catch the exception, something like:

try:
    import colorama
except ImportError:
    HAS_COLORAMA = False
else:
    try:
        colorama.init()
    except:
        # Would we want to at least log something here? 
        HAS_COLORAMA = False
    else:
        HAS_COLORAMA = True

comment:4 Changed 7 months ago by Claude Paroz

Triage Stage: UnreviewedAccepted
Type: UncategorizedBug

What about avoiding calling colorama.init() on non-Windows platforms?

comment:5 Changed 7 months ago by Carlton Gibson

Severity: NormalRelease blocker

PR with the try-catch thought... — Not at all sure how best to test that. Any thoughts welcome.

What about avoiding calling colorama.init() on non-Windows platforms?

Yes... maybe… not sure.

There's no reason to have colorama installed at all if you're not on Windows. So I'd prefer to push back a bit before we do too much: why is it getting installed at all?; why is mod_wsgi raising for just the closed check?; what do colorama say on this, since presumably they'd like to be safe on other platforms? 🤔

We can work around this, but I'm not sure it's our issue. It probably counts as a Release Blocker if we're going to address it though.

comment:6 Changed 7 months ago by Carlton Gibson

Hi Leon.

Graham replied on `mod_wsgi`:

For now try setting:

WSGIRestrictedStdout Off

He also points out there where colorama can catch the OSError (Matching colorama issue)

Can you give the PR on https://github.com/django/django/pull/14386 a go.
That should at least enable you to get past this with WSGIRestrictedStdout still on.

We're looking into whether we can move the colorama.init() off the import-time path. (It's meant to be a no-op on other platforms, but is hitting this OSError issue from mod_wsgi...)

comment:7 in reply to:  6 Changed 7 months ago by Leon Matthews

Replying to Carlton Gibson:

Can you give the PR on https://github.com/django/django/pull/14386 a go.
That should at least enable you to get past this with WSGIRestrictedStdout still on.

Thank you Carlton, I'm back at work now, so I'll check that out and get back to you soon.

We're looking into whether we can move the colorama.init() off the import-time path.

I agree that that would be cleanest approach. It would be entirely reasonable to hit this error if my production configuration used the feature (as my development configuration does).

comment:8 in reply to:  6 Changed 7 months ago by Leon Matthews

Replying to Carlton Gibson:

Can you give the PR on https://github.com/django/django/pull/14386 a go.
That should at least enable you to get past this with WSGIRestrictedStdout still on.

That seems to have done the trick!

BTW, it's a moot point now but I found out why colorama was getting installed in the first place. It wasn't a 3rd-party dependency like I thought it was. It turns out that Ubuntu 20.04 LTS installs a bunch of libraries into every Python virtualenv by default.

Thank you so much for your work on this Carlton, it's very much appreciated!

comment:9 Changed 7 months ago by Mariusz Felisiak

Owner: changed from nobody to Carlton Gibson
Status: newassigned
Triage Stage: AcceptedReady for checkin

comment:10 Changed 7 months ago by Mariusz Felisiak

Has patch: set

comment:11 Changed 7 months ago by Carlton Gibson <carlton.gibson@…>

Resolution: fixed
Status: assignedclosed

In c2e6047c:

Fixed #32740 -- Caught possible exception when initializing colorama.

comment:12 Changed 7 months ago by Carlton Gibson <carlton.gibson@…>

In a173202d:

[3.2.x] Fixed #32740 -- Caught possible exception when initializing colorama.

Backport of c2e6047c725e26987c87e2be59f2ab4bf9828fa5 from main

Note: See TracTickets for help on using tickets.
Back to Top