Running colorama.init() at import time causes deployment error

[Leon Matthews]

Ticket 31216 added terminal colour output on Windows. Unfortunately, there is a small problem with it that causes deployments to fail under Linux/Apache/mod_wsgi, leaving me stuck on Django 3.1.

I've tracked it down to line 16 of ​django/code/management/color.py. There, the function colorama.init() is being called, and being called at import time (with caveats, see below). My deployment setup uses Apache mod_wsgi with the setting WSGIRestrictStdout on (its recommended default). During import of the logging system colorama.init() accesses sys.stdout which is ​prohibited by this setting causing a dreaded 500 error.

Running code during import is ​best avoided if possible. I understand that colorama.init() must be called at some point, but could it please be moved into a function instead? Would the function colour_style() in the same module be suitable? Doing so would make it possible for me use my logging configuration to avoid the function call.

Some notes:

• My (production) logging configuration doesn't use colour, but the import chain forces a run of colorama.init() anyway.
• A work-around would be simply not install colorama, as init() is only called it's import succeeds. Unfortunately, colorama is being installed into my project's virtualenv by a 3rd-party dependency, and I don't want to drop that package. Also, I use colorama in the custom management commands of a couple of my projects.
• I apologise for not testing my deployments during the 3.2 beta. I'm kicking myself, because I did test upgrades of my main projects for the beta and release candidates. I fixed all the deprecation warnings and made sure my test suite passed - but I didn't try a test deployment. I fell prey to the classic it works on my computer blunder!

[Leon Matthews]

Traceback from server

[Carlton Gibson]

Hello again. I'm supposing we should fix this, but I'm a little intrigued by the last line of that traceback:

  File "/srv/websites/digitaladvisor.nz/env/lib/python3.8/site-packages/colorama/ansitowin32.py", line 59, in closed
26	    return stream.closed

The docs you link have this for justification:

A well behaved Python WSGI application should never attempt to write any data directly to sys.stdout or use the print statement without directing it to an alternate file object.

But ​`closed` is just checking the status of the stream. It's not writing data, so shouldn't trigger an error. To that extent it kind of looks like a bug in mod_wsgi. 🤔

could it please be moved into a function instead?

An earlier iteration of the PR did have this call nested inside a function.

[Carlton Gibson]

I've opened ​an issue on `mod_wsgi` to see what Graham's thoughts are there.

Even if we were to nest the call to colorama.init(), the same path would be hit, just later, and there'd still be a compatibility issue with mod_wsgi.

(I don't think that requiring the logging config to avoid the call is particularly robust...)

[Carlton Gibson]

Mariusz suggested in conversation that we might catch the exception, something like:

try:
    import colorama
except ImportError:
    HAS_COLORAMA = False
else:
    try:
        colorama.init()
    except:
        # Would we want to at least log something here? 
        HAS_COLORAMA = False
    else:
        HAS_COLORAMA = True

[Claude Paroz]

What about avoiding calling colorama.init() on non-Windows platforms?

[Carlton Gibson]

​PR with the try-catch thought... — Not at all sure how best to test that. Any thoughts welcome.

What about avoiding calling colorama.init() on non-Windows platforms?

Yes... maybe… not sure.

There's no reason to have colorama installed at all if you're not on Windows. So I'd prefer to push back a bit before we do too much: why is it getting installed at all?; why is mod_wsgi raising for just the closed check?; what do colorama say on this, since presumably they'd like to be safe on other platforms? 🤔

We can work around this, but I'm not sure it's our issue. It probably counts as a Release Blocker if we're going to address it though.

[Carlton Gibson]

Hi Leon.

​Graham replied on `mod_wsgi`:

For now try setting:

WSGIRestrictedStdout Off

He also points out there where colorama can catch the OSError (​Matching colorama issue)

Can you give the PR on ​https://github.com/django/django/pull/14386 a go.
That should at least enable you to get past this with WSGIRestrictedStdout still on.

We're looking into whether we can move the colorama.init() off the import-time path. (It's meant to be a no-op on other platforms, but is hitting this OSError issue from mod_wsgi...)

[Leon Matthews]

Replying to Carlton Gibson:

Can you give the PR on ​https://github.com/django/django/pull/14386 a go.
That should at least enable you to get past this with WSGIRestrictedStdout still on.

Thank you Carlton, I'm back at work now, so I'll check that out and get back to you soon.

We're looking into whether we can move the colorama.init() off the import-time path.

I agree that that would be cleanest approach. It would be entirely reasonable to hit this error if my production configuration used the feature (as my development configuration does).

[Leon Matthews]

Replying to Carlton Gibson:

Can you give the PR on ​https://github.com/django/django/pull/14386 a go.
That should at least enable you to get past this with WSGIRestrictedStdout still on.

That seems to have done the trick!

BTW, it's a moot point now but I found out why colorama was getting installed in the first place. It wasn't a 3rd-party dependency like I thought it was. It turns out that Ubuntu 20.04 LTS installs a bunch of ​libraries into every Python virtualenv by default.

Thank you so much for your work on this Carlton, it's very much appreciated!

[Carlton Gibson <carlton.gibson@…>]

In c2e6047c:

Fixed #32740 -- Caught possible exception when initializing colorama.

[Carlton Gibson <carlton.gibson@…>]

In a173202d:

[3.2.x] Fixed #32740 -- Caught possible exception when initializing colorama.

Backport of c2e6047c725e26987c87e2be59f2ab4bf9828fa5 from main