Queries within AppConfig.ready() can cause issues with some Django db commands

[Rich Rauenzahn]

I can recreate this in my AppConfig simply with the following inside of ready():

list(MyModel.objects.all())

If you then run manage dbshell you can see the connection still existing after invoking psql:

$ manage dbshell    
Settings: myproj.config.rich.DevelopmentSettings
Expanded display is used automatically.
Null display is "¤".
psql (9.2.24)
Type "help" for help.

myproj=> SELECT *
FROM pg_stat_activity;
-[ RECORD 1 ]----+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
datid            | 4855283
datname          | myproj
pid              | 2590
usesysid         | 16384
usename          | myproj
application_name | psql
client_addr      | 10.20.72.150
client_hostname  | ¤
client_port      | 53364
backend_start    | 2019-04-29 09:53:14.044483-07
xact_start       | 2019-04-29 09:53:15.280585-07
query_start      | 2019-04-29 09:53:15.280585-07
state_change     | 2019-04-29 09:53:15.28059-07
waiting          | f
state            | active
query            | SELECT *
                 | FROM pg_stat_activity;
-[ RECORD 2 ]----+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
datid            | 4855283
datname          | myproj
pid              | 2589
usesysid         | 16384
usename          | myproj
application_name | 
client_addr      | 10.20.72.150
client_hostname  | ¤
client_port      | 53362
backend_start    | 2019-04-29 09:53:14.013457-07
xact_start       | ¤
query_start      | 2019-04-29 09:53:14.018469-07
state_change     | 2019-04-29 09:53:14.020801-07
waiting          | f
state            | idle
query            | SELECT ...
myproj=> 

This can cause problems with trying to drop the database from the manage command as the database in still in use by another connection.

Work around seems to be adding this to the end of ready():

from django.db import connections                                      
connections.close_all()      

My guess is the underlying fork to psql isn't closing all file handles and psql is inheriting (but not managing) them.

[Simon Charette]

Performing database operations at module loading time should be avoided and is warned against in the ready() documentation.

​https://docs.djangoproject.com/en/2.2/ref/applications/#django.apps.AppConfig.ready

Performing such operations at ready() will have various side effects that are hard to enumerate as it's unexpected and I'm not convinced documenting this one is worthwhile.

[Rich Rauenzahn]

I didn't consider this a documentation defect -- I think the team may want to consider the merits of this bug instead whether they really want to leak file handles across the fork()/exec() system call in the management commands.

Python 34 changes the default to close file handles at fork, so this is only an issue for 2.7 support in 1.11. Maybe that's enough reason to close it.

​https://www.python.org/dev/peps/pep-0446/

Leaking file descriptors in child processes causes various annoying issues and is a known major security vulnerability. Using the subprocess module with the close_fds parameter set to True is not possible in all cases.

(This file handle leakage is a much more subtle issue than what the warning in the docs implies.)

Aside: Is there any supported way for running db commands at startup?

[Rich Rauenzahn]

Relevant SO for other (better supported?) methods of app startup code: ​https://stackoverflow.com/questions/6791911/execute-code-when-django-starts-once-only

[Carlton Gibson]

Sorry, this is a usage question. TicketClosingReasons/UseSupportChannels

Also TicketClosingReasons/DontReopenTickets

Thanks.