Using IPython for the manage.py REPL interface clobbers __main__, where the normal python REPL doesn't.

[Keryn Knight]

The only thing I can find of related note is #30588 which observes a similar thing but around auto-reloading*

Given the following file, which is ... I believe it has been called esoteric but works:

import django
from django.conf import settings
from django.http import HttpResponse
from django.urls import path

if not settings.configured:
    settings.configure(
        SECRET_KEY="?",
        DEBUG=True,
        INSTALLED_APPS=(),
        ROOT_URLCONF=__name__,
    )
    django.setup()


def test(request):
    return HttpResponse('test')


urlpatterns = [
    path("test/", test, name="test"),
    path("best/", test, name="test"),
]


if __name__ == "__main__":
    from django.core import management

    management.execute_from_command_line()
else:
    from django.core.wsgi import get_wsgi_application

    application = get_wsgi_application()

it is possible to reverse those url patterns, like so:

$ python whypython.py shell -ipython
Python 3.9.5 ...
>>> __name__
'builtins'
>>> import sys
>>> sys.modules['__main__']
<module '__main__' from '/path/to/whypython.py'>
>>> dir(sys.modules['__main__'])
['HttpResponse', ..., 'urlpatterns']
>>> sys.modules['builtins']
<module 'builtins' (built-in)>
>>> from django.urls import reverse
>>> reverse('test')
'/best/'
>>>

But doing the same via the IPython support:

$ python whypython.py shell -iipython
Python 3.9.5 ...
IPython 7.24.1 ...
In [1]: __name__
Out[1]: '__main__'

In [2]: import sys

In [3]: sys.modules['__main__']
Out[3]: <module '__main__'>

In [4]: dir(sys.modules['__main__'])
Out[4]:
['In',
 'Out',
...
 'quit',
 'sys']

In [5]: sys.modules['builtins']
Out[5]: <module 'builtins' (built-in)>

In [6]: from django.urls import reverse

In [7]: reverse('test')
...
ImproperlyConfigured: The included URLconf '__main__' does not appear to have any patterns in it [...]

This is because django.core.management.commands.shell.Command.ipython invokes start_python without supplying a user_ns, which means that down in IPython.core.interactiveshell.InteractiveShell.prepare_user_module this runs:

if user_module is None:
    user_module = types.ModuleType("__main__",
        doc="Automatically created module for IPython interactive environment")

and then IPython.core.interactiveshell.InteractiveShell.init_sys_modules patches that oddness in. I don't know why it does so, particularly.

The "fix" is to change the Django invocation to:

start_ipython(argv=[], user_ns={'__name__': '__ipython_main__'})

which leaves __main__ in peace, and inserts a DummyMod with that name into the sys.modules table. As far as I can tell, both are treated by IPython the same ... just a big __dict__ to patch things into.

I suppose an argument could be made for checking if __main__ is in sys.modules and only set the user_ns if it is.

You'd think that supplying user_module=... would also be an effective fix, but alas, IPython.terminal.ipapp.TerminalIPythonApp.init_shell doesn't pass that along, only user_ns. I don't know if it's possible to pass it along somehow because of the amount of magic in IPython (traits, observes, etc)

Thus, using user_ns, it all works:

In [1]: __name__
Out[1]: '__ipython_main__'

In [2]: import sys

In [3]: sys.modules['__main__']
Out[3]: <module '__main__' from '/path/to/whypython.py'>

In [4]: sys.modules[__name__]
Out[4]: <IPython.core.interactiveshell.DummyMod at 0x10659da90>

In [5]: dir(sys.modules['__main__'])
Out[5]:
['HttpResponse',
...
 'urlpatterns']

In [6]: dir(sys.modules[__name__])
['In',
 'Out',
...
 'quit',
 'sys']

In [7]: from django.urls import reverse

In [8]: reverse('test')
Out[8]: '/best/'

I've got little concrete idea about the knock-on effects on changing it -- I don't feel like there should be any, because as I mentioned, I think they're both just used as glorified dicts. But it does change the global __name__ and I don't know how to avoid that off the top of my head, I only know that I can't think of many reasons to be checking __name__ in the REPL, beyond pulling it from sys.modules for reasons (and there I think replacing/shadowing the real one is a very weird move for IPython to have made).

For historical reference, here's the pdb where output to get to the bit which triggers a change in behaviour:

  /path/to/whypython.py(29)<module>()
-> management.execute_from_command_line()
  /path/to/django/core/management/__init__.py(419)execute_from_command_line()
-> utility.execute()
  /path/to/django/core/management/__init__.py(413)execute()
-> self.fetch_command(subcommand).run_from_argv(self.argv)
  /path/to/django/core/management/base.py(363)run_from_argv()
-> self.execute(*args, **cmd_options)
  /path/to/django/core/management/base.py(407)execute()
-> output = self.handle(*args, **options)
  /path/to/django/core/management/commands/shell.py(112)handle()
-> return getattr(self, shell)(options)
  /path/to/django/core/management/commands/shell.py(36)ipython()
-> start_ipython(argv=[], user_ns={'__name__': '__ipython_main__'})
  /path/to/site-packages/IPython/__init__.py(126)start_ipython()
-> return launch_new_instance(argv=argv, **kwargs)
  /path/to/site-packages/traitlets/config/application.py(844)launch_instance()
-> app.initialize(argv)
  /path/to/site-packages/traitlets/config/application.py(87)inner()
-> return method(app, *args, **kwargs)
  /path/to/site-packages/IPython/terminal/ipapp.py(317)initialize()
-> self.init_shell()
  /path/to/site-packages/IPython/terminal/ipapp.py(331)init_shell()
-> self.shell = self.interactive_shell_class.instance(parent=self,
  /path/to/site-packages/traitlets/config/configurable.py(537)instance()
-> inst = cls(*args, **kwargs)
  /path/to/site-packages/IPython/terminal/interactiveshell.py(525)__init__()
-> super(TerminalInteractiveShell, self).__init__(*args, **kwargs)
  /path/to/site-packages/IPython/core/interactiveshell.py(647)__init__()
-> self.init_create_namespaces(user_module, user_ns)
  /path/to/site-packages/IPython/core/interactiveshell.py(1239)init_create_namespaces()
-> self.user_module, self.user_ns = self.prepare_user_module(user_module, user_ns)
> /path/to/site-packages/IPython/core/interactiveshell.py(1307)prepare_user_module()

Assigning it to myself on the off-chance it is accepted.

---

(* and I think in the process links to the wrong start_ipython, to no detriment. The actual one used is IPython.start_ipython rather than IPython.testing.globalipapp.start_ipython, whose implementation is different)

[Mariusz Felisiak]

Thanks for the ticket. Closing as "wontfix" based on discussion in #19737. Feel free to write to the DevelopersMailingList if you disagree with the conclusions of that ticket.