manage.py shell does not work with the latest IPython release (0.11.rc2)

[croach]

The latest release candidate for the IPython application has changed the way in which an IPython instance is embedded. According to the latest IPython documentation (​http://bit.ly/q1yj0O) an IPython instance can be embedded as follows:

from IPython import embed
embed()

[croach]

Fixes manage.py shell support for IPython 0.11rc2

[Aymeric Augustin]

Will this patch work with all versions of IPython? To put it differently, is it guaranteed that all versions of IPython provide either embed or IPShell?

Otherwise, we must try embed, fall back to TerminalInteractiveShell, and fall back again to IPShell.

See also r14895. Apparently, the IPython devs like changing their main public API every 6 months :(

[croach]

I've only replaced the portion that is responsible for IPython v0.11. The other fallbacks have been left untouched. So, this patch should only really fix the 0.11 portion of the code to work with the latest release of 0.11. Earlier stable releases should still function properly, but earlier 0.11 release candidates, alphas, etc. might not. This seems reasonable that Django would support stable versions and the latest release candidate. Also, since this is a release candidate and not an earlier alpha/beta, things should be slightly more settled (hopefully) as I believe the IPython devs are nearing a stable release of 0.11 very soon.

[minrk]

Does the previous code actually not work, or is this purely prompted by the IPython docs?

A few points:

1. You don't have to handle any fallbacks, there have been no releases of IPython without IPShell, and both TerminalInteractiveShell and embed() are introduced in 0.11, to be released this week. 0.11 is a major reorganization of IPython, so some imports were in flux early on, but these have not changed in a year.
2. embed() is for invoking a special InteractiveShell that uses the calling scope to initialize the user namespace (i.e. self will be defined if you use it as proposed here). Is that what you want? embed() is typically used in debugging, as it lets you have a full IPython shell at breakpoints.
3. manage.py shell appears to work just fine with IPython master today (to be 0.11 final very soon) and django release 1.3, but I'm not familiar with how IPython is supposed to be different than usual when invoked by django, so maybe that's not accurate.

The only change that should probably be made is that TerminalInteractiveShell should be imported from IPython.frontend.terminal.interactiveshell (where it's always been defined), not IPython.frontend.terminal.embed, which just happens to also import it.

[croach]

The reason that I added the fix that I did is because the manage.py shell command was not working with the latest IPython version. That said, I am running the master branch of the IPython 0.11 github repository and not one of the tagged versions. After much searching through the new documentation, I came across the code that you see in the patch as a way of starting up an IPython interactive session. It is possible that this is not the correct way to do this, but I've tried both what is in the current shell.py file and what you (minrk) mention at the end of your last comment and are failing at this time. For whatever reason, the import of TerminalInteractiveShell works just fine, but the instantiation of that class is not working. When I try to use the resultant shell variable, I get an object does not exist error. Calling the embed function, however, seems to work just fine.

minrk, is there anything else that you would suggest I try? I will checkout a tagged version of the repo and test against that and post my results here. If you can think of anything else, please post your ideas here and I'll give them a try.

[minrk]

I'm very sorry we are causing you trouble. There shouldn't be a significant difference between git master and any tagged rcs, other than small bugfixes, since we are trying to cut a release.

I am baffled by how the construction of the shell could not work for you, but does for me. How is this command supposed to be invoked? Both python manage.py shell and ./manage.py shell seem to work.

If embed works for you, go ahead and use it. The only reason not to is it will have access to self (unless you del self before calling embed()). I will continue to investigate why the other mechanism doesn't work, but since there are apparently some cases where it doesn't, using this patch as proposed is probably the safest approach.

[Jannis Leidel]

This needs a bit of more information than just saying "it doesn't work". Please provide a traceback and a step-by-step guide of how to reproduce this.

[Frankie Dintino]

I'm not sure whether this comment should go here, or if I should open a new ticket.

I'm doubtful that the ipython interactive shell was ever completely broken... however, the ipython shell as it is currently implemented doesn't import the user's iPython configuration, which changed in 0.11. In fact, as far as I can tell, that is the only difference between IPython.frontend.terminal.embed.InteractiveShellEmbed.mainloop() (what django currently calls) and IPython.frontend.terminal.embed.embed().

[Aymeric Augustin]

If I understand correctly, you're saying that we should switch from InteractiveShellEmbed.mainloop to embed so that the user's IPython configuration gets loaded? Or that we should keep InteractiveShellEmbed.mainloop?

From my (external) point of view, the best information on this ticket is provided by comment #3 (whose author seems to be an IPython developer), and it doesn't recommend switching to embed.

Any evidence in favor of one or the other will be appreciated — not everyone is very familiar with IPython. Thanks!

[Aymeric Augustin]

As far as I can tell, Django conforms to IPython's documention about embedding: ​http://ipython.org/ipython-doc/rel-0.11/interactive/reference.html#embedding => search "how to embed an IPython shell into a running program" in the page.

See also: ​http://ipython.org/ipython-doc/rel-0.11/whatsnew/version0.11.html

I just tested IPython's embedding with Django's latest revision and it works.

Since there's still no proof that it's broken (at least in some circumstances) I'm going to close this ticket again. If you want to suggest optimizations of IPython's embedding, please open separate tickets.

[matveym]

Using TerminalInteractiveShell.mainloop() indeed ignores IPython config settings, which in my case results in 'LightBG' color scheme used in Django shell vs 'Linux' color scheme in ipython. It is quite inconvenient to use 'LightBG' colors on black. embed() uses 'Linux' from config.

[matveym]

embed() is not a good solution - globals are not properly resolved: ​https://github.com/ipython/ipython/issues/62.
There's an easy way to load default config using IPython.frontend.terminal.ipapp.load_default_config - I will make a ticket about config not loaded and post the patch there.

[Aymeric Augustin]

Follow-up ticket is #17078.