Import of PyATS breaks runserver reloader

[Andy Norwood]

When importing from the pyats library the development server fails to start with the following error. A more complete description of the issue in listed here ​https://github.com/CiscoTestAutomation/pyats/issues/120#issuecomment-892857620

Django version 4.2.8, using settings 'config.settings.local'
Development server is running at http://0.0.0.0:8000/
Using the Werkzeug debugger (https://werkzeug.palletsprojects.com/)
Quit the server with CONTROL-C.
Traceback (most recent call last):
  File "/app/manage.py", line 31, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django_extensions/management/utils.py", line 62, in inner
    ret = func(self, *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django_extensions/management/commands/runserver_plus.py", line 338, in handle
    self.inner_run(options)
  File "/usr/local/lib/python3.11/site-packages/django_extensions/management/commands/runserver_plus.py", line 438, in inner_run
    self.extra_files |= set(filter(lambda filename: str(filename).endswith('.mo'), gen_filenames()))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/autoreload.py", line 304, in watched_files
    yield from iter_all_python_module_files()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/autoreload.py", line 120, in iter_all_python_module_files
    return iter_modules_and_files(modules, frozenset(_error_files))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: unhashable type: 'Configuration'

I have used the python manage.py shell to test the import and do not get an error after import

Python 3.11.7 (main, Dec 19 2023, 03:30:20) [GCC 12.2.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.19.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from pyats.topology import Testbed, Device

In [2]: 

As an aside, I found a similar issue reported but with the Netmiko library instead ​https://github.com/ktbyers/netmiko/issues/1955

I don't think this has been reported previously. I have limited Python skills but happy to assist where I can.

[Natalia Bidart]

Hello Andy, thanks for your report, though I'm having a hard time reproducing and thus understanding where Django is at fault.

I have read the linked GH report and while I understand the change (adding and isinstance(m, Hashable) to iter_modules_and_files), I'm not being able to reproduce with a minimal Django app (and less so with a test case). I have tried installing the pyats dependency and adding the imports to views.py modules without being able to trigger any error.

I'll be closing as worksforme, but please reopen if you can provide either a test case for the Django source (ideal), or a minimal pure Django project to reproduce otherwise.

Thanks, Natalia.

[Tim Graham]

I had no trouble reproducing with pip install pyats and adding from pyats.topology import Testbed to the tutorial's polls/views.py.

The question for me is why does pyats's Configuration({}) class appear in sys.modules. Is this a proper behavior that could reasonably be expected in other packages. Since pyats is closed-source, we may need some details from pyats developers on this.

[Natalia Bidart]

Thank you Tim, I double checked and while I did install pyats and added an adequate import to my views.py, due to a previous ticket triaging I had temporarily removed the corresponding urls.py from my Django project config :facepalm:.

Andy, I agree with Tim's comment in that we would need to understand why pyats is providing modules that are not hashable. I searched a bit and I couldn't find concrete Python docs regarding modules and their hashability, though I did find ​this post. I'm inclined to suggest that pyats may need to ensure their modules are hashable, or explain why they are not/can not be so we can evaluate how niche vs common that reason is.

[Andy Norwood]

Thank you both for looking at this.

I have contacted pyats support and pointed them to this ticket in the hope that they can share some insight that might help find a way forward.

[Andy Norwood]

An update from Cisco PyATS team:

---

Hi Andy,

Thanks for reaching out. I believe this was done to create a singleton object for the pyATS configuration data that is used by the pyATS code in several places.

Using the sys.modules structure for a non-module is a hack done a long time ago, we will review alternative options to handle this in the pyATS code.

Regards,

Dave

---

Sounds like the ball is in their court to resolve the 'hack'.

[Natalia Bidart]

Thank you a lot Andy for the follow up! I'll change the status to wontfix given the latest information, but feel free to comment if (for any reason) that needs to be changed.

[Mariusz Felisiak]

Marking as "invalid" because it's not an issue in Django so there is nothing to fix on our side.