django-admin's --template option fails to download from https via proxy

[Daniel Hahler]

Using Squid as $https_proxy, the following will fail:

django-admin.py startproject -v3 --template=​https://github.com/twoscoops/django-twoscoops-project/archive/develop.zip --extension=py,rst,html project

Rendering project template files with extensions: .html, .rst, .py
Rendering project template files with filenames: 
Downloading https://github.com/twoscoops/django-twoscoops-project/archive/develop.zip
Extracting /tmp/django_project_template_IKLof6_download/develop.zip
Traceback (most recent call last):
  File "/home/daniel/.virtualenvs/TMM/bin/django-admin.py", line 8, in <module>
    execfile(__file__)
  File "/home/daniel/projects/TMM/src/django/django/bin/django-admin.py", line 5, in <module>
    management.execute_from_command_line()
  File "/home/daniel/projects/TMM/src/django/django/core/management/__init__.py", line 397, in execute_from_command_line
    utility.execute()
  File "/home/daniel/projects/TMM/src/django/django/core/management/__init__.py", line 390, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/daniel/projects/TMM/src/django/django/core/management/base.py", line 244, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/daniel/projects/TMM/src/django/django/core/management/base.py", line 291, in execute
    output = self.handle(*args, **options)
  File "/home/daniel/projects/TMM/src/django/django/core/management/commands/startproject.py", line 31, in handle
    super(Command, self).handle('project', project_name, target, **options)
  File "/home/daniel/projects/TMM/src/django/django/core/management/templates.py", line 122, in handle
    base_subdir)
  File "/home/daniel/projects/TMM/src/django/django/core/management/templates.py", line 205, in handle_template
    return self.extract(absolute_path)
  File "/home/daniel/projects/TMM/src/django/django/core/management/templates.py", line 299, in extract
    archive.extract(filename, tempdir)
  File "/home/daniel/projects/TMM/src/django/django/utils/archive.py", line 49, in extract
    with Archive(path) as archive:
  File "/home/daniel/projects/TMM/src/django/django/utils/archive.py", line 58, in __init__
    self._archive = self._archive_cls(file)(file)
  File "/home/daniel/projects/TMM/src/django/django/utils/archive.py", line 181, in __init__
    self._archive = zipfile.ZipFile(file)
  File "/usr/lib/python2.7/zipfile.py", line 766, in __init__
    self._RealGetContents()
  File "/usr/lib/python2.7/zipfile.py", line 807, in _RealGetContents
    raise BadZipfile, "File is not a zip file"
zipfile.BadZipfile: File is not a zip file

Looking at the tempfile, it contains HTML from a Squid3 error page.

From looking at Squid3's logfile, urlretrieve appears to do a GET request to the https URL, while wget will use CONNECT.

I am using Django's current master and Python 2.7.5+ (Ubuntu 13.10, in a virtualenv).

As a workaround I can unset $https_proxy.

[Tim Graham]

Do you have a suggested fix (or an idea of how complex the fix might be)? Depending on the answer, it may be more effort than it's worth to maintain this code in Django - I'm unfamiliar with Squid so I could be completely wrong. One could also download the zip using something like wget and then use a local path to the zip file, right?

[Tim Graham]

Closing as needsinfo in absence of a proposal for what should be done here. Feel free to reopen if you'd like to provide more details.

[Daniel Hahler]

I have not investigated much into fixing this - I stopped when I arrived at the Python2/3-compatibility layer (django.utils.six).

I have fixed some proxy issue in an urllib based Python program before (IIRC), but cannot provide a fix here.

I agree that downloading the file via wget and/or disabling the https proxy is a good enough workaround. But I also believe that there should be a more descriptive error, instead of just a traceback and "File is not a zip file".
"

[aladagemre]

Problem should be that an additional dot (.) at the end of the URL is added when proxy is active.
Here is the downloaded file:

<head>
<title>Error response</title>
</head>
<body>
<h1>Error response</h1>
<p>Error code 400.
<p>Message: bad url https://github.com/rdegges/django-skel/zipball/master.
<p>Error code explanation: 400 = Bad request syntax or unsupported method.
</body>