Code

Opened 8 years ago

Closed 2 years ago

#1327 closed New feature (fixed)

Support nightly-build download

Reported by: xyb <xieyanbo@…> Owned by:
Component: *.djangoproject.com Version:
Severity: Normal Keywords: svn nightly snapshot sprintsept14
Cc: Triage Stage: Accepted
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

Django's evolution is keep walking, getting the lastest code must use subversion. Supporting nightly-build download could be more user-friendly.

Attachments (3)

nightly.diff (7.3 KB) - added by Jannis Leidel <jl@…> 7 years ago.
Initial approach of a nightly serving app for django_website using pysvn
nightly2.diff (7.3 KB) - added by Jannis Leidel <jl@…> 7 years ago.
Fixes a typo
nightly3.diff (7.4 KB) - added by Jannis Leidel <jl@…> 7 years ago.
Adds version labels to archive filenames to differ them when creating archives with the included cron utility

Download all attachments as: .zip

Change History (39)

comment:1 Changed 8 years ago by adrian

  • priority changed from normal to lowest
  • Severity changed from normal to trivial

comment:2 Changed 8 years ago by adrian

#1741 was a duplicate.

comment:3 Changed 8 years ago by anonymous

  • priority changed from lowest to high

comment:4 Changed 8 years ago by adrian

  • priority changed from high to low

comment:5 follow-up: Changed 8 years ago by jpaulofarias at gmail dot com

As an alternative to providing a nightly tarball, I've created a simple script to download code using http :o)

import os
import sys
import urllib
import sgmllib

class HtmlParser(sgmllib.SGMLParser):
    
    def reset(self):
        self.links = []
        sgmllib.SGMLParser.reset(self)
        
    def start_a(self, attrs):
        attrs = dict(attrs)
        if not attrs['href'].startswith('http://') and not attrs['href'] == '../':
            self.links.append(attrs['href'])
    

def download(url, basedir):
    fp = urllib.urlopen(url)
    data = fp.read()
    fp.close()
    parser = HtmlParser()
    parser.feed(data)
    parser.close()
    
    for link in parser.links:
        if link.endswith('/'):
            dirname = link[:-1]
            print 'entering', basedir + link
            try:
                os.mkdir(basedir + dirname)
            except:
                pass
            download(url + link, basedir + link)
        else:
            print 'downloading', url + link
            fp = urllib.urlopen(url + link)
            data = fp.read()
            fp.close()
            
            fp = open(basedir + link, 'w+')
            fp.write(data)
            fp.close()
    
def main():
    download('http://code.djangoproject.com/svn/django/trunk/',  './')
    
if __name__ == '__main__':
    main()

PS: Sorry adrian, I've changed it to high priority by mistake.

comment:6 Changed 7 years ago by anonymous

windows users need to change line 41 to:

fp = open(basedir + link, 'wb+')


to have the images download correctly.

comment:7 Changed 7 years ago by SmileyChris

  • Triage Stage changed from Unreviewed to Design decision needed

Is this going to happen? Someone with power, make a decision.

comment:8 Changed 7 years ago by mtredinnick

  • Triage Stage changed from Design decision needed to Accepted

This would be nice to have some day. It's not a high priority at the moment, but we have discussed it in the past and thought it would reasonable. The main reason not to dismiss it out of hand is that some corporate proxies are set up in a sufficiently b0rked fashion that they cannot be used for subversion access, so providing a tarball download is necessary.

To do this properly, we would actually need, say, one tarball for each of the last N days (N = 3 or 5, say), so that if one day has a broken tarball, there is still a backup.

Changed 7 years ago by Jannis Leidel <jl@…>

Initial approach of a nightly serving app for django_website using pysvn

comment:9 Changed 7 years ago by Jannis Leidel <jl@…>

  • Summary changed from Support nightly-build download to [PATCH] Support nightly-build download

Changed 7 years ago by Jannis Leidel <jl@…>

Fixes a typo

Changed 7 years ago by Jannis Leidel <jl@…>

Adds version labels to archive filenames to differ them when creating archives with the included cron utility

comment:10 Changed 7 years ago by Marc Fargas <telenieko@…>

  • Has patch set

comment:11 Changed 7 years ago by Brian Rosner <brosner@…>

I really would like to see a nightly tarball for download. Would rather wget http://nightlytarball than svn export and create my own ;)

Wouldn't this be better as a post commit hook and provide an archive of the last five commits?

comment:12 Changed 7 years ago by Jannis Leidel <jl@…>

  • Keywords svn nightly snapshot added

Good point, running a script after a commit should be straight forward with the current code in build_nightlies.py. Just run the build_nightlies.py script and the daily nightly (??) will be build.

But maby it makes more sense to come up with snapshots instead of nightlies anyway, tagging the archive files with the revision number.
Can we can please get a signal from one of the main devs?

comment:13 Changed 7 years ago by mtredinnick

It's not going to be a good idea to generate a new tarball after every commit. Commits are often very small changes, sometimes just one, two or five lines and there can be many of them close together. So sometimes we'll be generating tarballs more or less continuall and they won't be that different under that model.

Generating a tarball every 24 hours is a more reasonable interval and we can just keep the last five or so hanging around at any given moment (in case the most recent one is busted for some reason, which can easily happen).

comment:14 Changed 7 years ago by Jannis Leidel <jl@…>

Fair enough, the current patch creates the archives (tar.gz, tar.bz2 and zip) from the current trunk of every DocumentRelease object from the django_website project.
I hope you feel like really using this, I think it is very useful.

comment:15 Changed 7 years ago by Simon G. <dev@…>

  • Triage Stage changed from Accepted to Ready for checkin

I've marked this as ready to checkin, since the last patch here seems to take into account all the comments and is good to go. Setting it up on the website, however, might take some time. I do think it's a good idea to get this in action soon, as it's probably less use post-1.0


comment:16 Changed 7 years ago by anonymous

  • Keywords sprintsept14 added

comment:17 Changed 6 years ago by PJCrosier

  • Summary changed from [PATCH] Support nightly-build download to Support nightly-build download

comment:18 Changed 6 years ago by jezdez

  • Owner changed from nobody to jezdez

Ha! I had already forgotten this patch. Thanks PJCrosier :)

comment:19 Changed 6 years ago by jacob

  • Triage Stage changed from Ready for checkin to Someday/Maybe


comment:20 in reply to: ↑ 5 Changed 6 years ago by anonymous

Replying to jpaulofarias at gmail dot com:

Replace the last two function calls with this to specify a folder to download to at the command prompt.

def main(loc):
    download('http://code.djangoproject.com/svn/django/trunk/',  loc)
    
if __name__ == '__main__':
    try:
        loc = sys.argv[1]
    except IndexError:
	loc = "./"
    main(loc)

comment:21 Changed 6 years ago by jezdez

So, what's the status on that? Do we really need nightlies now that we have a roadmap?

comment:22 Changed 6 years ago by Karen Tracey <kmtracey@…>

Actually I think this idea might be more important as the project gets closer to 1.0 release. Having a daily tarball that is produced by the same tools as will produce the official release and made available for people to test would give wider opportunity for finding bugs in the tarball-creation process plus in the install-from-tarball scenario which few regular contributors ever exercise.

comment:23 Changed 6 years ago by jezdez

Interesting point! I'm not sure what's the process of producing a official release though, while I imagine it's just running python setup.py sdist. But I like the idea to have that process automated. IIRC the last patch would need some work then, even if it already shares some code with distutils.

comment:24 Changed 6 years ago by jezdez

  • Status changed from new to assigned

comment:25 Changed 5 years ago by gav

We're long past 1.0 (1.1 even!), and no one's asked for this in over a year. Is this still of interest? It'd be nice to see yet one more low bug number closed. :)

comment:26 Changed 4 years ago by jezdez

  • Owner jezdez deleted
  • Status changed from assigned to new

comment:27 Changed 4 years ago by adamnelson

Nobody has complained about potentially closing this for close to a year and nowadays just using svn (or a git/mercurial clone) is so much more sensible than a tarball. setup.py can be run much more cleanly from trunk with:

pip install -U -e svn+http://code.djangoproject.com/svn/django/trunk/#egg=django

Can somebody just close it? Can I just close it?

comment:28 Changed 4 years ago by SmileyChris

You can always bring it up on django-dev group if you're unsure (personally I think that spending any time on this would be redundant).

comment:29 Changed 4 years ago by russellm

IMHO, this is still a useful "someday, maybe" goal. Not because a nightly tarball is a particularly useful artefact -- I agree that a (D)VCS checkout is much more appropriate way of handling regular updates. However, the tooling and processes required to get a nightly tarball is a useful resource. In order to have a nightly tarball, we must have a completely automated build and packing process. That means that the packing bugs (such as the accidental packaging of .DS_Store in #11520) can be avoided by process, rather than by convention. It also means that we are *always* in a position to cut a release - we just take the tarball on the top of the pile and relabel it (or push a slightly different button on the nightly build process).

"Someday, maybe" is the best description for this. It would be nice if it were to happen, but it's not a huge priority, and we can live with what we have for the moment.

comment:30 Changed 4 years ago by ericholscher

Would it be useful to note in the documentation that http://github.com/django/django/archives/master is a place to get a tarball of the latest bits of django? This wouldn't necessarily solve the part about the automated build process, but it's at least a tarball we could offer people of the latest bits.

I can talk to james about this and see if we can set up a fabric script that would handle the release process and then get that automate as well.

comment:31 Changed 3 years ago by lrekucki

  • Severity changed from trivial to Normal
  • Type changed from enhancement to New feature

comment:32 Changed 2 years ago by aaugustin

  • UI/UX unset

Change UI/UX from NULL to False.

comment:33 Changed 2 years ago by aaugustin

  • Easy pickings unset

Change Easy pickings from NULL to False.

comment:34 Changed 2 years ago by aviraldg

  • Triage Stage changed from Someday/Maybe to Accepted

This should be trivial to fix, now that Django is hosted on Github. (They have an automatic "nightly" link.)

comment:35 Changed 2 years ago by aaugustin

IMO creating a zip of the repo and making a release are different tasks. This ticket is about the former.

As suggested by aviraldg, I'm going to just add a link to the zip provided by GitHub. We have a precedent for this — the PDF of the docs provided by RTD.

comment:36 Changed 2 years ago by aaugustin

  • Resolution set to fixed
  • Status changed from new to closed

I've updated the download page: https://www.djangoproject.com/download/

Add Comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
as The resolution will be set. Next status will be 'closed'
The resolution will be deleted. Next status will be 'new'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.