Changes between Version 2 and Version 3 of OpenData

Timestamp:

Sep 7, 2011, 11:44:04 AM (14 years ago)

Author:

Jacob

Comment:

--

OpenData

@@ -10 +10 @@
 .. contents:: What's available:
 
-Please take it, mash it up, and let show us the results!
+Please take it, mash it up, and show us the results!
 
 If there's other data you'd like to see, please get in touch (``jacob -at- jacobian.org``) and let me know what you'd like to see. I'll do my best!
@@ -17 +17 @@
 ===============
 
-Data dumps out of Trac, our ticket tracking software.
+Data dumps out of Trac, our ticket tracking software. You could use this to get
+information about our ticket workflow, patches, etc.
 
 There's two ways to access the data: `Trac's RPC interface`_ and the `daily
@@ -26 +27 @@
 
 These are direct data dumps of the Trac database, collected nightly, in various
-formats. They're sanitized to remove some tables with senstive info (session
+formats. They're sanitized to remove some tables with sensitive info (session
 data, etc.) but are otherwise complete.
 
@@ -81 +82 @@
 .. _xmlrpclib: http://docs.python.org/library/xmlrpclib.html
 
+Repository data/dumps
+=====================
+
+Data and dumps from our source control repository. You could use this to mine
+information about who's committing code, when, etc.
+
+There are a few ways of accessing this data: `Querying the SVN repo`_, 
+`the GitHub API`_, and `SVN data dumps`_ in a variety of formats.
+
+Querying the SVN repo
+---------------------
+
+Django's SVN repository is at http://code.djangoproject.com/svn/django/;
+you can use the ``svn`` client binary to interact with this as a sort of "API".
+In particular, most ``svn`` commands take a ``--xml`` argument to return data
+in XML. For example, to get information about a particular commit you might
+do something like::
+
+    $ svn log http://code.djangoproject.com/svn/django/trunk -r1234 --xml
+    <?xml version="1.0"?>
+    <log>
+    <logentry
+       revision="1234">
+    <author>jacob</author>
+    <date>2005-11-14T18:50:13.298556Z</date>
+    <msg>Added NOINDEX tag to debug 500 page (for robots)</msg>
+    </logentry>
+    </log>
+
+There are also a number of libraries in Python (and other languages) that can
+access SVN directly. `pysvn`_ seems to be a popular choice.
+
+.. _pysvn: http://pysvn.tigris.org/
+
+The GitHub API
+--------------
+
+Django's repository is mirrored onto GitHub (http://github.com/django/django),
+which means you can use `GitHub's API`_ to to pull commit data. For example::
+
+    $ curl -i https://api.github.com/repos/django/django/git/commits/a0d59b49019d65b38c5612eb0b4fab0bb37271ae
+    HTTP/1.1 200 OK
+    Server: nginx/1.0.4
+    Date: Wed, 07 Sep 2011 16:38:12 GMT
+    Content-Type: application/json
+    Connection: keep-alive
+    Status: 200 OK
+    X-RateLimit-Limit: 5000
+    X-RateLimit-Remaining: 4994
+    Content-Length: 995
+    
+    {
+      "parents": [
+        {
+          "url": "https://api.github.com/repos/django/django/git/commits/6465e005fd564bd75ba64f2f09d5824ed2455c9c",
+          "sha": "6465e005fd564bd75ba64f2f09d5824ed2455c9c"
+        }
+      ],
+      "committer": {
+        "date": "2005-11-14T10:50:13-08:00",
+        "name": "jacob",
+        "email": "jacob@bcc190cf-cafb-0310-a4f2-bffc1f526a37"
+      },
+      "author": {
+        "date": "2005-11-14T10:50:13-08:00",
+        "name": "jacob",
+        "email": "jacob@bcc190cf-cafb-0310-a4f2-bffc1f526a37"
+      },
+      "message": "Added NOINDEX tag to debug 500 page (for robots)\n\ngit-svn-id: http://code.djangoproject.com/svn/django/trunk@1234 bcc190cf-cafb-0310-a4f2-bffc1f526a37\n",
+      "url": "https://api.github.com/repos/django/django/git/commits/a0d59b49019d65b38c5612eb0b4fab0bb37271ae",
+      "sha": "a0d59b49019d65b38c5612eb0b4fab0bb37271ae",
+      "tree": {
+        "url": "https://api.github.com/repos/django/django/git/trees/a5d296a396f5bbf70d074ce09fa947f95cd91523",
+        "sha": "a5d296a396f5bbf70d074ce09fa947f95cd91523"
+      }
+    }
+
+.. _github's api: http://developer.github.com/v3/
+
+SVN data dumps
+--------------
+
+Finally, for convenience, we provide a couple of full dumps of repository data
+for off-line processing:
+
+    * `Complete SVN log`_ (bzipped XML; ~1 MB). This is the complete output of
+      ``svn log --xml``.
+
+    * `Full SVN dump`_ (bziiped SVN dump; ~200 MB, expands to ~ 1.8 GB). This
+      is the result of a ``svnadmin dump``.
+
+Each dump is updated nightly.
+
+.. _complete svn log: https://www.djangoproject.com/m/data/django-svn-log.xml.bz2
+.. _full svn dump: https://www.djangoproject.com/m/data/django-svn.svndump.bz2
+
 Mashups
 =======