moria.org.uk

Sat, 26 Mar 2005

Downloading source tarballs with zsync

Another interesting application for zsync: downloading updated source tarballs. For regular software updaters, this has to be a big win. I wrote a small wrapper script for the fetch operation in the FreeBSD ports system, and was able to cut the total download for updating XFCE by roughly a third.

% make FETCH_CMD="/home/cph/src/zsync-fetch -ARr"
===>  Vulnerability check disabled, database not found
===>  Found saved configuration for xfce4-wm-4.2.1
=> xfwm4-4.2.1.tar.gz doesn't seem to exist in /usr/ports/distfiles/xfce4.
=> Attempting to fetch from http://www.us.xfce.org/archive/xfce-4.2.1/src/.
Downloading xfwm4-4.2.1.tar.gz; old versions ./xfce4/xfwm4-4.0.6.tar.gz found.
#################### 100.0% 21.2 kBps DONE    
Read /usr/ports/distfiles/./xfce4/xfwm4-4.0.6.tar.gz. Target 27.4% complete.
downloading from http://www.us.xfce.org/archive/xfce-4.2.1/src/xfwm4-4.2.1.tar.g
z:
#################### 100.0% 47.3 kBps DONE    
verifying download...checksum matches OK
used 1681408 local, fetched 857709
===>  Extracting for xfce4-wm-4.2.1
=> Checksum OK for xfce4/xfwm4-4.2.1.tar.gz.
…
=> libxfcegui4-4.2.1.tar.gz doesn't seem to exist in /usr/ports/distfiles/xfce4.
=> Attempting to fetch from http://www.us.xfce.org/archive/xfce-4.2.1/src/.
Downloading libxfcegui4-4.2.1.tar.gz; old versions ./xfce4/libxfcegui4-4.0.6.tar
.gz found.
#################### 100.0% 15.2 kBps DONE
Read /usr/ports/distfiles/./xfce4/libxfcegui4-4.0.6.tar.gz. Target 33.8% complete. 
downloading from http://www.us.xfce.org/archive/xfce-4.2.1/src/libxfcegui4-4.2.1.tar.gz:
#################### 100.0% 48.8 kBps DONE
verifying download...checksum matches OK
used 1582080 local, fetched 548199
===>  Extracting for libxfce4gui-4.2.1
=> Checksum OK for xfce4/libxfcegui4-4.2.1.tar.gz.
…

In total, a 6.1MB download was reduced to 4.0MB. You can see the benefits of the recompression support added in 0.3.2 now — it reconstructs the original .gz file, allowing it to pass the checksum applied by the ports framework. This was on a fairly large update, from 4.0.x to 4.2.x, so a smaller update would probably see a larger benefit; this has to be a big application of zsync, I think.

For the moment I am running a CGI on moria.org.uk which will automatically generate .zsync files for FreeBSD package sources. And I am providing the zsync-fetch wrapper script which you see in use above. Feel free to try it out (note that you should upgrade to zsync-0.3.3 at least before using it, though, as 0.3.2 had some glitches in the recompression support) — it obviously only kicks in if you have an older version of a file to update from, and it only works for ports with gzip-compressed source tarballs (it just falls back to fetch for .tbz, .tar.bz, and files that moria.org.uk rejects or is unable to download). This is highly experimental, but I hope gives a good idea of the potential that zsync has.

[20:28] | [/computers/zsync] | #

Colin Phipps.
Archives
January 2007
November 2006
October 2006
September 2006
August 2006
July 2006
June 2006
May 2006
April 2006
March 2006
February 2006
January 2006
December 2005
November 2005
October 2005
July 2005
June 2005
May 2005
April 2005
March 2005
February 2005
January 2005
December 2004
November 2004
October 2004
Web Sites
zsync
PrBoom
About Kye
Credits
Blosxom
Powered by
Blogs that link here
[Valid Atom]