Mon blog-notes à moi que j'ai

Blog personnel d'un sysadmin, tendance hacker

Automatically watch for new upstream software releases

Nowadays, many software editors do not properly advertize new software releases. They do not provide simple way to watch new upstream version like RSS/Atom feeds. They usually advertize new versions only on social networks like Twitter or sometimes by email.
Problem is that you, as a sysadmin, have to manually go to the (in)famous “Download” page and click.

From time to time, you apply the same and boring procedure to get new version of your software. But you’re lazy, right ? At least I’m so I looked around to find how to automate this :)

Do not reinvent the wheel

In fact, each package maintainer for every Linux distribution has exactly the same problem. Hopefully, most distributions have developped utilities to assume these repetitive tasks. On Debian, you have uscan from devscripts packages.

The good thing is that you can use uscan outside of the traditional debian packaging workflow.

How it works

uscan utility is provided by devscripts package

apt-get install devscripts

With the traditional Debian packaging workflow, uscan needs 2 files:

  • debian/changelog which is used to determine the source package name and the current version
  • debian/watch which gives the rules uscan will have to follow to detect new upstream version

Hopefully, you can provide both package (with option --package) and current version (with option --upstream-version), as parameters to uscan.
This way, you only have to focus on rules in the watch file.

Our command “prototype” looks like

uscan --package my_fake_package_name --upstream-version 0

You can also provide a custom watch file name

uscan --package my_fake_package_name --upstream-version 0 --watchfile my_watch_file

By default, uscan will download new upstream version in parent directory ../ and create a symlink <package_name>_<version>.orig.tar.gz to the real archive.
uscan will also check whether the choosen version has already been downloaded.

You can also provide many other options. You can refer to the man page of uscan for details

The watch file

Once uscan knows about package name and current version, it’ll read the watch file and apply specified rules.

Once more, uscan man page gives a lot of information.
I’ll only give some quite basic examples:

Tableau Python SDK

Tableau allows you to build reports in a simple way. Tableau software, the company behind Tableau, provides a SDK for Python. Unfortunately, new versions are not advertized and the download link is hidden deep into online help pages.

uscan rule is really simple

Go to a specific page, look for a link pattern, extract version number, pick to biggest one and download it

watch file looks like:

version=4 https://onlinehelp.tableau.com/current/api/sdk/en-us/SDK/tableau_sdk_installing.htm \ .*/tssoftware/Tableau-SDK-Python-Linux-64Bit-([\d-]+).tar.gz

The first line provides the uscan rules’ version.
The second provides the start URL uscan will have to download, followed by the link pattern to look for into that page.
Upstream version will be catched by the regex ([\d-]+).

Dataiku studio

Dataiku Science Studio is an analytics platform. A community edition is available for free but this page is usually not up-to-date. More, this page requires Javascript.
You better have to look at http://downloads.dataiku.com/public/studio, click on the relevant version number and finally download the new version.

Though still basic, rule is a bit more subtle:

version=4
https://downloads.dataiku.com/public/studio/([\d\.]+)/ \
        dataiku-dss-([\d\.]+).tar.gz

You will notice that the version number is also included in the start URL. More, the start URL ends with /.
uscan will open the parent URL, here https://downloads.dataiku.com/public/studio/, look for a directory link matching first pattern ([\d\.]+), open the relevant page, look for the “real” link pattern dataiku-dss-([\d\.]+).tar.gz in that page and finally download it.

Let’s try

Create 2 directories downloads and watch. In watch create a dataiku-dss file with the example above.

Rules are as follow:

  • Fake Debian package name is provided by watch file name. In the example dataiku-dss
  • Current version will be determined looking into downloads directory. If none found, 0 will be used

Run the command

uscan --download \
      --package dataiku-dss \
      --upstream-version 0 \
      --destdir downloads \
      --watchfile watch/dataiku-dss

Output should be something like

uscan: Newest version of dataiku-dss on remote site is 3.0.3, local version is 0
uscan:    => Newer package available from
      https://downloads.dataiku.com/public/studio/3.0.3/dataiku-dss-3.0.3.tar.gz

Exit code should be 0 (because uscan found and downloaded new version. If not, exit code will be 1) and the tarball should be in downloads directory

> ls -l downloads/
total 342996
lrwxrwxrwx 1 jbfavre jbfavre        24 juin  11 09:17 dataiku-dss_3.0.3.orig.tar.gz -> dataiku-dss-3.0.3.tar.gz
-rw-r--r-- 1 jbfavre jbfavre 270791663 juin  11 09:17 dataiku-dss-3.0.3.tar.gz

Automate all the things!

You can quite easily automate the watch for multiple upstream versions, using different watch files and a shell script.

Fake Debian package name will be determined from watch file name. In shell

for package in $(ls -1 watch/);
do
    current_version=$(check_version ${package})
    […]do_stuff[…]
done

Current version will be determined from downloads directory lookup

function check_version() {
    package=$1
    last_archive=$(ls -1 downloads/${package}_*.orig.tar.gz 2>/dev/null)
    version=${last_archive#downloads/${package}_}
    version=${version%.orig.tar.gz}
    if [ "X${version}" == "X" ];
    then
        version=0
    fi
    echo ${version}
}

Full uscan command is

uscan --download \
      --package ${package} \
      --upstream-version ${current_version} \
      --watchfile watch/${package} \
      --destdir ./downloads/

Some more config examples are available under GPLv3 on the irma-uscan github repository.

You will no longer have any excuse for missing a new upstream version of your software ! Enjoy :)