The Yubikey is a nice little device. It’s quite simple in design and operation.

The key actually emulating a USB keyboard, which makes it instantly usable on any modern OS. You just press the button on the key to generate a one-time-password (OTP) to validate you. The method works by typing in your password, but before hitting the return key, you press the Yubikey button to finish it off. At the end of the OTP generation, it sends a carriage return itself.

The OTP is then sent to a validation server, either hosted by Yubico themselves, or you can host your own.

I’m going to walk through how you can set the infrastructre for doing two-factor authentication on Debian. In my specific case, the requirement was two-factor with an Active Directory username/password combination and the Yubikey as the second factor.

Unfortunately, the documentation from Yubico is quite average. To top it off, they insist on using multiple Google Code project sites for hosting their software.

This would normally be fine, but in this case, they have a Google Code project for every single little piece of code. Much of the documentation I found relates to older projects which are not supported by Yubico. This makes working out exactly what you need difficult. Within the Google Code project sites, documentation often runs in circles between projects.

In this document, I’ll look at using PAM to auth again the Yubico auth servers first. Once that’s working, I’ll move onto flashing the Yubikey with a new key and using our own Validation System.

NOTE: This is just some rough notes I put together. You should definitely read the Yubico documentation for this to really make sense.

Authenticating with the Yubikey with PAM

Get some dependencies

apt-get install libpam-dev libcurl4-openssl-dev libpam-radius-auth

Make ourselves a source directory

mkdir ~/yubikey; cd ~/yubikey

Get the current tarball of libyubikey, and install it

wget http://yubico-c.googlecode.com/files/libyubikey-1.5.tar.gz
tar xf libyubikey-1.5.tar.gz
cd libyubikey-1.5
./configure
make check install

Get the current tarball of the Yubico C client, and install it

wget http://yubico-c-client.googlecode.com/files/ykclient-2.3.tar.gz
tar -xf ykclient-2.3.tar.gz
cd ykclient-2.3
./configure
make
make install

Get the current tarball of the Yubico PAM module, and install it

wget http://yubico-pam.googlecode.com/files/pam_yubico-2.3.tar.gz
tar -xf pam_yubico-2.3.tar.gz
cd pam_yubico-2.3
./configure
make
make install

You should end up with your Yubico PAM module ‘/usr/local/lib/security/pam_yubico.so’

We’ll refer to this in our PAM config /etc/pam.d/openvpn

#
# /etc/pam.d/openvpn - OpenVPN pam configuiration
#
# We fall back to the system default in /etc/pam.d/common-*
#
auth required /usr/local/lib/security/pam_yubico.so id=1 debug authfile=/etc/yubikeyid
auth required pam_radius_auth.so no_warn try_first_pass
@include common-account
@include common-password
@include common-session

This configuration will tell PAM to hit the Yubico module first. This splits apart your password field into your password and OTP. The OTP is validated against the Validation Servers, and the password is then passed onto the next module. This configuration will use the Yubico auth servers to check your token.

Once you have a working config, we’ll move to setting up our own Validation Servers. We’ll need to specify the URL for that in this config later on.

In that case, we’re also using RADIUS. This could be LDAP if you had an LDAP server available. You should be able to use the standard UNIX credentials (/etc/password, /etc/shadow) also.

The other important piece to note here is the authfile, /etc/yubikeyid

This file lists the mapping between username and the fixed part of your Yubikey. This is the first 12 chars of the Yubikey OTP (e.g. when you press the button)

abotting:vvcnrdkvevtj

FreeRADIUS authenticating against Active Directory 2008.

I banged my head against a wall for a while on this one. The trick is that you need at least FreeRADIUS 2.1.6 for AD authentication to work properly.

Add Debian backports to your /etc/apt/sources.list

deb http://www.backports.org/debian lenny-backports main contrib non-free

Import the backports key

wget -O - http://backports.org/debian/archive.key | apt-key add -

Update and install the new freeradius

apt-get update
apt-get -t lenny-backports install freeradius freeradius-ldap

In your radiusd.conf

ldap {
    # Define the LDAP server and the base domain name
    server = "ad.yourcompany.com"
    basedn = "dc=ad, dc=yourcompany, dc=com"

    # Active Directory doesn't allow for Anonymous Binding
    identity = "ldap_bind_user@ad.yourcompany.com"
    password = password

    password_attribute = "userPassword"
    filter = "(&(sAMAccountname=%{Stripped-User-Name:-%{User-Name}})(memberOf=CN=Users,DC=ad,DC=yourcompany,DC=com))"

    # This fixes Active Directory 2008 access
    chase_referrals = yes
    rebind = yes

    # The following are RADIUS defaults
    start_tls = no
    dictionary_mapping = ${raddbdir}/ldap.attrmap
    ldap_connections_number = 5
    timeout = 4
    timelimit = 3
    net_timeout = 1
}

In our FreeRADIUS client file /etc/freeradius/clients.conf:

client localhost {
    ipaddr = 127.0.0.1
    secret = testing123
    nastype = other
}

Use radtest to test our RADIUS is authenticating properly

radtest <username> <password> localhost 1 testing123

Should return Accept.

Set the address and shared secret of the radius server in /etc/pam_radius_auth.conf. The password of testing123 was defined in our RADIUS client config.

# server[:port] shared_secret   timeout (s)
127.0.0.1       testing123      1

OpenVPN has an issue with PAM loading the Yubikey module, so we have to LD_PRELOAD the pam module before starting OpenVPN.

export LD_PRELOAD=/lib/libpam.so.0.81.12; openvpn --config openvpn.conf

For a permanent fix, at the end of the start_vpn function in /etc/init.d/openvpn, just before the $DAEMON line:

    export LD_PRELOAD=/lib/libpam.so.0.81.12
    $DAEMON $OPTARGS --writepid /var/run/openvpn.$NAME.pid \
        $DAEMONARG $STATUSARG --cd $CONFIG_DIR \
        --config $CONFIG_DIR/$NAME.conf || STATUS=1

Change the path of /lib/libpam.so.0.81.12 to suit your own system.

I won’t go into the OpenVPN configuration, except that for PAM authentication you need these options in your server config:

plugin /usr/lib/openvpn/openvpn-auth-pam.so openvpn
username-as-common-name
ns-cert-type server
client-cert-not-required

Personalising your Yubikey

To host your own Yubikey validation system, you require the secret AES key of your Yubikey. In the past, Yubico could provide this to you. Now, you’re required to flash your Yubikey yourself which will generate a new AES key.

Yubico provide a personalisation tool for Linux, Mac and Windows. If you’re on Windows, you get a nice little GUI. For Linux and Mac, you have a CLI based tool. It’s worth having a look at the ‘Personalization Tool’ page at: http://www.yubico.com/developers/personalization/

Installing the Personalisation Tool

Install some dependencies:

apt-get install libusb-1.0.0-dev

Grab the latest Pesonalisation Tool tarball from: http://code.google.com/p/yubikey-personalization/

cd ~/yubikey
wget http://yubico-c.googlecode.com/files/libyubikey-1.5.tar.gz

Extract, build and install libyubikey

tar xf libyubikey-1.5.tar.gz
cd libyubikey-1.5
./configure
make
make install

You’ll need to provide a UID value for flashing your Yubikey. It needs to be 6 characters, and in hexadecimal. You can use this command to generate one for you.

dd if=/dev/urandom of=/dev/stdout count=100 2>/dev/null | xargs -0 modhex | cut -c 1-10 | awk '{print "vv" $1}'
74657374696e

You must provide the public name (fixed) parameter in modhex format. The modhex format is a special encoding used to ensure characters sent by the key are always correctly interpreted whatever keyboard layout you use.

You also need to generate yourself a public name for your key. This is known as the ‘fixed’ part, and it’ll be the first 16 chars when you generate your OTP. This will identify your key from anybody else’s.

dd if=/dev/urandom of=/dev/stdout count=100 2>/dev/null | xargs -0 modhex | cut -c 1-10 | awk '{print "vv" $1}'
vvcnrdkvevtj

This comamnd generate some random text, does a modhex operation, grabs the first 10 chars, then adds ‘vv’ to the front to make it up to 12.

You’ll be prompted for a passphrase on your AES key. I leave mine blank, but if you do set one, don’t ever lose it. I believe it’ll stop you from re-personalising your Yubikey.

ykpersonalize -ouid=74657374696e -ofixed=vvcnrdkvevtj
Firmware version 2.1.2 Touch level 1793 Program sequence 1
Passphrase to create AES key:
Configuration data to be written to key configuration 1:
fixed: m:vvcnrdkvevtj
uid: h:74657374696e
key: h:fcaad309a20ne1809c2db2f7f0e8d6ea
acc_code: h:000000000000
ticket_flags: APPEND_CR
config_flags:

Commit? (y/n) [n]: y

Save this information, as we’ll need it later.

Setting up yor own YubiKey OTP Validation Server

You need to install two things: The Key Storage Module and the Yubico Validation Server. The Key Storage Module (KSM) holds the secret AES key of your Yubikey token, while the Validation Server does the OTP check against the KSM.

In their 2.0 architecture, you can have multiple KSM’s and Validation servers with work together for reduncancy.

KSM Installation

Make a working directory, and get the KSM package

mkdir ~/yubikey && cd ~/yubikey
wget http://yubikey-ksm.googlecode.com/files/yubikey-ksm-1.3.tgz
tar xfz yubikey-ksm-1.3.tgz

Install the KSM files

cd yubikey-ksm-1.3
make install

Install Apache2 and PHP

Install Apache2, PHP and MySQL

apt-get install apache2 php5 php5-mcrypt php5-curl mysql-server php5-mysql libdbd-mysql-perl

Create the ykksm table

echo "CREATE DATABASE ykksm;" | mysql -u root -p

Import the DB schema

mysql -u root -p ykksm < /usr/share/doc/ykksm/ykksm-db.sql

Set up some MySQL permissions

CREATE USER 'ykksmreader';
GRANT SELECT ON ykksm.yubikeys TO 'ykksmreader'@'localhost';
SET PASSWORD FOR 'ykksmreader'@'localhost' = PASSWORD('hYea3Inb');

CREATE USER 'ykksmimporter';
GRANT INSERT ON ykksm.yubikeys TO 'ykksmimporter'@'localhost';
SET PASSWORD FOR 'ykksmimporter'@'localhost' = PASSWORD('ikSab29');

FLUSH PRIVILEGES;

Include path configuration

Set the include path by creating a file /etc/php5/conf.d/ykksm.ini

cat > /etc/php5/conf.d/ykksm.ini << EOF
include_path = "/etc/ykksm:/usr/share/ykksm"
EOF

Make a web server symlink

make -f /usr/share/doc/ykksm/ykksm.mk symlink

Set your configuration settings in /etc/ykksm/ykksm-config.php

<?php
  $db_dsn      = "mysql:dbname=ykksm;host=127.0.0.1";
  $db_username = "ykksmreader";
  $db_password = "hYe63Inb";
  $db_options  = array();
  $logfacility = LOG_LOCAL0;
?>

Restart Apache2

/etc/init.d/apache2 restart

Test the KSM Server

Try this URL:

curl 'http://localhost/wsapi/decrypt?otp=dteffujehknhfjbrjnlnldnhcujvddbikngjrtgh'
ERR Unknown yubikey

It should return ‘Unknown Key’ until we have imported our Yubikey into the database.

Install the Yubico Validation Server

The latest version, and documentation can be found at: http://code.google.com/p/yubikey-val-server-php/

Installation

Go to our working source directory, and grab the package

cd ~/yubikey
wget http://yubikey-val-server-php.googlecode.com/files/yubikey-val-2.4.tgz

Extract, build and install the server

tar -zxf yubikey-val-2.4.tgz
cd yubikey-val-2.4
make install

Create the ykval database and import the schema

echo 'create database ykval' | mysql -u root -p
mysql -u root -p ykval < /usr/share/doc/ykval/ykval-db.sql

Install the symlink

make symlink

Include path configuration

cat > /etc/default/ykval-queue << EOF
DAEMON_ARGS="/etc/ykval:/usr/share/ykval
EOF

Create a htaccess file: /var/www/wsapi/2.0/.htaccess

RewriteEngine on
RewriteRule ^([^/\.\?]+)(\?.*)?$ $1.php$2 [L]

php_value include_path ".:/etc/ykval:/usr/share/ykval"

Symlink the htaccess file

cd /var/www/wsapi; ln -s 2.0/.htaccess /var/www/wsapi/.htaccess

Copy the template config file for the Validation Server

cp /etc/ykval/ykval-config.php-template /etc/ykval/ykval-config.php

Edit the file and configure settings in /etc/ykval/ykval-config.php

<?php

  # For the validation interface.
  $baseParams = array ();
  $baseParams['__YKVAL_DB_DSN__'] = "mysql:dbname=ykval;host=127.0.0.1";
  $baseParams['__YKVAL_DB_USER__'] = 'ykvalverifier';
  $baseParams['__YKVAL_DB_PW__'] = 'password';
  $baseParams['__YKVAL_DB_OPTIONS__'] = array();

  # For the validation server sync
  $baseParams['__YKVAL_SYNC_POOL__'] = array("http://localhost/wsapi/2.0/sync");

  # An array of IP addresses allowed to issue sync requests
  # NOTE: You must use IP addresses here.
  $baseParams['__YKVAL_ALLOWED_SYNC_POOL__'] = array("127.0.0.1");

  # Specify how often the sync daemon awakens
  $baseParams['__YKVAL_SYNC_INTERVAL__'] = 10;

  # Specify how long the sync daemon will wait for response
  $baseParams['__YKVAL_SYNC_RESYNC_TIMEOUT__'] = 30;

  # Specify how old entries in the database should be considered aborted attempts
  $baseParams['__YKVAL_SYNC_OLD_LIMIT__'] = 10;

  # These are settings for the validation server.
  $baseParams['__YKVAL_SYNC_FAST_LEVEL__'] = 1;
  $baseParams['__YKVAL_SYNC_SECURE_LEVEL__'] = 40;
  $baseParams['__YKVAL_SYNC_DEFAULT_LEVEL__'] = 60;
  $baseParams['__YKVAL_SYNC_DEFAULT_TIMEOUT__'] = 1;

  // otp2ksmurls: Return array of YK-KSM URLs for decrypting OTP for
  // CLIENT.  The URLs must be fully qualified, i.e., contain the OTP
  // itself.
  function otp2ksmurls ($otp, $client) {
    return array("http://localhost/wsapi/decrypt?otp=$otp",);
  }
?>

In the above configuration, we’re only expecting to use one Validation Server and one KSM. If you’re planning on having multiple Validation servers and KSM’s, then you’ll be including the other Validation Servers in the SYNC_POOL, and your KSM’s in the URLs at the bottom, returned by the otp2ksmurls function.

Enable the mod_rewrite

a2enmod rewrite

Create the ykval database user

CREATE USER 'ykvalverifier'@'localhost' IDENTIFIED BY  'password';
GRANT ALL PRIVILEGES ON `ykval`. * TO  'ykvalverifier'@'localhost';

Fix some privileges on our config file

chgrp www-data /etc/ykval/ykval-config.php

The Sync Daemon uses the PEAR module System_Daemon so you need to install it:

apt-get install php-pear
pear install System_Daemon-0.9.2

Install the init.d script

ykval-queue install
update-rc.d -f ykval-queue defaults

Start the daemon

/etc/init.d/ykval-queue start

Testing

Use CURL to test our server is working

curl 'http://localhost/wsapi/verify?id=1&otp=vvcnrdkvevtefjbrjnlnldnhcujvddbikngjrtgh'

It should return something like this:

h=aPCQ4kWJilDgriyEii3j8J8lfuY=
t=2009-04-27T19:08:51Z0100
status=NO_SUCH_CLIENT

Once we import our Yubikey into the database, we should get a nice ’status=OK’ message.

Importing your keys into the KSM server

Refer back to the output from personalising your Yubikey. You’ll need the fixed part (referred to as publicname in the DB), internal name (UID) and our AES key.

This is an entry for our newly personalised Yubikey.

USE ykksm;
INSERT INTO `yubikeys` (`serialnr`, `publicname`, `created`, `internalname`, `aeskey`, `lockcode`, `creator`, `active`, `hardware`)
VALUES (101209, 'vvcnrdkvevtj', '2010-05-07 15:18:40', '74657374696e', 'fcaad309a20ne1809c2db2f7f0e8d6ea', '000000000000', '', 1, 1);

This entry is required for our systems to authenticate against the Validation server. I’m not exactly sure about this, as the documentation is somewhat bare. I think you need an administrator-type person’s key details in here. The imporant part is the ID. This values corresponds the the ‘id=’ value in our CURL requests and in our PAM config.

USE ykval;
INSERT INTO `clients`
(`id`, `active`, `created`, `secret`, `email`, `notes`, `otp`)
VALUES
(1, 1, 1, 'fcaad309a20ne1809c2db2f7f0e8d6ea', 'your@email.addr', 'Any text your want', 'vvcnrdkvevterfbtelvnvkkueenecrlfnlhdjetrhgnk');

We’ll hit our new Validation Server to make sure it’s working

curl "http://localhost/wsapi/2.0/verify?id=1&nonce=askjdnvajsndjkasndvjsnad&otp=vvcnrdkvevtjkreuvvlhtubjecbrticjneckgrigkck"
h=KLEb3gOJ4KqQaCVbh8cEvXjH50U=

It should return something like this:

t=2010-05-20T11:24:53Z0051
otp=vvvcnrdkvevtjkreuvvlhtubjecbrticjneckgrigkck
nonce=askjdnvajsndjkasndvjsnad
sl=100
status=OK

In this URL, we’ve added the ‘nonce’ parameter. This just a test to make sure the v2.0 API is working. ’status=OK’ means it’s all good! If you get ‘NOT_ENOUGH_ANSWERS’, it means it has trouble trying to sync with other Validation Servers.

We’ll get PAM using our new Validation Servers for auth

/etc/pam.d/openvpn

auth required /usr/local/lib/security/pam_yubico.so id=1 authfile=/etc/yubikeyid url=http://10.68.130.198/wsapi/verify?id=%d&otp=%s debug

If you watch /var/log/auth.log, you should see the PAM module spitting out some debugging information which may be useful. It also spits out your plain text password too, while you have the debug option on. Make sure you remove this later.

Problems

If you see an error like this:

PAM unable to dlopen(/lib/security/pam_yubico.so): /lib/security/pam_yubico.so: undefined symbol: pam_set_data

you’ll need the LD_PRELOAD trick from above. Something to do with dlopening the PAM module I believe.

Following on from my post about building Debian virtual machines with libvirt, I’ve now got automated installations of Debian Lenny using the preseeding method. Coupling this with using virt-install, I can have a Debian virtual machine installation in only a few minutes. No questions asked.

The virt-install command contains an extra-args argument, where you can fill-in the specific parts of the preseeding. I don’t want to set an IP address in the file as it’s going to be used to build lots of machines, so I just specify that at install time. The URL part is where out preseed config file is stored. This obviously means that the machine needs to able to contact with webserver at install time to download the config.

$ NAME=debian-test
virt-install --name=${NAME} \
--ram=512 --file=/var/lib/xen/images/${NAME}.img \
--file-size 8 \
--nographics \
--paravirt \
--network=bridge:br0 \
--location=http://mirrors.uwa.edu.au/debian/dists/lenny/main/installer-i386 \
--extra-args="auto=true interface=eth0 hostname=${NAME} domain=vpac.org netcfg/get_ipaddress=192.168.1.2 netcfg/get_netmask=255.255.255.0 netcfg/get_gateway=192.168.1.1 netcfg/get_nameservers=192.168.1.1 netcfg/disable_dhcp=true url=http://webserver/preseed.cfg"

To get an idea of the contents of the preseed config file, the best place to start is the Debian stable example preseed file. It lists lots of different options, with plenty of comments so you can understand what’s going on.

For me to get a fully-automated install, I used these options. It’s fairly standard, but definitely worth reading the comments about each line.

$ egrep -v "(^#|^$)" preseed.cfg
d-i debian-installer/locale string en_AU
d-i console-keymaps-at/keymap select us
d-i netcfg/choose_interface select eth0
d-i netcfg/disable_dhcp boolean true
d-i netcfg/dhcp_options select Configure network manually
d-i netcfg/confirm_static boolean true
d-i mirror/protocol string http
d-i mirror/country string manual
d-i mirror/http/hostname string mirrors.uwa.edu.au
d-i mirror/http/directory string /debian
d-i mirror/http/proxy string
d-i clock-setup/utc boolean true
d-i time/zone string Australia/Melbourne
d-i clock-setup/ntp boolean true
d-i clock-setup/ntp-server string ntp.vpac.org
d-i partman-auto/method string regular
d-i partman-lvm/device_remove_lvm boolean true
d-i partman-md/device_remove_md boolean true
d-i partman-lvm/confirm boolean true
d-i partman-auto/choose_recipe select atomic
d-i partman/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i passwd/make-user boolean false
d-i passwd/root-password-crypted password [MD5 Sum of the password]
tasksel tasksel/first multiselect standard
d-i pkgsel/include string openssh-server vim puppet
popularity-contest popularity-contest/participate boolean false
d-i grub-installer/only_debian boolean true
d-i grub-installer/with_other_os boolean false
d-i finish-install/reboot_in_progress note

Some good resources I found, which might help you are:

• The Preseeding d-i page on the Debian wiki
• Mike Renfro’s Unattended Debian Installations (or How I Learned to Stop Worrying and Love the preseed.cfg)

It seems that Adobe, after issuing a press release claiming they would be opening up the RTMP protocol in the ‘first half of 2009′, have issued a DMCA take down request for an open source implementation of the protocol, RTMPdump. The SourceForge project site for RTMPdump now shows ‘Invalid Project’.

This is going to mean it’s going to become much harder to get RTMPdump for downloading copies of ABC’s iView files, which I previously posted about. This might also have interesting consequences for XBMC and Boxee which both include this code for supporting streaming media from BBC’s iPlayer.

This is pretty disappointing from Adobe, especially after claiming they would be in the process of opening up the protocol.

A plugin for ABC iView on XBMC has been released. See this page for progress of ABC iView on XBMC.

I just rewrote the iView plugin for XBMC.. and it’s far more robust. There is still lots to finish, but it kinda works. This was spurred on by someone actually trying it out, and then me finding out that ABC changed their XML.

You’ll need the RTMP patch for XBMC, and version 0.2 of the ABC iView plugin for XBMC.

Some things that still need improving are:

• Auth token still times out. That means that if you watch something, you’ll need to go back to the channels list and then back into the channel to list the programs again and get a new auth token. Annoying.
• No thumbnails or extended metadata available for channels or programs.
• Some programs have funny names. Pretty minor, but annoying.
• Programs are streamed in 4:3, but are actually produced in 16:9. I set XBMC to 16:9 Stretch mode.

For more info about the plugin, see this other entry I wrote.

Don’t forget to vote for an iView plugin for Boxee at the Customer Support Community for boxee. It might might help get iView into Boxee!

A plugin for ABC iView on XBMC has been released. See this page for progress of ABC iView on XBMC.

Following on from the last post about using rtmpdump to grab ABC’s iView programs, I’ve made a start on an XBMC plugin.. with the hope of eventually working on a Boxee plugin also.

To start with, you’ll need my patch to all you to specify the tcurl of an rtmp stream from with the XBMC API. This is needed because XBMC makes some assumptions about RTMP urls, based on other streams like Hulu and BBC’s iPlayer. ABC’s method is similar, but a little different. I’ll be trying to get the patch sent upstream, but it may need a little more work.

Now you’re going to have to compile XBMC yourself from source. I’ve only done it on Linux, so I can’t help you with Mac, Windows or Xbox versions. For information about compiling it on Ubuntu, you can check out the page on the XBMC wiki. You just need to do ‘cd’ into the XBMC directory you did your SVN checkout on, and then:

patch -p0 < /path/to/abc-iview-rtmp-tcurl-fix.patch

Hopefully you shouldn’t see any errors.

You can then grab my very basic iView plugin for XBMC. It’ll need to be extracted into your plugins/video directory of your XBMC installation.

This plugin has some serious limitations right now..

Firstly, some shows are listed as just ‘Episode 1′. It seems that in the XML files describing the shows, the data is very inconsistent. I’ll be looking at this in the next version of the plugin.

Next, because of the nature of the auth token that is generated, if you watch a program and then go back to the list of programs, if you try another, it will fail to play, as the token has timed out. You need to go back another level to the channels, then click the channel you want. This means that the URLS listed will generate a new token which will be valid again.

Last, the shows are all broadcasted in 16:9 on the TV, but streamed at 640×480 (4:3). This is really silly, but you can fix it by setting your XBMC view to use ‘Stretch 16:9′. Not ideal, but I’ll be looking into automatically setting the view if it’s exposed in the XBMC API.

It’s still very rough, but a start. Boxee has just announced a new API which I’ll be taking a look at shortly.

UPDATE: Version 0.2 of the plugin is out. See here.

A plugin for ABC iView on XBMC has been released. See this page for progress of ABC iView on XBMC.

I’ve done a little bit of work since my last post on this, and a couple of people have asked for my stuff. Here goes.

Firstly, you can use RTMPdump to download the iView stream on your Linux box. You’ll need to download rtmpdump 1.4 and compile it yourself. It should just take a ‘make’ as long as you have all the requirements.

When iView starts, it first requests an XML config file, from the URL http://www.abc.net.au/iview/iview_config.xml

<?xml version="1.0" encoding="utf-8"?>
<config>
<param name="authenticate_path"   value="http://202.125.43.119/iview.asmx/isp" />
<param name="media_path"          value="flash/playback/_definst_/" />
<param name="media_path_mp4"      value="flash:mp4/playback/_definst_/" />
<param name="server_streaming"    value="rtmp://cp53909.edgefcs.net/ondemand" />
<param name="server_speedtest"    value="rtmp://cp44823.edgefcs.net/ondemand" />
<param name="xml_help"            value="iview_help.xml" />
<param name="xml_channels"        value="iview_channels.xml" />
<param name="xml_series"          value="http://www.abc.net.au/playback/xml/rmp_series_list.xml" />
<param name="xml_thumbnails"      value="http://www.abc.net.au/playback/xml/thumbnails.xml" />
<param name="xml_classifications" value="http://www.abc.net.au/playback/xml/classifications.xml" />
<param name="xml_feature"         value="http://www.abc.net.au/playback/xml/iview_feature.xml" />
<param name="xml_feature_home"    value="http://www.abc.net.au/playback/xml/iview_homepage.xml" />
<param name="server_time"         value="http://www.abc.net.au/iview/time.htm" />
<param name="thumbs_path"         value="http://www.abc.net.au/playback/thumbs/" />
<param name="base_url"            value="http://www.abc.net.au/iview" />
<param name="channel_id_arts"     value="2260366" />
<param name="channel_id_news"     value="2186765" />
<param name="channel_id_docs"     value="2176127" />
<param name="channel_id_shop"     value="2186639" />
<param name="channel_id_catchup"  value="2172737" />
<param name="channel_id_kazam"    value="2288241" />
<param name="channel_id_faves"    value="2478452" />
<param name="channels_main"       value="catchup,news,docs,arts,shop" />
<param name="channels_kids"       value="kazam,faves" />
</config>

From this file, you can find out which other XML files you need for the channels and program descriptions. Firstly though, you need a special token, which is like an authorisation string. It’s done by doing a HTTP GET on the authenticate_path, which will return something like:

<?xml version="1.0" encoding="utf-8"?>
<iview xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="iview.abc.net.au">
<ip>124.168.17.31</ip>
<isp>iiNet</isp>
<desc>iiNet Limited</desc>
<host>Akamai</host>
<server />
<bwtest />
<token>daEdOckcEbtaqdmdLasbhcBbCbobAbOaxa5-bjOn1r-8-jml_rFAnL&amp;aifp=v001</token>
<text>iView is unmetered for &lt;a href=”http://www.iinet.net.au/” target=”_blank”&gt;iiNet&lt;/a&gt; customers.</text>
<free>yes</free>
<count>5557</count>
<init>false</init>
</iview>

This is doing a lookup of my IP address, to ensure I’m in Australia, and pass me the token. The token has a short lifetime also, only a few minutes. You then need this token to help you build the URL to request the video stream you want.

To find the programs of a particular channel, you need to grab a URL like this: http://www.abc.net.au/playback/xml/output/catchup.xml.

<?xml version="1.0"?>
<rmp-content xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<channel enabled="true" id="2172737">
<name>ABC CatchUp</name>
<description><![CDATA[Recent best of ABC1 & ABC2 TV]]></description>
<intro></intro>
<ident></ident>
<channel-logo>http://www.abc.net.au/playback/img/chl_catchup.png</channel-logo>
<image id=”258433″ order=”1″>
<title><![CDATA[ABC Catchup Background 09]]></title>
<version id=”1071615″>
<title><![CDATA[1230x564jpg]]></title>
<url>http://www.abc.net.au/reslib/200806/r258433_1071615.jpg</url>
</version>
</image>
<image id=”257912″ order=”2″>
<title><![CDATA[ABC Catchup background 06]]></title>
<version id=”1068909″>
<title><![CDATA[1230x564jpg]]></title>
<url>http://www.abc.net.au/reslib/200806/r257912_1068909.jpg</url>
</version>
</image>
<program-title-list>
<program-title id=”352699″ promo=”false” order=”9″>
<title><![CDATA[Catalyst Series 10 Episode 8]]></title>
<short-title></short-title>
<synopsis><![CDATA[Malaria jumps the gap from monkey to man; could bubbles be a solution to the hard hit  mining industry? And see how a horse trainer applies his skill to the training of elephants, with remarkable success.]]></synopsis>
<publish-date>03/04/2009 12:00:00</publish-date>
<expire-date>17/04/2009 00:00:00</expire-date>
<transmission-date>02/04/2009 00:00:00</transmission-date>
<censorship>G</censorship>
<censorship-warning></censorship-warning>
<website>Go to website</website>
<website-url>http://www.abc.net.au/catalyst/</website-url>
<video-download></video-download>
<video-download-url>http://www.abc.net.au/tv/geo/catalyst/vodcast/default.htm</video-download-url>
<shop></shop>
<shop-url></shop-url>
<category>Science and Technology</category>
<cue-points>
</cue-points>
<video-asset id=”1619127″ order=”0″>
<title><![CDATA[1850flv]]></title>
<url>catch_up/catalyst_09_10_08.flv</url>
<unc-path>catalyst_09_10_08.flv</unc-path>
<duration>27.00</duration>
<file-size>135</file-size>
<thumb>abc_catchup.jpg</thumb>
</video-asset>
</program-title>
<program-title id=”….”>
…more programs…
</program-title>
</program-title-list>
</channel>
</rmp-content>


I’ve shortened the output of the ‘Catch Up’ channel here. This is what you’re likely to see when you get the channel XML file. As you can see, this is describing an episode of Catalyst, and I’ve marked the URL in bold.

TOKEN=`curl -q http://202.125.43.119/iview.asmx/isp | grep token | sed 's/<token>//g' | sed 's/\&amp;/\&/g' | sed 's,</token>,,g' | sed 's/ //g'`; ./rtmpdump --rtmp "rtmp://203.206.129.37:1935////flash/playback/_definst_/catch_up/catalyst_09_10_08.flv" --auth "auth=${TOKEN}" -t "rtmp://cp53909.edgefcs.net/ondemand?auth=${TOKEN}" -o test.flv

This horrible command is getting the token, and stripping out all unncessesary characters, and then passing it onto rtmpdump. You might have also noticed in the command above, I have four slashes in the RTMP url. This is to work around some assumptions that rtmpdump makes about the path. I had made a patch, but in rtmpdump 1.4, you can just use 4 slashes to make it work.

Most of this data came from doing Wireshark packet traces while working with the flash-based iView interface. Also important to note that the programs have an expiry date also. If the command above returns a ’stream not found’ message, you’ll probably need a newer episode.

In the next post, I’ll be posting the code for the XBMC plugin.

So it seems that my CentOS 5 Dom0 wasn’t stable. When building new virtual machines, the machine would hang and I’ve have to go back into the machine room to reboot it.

I have suspicions that it was due to the 3ware 8006 RAID controller, but instead of messing around with that, I’ve installed Debian Lenny as the Dom0 (using kernel 2.6.26 as opposed to kernel 2.6.18 with CentOS).

With this machine, I wanted to find the best way to support both Debian and CentOS machine, using a common method of installation. There seem to be two main ways to accomplish this.

You could use the (older) Debian route and use xen-create-image from xen-tools, which does a bootstrap of the OS on the filesystem, or use the newer virt-install from libvirt, to do an actual OS install. Libvirt seems like it’s the preferred method these days, which many of the distro’s now using for managing virtual machines using Xen, KVM or QEMU.

Using xen-create-image for Debian virtual machines has worked for me for a long time, but trying to use it for CentOS failed. The machine built, but I believe there were some packages missing from the install. I really didn’t want to have to mess around with the package lists, so I tried to use both xen-create-image for Debian and virt-install for CentOS. One problem with this is that virt-install doesn’t install the Xen config files into /etc/xen like the other tools do. Instead, it manages its own list, and contacts Xen directly using a Unix socket.

This would make management a pain, because you would have to use xm create <domain>to start a Debian VM, but then use virsh start <domain> for CentOS. I needed something simpler.

Then I discovered that Debian Lenny now has para-virtualisation support built into the Debian Installer.

This means that I could use virt-install to build Debian Lenny virtual machines, using the actual Debian installer.

With a quick install of the libvirt packages in the Debian Lenny’s repository:

apt-get install libvirt-bin virtinst

You’ve got all the libvirt stuff you need. Then, to create a Debian virtual machine using virt-install:

virt-install \
--name=debian-test \
--ram=512 \
--file-size=8 \
--nographics \
--paravirt \
--file=/var/lib/xen/images/debian-test.img \
--location=http://mirrors.uwa.edu.au/debian/dists/lenny/main/installer-i386

The important part is that last line. You can actually just throw a path to the install images of a Debian mirror, and virt-install is smart enough to boot a new VM from that. This then begins a Debian install, identical to what you would use on a standard machine. This also gives you full access to use the nice virt-manager. You can install virt-manager by doing:

apt-get install virt-manager

So I just need to remember now that if I want to start a VM, I need to use virsh start <domain>

Although, once started, you can use the standard xm tools.

So finally, I have reached open-source para-virtualisation nirvana. Now if only Debian did Kickstart…

A plugin for ABC iView on XBMC has been released. See this page for progress of ABC iView on XBMC.

I think it would be really neat to use ABC’s iView on the Xbox Media Centre (XBMC) and/or Boxee. Honestly, who really wants to watch TV on their computers? Haven’t we evolved from that now?

I’ve got a modded XBOX running XBMC, and I have various Linux boxes running XBMC and Boxee and I think they’re the perfect platform for something like iView.. especially because it’s unmetered traffic on iiNet, Internode and other great ISP’s.

I did a little research, and they seem to use Adobe’s Real Time Message Protocol (RTMP) to stream the video from their server to the iView client, which is written in Flash. Recent versions of XBMC and Boxee have code to support RTMP, which is also used by other digital content providers like NBC’s Hulu, and the BBC’s iPlayer.

I have managed to work out most of the iView’s XML stuff, which describes channels, programs, thumbnails, etc but not quite got there with the actual streaming part. I’m playing with rtmpdump, which is based on the rtmp code from XBMC/Boxee, and have almost worked out the URL part to get the server to stream. I just keep getting a message about not being able to find the specified stream.

If anyone out there on the interwhizzle has worked this stuff out, I’d love to hear from them. My googling hasn’t really revealed anything like what I’m after. If you’re interested in using iView on XBMC or Boxee, I’d like to hear from you also.

UPDATE: Please vote for an iView plugin for Boxee at the Customer Support Community for boxee. It might might help get iView into Boxee!

UPDATE 2: I’ve uploaded a basic plugin for XBMC. See: http://www.andybotting.com/wordpress/iview-for-xbmc-plugin-v02 for more info.

The aim of this was to use the standard CentOS/RHEL Xen Dom0 tools to boot a Debian Lenny DomU.

I found plenty of instructions for doing CentOS DomU under a Debian Dom0, but not the other way around. So, this is a little how-to documenting the little things that need to be overcome.

I also wanted the Debian virtual machines to have their filesystems in a file, in the same standard way that the RHEL virt-install creates.

Steps involved:

• Use virt-install to build a standard CentOS virtual machine
• Use debootstrap to build a Debian Lenny base install for transplanting
• Break apart a CentOS filesystem-in-a-file, and move the Debian install into it
• Modify Debian config for booting the CentOS kernel

Use virt-install to build a standard CentOS virtual machine

I created a new virtual machine, using virt-install.

virt-install -n newvm -r 512 -f /var/lib/xen/images/debian.img -s 8 -l http://ftp.monash.edu.au/pub/linux/CentOS/5/os/i386/ -p --nographics -x

I needed some CentOS virtual machines anyway, so I let the install go through and do its thing. If you didn’t need it, you could probably kill the install before it started installing packages. We just needed the config file for the VM in /etc/xen and the filesystem image.

Use debootstrap to build a Debian Lenny base install for transplanting

I actually had a Debian Xen Dom0 with the xen-tools package installed. I used this to create a new Debian Lenny install, and also do some of the nice hook scripts with you would otherwise have to do by hand.

# xen-create-image --hostname=vanila --size=8Gb --dist=lenny --memory=512M --ide --dhcp

This meant I had a hostname file, libc6-xen and other things already done for me.

This was installed into an LVM partition, so after building it, I mounted the LVM partition, and used tar to compress it.

# mount /dev/mapper/vg-vanilla--disk /mnt
# tar zc -C /mnt/ . > /tmp/vanilla-debian.tar.gz

Break apart a CentOS filesystem-in-a-file, and move the Debian install into it

Set up the loop device
# losetup -f /var/lib/xen/images/debian.img

Map the partitions inside the loop device
# kpartx -av  /dev/loop0
add map loop0p1 : 0 208782 linear /dev/loop0 63
add map loop0p2 : 0 16032870 linear /dev/loop0 208845

At this point, you should have /dev/mapper/loop0p1 which is the root filesystem of your new VM. You’ll need to format the filesystem with:
# mkfs.ext3 /dev/mapper/loop0p1

Mount the newly formatted filesystem
# mount /dev/mapper/loop0p1 /mnt

Extract our Debian Lenny install into the filesystem
# cd /mnt
# tar xf /tmp/vanilla-debian.tar.gz

Modify Debian config for booting the CentOS kernel

We want to use CentOS/RHEL’s pygrub bootloader, just because it’s nice.

First, you’ll need to copy the CentOS kernel into your Debian install. You’ll need the config, kernel and initrd files from /boot of a DomU (or maybe the Dom0..)
# cd /boot
# cp config-2.6.18-92.1.22.el5xen vmlinuz-2.6.18-92.1.22.el5xen initrd-2.6.18-92.1.22.el5xen.img /mnt/boot

Rename the initrd to drop the .img from the end. It doesn’t work with the update-grub script in Debian
# mv initrd-2.6.18-92.1.22.el5xen.img initrd-2.6.18-92.1.22.el5xen

Copy the kernel modules to your new VM too:
# cp -r /lib/modules/2.6.18-92.1.22.el5xen /mnt/lib/modules

If you don’t have a /boot/grub directory in your Debian DomU, then you’ll need create one. You only really need three files: menu.lst and device.map. We’ll need to set it up properly so that both the update-grub script in Debian and the pyGrub bootloader work happily.

Edit the /boot/grub/device.map file. Make sure your hd0 is set to /dev/xvda:
(hd0)   /dev/xvda

The pyGrub script reads grub.conf, and not menu.lst, so symlink it
cd /boot; ln -s menu.lst grub.conf

Here’s the contents of my /boot after I’m finished:
/boot/config-2.6.18-92.1.22.el5xen
/boot/initrd-2.6.18-92.1.22.el5xen
/boot/vmlinuz-2.6.18-92.1.22.el5xen
/boot/grub
/boot/grub/default
/boot/grub/menu.lst
/boot/grub/device.map
/boot/grub/grub.conf


You’ll need to fix your inittab to use the xvc0 as your console. If you don’t you lose access to log into the console. In the file /etc/inittab, edit the tty1 line to be xvc0 instead.
1:2345:respawn:/sbin/getty 38400 xvc0

Your first tty should be changed to xvc0, and the others (tty2-6) should be commented out (if they’re not already)

Unmap the partitions and destroy our loop device
# kpartx -d /dev/loop0
# losetup -d /dev/loop0

Start the new Debian Lenny virtual machine

# xm create -c debian

You should see PyGrub come up, and let you pick the kernel.
pyGRUB version 0.6
==========================================================================
| Debian GNU/Linux, kernel 2.6.18-92.1.22.el5xen                         |
| Debian GNU/Linux, kernel 2.6.18-92.1.22.el5xen (single-user mode)      |
|                                                                        |
|                                                                        |
|                                                                        |
|                                                                        |
|                                                                        |
|                                                                        |
==========================================================================
Use the ^ and v keys to select which entry is highlighted.
Press enter to boot the selected OS. 'e' to edit the
commands before booting, 'a' to modify the kernel arguments
before booting, or 'c' for a command line.

Will boot selected entry in  4 seconds

Hopefully, it works for you too

I’ve made one vanilla debian install, and just make a copy of that image file for each new VM I need to create. I have eth0 in the interfaces file commented out, so I just put the new IP in, set the hostname and I’m ready to go.

I may have missed a step in here, so if you’re trying this out, please comment to let us know how it goes.

Once we realised how much Stable lodge was really costing us, we upgraded our accommodation for the four nights we had in Bangkok. We booked a serviced apartment at Citiadines in Sukhumvit 8, just down the road from Stable Lodge.

Messy bed at Citadines

We booked it on Wotif and it cost us a little bit more than what Stable Lodge was. It was totally worth it for the comfortable bed alone, especially after sleeping on really hard and really soft beds at the other places.

Nice TV at Citadines

Somehow we ended up wth four nights in Bangkok with was way too much. If you’re going to Thailand, only spend a few days in Bangers at a maximum. It’s just not that exciting. It’s too similar to Melbourne really. Trains, shopping centers, etc.

We checked out the massive MBK shopping center and the King Power duty free place too. If you see King Power anywhere.. avoid it. Don’t waste your time, it’s fancy stuff that is way overpriced.

The last night in Bangkok, we found out about the semi-permanent beer gardens that get set up outside the CentralWorld shopping center. The beer garden for the Thai beer Singha, must have had some association with the Japanese beer Asahi, which just happens to be my favourite beer. So, you can imagine my excitement to find out they were serving it there.

The Asahi tower

Bek and I polished off a tower of Asahi. The tower is a 3 litre tube full of beer with a column of ice down the middle to keep it cold and a tap on the bottom to pour. It was awesome. I thought about how good it would be to do a similar thing in Melbourne, but I realised that it just wouldn’t work because it would get abused. People would be getting smashed and then smashing each other (like going to any pub in the city these days). The Thai people don’t drink that much and are very passive. It’s nice to walk around at any time and feel totally safe.

All gone

We had trouble getting a taxi to the airport when it was time to leave. The guy at the hotel mentioned something about a bomb, but we didn’t know anything else. Finally a taxi arrived who was willing to take us. This guy was crazy. He had this strange twitch in his seat while he was driving. He was doing 130 km/h down the freeway, which had 80 signs. Weaving through traffic and flashing his lights at anyone slowing him down. We also didn’t have any seat belts.

Closer to the airport, we started seeing lots of people in yellow shirts with plastic hand clappers. I had no idea who they were until later. They were protesters heading towards the airport to shut it down.

Protester convoy

We managed to get most of the way to the airport before traffic stopped moving. Bek and I had to put our backpacks on and walk the last km to the terminal. It was pretty exciting actually. We were so lucky to get our fight, because not long after, the protesters stormed the terminal and all flights got shut down. It must have been only an hour or two after we left.

Their aim was to stop their prime minister from getting into the country, from Peru where the APEC summit was held. It seems they don’t like him very much… and it’s a long story.

A nine hour flight and we touched down in Melbourne. First stop, the Classic Curry Company

All the photos have been uploaded to my Google Picasa account.