ZFS on Linux

ZFS is a fantastic filesystem developed by Sun. Compared to other filesystems, it’s quite interesting as it combines both a filesystem and a logical volume manager. This allows you to get great flexibility, features and performance. It supports things like integrated snapshots, native NFSv4 ACL support and clever data integrity checking.

I’m now running a HP ProLiant MicroServer N36L which is a small NAS unit containing a 4-bay SATA enclosure. It has a low-performance AMD CPU, and comes with 1GB RAM and a 250GB harddisk. I’ve upgraded mine to 4GB of RAM and 4 x 2TB Seagate Barracuda drives.

The benefit of these units are that they’re a standard x86 machine allowing you to easily install any OS you like. They’re also really cheap and often have cash-back promotions.

I bought mine when I was in the UK and I brought it back with me to Australia. I waited until I got back to upgrade it so save me the trouble of shipping the extra harddisks on the ship.

In this post, I’ll document how to easily install ZFS on Debian Wheezy and some basic ZFS commands you’ll need to get started.


UPDATE: ZFS on Linux now has their own Debian Wheezy repository! http://zfsonlinux.org/debian.html

Install the ZFS packages

# apt-get install debian-zfs

This should use DKMS to build some new modules specific to your running kernel and install all the required packages.

Pull the new module into the kernel
# modprobe zfs

If all went well, you should see that spl and zfs have been loaded into the kernel.


Prepare disks

ZFS works best if you give it full access to your disks. I’m not going to run ZFS on my root filesystem, so this makes things much simpler.

Find our ZFS disks. We use the disk ID’s instead of the standard /dev/sdX naming because it’s more stable.
# ls /dev/disk/by-id/ata-*
lrwxrwxrwx 1 root root 9 Jan 21 19:18 /dev/disk/by-id/ata-ST2000DM001-1CH164_Z1E1GYH5 -> ../../sdd
lrwxrwxrwx 1 root root 9 Jan 21 08:55 /dev/disk/by-id/ata-ST2000DM001-9YN164_Z1E2ACRM -> ../../sda
lrwxrwxrwx 1 root root 9 Jan 21 08:55 /dev/disk/by-id/ata-ST2000DM001-9YN164_Z1F1SHN4 -> ../../sdb

Create partition tables on the disks so we can use them in a zpool:
# parted /dev/disk/by-id/ata-ST2000DM001-9YN164_Z1E2ACRM mklabel gpt
# parted /dev/disk/by-id/ata-ST2000DM001-9YN164_Z1F1SHN4 mklabel gpt
# parted /dev/disk/by-id/ata-ST2000DM001-1CH164_Z1E1GYH5 mklabel gpt


Create a new pool

ZFS uses the concept of pools in a similar way to how LVM would handle volume groups.

Create a pool called mypool, with the initial member being a RAIDZ composed of the remaining three drives.
# zpool create -m none -o ashift=12 mypool raidz /dev/disk/by-id/ata-ST2000DM001-1CH164_Z1E1GYH5/dev/disk/by-id/ata-ST2000DM001-9YN164_Z1E2ACRM/dev/disk/by-id/ata-ST2000DM001-9YN164_Z1F1SHN4

RAIDZ is a little like RAID-5. I’m using RAID-Z1, meaning that from a 3-disk pool, I can lose one disk while maintaining the data access.

NOTE: Unlike RAID, once you build your RAIDZ, you cannot add new individual disks. It’s a long story.

The -m none means that we don’t want to specify a mount point for this pool yet.

The -o ashift=12 forces ZFS to use 4K sectors instead of 512 byte sectors. Many new drives use 4K sectors, but lie to the OS about it for ‘compatability’ reasons. My first ZFS filesystem used the 512-byte sectors in the beginning, and I had shocking performance (~10Mb/s write).

See http://zfsonlinux.org/faq.html#HowDoesZFSonLinuxHandlesAdvacedFormatDrives for more information about it.

# zpool list
mypool 5.44T 1.26T 4.18T 23% 1.00x ONLINE -

Disable atime for a small I/O boost
# zfs set atime=off mypool

Deduplication is probably not worth the CPU overheard on my NAS.
# zfs set dedup=off mypool

Our pool is now ready for use.


Create some filesystems

Create our documents filesystem, mount and share it by NFS
# zfs create mypool/documents
# zfs set mountpoint=/mnt/documents mypool/documents
# zfs set sharenfs=on mypool/documents

Create our photos filesystem, mount and share it by NFS
# zfs create mypool/photos
# zfs set mountpoint=/mnt/photos mypool/photos
# zfs set sharenfs=on mypool/photos

Photos are important, so keep two copies of them around
# zfs set copies=2 mypool/photos

Documents are really important, so we’ll keep three copies of them on disk
# zfs set copies=3 mypool/documents

Documents are mostly text, so we’ll compress them.
# zfs set compression=on mypool/documents


ZFS pools should be scrubbed at least once a week. It helps balance the data across the disks in your pool and to fix up any data integrity errors it might find.
# zpool scrub <pool>

To do automatic scrubbing once a week, set the following line in your root crontab
# crontab -e
30 19 * * 5 zpool scrub <pool>

Coming soon is a follow-up to this post with some disk fail/recovery steps.

Posted in Linux at October 7th, 2013. 9 Comments.

Using pkgutil on Solaris with Puppet for easy package management

I’ve been using Puppet on Linux systems for some time now, but I’ve only just started using it in a Solaris environment.

I think one of the killer functions of Puppet is being able to easily install packages and manage services on a system. Most Linux distros these days have tools for working with repositories of packages, like Yum on Fedora/RedHat/CentOS and Apt on Debian and Ubuntu. These work really well with Puppet, because you can easily script a class which requires a specific package, and Puppet will just call the package tool and it’ll install the right package and all of the required dependencies.

Using Solaris feels like a step back from Linux, not having an official repository tool like Yum and Apt. Its package system seems quite primitive which can suffer from the dependency hell that we used to have with RPM before it was wrapped up with Yum. Enter: pkgutil.

Pkgutil is like Yum for Solaris, written in Perl by Peter Bonivart. It was designed for OpenCSW, which is a repository for Open Source packages on Solaris – and also the best place to install Puppet from. With a few simple steps, you can actually build an OpenCSW compatible repository of Solaris packages and tell pkgutil to use it, rather than the standard OpenCSW one.

Puppet has almost gained a proper package provider for Pkgutil (See Puppet issue #4258: Add pkgutil provider), which should be available in Puppet 2.6.4 maybe. In the mean time, we can just install it into our Ruby path to make use of it right now.

Steps involved are:

  • Install pkgutil
  • Install Puppet on Solaris
  • Install the pkgutil provider
  • Build an OpenCSW-compatible repository of your own packages
  • Define pkgutil as a provder in your Puppet configuration
  • Install some packages!

Install pkgutil

Before we do anything, we should install pkgutil. This handy one-liner will install it for Solaris 10 and OpenSolaris.

# pkgadd -d http://mirror.opencsw.org/opencsw/pkgutil-`uname -p`.pkg

For Solaris 8 and 9, take a look at the pkgutil installation page for more details.

Install Puppet

Now that pkgutil is installed, installing Puppet is a breeze!

# /opt/csw/bin/pkgutil --install puppet

This will resolve all the dependencies and install everything just like the Linux package management tools do.

Install the pkgutil provider

I’m using a version of pkgutil from Dominic Cleal’s git repository.

# wget --no-check-certificate https://github.com/domcleal/puppet/raw/143fc744a839affd328234fca26246d49d15d3d8/lib/puppet/provider/package/pkgutil.rb -O /opt/csw/lib/ruby/site_ruby/1.8/puppet/provider/package/pkgutil.rb

This wget will download it, and copy into the right place in the filesystem for Puppet to pick it up.

Build an OpenCSW-compatible repository

As part of OpenCSW, Peter Bonivart has released a tool for creating OpenCSW repositories, called bldcat. You can find it as part of the pkgutilplus package from OpenCSW.

Create yourself a new directory for your packages on your webserver. For me, I needed OpenSolaris 2009.06 and Solaris 10 support, so:

# mkdir -p repo/solaris/i386/5.11/
# mkdir -p repo/solaris/i386/5.10/

Then just put all your packages into that directory, and run bldcat:

# bldcat .

This will generate the catalog, and descriptions file needed for pkgutil. Once you make this directory available by HTTP, you can add the URL into the pkgutil.conf file.

One thing to remember is that you’ll need to do this on a Solaris machine. Although bldcat will work on Linux, it requires some of the Solaris package tools, which won’t be available. For me, I just did it NFS mounted from a Linux server.

Now, set the mirror and noncsw entries like this:


For my situation, I had to include a few packages that we provided as our standard environment, and the package names weren’t prefixed with CSW, to the ‘noncsw’ option needs to be set.

Because I wanted a mix of OpenCSW packages and our corporate standard packages, I copied in the OpenCSW packages (and dependencies) along with the corporate ones into the one repository. You can put Puppet in there also.

NOTE: All your packages need to be *.pkg.gz format, so make sure you compress any packages that aren’t already gzipped!

Define pkgutil as a provider in your Puppet configuration

In the site.pp file on my Puppet Master, I include this definition:

Package {
    provider => $operatingsystem ? {
        redhat => yum,
        centos => yum,
        sles => zypper,
        solaris => pkgutil,

To see this in action, I’ve used Nagios’s NRPE as an example.

package { nrpe_package:
  name => $operatingsystem ? {
    Solaris => 'CSWnrpe'
    CentOS  => 'nrpe',
    SLES    => 'nagios-nrpe',
    Debian  => 'nagios-nrpe-server',
  ensure => installed,

So with pkgutil, installing packages on Solaris can be as easy as Linux with Puppet.

Posted in Geek at December 10th, 2010. 6 Comments.

Tram Hunter: the blog post

I think this post has been a long time in the making, but I thought it might be time to share this little story.

Tram Hunter is a project I started nearly 2 years ago. It’s an Android client to the Yarra Trams TramTracker web service, which their iPhone client leaverages to provide real-time tram arrival information to users of trams in Melbourne.

I’m not sure what it is about Trams, but I’m almost enchanted by them. They’re slow, many are really old and usually it’s a pretty rough ride, but they also have much more character than buses and trains.

A friend and I made a mashup of Google Maps with tram stops once, and using timetable information, we plotted approximated locations of trams along a line. The trams even moved along the line, although it wasn’t really realistic, it was fun to watch. I spoke to Yarra Trams about what we had done, and I was invited to come and see the Operations Centre in South Melbourne, which was quite interesting. They offered me a job working with their development team on some .NET/Windows web services stuff (which turned out to be the TramTracker service), but I just couldn’t leave VPAC at the time.

Tram Hunter Stop Details

Real Time Departures

Tram Hunter Menu

Application Menu

So once Android was finally released, I bought their ADP1 development phone as quickly as I could. It cost a fortune, as the Australian dollar was quite weak at the time, but was pretty exciting. The idea of an Open Source phone to finally kick start some innovation in the mobile industry really appealed to me. I started messing with the Android API soon after.

I started working on Tram Hunter but got a bit stuck. I ended up shelving the project because I couldn’t figure out a problem I had, and moved on to other projects. It wasn’t until later (and I had moved to London), I was speaking to a friend of mine who was doing some Android development and he offered to help with the project. I proceeded to clean up the code, so it was in a compile-able state for someone else to look at. Somehow I managed to solve the issue and get something working. Everything seemed to just fall into place, and I had a working first version done.

I came across another project by accident by a couple of guys looking to do the same thing. I emailed them, and suddenly we had three developers and another joined soon after. I opened a Google Code project, put all our stuff into SVN and released version 0.1 to the Android Market. I later started a Google Groups mailing list for the project also.

The Tram Tracker iPhone application is slow and takes many taps to get to the information you want. Their interface has been designed to mimic the information screens at tram stops which is a nice idea, but actually provides an irritating user experience.

In comparison, the goal of Tram Hunter is to bring as many useful features as we can, without compromising the interface. I wanted to provide users the ability to get the information they want, with the least amount of clicks.

By using all the standard Android UI features, we gain a lot without needing to write a lot of code. Google Maps, location information by GPS, Network and Wifi, UI and search are all provided in the API so we don’t need to write this stuff ourselves. It also means it’s fast and simple.

Since the first version, we’ve introduced a few new features and have been fixing bugs. We’re on version 0.5 right now, and there’ll be a new one just around the corner.

The latest stats from the Android Market show 4325 total installs, with 3128 active installs (72%). Not bad considering the slow uptake of Android in Melbourne, and the limited number of tram users in Melbourne.

In version 0.4 of Tram Hunter, I introduced some code which (when only specifically enabled by the user) would send some usage information to a Google App Engine site I have set up. Tram Hunter will provide information about the user’s handset and Tram Hunter settings (e.g. What device is being used, what version of Tram Hunter is installed, which mobile network are we using, etc). It will also send information about which stops a user is requesting, and their location when they make the request.

Melbourne Heat Map

I’m currently in the process of generating heat maps, to indicate the location of Tram Hunter requests. Unfortunately, the code isn’t finished so I can’t release them out in the open yet. I have some Google App Engine bit to sort out first, but I’ll be releasing all the interesting statistics to the Android community.

UPDATE: The heat map is now running well on App Engine. The totally new Tram Hunter web site is now up and running with lots of cool graphs and stuf.

What’s next?

For Tram Hunter, I’m still taking feature requests and bug reports at our issue tracker, but I think development of this is starting to slow down.

I have been throwing around the possibility of porting it to Maemo/Meego to support the Nokia N900 (although something similar already exists) and possibly to BlackBerry devices. BlackBerry also uses Java, so it should be quite easy to reuse a lot of code.

I’m also looking into developing another application for timetable information. I have had many requests for an app for buses and trains, so I’m looking to leaveraging some Google Transit code and proving users with an ability to download specially formatted timetables to their handset and use many of the features of Tram Hunter, but in an offline fashion. The idea is that it’ll be generic enough that it can be used for any type of timetable information for anywhere in the world, as long as people are willing to help port the timetable information.

Posted in Geek at September 14th, 2010. 5 Comments.

Automating Debian installs with Preseeding

Following on from my post about building Debian virtual machines with libvirt, I’ve now got automated installations of Debian Lenny using the preseeding method. Coupling this with using virt-install, I can have a Debian virtual machine installation in only a few minutes. No questions asked.

The virt-install command contains an extra-args argument, where you can fill-in the specific parts of the preseeding. I don’t want to set an IP address in the file as it’s going to be used to build lots of machines, so I just specify that at install time. The URL part is where out preseed config file is stored. This obviously means that the machine needs to able to contact with webserver at install time to download the config.

$ NAME=debian-test
virt-install --name=${NAME} \
--ram=512 --file=/var/lib/xen/images/${NAME}.img \
--file-size 8 \
--nographics \
--paravirt \
--network=bridge:br0 \
--location=http://mirrors.uwa.edu.au/debian/dists/lenny/main/installer-i386 \
--extra-args="auto=true interface=eth0 hostname=${NAME} domain=vpac.org netcfg/get_ipaddress= netcfg/get_netmask= netcfg/get_gateway= netcfg/get_nameservers= netcfg/disable_dhcp=true url=http://webserver/preseed.cfg"

To get an idea of the contents of the preseed config file, the best place to start is the Debian stable example preseed file. It lists lots of different options, with plenty of comments so you can understand what’s going on.

For me to get a fully-automated install, I used these options. It’s fairly standard, but definitely worth reading the comments about each line.

$ egrep -v "(^#|^$)" preseed.cfg
d-i debian-installer/locale string en_AU
d-i console-keymaps-at/keymap select us
d-i netcfg/choose_interface select eth0
d-i netcfg/disable_dhcp boolean true
d-i netcfg/dhcp_options select Configure network manually
d-i netcfg/confirm_static boolean true
d-i mirror/protocol string http
d-i mirror/country string manual
d-i mirror/http/hostname string mirrors.uwa.edu.au
d-i mirror/http/directory string /debian
d-i mirror/http/proxy string
d-i clock-setup/utc boolean true
d-i time/zone string Australia/Melbourne
d-i clock-setup/ntp boolean true
d-i clock-setup/ntp-server string ntp.vpac.org
d-i partman-auto/method string regular
d-i partman-lvm/device_remove_lvm boolean true
d-i partman-md/device_remove_md boolean true
d-i partman-lvm/confirm boolean true
d-i partman-auto/choose_recipe select atomic
d-i partman/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i passwd/make-user boolean false
d-i passwd/root-password-crypted password [MD5 Sum of the password]
tasksel tasksel/first multiselect standard
d-i pkgsel/include string openssh-server vim puppet
popularity-contest popularity-contest/participate boolean false
d-i grub-installer/only_debian boolean true
d-i grub-installer/with_other_os boolean false
d-i finish-install/reboot_in_progress note

Some good resources I found, which might help you are:

Posted in Geek, Linux, Work at September 18th, 2009. 3 Comments.

iView for XBMC plugin v0.2

A plugin for ABC iView on XBMC has been released. See this page for progress of ABC iView on XBMC.

I just rewrote the iView plugin for XBMC.. and it’s far more robust. There is still lots to finish, but it kinda works. This was spurred on by someone actually trying it out, and then me finding out that ABC changed their XML.

You’ll need the RTMP patch for XBMC, and version 0.2 of the ABC iView plugin for XBMC.

Some things that still need improving are:

  • Auth token still times out. That means that if you watch something, you’ll need to go back to the channels list and then back into the channel to list the programs again and get a new auth token. Annoying.
  • No thumbnails or extended metadata available for channels or programs.
  • Some programs have funny names. Pretty minor, but annoying.
  • Programs are streamed in 4:3, but are actually produced in 16:9. I set XBMC to 16:9 Stretch mode.

For more info about the plugin, see this other entry I wrote.

Don’t forget to vote for an iView plugin for Boxee at the Customer Support Community for boxee. It might might help get iView into Boxee!

Posted in Geek at May 3rd, 2009. 28 Comments.

ABC’s iView on XBMC.. update 2

A plugin for ABC iView on XBMC has been released. See this page for progress of ABC iView on XBMC.

Following on from the last post about using rtmpdump to grab ABC’s iView programs, I’ve made a start on an XBMC plugin.. with the hope of eventually working on a Boxee plugin also.

To start with, you’ll need my patch to all you to specify the tcurl of an rtmp stream from with the XBMC API. This is needed because XBMC makes some assumptions about RTMP urls, based on other streams like Hulu and BBC’s iPlayer. ABC’s method is similar, but a little different. I’ll be trying to get the patch sent upstream, but it may need a little more work.

Now you’re going to have to compile XBMC yourself from source. I’ve only done it on Linux, so I can’t help you with Mac, Windows or Xbox versions. For information about compiling it on Ubuntu, you can check out the page on the XBMC wiki. You just need to do ‘cd’ into the XBMC directory you did your SVN checkout on, and then:

patch -p0 < /path/to/abc-iview-rtmp-tcurl-fix.patch

Hopefully you shouldn’t see any errors.

You can then grab my very basic iView plugin for XBMC. It’ll need to be extracted into your plugins/video directory of your XBMC installation.

This plugin has some serious limitations right now..

Firstly, some shows are listed as just ‘Episode 1’. It seems that in the XML files describing the shows, the data is very inconsistent. I’ll be looking at this in the next version of the plugin.

Next, because of the nature of the auth token that is generated, if you watch a program and then go back to the list of programs, if you try another, it will fail to play, as the token has timed out. You need to go back another level to the channels, then click the channel you want. This means that the URLS listed will generate a new token which will be valid again.

Last, the shows are all broadcasted in 16:9 on the TV, but streamed at 640×480 (4:3). This is really silly, but you can fix it by setting your XBMC view to use ‘Stretch 16:9’. Not ideal, but I’ll be looking into automatically setting the view if it’s exposed in the XBMC API.

It’s still very rough, but a start. Boxee has just announced a new API which I’ll be taking a look at shortly.

UPDATE: Version 0.2 of the plugin is out. See here.

Posted in Geek at April 15th, 2009. 8 Comments.

ABC’s iView on XBMC.. update 1

A plugin for ABC iView on XBMC has been released. See this page for progress of ABC iView on XBMC.

I’ve done a little bit of work since my last post on this, and a couple of people have asked for my stuff. Here goes.

Firstly, you can use RTMPdump to download the iView stream on your Linux box. You’ll need to download rtmpdump 1.4 and compile it yourself. It should just take a ‘make’ as long as you have all the requirements.

When iView starts, it first requests an XML config file, from the URL http://www.abc.net.au/iview/iview_config.xml

<?xml version="1.0" encoding="utf-8"?>
<param name="authenticate_path"   value="" />
<param name="media_path"          value="flash/playback/_definst_/" />
<param name="media_path_mp4"      value="flash:mp4/playback/_definst_/" />
<param name="server_streaming"    value="rtmp://cp53909.edgefcs.net/ondemand" />
<param name="server_speedtest"    value="rtmp://cp44823.edgefcs.net/ondemand" />
<param name="xml_help"            value="iview_help.xml" />
<param name="xml_channels"        value="iview_channels.xml" />
<param name="xml_series"          value="http://www.abc.net.au/playback/xml/rmp_series_list.xml" />
<param name="xml_thumbnails"      value="http://www.abc.net.au/playback/xml/thumbnails.xml" />
<param name="xml_classifications" value="http://www.abc.net.au/playback/xml/classifications.xml" />
<param name="xml_feature"         value="http://www.abc.net.au/playback/xml/iview_feature.xml" />
<param name="xml_feature_home"    value="http://www.abc.net.au/playback/xml/iview_homepage.xml" />
<param name="server_time"         value="http://www.abc.net.au/iview/time.htm" />
<param name="thumbs_path"         value="http://www.abc.net.au/playback/thumbs/" />
<param name="base_url"            value="http://www.abc.net.au/iview" />
<param name="channel_id_arts"     value="2260366" />
<param name="channel_id_news"     value="2186765" />
<param name="channel_id_docs"     value="2176127" />
<param name="channel_id_shop"     value="2186639" />
<param name="channel_id_catchup"  value="2172737" />
<param name="channel_id_kazam"    value="2288241" />
<param name="channel_id_faves"    value="2478452" />
<param name="channels_main"       value="catchup,news,docs,arts,shop" />
<param name="channels_kids"       value="kazam,faves" />

From this file, you can find out which other XML files you need for the channels and program descriptions. Firstly though, you need a special token, which is like an authorisation string. It’s done by doing a HTTP GET on the authenticate_path, which will return something like:

<?xml version="1.0" encoding="utf-8"?>
<iview xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="iview.abc.net.au">
<desc>iiNet Limited</desc>
<server />
<bwtest />
<text>iView is unmetered for &lt;a href="http://www.iinet.net.au/" target="_blank"&gt;iiNet&lt;/a&gt; customers.</text>

This is doing a lookup of my IP address, to ensure I’m in Australia, and pass me the token. The token has a short lifetime also, only a few minutes. You then need this token to help you build the URL to request the video stream you want.

To find the programs of a particular channel, you need to grab a URL like this: http://www.abc.net.au/playback/xml/output/catchup.xml.

<?xml version="1.0"?>
<rmp-content xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<channel enabled="true" id="2172737">
<name>ABC CatchUp</name>
<description><![CDATA[Recent best of ABC1 & ABC2 TV]]></description>
<image id="258433" order="1">
<title><![CDATA[ABC Catchup Background 09]]></title>
<version id="1071615">
<image id="257912" order="2">
<title><![CDATA[ABC Catchup background 06]]></title>
<version id="1068909">
<program-title id="352699" promo="false" order="9">
<title><![CDATA[Catalyst Series 10 Episode 8]]></title>
<synopsis><![CDATA[Malaria jumps the gap from monkey to man; could bubbles be a solution to the hard hit  mining industry? And see how a horse trainer applies his skill to the training of elephants, with remarkable success.]]></synopsis>
<publish-date>03/04/2009 12:00:00</publish-date>
<expire-date>17/04/2009 00:00:00</expire-date>
<transmission-date>02/04/2009 00:00:00</transmission-date>
<website>Go to website</website>
<category>Science and Technology</category>
<video-asset id="1619127" order="0">
<program-title id="....">
...more programs...

I’ve shortened the output of the ‘Catch Up’ channel here. This is what you’re likely to see when you get the channel XML file. As you can see, this is describing an episode of Catalyst, and I’ve marked the URL in bold.

TOKEN=`curl -q | grep token | sed 's/<token>//g' | sed 's/\&amp;/\&/g' | sed 's,</token>,,g' | sed 's/ //g'`; ./rtmpdump --rtmp "rtmp://" --auth "auth=${TOKEN}" -t "rtmp://cp53909.edgefcs.net/ondemand?auth=${TOKEN}" -o test.flv

This horrible command is getting the token, and stripping out all unncessesary characters, and then passing it onto rtmpdump. You might have also noticed in the command above, I have four slashes in the RTMP url. This is to work around some assumptions that rtmpdump makes about the path. I had made a patch, but in rtmpdump 1.4, you can just use 4 slashes to make it work.

Most of this data came from doing Wireshark packet traces while working with the flash-based iView interface. Also important to note that the programs have an expiry date also. If the command above returns a ‘stream not found’ message, you’ll probably need a newer episode.

In the next post, I’ll be posting the code for the XBMC plugin.

Posted in Geek at April 15th, 2009. 10 Comments.

ABC iView on XBMC/Boxee

A plugin for ABC iView on XBMC has been released. See this page for progress of ABC iView on XBMC.

I think it would be really neat to use ABC’s iView on the Xbox Media Centre (XBMC) and/or Boxee. Honestly, who really wants to watch TV on their computers? Haven’t we evolved from that now?

I’ve got a modded XBOX running XBMC, and I have various Linux boxes running XBMC and Boxee and I think they’re the perfect platform for something like iView.. especially because it’s unmetered traffic on iiNet, Internode and other great ISP’s.

I did a little research, and they seem to use Adobe’s Real Time Message Protocol (RTMP) to stream the video from their server to the iView client, which is written in Flash. Recent versions of XBMC and Boxee have code to support RTMP, which is also used by other digital content providers like NBC’s Hulu, and the BBC’s iPlayer.

I have managed to work out most of the iView’s XML stuff, which describes channels, programs, thumbnails, etc but not quite got there with the actual streaming part. I’m playing with rtmpdump, which is based on the rtmp code from XBMC/Boxee, and have almost worked out the URL part to get the server to stream. I just keep getting a message about not being able to find the specified stream.

If anyone out there on the interwhizzle has worked this stuff out, I’d love to hear from them. My googling hasn’t really revealed anything like what I’m after. If you’re interested in using iView on XBMC or Boxee, I’d like to hear from you also.

UPDATE: Please vote for an iView plugin for Boxee at the Customer Support Community for boxee. It might might help get iView into Boxee!

UPDATE 2: I’ve uploaded a basic plugin for XBMC. See: http://www.andybotting.com/wordpress/iview-for-xbmc-plugin-v02 for more info.

Posted in Geek at February 27th, 2009. 13 Comments.

Running a Debian Lenny DomU under a CentOS 5 Dom0

The aim of this was to use the standard CentOS/RHEL Xen Dom0 tools to boot a Debian Lenny DomU.

I found plenty of instructions for doing CentOS DomU under a Debian Dom0, but not the other way around. So, this is a little how-to documenting the little things that need to be overcome.

I also wanted the Debian virtual machines to have their filesystems in a file, in the same standard way that the RHEL virt-install creates.

Steps involved:

  • Use virt-install to build a standard CentOS virtual machine
  • Use debootstrap to build a Debian Lenny base install for transplanting
  • Break apart a CentOS filesystem-in-a-file, and move the Debian install into it
  • Modify Debian config for booting the CentOS kernel

Use virt-install to build a standard CentOS virtual machine

I created a new virtual machine, using virt-install.

virt-install -n newvm -r 512 -f /var/lib/xen/images/debian.img -s 8 -l http://ftp.monash.edu.au/pub/linux/CentOS/5/os/i386/ -p --nographics -x

I needed some CentOS virtual machines anyway, so I let the install go through and do its thing. If you didn’t need it, you could probably kill the install before it started installing packages. We just needed the config file for the VM in /etc/xen and the filesystem image.

Use debootstrap to build a Debian Lenny base install for transplanting

I actually had a Debian Xen Dom0 with the xen-tools package installed. I used this to create a new Debian Lenny install, and also do some of the nice hook scripts with you would otherwise have to do by hand.

# xen-create-image --hostname=vanila --size=8Gb --dist=lenny --memory=512M --ide --dhcp

This meant I had a hostname file, libc6-xen and other things already done for me.

This was installed into an LVM partition, so after building it, I mounted the LVM partition, and used tar to compress it.

# mount /dev/mapper/vg-vanilla--disk /mnt
# tar zc -C /mnt/ . > /tmp/vanilla-debian.tar.gz

Break apart a CentOS filesystem-in-a-file, and move the Debian install into it

Set up the loop device
# losetup -f /var/lib/xen/images/debian.img

Map the partitions inside the loop device
# kpartx -av  /dev/loop0
add map loop0p1 : 0 208782 linear /dev/loop0 63
add map loop0p2 : 0 16032870 linear /dev/loop0 208845

At this point, you should have /dev/mapper/loop0p1 which is the root filesystem of your new VM. You’ll need to format the filesystem with:
# mkfs.ext3 /dev/mapper/loop0p1

Mount the newly formatted filesystem
# mount /dev/mapper/loop0p1 /mnt

Extract our Debian Lenny install into the filesystem
# cd /mnt
# tar xf /tmp/vanilla-debian.tar.gz

Modify Debian config for booting the CentOS kernel

We want to use CentOS/RHEL’s pygrub bootloader, just because it’s nice.

First, you’ll need to copy the CentOS kernel into your Debian install. You’ll need the config, kernel and initrd files from /boot of a DomU (or maybe the Dom0..)
# cd /boot
# cp config-2.6.18-92.1.22.el5xen vmlinuz-2.6.18-92.1.22.el5xen initrd-2.6.18-92.1.22.el5xen.img /mnt/boot

Rename the initrd to drop the .img from the end. It doesn’t work with the update-grub script in Debian
# mv initrd-2.6.18-92.1.22.el5xen.img initrd-2.6.18-92.1.22.el5xen

Copy the kernel modules to your new VM too:
# cp -r /lib/modules/2.6.18-92.1.22.el5xen /mnt/lib/modules

If you don’t have a /boot/grub directory in your Debian DomU, then you’ll need create one. You only really need three files: menu.lst and device.map. We’ll need to set it up properly so that both the update-grub script in Debian and the pyGrub bootloader work happily.

Edit the /boot/grub/device.map file. Make sure your hd0 is set to /dev/xvda:
(hd0)   /dev/xvda

The pyGrub script reads grub.conf, and not menu.lst, so symlink it
cd /boot; ln -s menu.lst grub.conf

Here’s the contents of my /boot after I’m finished:

You’ll need to fix your inittab to use the xvc0 as your console. If you don’t you lose access to log into the console. In the file /etc/inittab, edit the tty1 line to be xvc0 instead.
1:2345:respawn:/sbin/getty 38400 xvc0

Your first tty should be changed to xvc0, and the others (tty2-6) should be commented out (if they’re not already)

Unmap the partitions and destroy our loop device
# kpartx -d /dev/loop0
# losetup -d /dev/loop0

Start the new Debian Lenny virtual machine

# xm create -c debian

You should see PyGrub come up, and let you pick the kernel.
pyGRUB version 0.6
| Debian GNU/Linux, kernel 2.6.18-92.1.22.el5xen                         |
| Debian GNU/Linux, kernel 2.6.18-92.1.22.el5xen (single-user mode)      |
|                                                                        |
|                                                                        |
|                                                                        |
|                                                                        |
|                                                                        |
|                                                                        |
Use the ^ and v keys to select which entry is highlighted.
Press enter to boot the selected OS. 'e' to edit the
commands before booting, 'a' to modify the kernel arguments
before booting, or 'c' for a command line.

Will boot selected entry in  4 seconds

Hopefully, it works for you too 🙂

I’ve made one vanilla debian install, and just make a copy of that image file for each new VM I need to create. I have eth0 in the interfaces file commented out, so I just put the new IP in, set the hostname and I’m ready to go.

I may have missed a step in here, so if you’re trying this out, please comment to let us know how it goes.

Posted in Geek, Work at February 26th, 2009. 4 Comments.

How do you clone an LVM partition?

It’s actually more difficult than you might think. From the bit of googling that I did, it seems that you can’t just ‘clone’ and LVM logical volume, while it’s running.

One method I found was to use the ‘snapshot’ feature of LVM to create a ‘frozen image’ copy of the logical volume, which is then suitable for copying to a new logical volume, while leaving the original intact.

Here’s our original logical volume that we want to clone.

# lvdisplay
--- Logical volume ---
LV Name                /dev/vg/host-disk
VG Name                vg
LV UUID                UK1rjH-LS3l-f7aO-240S-EwGw-0Uws-5ldhlW
LV Write Access        read/write
LV Status              available
# open                 1
LV Size                9.30 GB
Current LE             2382
Segments               1
Allocation             inherit
Read ahead sectors     0
Block device           254:0

Let’s now create our snapshot logical volume. For the size, it should only need 10 – 20% of the original size, as we’re only mirroring the real volume.

# lvcreate --size 2G --snapshot --name host-disk-snap /dev/vg/host-disk

Let’s take a look at our new volume

# lvdisplay
--- Logical volume ---
LV Name                /dev/vg/host-disk
VG Name                vg
LV UUID                UK1rjH-LS3l-f7aO-240S-EwGw-0Uws-5ldhlW
LV Write Access        read/write
LV snapshot status     source of /dev/vg/host-disk-snap [active]
LV Status              available
# open                 1
LV Size                9.30 GB
Current LE             2382
Segments               1
Allocation             inherit
Read ahead sectors     0
Block device           254:0
--- Logical volume ---
LV Name                /dev/vg/host-disk-snap
VG Name                server1
LV UUID                9zR5X5-OhM5-xUI0-OolP-vLjG-pexO-nk36oz
LV Write Access        read/write
LV snapshot status     active destination for /dev/vg/host-disk
LV Status              available
# open                 1
LV Size                9.30 GB
Current LE             2382
COW-table size         10.00 GB
COW-table LE           2560
Allocated to snapshot  0.01%
Snapshot chunk size    8.00 KB
Segments               1
Allocation             inherit
Read ahead sectors     0
Block device           254:5

From the output, you should be able to see that we’ve now got some snapshot fields shown in our output. We’ll create another logical volume, which will be our final target for our new virtual machine.

# lvcreate --size 10G --name newhost-disk vg

With our source and target partitions ready to go, we need to begin copying the data. You have some choices here, depending on your setup.

If you’re using the same size partitions you could use dd, or even xfs_copy if you’re using XFS.

If you’re like me, I wanted the new target partition to be a smaller size than the original. Also, if you wanted to use a different filesystem, the only real way to do it is to copy the files.

We’ll need to make the new filesystem on our target partiton

# mkfs.xfs /dev/vg/newhost-disk

and mount our filesystems

# mkdir /mnt/host-disk-snap
# mount -o ro /dev/vg/host-disk-snap /mnt/host-disk-snap
# mkdir /mnt/newhost-disk
# mount /dev/vg/newhost-disk /mnt/newhost-disk

I wasn’t sure about how changes to the filesystem would affect the original, so I thought I’d stick to the safe side, and mount it as read-only.

Change into the source filesystem

# cd /mnt/host-disk-snap

Using a mix of find and cpio, copy the files

# find . -mount -print | cpio -pdm /mnt/newhost-disk

Wait a few minutes, depending on your filesystem size, and you’re done.

When you’re satisfied, you can just use lvremove to remove your snapshot volume.

# umount /mnt/host-disk-snap
# lvremove /dev/vg/host-disk-snap

After all that, you should finally have a cloned filesystem to use. I’m sure there’s an easier way, but this worked for me.

Posted in Linux at October 17th, 2008. 13 Comments.