Thus, I present a VIM plugin called TangledStrings, which I’m distributing as a Vimball (.vba) you can download from its project page on Launchpad. For more information on Vimball formatted plugins, see this page. To install the plugin, simply:

• vim tangledstrings.vba
• Follow the instructions from Vimball to type: :so %

By default, TangledStrings will show a (configurable) warning message when you load a Puppet-owned file:

This message can be disabled, and you can choose to enable a persistent message in the VIM status line instead:

(or you could choose to enable both of these methods).

For more information, see the documentation included in the Vimball which you can display with the VIM command:

:help TangledStrings

Suggestions, improvements, patches, etc. are most welcome! Email me or use Launchpad to file bugs and propose merges.

I’m sure that a more seasoned Puppet veteran would have had this running in no time, but since it introduced some new concepts for me, I thought I’d throw up some notes of how I’m using it. I was particularly interested in an example usage I saw which lists the puppet modules a system is using in its /etc/motd, but because of the way Ubuntu handles constructing the motd, I needed to slightly rework the example. In Ubuntu, the /etc/motd file is constructed dynamically when you log in – this is done by pam_motd which executes the scripts in /etc/update-motd.d/. One of those scripts (99-footer) will simply append the contents of /etc/motd.tail to /etc/motd after everything else – my example will take advantage of this. If you are already using motd.tail, you could just have this puppet system write to a different file and then drop another script into /etc/update-motd.d/ to append the contents of that different file.

This is what I did:

• git clone http://github.com/ripienaar/puppet-concat.git
• Move the resulting git branch to /etc/puppet/modules/concat and add it to my top-level site manifest that includes modules
• Create a class to manage /etc/motd.tail. In my setup this ends up being /etc/puppet/manifests/classes/motd.pp, which is included by my default node, but your setup is probably different. This is what my class looks like:

class motd {
       include concat::setup
       $motdfile = "/etc/motd.tail"

       concat{$motdfile:
               owner => root,
               group => root,
               mode => 644
       }

       concat::fragment{"motd_header":
               target => $motdfile,
               content => "\nPuppet modules: ",
               order => 10,
       }

       concat::fragment{"motd_footer":
               target => $motdfile,
               content => "\n\n",
               order => 90,
       }
}

# used by other modules to register themselves in the motd
define motd::register($content="", $order=20) {
   if $content == "" {
      $body = $name
   } else {
      $body = $content
   }

   concat::fragment{"motd_fragment_$name":
      target  => "/etc/motd.tail",
      content => "$body ",
      order => $order
   }
}

So that’s quite a mouthful. Let’s break it down:

• We have to include concat::setup so the concat module can…set… up :)
• We then set a variable pointing at the location of the file we want to manage
• We then instantiate the concat module for the file we want to manage and set properties like the ownership/mode
• We then call the concat::fragment function for two specific fragments we want in the output – a header and a footer (although I do this on a single line, so it’s the phrase “Puppet modules” and “\n\n” respectively). They’re forced to be header/footer by the “order” parameter – by making sure we use a low number for the header and a high number for the footer, we get the layout we expect.
• Outsite this class we define a function motd::register which other modules will call and the content they supply will be handed to concat::fragment with a default order parameter of 20 (which is higher than the value we used for the header and lower than the footer one).

Finally, in each of my modules I include the line:

motd::register{"someawesomemodule":}

and now when I ssh to a node, I see a line like:

Puppet modules: web ssh

It’s a fairly simple little thing, but quite pleasing and from here out it’s almost zero effort – just adding the motd::register calls to each module.

The first post is coming up immediately, I’m just writing this post as an opening bookend :)

Simple really, but it hadn’t occurred to me.

I’m in constant need of debugging information and in Terminator particularly where we have lots of objects all interacting and reparenting all the time. We’ve had a simple dbg() method for a long time, but I was getting very bored of typing out dbg(‘Class::method:: Some message about %d’ % foo), so I decided to see what could be done about inferring the Class and method parts of the message.

It turns out that python is very good at introspecting its own runtime, so back in January, armed with my own stupidity and some help from various folks on the Internet, I came up with the following:

# set this to true to enable debugging output
DEBUG = False
# set this to true to additionally list filenames in debugging
DEBUGFILES = False
# list of classes to show debugging for. empty list means show all classes
DEBUGCLASSES = []
# list of methods to show debugging for. empty list means show all methods
DEBUGMETHODS = []

def dbg(log = ""):
    """Print a message if debugging is enabled"""
    if DEBUG:
        stackitem = inspect.stack()[1]
        parent_frame = stackitem[0]
        method = parent_frame.f_code.co_name
        names, varargs, keywords, local_vars = inspect.getargvalues(parent_frame)
        try:
            self_name = names[0]
            classname = local_vars[self_name].__class__.__name__
        except IndexError:
            classname = "noclass"
        if DEBUGFILES:
            line = stackitem[2]
            filename = parent_frame.f_code.co_filename
            extra = " (%s:%s)" % (filename, line)
        else:
            extra = ""
        if DEBUGCLASSES != [] and classname not in DEBUGCLASSES:
            return
        if DEBUGMETHODS != [] and method not in DEBUGMETHODS:
            return
        try:
            print >> sys.stderr, "%s::%s: %s%s" % (classname, method, log, extra)
        except IOError:
            pass

How’s about that for shockingly bad? ;)
It also adds a really impressive amount of overhead to the execution time.
I added the DEBUGCLASSES and DEBUGMETHODS lists so I could cut down on the huge amount of output – these are hooked up to command line options, so you can do something like “terminator -d –debug-classes=Terminal” and only receive debugging messages from the Terminal module.

I’m not exactly sure what I hope to gain from this post, other than ridicule on the Internet, but maybe, just maybe, someone will pop up and point out how stupid I am in a way that turns this into a 2 line, low-overhead function :D

This website shares a server with various other network services that form the foundation of my online life (i.e. IRC and Email) and I’ve been running into capacity issues in the last few months, so I’m running an experiment whereby I upgrade to brand new hardware (Quad Core i7, 8GB of RAM) and partition the available resources across virtual machines so the various network services are isolated into logical security zones.

Whining

I have plenty of experience using Xen for this sort of thing, but that’s becoming more and more irrelevant in newer kernels/distributions. As much as I think that’s a shame and a stupid upstream decision, I can’t change it, so I need to move on to KVM and libvirt.

Resolution

So, with the beefy new server booted up in a -server kernel and a big, empty LVM Volume Group I got to work creating some virtual machines. This post is mainly a reminder to myself of the things I need to do for each VM :)

Action

These are the steps I used to make a VM with 1GB of RAM, 10GB / and 1GB of swap:

Create an LVM Logical Volume

lvcreate -L11G -n somehostname VolumeGroup

Create a VM image and libvirt XML definition

ubuntu-vm-builder kvm lucid --arch amd64 --mem=1024 --cpus=1

--raw=/dev/VolumeGroup/somehostname --rootsize=10240 --swapsize=1024

--kernel-flavour=server --hostname=somehostname

--mirror=http://archive.ubuntu.com/ubuntu/ --components=main,universe

--name 'Chris Jones' --user cmsj --pass 'ubuntu' --bridge virbr0

--libvirt qemu:///system --addpkg vim --addpkg ssh --addpkg ubuntu-minimal

Catchy command, huh? ;)

Wait

(building the VM will take a few minutes)

Modify the libvirt XML definition for performance

The best driver for disk/networking is the paravirtualised “virtio” driver. I found that ubuntu-vm-builder had already configured the networking to use this, but not the disk, so I modified the disk section to look like this:

<disk type='block' device='disk'>

  <source dev='/dev/VolumeGroup/somehostname'/>

  <target dev='vda' bus='virtio'/>

</disk>

Modify the libvirt XML definition for emulated serial console

<serial type='pty'>
  <target port='0'/>
</serial>
<console type='pty'>
  <target port='0'/>
</console>

Modify the libvirt XML definition for a better CPU

I’m running this on an Intel Core i7 (Nehalem), but libvirt’s newest defined CPU type is a Core2Duo, so we’ll go with that in the root of the <domain> section:

<cpu match='minimum'>

  <model>core2duo</model>

</cpu>

Import the XML definition into the running libvirt daemon

virsh define /etc/libvirt/qemu/somehostname.xml

Mount the VM’s root filesystem

mkdir /mnt/tmpvmroot

mount /dev/mapper/VolumeGroup-somehostnamep1 /mnt/tmpvmroot

Fix fstab in the VM

Configure serial console in the VM

# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.
start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]
respawn
exec /sbin/getty -L 115200 ttyS0 xterm

# defoptions=console=ttyS0 console=tty0

Unmount the VM’s root filesystem

umount /mnt/tmpvmroot

rmdir /mnt/tmpvmroot

Start the VM

virsh start somehostname

SSH into the VM

I didn’t specify any networking details to ubuntu-vm-builder, so the machine will boot and try to get an address from DHCP. By default you’ll have a bridge device for libvirt called virbr0 and dnsmasq will be running, so watch syslog for the VM getting its address.

ssh cmsj@192.168.122.xyz

you should now be in your VM! Now all you need to do is configure it to do things and then fix its networking. My plan is to switch the VMs to static IPs and then use NAT to forward connections from public IPs to the VMs, but you could bridge them onto the host’s main ethernet device and assign public IPs directly to the VMs.

Is most of my music collection from the 90s?

Obviously what I want to do here is examine the ID3 tags of the files in my music collection and see how they’re distributed. A quick search with apt showed that Ubuntu 10.04 has two python libraries for dealing with ID3 tags and a quick play with each suggested that the one with the API most relevant to my interests was eyeD3. After a few test iterations of the script I was getting bored of waiting for it to silently scan the roughly 4000 MP3s I have, so I did another quick search and found a progress bar library.

So that’s all of the motive and opportunity established, now let’s examine the means to the end. If you want to follow along at home, the whole script is here.

try:

 import eyeD3

 import progressbar as pbar

except ImportError:

print("You should make sure python-eyed3 and python-progressbar \

are installed")

 sys.exit(1)

First off this is the section where I’m importing the two non-default python libraries that I depend on. I want to provide a good experience when they’re not installed, so I catch the exception and tell people the Debian/Ubuntu package names they need, and exit gracefully. I rename the progressbar module as I import it just because “progressbar” is annoyingly long as a name, and I don’t like doing “from foo import *”.

Skipping further on, we find the code that extracts the ID3 year tag:

year = tag.getYear() or 'Unknown'

This is something I’m really not sure about the “correctness” of; One of the reasons I went with the eyeD3 library was that the getYear() method returns None if it can’t find any data, but I don’t really want to capture the result, then test the result explicitly and if it’s None set the value to “Unknown”, so I went with the above code which only needs a single line and is (IMHO) highly readable.

This is ultimately the crux of the entire program – we’ve now collected the year, so we can work out which decade it’s from:

if year is not 'Unknown':
 year = "%s0s" % str(year)[:3]

If this isn’t an unknown year we chop the final digit off the year and replace it with a zero. Job done!

Next up, another style question. Rather than store the year we just processed I want to know how many of each decade have been found, so the obvious choice is a dict where the keys are the decades and the values are the number of times each decade has been found. One option would be to pre-fill the dict with all the decades, each with a value of zero, but that seems redundant and ugly, so instead I start out with an empty dict. This presents a challenge – if we find a decade that isn’t already a key in the dict (which will frequently be the case) we need to notice that and add it. We could do this by pre-emptively testing the dict with its has_key() method, but that struck me as annoyingly wordy, so I went with:

try:

 years[year] += 1

except KeyError:

 years[year] = 1

If we are incrementing a year that isn’t already in the dict, python will raise a KeyError, at which point we know what’s happened and know the correct value is 1, so we just set it explicitly. Seems like the simplest solution, but is it the sanest?

The only other thing I wanted to say is a complaint – having built up the dict I then want to print it nicely, so I have a quick list comprehension to produce a list of strings of the format “19xx: yy” (i.e. the decade and the final number of tracks found for that decade), which I then join together using:

', '.join(OUTPUT)

which I hate! Why can’t I do:

OUTPUT.join(', ')

(where “OUTPUT” is the list of strings). If that were possible, what I’d actually do is tack the .join() onto the end of the list comprehension and a single line would turn the dict into a printable string.

So there we have it, my thoughts on the structure of my script. I’d also add that I’ve become mildly obsessive about getting good scores from pylint on my code, which is why it’s rigorously formatted, docstring-ed and why the variable names in the __main__ section are in capitals.

What are your thoughts?

Oh, and the answer is no, most of my music is from the 2000s. The 1990s come in second :)

We did a little digging and realised I had an icon theme cache file in /usr/local/ that was overriding the one in /usr/. A bit of deleting later and it’s back, but in the process we whipped up a little bit of python to print out the filename of an icon given an icon name.

#!/usr/bin/python
# gtk-find-icon by Chris Jones <cmsj@tenshu.net>
# Copyright 2010. GPL v2.

import sys
import gtk

THEME = gtk.IconTheme()
ICON = THEME.lookup_icon(sys.argv[1],
 gtk.ICON_SIZE_MENU,
 gtk.ICON_LOOKUP_USE_BUILTIN)

if not ICON:
 print "None found"
else:
 print(ICON.get_filename())

One of the features of the new 0.9x series of Terminator releases that hasn’t had a huge amount of announcement/discussion yet is the plugin system. I’ve posted previously about the decisions that went into the design of the plugin framework, but I figured now would be a good time to look at how to actually take advantage of it.

While the plugin system is really generic, so far there are only two points in the Terminator code that actually look for plugins – the Terminal context menu and the default URL opening code. If you find you’d like to write a plugin that interacts with a different part of Terminator, please let me know, I’d love to see some clever uses of plugins and I definitely want to expand the number of points that plugins can hook into.

The basics of a plugin

A plugin is a class in a .py file in terminatorlib/plugins or ~/.config/terminator/plugins, but not all classes are automatically treated as plugins. Terminator will examine each of the .py files it finds for a list called ‘available’ and it will load each of the classes mentioned therein.

Additionally, it would be a good idea to import terminatorlib.plugin as that contains the base classes that other plugins should be derived from.

A quick example:

import terminatorlib.plugin as plugin
available = ['myfirstplugin']
class myfirstplugin(plugin.SomeBasePluginClass):
  etc.

So now let’s move on to the simplest type of plugin currently available in Terminator, a URL handler.

URL Handlers

This type of plugin adds new regular expressions to match text in the terminal that should be handled as URLs. We ship an example of this with Terminator, it’s a handler that adds support for the commonly used format for Launchpad. Ignoring the comments and the basics above, this is ultimately all it is:

class LaunchpadBugURLHandler(plugin.URLHandler):
  capabilities = ['url_handler']
  handler_name = 'launchpad_bug'
  match = '\\b(lp|LP):?\s?#?[0-9]+(,\s*#?[0-9]+)*\\b'

  def callback(self, url):
    for item in re.findall(r'[0-9]+', url):
      return('https://bugs.launchpad.net/bugs/%s' % item)

That’s it! Let’s break it down a little to see the important things here:

• inherit from plugin.URLHandler if you want to handle URLs.
• include ‘url_handler’ in your capabilities list
• URL handlers must specify a unique handler_name (no enforcement of uniqueness is performed by Terminator, so use some common sense with the namespace)
• Terminator will call a method in your class called callback() and pass it the text that was matched. You must return a valid URL which will probably be based on this text.

and that’s all there is to it really. Next time you start terminator you should find the pattern you added gets handled as a URL!

Context menu items

This type of plugin is a little more involved, but not a huge amount and as with URLHandler we ship an example in terminatorlib/plugins/custom_commands.py which is a plugin that allows users to add custom commands to be sent to the terminal when selected. This also brings a second aspect of making more complex plugins – storing configuration. Terminator’s shiny new configuration system (based on the excellent ConfigObj) exposes some API for plugins to use for loading and storing their configuration. The nuts and bolts here are:

import terminatorlib.plugin as plugin
from terminatorlib.config import Config
available = ['CustomCommandsMenu']
class CustomCommandsMenu(plugin.MenuItem):
  capabilities = ['terminal_menu']
  config = None
  def __init__(self):
    self.config = Config()
    myconfig = self.config.plugin_get_config(self.__class__.__name__)
    # Now extract valid data from sections{}
  def callback(self, menuitems, menu, terminal):
    menuitems.append(gtk.MenuItem('some jazz'))

This is a pretty simplified example, but it’s sufficient to insert a menu item that says “some jazz”. I’m not going to go into the detail of hooking up a handler to the ‘activate’ event of the MenuItem or other PyGTK mechanics, but this gives you the basic detail. The method that Terminator will call from your class is again “callback()” and you get passed a list you should add your menu structure to, along with references to the main menu object and the related Terminal. As the plugin system expands and matures I’d like to be more formal about the API that plugins should expect to be able to rely on, rather than having them poke around inside classes like Config and Terminal. Suggestions are welcome :)

Regarding the configuration storage API – the value returned by Config.plugin_get_config() is just a dict, it’s whatever is currently configured for your plugin’s name in the Terminator config file. There’s no validation of this data, so you should pay attention to it containing valid data. You can then set whatever you want in this dict and pass it to Config().plugin_set_config() with the name of your class and then call Config().save() to flush this out to disk (I recommend that you be quite liberal about calling save()).

Wrap up

Right now that’s all there is to it. Please get in touch if you have any suggestions or questions – I’d love to ship more plugins with Terminator itself, and I can think of some great ideas. Probably the most useful thing would be something to help customise Terminator for heavy ssh users (see the earlier fork of Terminator called ’ssherminator’)

I dislike ordering hardware online because of the shipping lag and because I’m a sucker for the retail experience, so I was checking out which devices would work with Ubuntu and which devices were on sale in my local computer supermarket. The latter was a depressingly short list, and the former was getting annoying to search for, but I stumbled on the idea of a multi-function printer. It turns out that it’s cheaper to buy a scanner as part of a printer than it is to buy a scanner on its own (granted the resolution of the scanner isn’t quite as good, but it’s more than sufficient for my needs). The reason for this is undoubtedly that the manufacturers are expecting to make up their money by selling me ink cartridges every few months.

As I started to look at models of multi-function printers, one thing became apparent almost immediately – HP have done a lot of leg work on this. I quickly found a bunch of info on their site about how they basically support all of their stuff on Linux, including a page which specifically listed popular distros and which versions worked with which printers.

I decided pretty much immediately that I wanted to support this, so off I went to the shop to buy an HP. They had the decent looking F4580 for around £40, so I nabbed that and set off home.

When I got home I fired up my laptop running Lucid and plugged the new device in. Less than 10 seconds later I was told it was ready for printing, and I fired up Robert Ancell’s excellent new Simple Scan to see what configuration I would need to do to make that work…. the answer being none, it scanned a page first time.

Now smug with the ease with which that had worked I started installing the HP driver software on a popular proprietary operating system so I could use it to configure the printer’s WiFi feature (something I assumed I couldn’t do from within Ubuntu – an assumption that turns out to have been wrong). Ten minutes later it was still finishing off the install process, but eventually I did get the printer hooked up to our wireless network.

Back to the Lucid machine, I told it to add a new printer, it immediately saw the HP announcing itself on the network and let me quickly add that and I could print over wifi. Pretty nifty stuff.

Then I started poking around with HP’s Linux Imaging and Printing software (HPLIP) and noticed that there was an “hp-toolbox” that wasn’t installed. This is the tool I should have used to configure the wifi network on the printer; It also shows the ink levels and lets you kick off scanning/printing/cleaning type jobs. Out of sheer curiosity I went into hp-toolbox’s preferences and changed it from using xsane to simple-scan, and told it to start a Scan. I wasn’t expecting it to work because the device wasn’t connected via USB, but it turns out that not only does the device support scanning over WiFi, it works in Linux. It’s not quite as fast as a direct hookup, but it’s certainly significantly more convenient!

So there we have it, out of the box I was up and running within 10 seconds of plugging the device in, and if I’d known to just install hp-toolbox I would have had everything running wirelessly a few minutes later. This being compared to installing CDs and dealing with great gobs of driver/application mess (I’ve seen HP’s Windows drivers and it’s no fun trying to persuade them to update themselves, or to persuade them not to prompt you to register every week). A huge, epic victory for Linux and Ubuntu – and one that I seem to find with much random consumer hardware these days. A few years ago this post would have been full of complicated commands and scripts and compilation as I described how to make the device work, but now all I can do is be smug about how easy it was :D
Win.

Now a bunch of bug fixing and a config converter and we can release!
Thanks to everyone who has been testing so far.

• Fix a bug in handling URLs dropped on the window
• Implement directional navigation
• Implement geometry hinting
• Fix a bug in group emitting that cause “Broadcast off” and “Broadcast to all” to become inverted
• Implement WM_URGENT bell handler

I’m really happy with how this is going. All that is left to have feature parity with trunk, I think, is some keyboard shortcut handlers.

I’d still love to get more testing results to make sure I haven’t missed anything, but at this rate I’m expecting to be able to land the epic-refactor branch on trunk this weekend, after five and a half months.

Then I’m going to write a tool to convert old config files and we can think about putting out a 0.90 beta release. Exciting stuff!