I’m in constant need of debugging information and in Terminator particularly where we have lots of objects all interacting and reparenting all the time. We’ve had a simple dbg() method for a long time, but I was getting very bored of typing out dbg(‘Class::method:: Some message about %d’ % foo), so I decided to see what could be done about inferring the Class and method parts of the message.

It turns out that python is very good at introspecting its own runtime, so back in January, armed with my own stupidity and some help from various folks on the Internet, I came up with the following:

# set this to true to enable debugging output
DEBUG = False
# set this to true to additionally list filenames in debugging
DEBUGFILES = False
# list of classes to show debugging for. empty list means show all classes
DEBUGCLASSES = []
# list of methods to show debugging for. empty list means show all methods
DEBUGMETHODS = []

def dbg(log = ""):
    """Print a message if debugging is enabled"""
    if DEBUG:
        stackitem = inspect.stack()[1]
        parent_frame = stackitem[0]
        method = parent_frame.f_code.co_name
        names, varargs, keywords, local_vars = inspect.getargvalues(parent_frame)
        try:
            self_name = names[0]
            classname = local_vars[self_name].__class__.__name__
        except IndexError:
            classname = "noclass"
        if DEBUGFILES:
            line = stackitem[2]
            filename = parent_frame.f_code.co_filename
            extra = " (%s:%s)" % (filename, line)
        else:
            extra = ""
        if DEBUGCLASSES != [] and classname not in DEBUGCLASSES:
            return
        if DEBUGMETHODS != [] and method not in DEBUGMETHODS:
            return
        try:
            print >> sys.stderr, "%s::%s: %s%s" % (classname, method, log, extra)
        except IOError:
            pass

How’s about that for shockingly bad? ;)
It also adds a really impressive amount of overhead to the execution time.
I added the DEBUGCLASSES and DEBUGMETHODS lists so I could cut down on the huge amount of output – these are hooked up to command line options, so you can do something like “terminator -d –debug-classes=Terminal” and only receive debugging messages from the Terminal module.

I’m not exactly sure what I hope to gain from this post, other than ridicule on the Internet, but maybe, just maybe, someone will pop up and point out how stupid I am in a way that turns this into a 2 line, low-overhead function :D

Is most of my music collection from the 90s?

Obviously what I want to do here is examine the ID3 tags of the files in my music collection and see how they’re distributed. A quick search with apt showed that Ubuntu 10.04 has two python libraries for dealing with ID3 tags and a quick play with each suggested that the one with the API most relevant to my interests was eyeD3. After a few test iterations of the script I was getting bored of waiting for it to silently scan the roughly 4000 MP3s I have, so I did another quick search and found a progress bar library.

So that’s all of the motive and opportunity established, now let’s examine the means to the end. If you want to follow along at home, the whole script is here.

try:

 import eyeD3

 import progressbar as pbar

except ImportError:

print("You should make sure python-eyed3 and python-progressbar \

are installed")

 sys.exit(1)

First off this is the section where I’m importing the two non-default python libraries that I depend on. I want to provide a good experience when they’re not installed, so I catch the exception and tell people the Debian/Ubuntu package names they need, and exit gracefully. I rename the progressbar module as I import it just because “progressbar” is annoyingly long as a name, and I don’t like doing “from foo import *”.

Skipping further on, we find the code that extracts the ID3 year tag:

year = tag.getYear() or 'Unknown'

This is something I’m really not sure about the “correctness” of; One of the reasons I went with the eyeD3 library was that the getYear() method returns None if it can’t find any data, but I don’t really want to capture the result, then test the result explicitly and if it’s None set the value to “Unknown”, so I went with the above code which only needs a single line and is (IMHO) highly readable.

This is ultimately the crux of the entire program – we’ve now collected the year, so we can work out which decade it’s from:

if year is not 'Unknown':
 year = "%s0s" % str(year)[:3]

If this isn’t an unknown year we chop the final digit off the year and replace it with a zero. Job done!

Next up, another style question. Rather than store the year we just processed I want to know how many of each decade have been found, so the obvious choice is a dict where the keys are the decades and the values are the number of times each decade has been found. One option would be to pre-fill the dict with all the decades, each with a value of zero, but that seems redundant and ugly, so instead I start out with an empty dict. This presents a challenge – if we find a decade that isn’t already a key in the dict (which will frequently be the case) we need to notice that and add it. We could do this by pre-emptively testing the dict with its has_key() method, but that struck me as annoyingly wordy, so I went with:

try:

 years[year] += 1

except KeyError:

 years[year] = 1

If we are incrementing a year that isn’t already in the dict, python will raise a KeyError, at which point we know what’s happened and know the correct value is 1, so we just set it explicitly. Seems like the simplest solution, but is it the sanest?

The only other thing I wanted to say is a complaint – having built up the dict I then want to print it nicely, so I have a quick list comprehension to produce a list of strings of the format “19xx: yy” (i.e. the decade and the final number of tracks found for that decade), which I then join together using:

', '.join(OUTPUT)

which I hate! Why can’t I do:

OUTPUT.join(', ')

(where “OUTPUT” is the list of strings). If that were possible, what I’d actually do is tack the .join() onto the end of the list comprehension and a single line would turn the dict into a printable string.

So there we have it, my thoughts on the structure of my script. I’d also add that I’ve become mildly obsessive about getting good scores from pylint on my code, which is why it’s rigorously formatted, docstring-ed and why the variable names in the __main__ section are in capitals.

What are your thoughts?

Oh, and the answer is no, most of my music is from the 2000s. The 1990s come in second :)

We did a little digging and realised I had an icon theme cache file in /usr/local/ that was overriding the one in /usr/. A bit of deleting later and it’s back, but in the process we whipped up a little bit of python to print out the filename of an icon given an icon name.

#!/usr/bin/python
# gtk-find-icon by Chris Jones <cmsj@tenshu.net>
# Copyright 2010. GPL v2.

import sys
import gtk

THEME = gtk.IconTheme()
ICON = THEME.lookup_icon(sys.argv[1],
 gtk.ICON_SIZE_MENU,
 gtk.ICON_LOOKUP_USE_BUILTIN)

if not ICON:
 print "None found"
else:
 print(ICON.get_filename())

One of the features of the new 0.9x series of Terminator releases that hasn’t had a huge amount of announcement/discussion yet is the plugin system. I’ve posted previously about the decisions that went into the design of the plugin framework, but I figured now would be a good time to look at how to actually take advantage of it.

While the plugin system is really generic, so far there are only two points in the Terminator code that actually look for plugins – the Terminal context menu and the default URL opening code. If you find you’d like to write a plugin that interacts with a different part of Terminator, please let me know, I’d love to see some clever uses of plugins and I definitely want to expand the number of points that plugins can hook into.

The basics of a plugin

A plugin is a class in a .py file in terminatorlib/plugins or ~/.config/terminator/plugins, but not all classes are automatically treated as plugins. Terminator will examine each of the .py files it finds for a list called ‘available’ and it will load each of the classes mentioned therein.

Additionally, it would be a good idea to import terminatorlib.plugin as that contains the base classes that other plugins should be derived from.

A quick example:

import terminatorlib.plugin as plugin
available = ['myfirstplugin']
class myfirstplugin(plugin.SomeBasePluginClass):
  etc.

So now let’s move on to the simplest type of plugin currently available in Terminator, a URL handler.

URL Handlers

This type of plugin adds new regular expressions to match text in the terminal that should be handled as URLs. We ship an example of this with Terminator, it’s a handler that adds support for the commonly used format for Launchpad. Ignoring the comments and the basics above, this is ultimately all it is:

class LaunchpadBugURLHandler(plugin.URLHandler):
  capabilities = ['url_handler']
  handler_name = 'launchpad_bug'
  match = '\\b(lp|LP):?\s?#?[0-9]+(,\s*#?[0-9]+)*\\b'

  def callback(self, url):
    for item in re.findall(r'[0-9]+', url):
      return('https://bugs.launchpad.net/bugs/%s' % item)

That’s it! Let’s break it down a little to see the important things here:

• inherit from plugin.URLHandler if you want to handle URLs.
• include ‘url_handler’ in your capabilities list
• URL handlers must specify a unique handler_name (no enforcement of uniqueness is performed by Terminator, so use some common sense with the namespace)
• Terminator will call a method in your class called callback() and pass it the text that was matched. You must return a valid URL which will probably be based on this text.

and that’s all there is to it really. Next time you start terminator you should find the pattern you added gets handled as a URL!

Context menu items

This type of plugin is a little more involved, but not a huge amount and as with URLHandler we ship an example in terminatorlib/plugins/custom_commands.py which is a plugin that allows users to add custom commands to be sent to the terminal when selected. This also brings a second aspect of making more complex plugins – storing configuration. Terminator’s shiny new configuration system (based on the excellent ConfigObj) exposes some API for plugins to use for loading and storing their configuration. The nuts and bolts here are:

import terminatorlib.plugin as plugin
from terminatorlib.config import Config
available = ['CustomCommandsMenu']
class CustomCommandsMenu(plugin.MenuItem):
  capabilities = ['terminal_menu']
  config = None
  def __init__(self):
    self.config = Config()
    myconfig = self.config.plugin_get_config(self.__class__.__name__)
    # Now extract valid data from sections{}
  def callback(self, menuitems, menu, terminal):
    menuitems.append(gtk.MenuItem('some jazz'))

This is a pretty simplified example, but it’s sufficient to insert a menu item that says “some jazz”. I’m not going to go into the detail of hooking up a handler to the ‘activate’ event of the MenuItem or other PyGTK mechanics, but this gives you the basic detail. The method that Terminator will call from your class is again “callback()” and you get passed a list you should add your menu structure to, along with references to the main menu object and the related Terminal. As the plugin system expands and matures I’d like to be more formal about the API that plugins should expect to be able to rely on, rather than having them poke around inside classes like Config and Terminal. Suggestions are welcome :)

Regarding the configuration storage API – the value returned by Config.plugin_get_config() is just a dict, it’s whatever is currently configured for your plugin’s name in the Terminator config file. There’s no validation of this data, so you should pay attention to it containing valid data. You can then set whatever you want in this dict and pass it to Config().plugin_set_config() with the name of your class and then call Config().save() to flush this out to disk (I recommend that you be quite liberal about calling save()).

Wrap up

Right now that’s all there is to it. Please get in touch if you have any suggestions or questions – I’d love to ship more plugins with Terminator itself, and I can think of some great ideas. Probably the most useful thing would be something to help customise Terminator for heavy ssh users (see the earlier fork of Terminator called ’ssherminator’)

Now a bunch of bug fixing and a config converter and we can release!
Thanks to everyone who has been testing so far.

• Fix a bug in handling URLs dropped on the window
• Implement directional navigation
• Implement geometry hinting
• Fix a bug in group emitting that cause “Broadcast off” and “Broadcast to all” to become inverted
• Implement WM_URGENT bell handler

I’m really happy with how this is going. All that is left to have feature parity with trunk, I think, is some keyboard shortcut handlers.

I’d still love to get more testing results to make sure I haven’t missed anything, but at this rate I’m expecting to be able to land the epic-refactor branch on trunk this weekend, after five and a half months.

Then I’m going to write a tool to convert old config files and we can think about putting out a 0.90 beta release. Exciting stuff!

• -e and -x command line options
• all forms of drag & drop
• directional navigation
• some keyboard shortcuts

I’m pleased to say that the first two of these are now taken care of, but the latter two are still to be done. I’m less pleased to say that I haven’t had much external feedback about this branch yet, but I suspect that most people who might be interested probably don’t read my blog ;)

So if you know people who like Terminator and enjoy testing things out, all they need to do is:

bzr branch lp:~cmsj/terminator/epic-refactor
cd epic-refactor
./terminator

and give some feedback!

I think it’s finally time that we get some more eyeballs on it, mainly so I can check that I haven’t massively screwed something up. I know there is lots of missing functionality right now, and probably a bunch of subtle bugs, but I could use your help quantifying exactly what these are!

If you’re inclined to help, please branch lp:~cmsj/terminator/epic-refactor, cd into it and run ./terminator, then use it like you always would and file bugs, preferably indicating clearly in the bug that you’re using this branch and not trunk (maybe tag the bug ‘epicrefactor‘).

Things I know are broken right now:

• -e and -x command line options
• all forms of drag & drop
• directional navigation
• some keyboard shortcuts

Things I know are missing because they’re not coming back:

• Extreme tabs mode (sorry, it’s just too insane to support)
• GNOME Terminal profile reading (I’m trying to simplify our crazy config system and dropping GConf is a good way to achieve that)
• Config file reading. At some point I’ll write something that migrates old Terminator configs to the new format, but for now you’ll have to live without your old config file. The new one isn’t documented yet either, but it is a whole bunch better!

Now would also be a great time to start writing plugins for Terminator and telling me about them. I’m happy to ship good plugins, but more importantly I want feedback about the weaknesses/strengths of our plugin system. Right now you can only hook into URL mangling and the terminal context menu, but the latter of those gives you pretty serious flexibility I think. Obviously one massive weakness is a lack of documentation about the plugin API, but I’ll get to that, I promise!

So there we have it, another step along the way to me being able to merge this branch into trunk and put out a real release of 0.90 and then eventually 1.0!

Sometimes we get feature requests and merge proposals for features that are clearly useful for someone, but not appropriate for the general use cases. It’s always unfortunate to have to say no to these folks, but we have a slim menu UI and I’m wary of cluttering it with niche features. Still, turning away legitimate users is something I don’t like doing, so for a while we’ve been considering how to fix this.

The obvious answer is that we should support plugins, and I’ve been working on such a system for my epic refactoring. This is a quick wander through some thoughts I’ve had.

I started out by googling for python plugin systems; One if the top hits was this page by Armin Ronacher . In it he demonstrates a plugin system in under 40 lines of Python. It’s simple and flexible, but there are some issues, like it makes doctest very sad.

I asked about this in #python and was politely informed that I was Doing It Wrong. I chatted for a while with the helpful residents and came away with a list of plugin frameworks to look at, namely twisted.plugin and zope.interface.

Pulling in external dependencies is a big deal for us – many of our users are on Ubuntu or similar desktops with lots of python packages already installed, but some are not using GNOME or a Linux desktop at all, so we have to be sure that we need a library before we depend on it.

After playing a little with both of the options I came to the conclusion that while they are both really well made and capable, they are far more formal than we need, and the added dependency issues continued to concern me.

I revisited Armin’s plugin system and removed the use of .__subclasses__() that was breaking doctest and offending #python instead having a list in each .py file which the plugin system extracts and treats any classes mentioned in that list as plugins. I also extended it to always instantiate the plugins and look for the plugin files in both the system directories and the user’s home directory.

This plugin system is currently hooked into two places in the branch, URL mangling and the context menu. This allows plugins to add support for new URL types (e.g. we just added support for Launchpad code URLs like lp:~cmsj/+junk/terminator-epic-refactor), and insert new options into the context menu. I’m not sure if we need to go further, but if you would like to hook into other parts let me know – it’s pretty easy to arrange now :)

This is going to be a multi-part post so I at least get something out there, rather than leaving it to rot forever in my drafts folder.

1. Solving global warming^W variables

I have objects that represent terminal widgets, objects that represent widgets that contain terminals, objects that contain configuration, and one master object that functions as the brains of the operation.

Inevitably these objects need to know about each other, but how to achieve that? The brain object is simply called ‘Terminator’ and almost every other part of the system needs to know about it, same with the config object, and Terminator needs to know about all of the terminal objects, etc. The dependencies are all over the place and one aim of the re-factor was to separate all these parts out and decouple them, but ultimately I was never going to get away from different objects needing to know about each other.

So how to go about it? As far as I know the options are:

• pass around object references (every time you create something, pass it your references to all the bits it needs)
  • Pros: no hacks or tricks involved
  • Cons: makes every __init__() more complicated, means passing references that an object doesn’t need other than to pass to its children.
• use global variables
  • Pros: they’re global
  • Cons: everyone seems to hate global variables, perhaps because it’s an implicit dependency not an explicit one, or because of potential namespace collisions, or maybe other reasons.
• use singletons
  • Pros: explicit dependency
  • Cons: often seems to involve hackery to get the singleton object reference

In my searching around I came across a fourth option that somewhat relates to singletons… the Borg pattern.

This is a very simple idea – it’s a class that always instantiates to the same thing. You don’t need a factory or function or something that gives you a reference to the singleton, you just instantiate a class and it’s the same as all of the others you’ve instantiated of the same class.

Best of all, the Borg pattern is incredibly simple in Python. Like, really simple. Don’t believe me? Click here. Yep, four lines of code. Technically it’s probably a bit ugly, but the resulting code feels very clean.

So now I have the Borg pattern in use for the main class, a class that provides all the configuration, a class that discovers plugins and lets them be referenced, and a fairly new class I’m experimenting with that acts as a factory for all of my classes, as a way to break any possibility of circular module dependencies.

Reality has to bite though, the Borg isn’t a panacea; One has to be very careful about how one creates Borg objects. I chose to create a base class called Borg which Terminator, Config, Factory and PluginRegistry all derive from, but this turns out to have been a very short sighted decision to abstract out the common 4 lines. It wasn’t until I started building Config to have functions that allow it to be accessed as a dict that I realised all of my Terminator, Config, Factory and PluginRegistry instances were the same thing as opposed to each type being distinct. It’s also terrifyingly important that the subclasses of Borg not use class attributes. Any attributes defined by these classes *must* be instantiated as None so they are instance variables, and *after* you’ve called Borg.__init__(self) in your own __init__() you can then set up your attributes however you want because they are then part of the shared state.

On the whole I am happy with the Borg pattern. I’ve written test code to ensure that all of the assumptions I explicitly made are guaranteed, and all of the implicit assumptions I’ve discovered I made are also safe. Nonetheless, it’s not a completely clean solution and I find myself wishing it was somehow a primitive of the language.