Python wanderings, part two

Posted on Thursday, 31 December 2009

2. Plugging it all in

Sometimes we get feature requests and merge proposals for features that are clearly useful for someone, but not appropriate for the general use cases. It's always unfortunate to have to say no to these folks, but we have a slim menu UI and I'm wary of cluttering it with niche features. Still, turning away legitimate users is something I don't like doing, so for a while we've been considering how to fix this.

The obvious answer is that we should support plugins, and I've been working on such a system for my epic refactoring. This is a quick wander through some thoughts I've had.

I started out by googling for python plugin systems; One if the top hits was this page by Armin Ronacher . In it he demonstrates a plugin system in under 40 lines of Python. It's simple and flexible, but there are some issues, like it makes doctest very sad.

I asked about this in #python and was politely informed that I was Doing It Wrong. I chatted for a while with the helpful residents and came away with a list of plugin frameworks to look at, namely twisted.plugin and zope.interface.

Pulling in external dependencies is a big deal for us - many of our users are on Ubuntu or similar desktops with lots of python packages already installed, but some are not using GNOME or a Linux desktop at all, so we have to be sure that we need a library before we depend on it.

After playing a little with both of the options I came to the conclusion that while they are both really well made and capable, they are far more formal than we need, and the added dependency issues continued to concern me.

I revisited Armin's plugin system and removed the use of .__subclasses__() that was breaking doctest and offending #python instead having a list in each .py file which the plugin system extracts and treats any classes mentioned in that list as plugins. I also extended it to always instantiate the plugins and look for the plugin files in both the system directories and the user's home directory.

This plugin system is currently hooked into two places in the branch, URL mangling and the context menu. This allows plugins to add support for new URL types (e.g. we just added support for Launchpad code URLs like lp:~cmsj/+junk/terminator-epic-refactor), and insert new options into the context menu. I'm not sure if we need to go further, but if you would like to hook into other parts let me know - it's pretty easy to arrange now :)


Random musical linkage

Posted on Wednesday, 23 December 2009

I listen to a lot of electronic music and one of the nice things about that is the intermingling of other music by way of samples.

Case in point, I'm currently watching The Ballet Boyz's production of The Rite Of Spring with Rike, and I suddenly realised that the string motif playing at the entrance of The Elders is sampled and looped to great effect by the hypnotic Physical World on Freeland's album Now & Them.

This kind of thing makes me smile :)


Python wanderings, part one

As mentioned in my earlier post about refactoring Terminator, I want to talk about some of the things I've learned about Python and programming in the last few months. If I were you I wouldn't place any great significance in anything I'm about to say - after all I'm a rank amateur in the field of programming.

This is going to be a multi-part post so I at least get something out there, rather than leaving it to rot forever in my drafts folder.

1. Solving global warming^W variables

I have objects that represent terminal widgets, objects that represent widgets that contain terminals, objects that contain configuration, and one master object that functions as the brains of the operation.

Inevitably these objects need to know about each other, but how to achieve that? The brain object is simply called 'Terminator' and almost every other part of the system needs to know about it, same with the config object, and Terminator needs to know about all of the terminal objects, etc. The dependencies are all over the place and one aim of the re-factor was to separate all these parts out and decouple them, but ultimately I was never going to get away from different objects needing to know about each other.

So how to go about it? As far as I know the options are:

  • pass around object references (every time you create something, pass it your references to all the bits it needs)

    • Pros: no hacks or tricks involved

    • Cons: makes every __init__() more complicated, means passing references that an object doesn't need other than to pass to its children.

  • use global variables

    • Pros: they're global

    • Cons: everyone seems to hate global variables, perhaps because it's an implicit dependency not an explicit one, or because of potential namespace collisions, or maybe other reasons.

  • use singletons

    • Pros: explicit dependency

    • Cons: often seems to involve hackery to get the singleton object reference

In my searching around I came across a fourth option that somewhat relates to singletons... the Borg pattern.

This is a very simple idea - it's a class that always instantiates to the same thing. You don't need a factory or function or something that gives you a reference to the singleton, you just instantiate a class and it's the same as all of the others you've instantiated of the same class.

Best of all, the Borg pattern is incredibly simple in Python. Like, really simple. Don't believe me? Click here. Yep, four lines of code. Technically it's probably a bit ugly, but the resulting code feels very clean.

So now I have the Borg pattern in use for the main class, a class that provides all the configuration, a class that discovers plugins and lets them be referenced, and a fairly new class I'm experimenting with that acts as a factory for all of my classes, as a way to break any possibility of circular module dependencies.

Reality has to bite though, the Borg isn't a panacea; One has to be very careful about how one creates Borg objects. I chose to create a base class called Borg which Terminator, Config, Factory and PluginRegistry all derive from, but this turns out to have been a very short sighted decision to abstract out the common 4 lines. It wasn't until I started building Config to have functions that allow it to be accessed as a dict that I realised all of my Terminator, Config, Factory and PluginRegistry instances were the same thing as opposed to each type being distinct. It's also terrifyingly important that the subclasses of Borg not use class attributes. Any attributes defined by these classes *must* be instantiated as None so they are instance variables, and *after* you've called Borg.__init__(self) in your own __init__() you can then set up your attributes however you want because they are then part of the shared state.

On the whole I am happy with the Borg pattern. I've written test code to ensure that all of the assumptions I explicitly made are guaranteed, and all of the implicit assumptions I've discovered I made are also safe. Nonetheless, it's not a completely clean solution and I find myself wishing it was somehow a primitive of the language.


Epic Terminator refactoring afoot

Posted on Saturday, 19 December 2009

The current bzr repository for Terminator began its life in November 2006 with the simplest possible implementation of the concept of packing multiple terminals into one window. In the 3 years since then we have expanded and extended the code in a variety of directions to produce a moderately compelling feature set, but one that is really obviously incomplete.

In the same time period we've also seen a really gratifying amount of adoption - I believe our active userbase numbers in the thousands if not tens of thousands. I am forced to largely estimate these numbers for all the usual FOSS reasons, but it's all based on one real metric - Ubuntu has about a million users reporting popcon data and over 10,000 of those have Terminator installed. I don't actually think that they all use it, but nonetheless it's the kind of number that makes you think "hey maybe I need to be doing more for these folks".

And I do think that, and I am trying to do more.

Back in August I took a serious look at where we are and came to the same old conclusions - we lack one or two headline features that people keep asking for (barely a week goes by when I don't get asked how someone can save a particular layout of terminals). These features are very subtle and deeply problematic with the existing code architecture - we've just been hacking in features as we can without any regard for architecture or future maintainability.

I decided that I'd had enough of being confused and frustrated by the status quo and so I started a side branch in my Launchpad /+junk/ folder called "epic-refactor" with the aim of refactoring all of Terminator from scratch. I'd read through every line of existing code and figure out what we were actually doing and how it could fit together more sensibly, then sketch that out in the barest form possible while I experimented with various Python techniques to arrive at an architecture that makes sense for our project, then port over the existing code feature-by-feature to the new architecture.

It's now a little over four months since I started the epic refactor and looking at where I stand today I am really happy. It's not ready to be merged into trunk yet, but the amount of work to get it there is less than the amount of work I've done on it so far. I don't want to put a timescale on it, but I hope to be calling for some wider testing in 2-3 months or less.

Once we are acceptably close to feature parity with the current releases I'll merge the epic-refactor branch over  and we can start to push forward with implementing the features that everyone wants, and finally get to the point of being able to comfortably release a 1.0 version.

I'd always thought that I'd hand over maintainership after a 1.0 version, but the last 4 months have been a whirlwind of programming discovery, so I might very well just stick around and see what people want on the road to a 2.0 release. Alternatively, the work I've been doing in the last few days on a plugin system might mean that I can kick back and watch everyone else implement crazy, awesome and sublime features I'd never thought of!

I'll be back with more when I have written a configuration subsystem for epic-refactor, because by then I'll be wanting your help to test!

I'm also going to write a separate post shortly about some of the interesting Python paradigms and ideas I've hit upon along the way. I'm sure none of it will be a revelation to anyone with serious programming chops, but for a rank amateur like me it would have been useful to have read four months ago ;)


Terminator 0.14 released!

Posted on Thursday, 3 December 2009

This has been another long gap release unfortunately, but we do have some goodies for you.
Stephen Boddy is back with some excellent re-working of the UI relating to grouping terminals, Kees Cook brings us some clever work relating to window geometry and myself and others have been fixing bugs for you. We hope you enjoy this release, especially those of you who hated the "I'll be back" notifications from 0.13!

Home Page: http://www.tenshu.net/terminator
Launchpad Page: http://launchpad.net/terminator
Ubuntu PPAs: https://launchpad.net/~gnome-terminator/+archive/ppa


Rise of the Floating Fonters

Posted on Tuesday, 20 October 2009

For about two years now I've been using a 127dpi laptop screen as my primary computer display. It's a comfortable thing to be looking at, and after much playing around I've settled on 6.5pt as my ideal application font size.

No problems with that, right? Fontconfig says font sizes are a double (a high precision floating point number), but not all libraries/applications follow this.

In my testing of Karmic I've found two such things that particularly stick out:

  • notify-osd

    • Assumes font sizes are whole numbers, so ends up using a tiny font

  • Gwibber

    • Assumes font sizes are integers and completely fails to run if they are not

Obviously this won't do, so I've checked that we have filed bugs (and in the case of Gwibber, a patch), but I seem to be meeting some resistance, or this just isn't considered to be a high priority.

Thus a new Launchpad team is born, The Floating Fonters, for exiles such as myself who won't kowtow to the integers. We even have a PPA with fixed versions of notify-osd and Gwibber, but no guarantees are included!


Thinkpad USB keyboard

Posted on Saturday, 17 October 2009

Yesterday I took delivery of a new Thinkpad USB keyboard because I've started putting my laptop up on a stand to be next to it's external monitor.
In so doing I needed a keyboard and the ten quid Logitech was making me very sad, hence the Thinkpad one.

If you've ever typed on a thinkpad you immediately know why this keyboard is awesome :)


Double posts

Posted on Thursday, 24 September 2009

I've just chucked in a new wordpress plugin to syndicate my Launchpad project announcements (so Lifesaver and Terminator) as regular posts here so I don't need to keep writing release announcements in two places. This does mean that all the old Terminator announcements are doubled up here. I'm not sure if I'll leave it or go back and remove the dupes. Probably nobody cares anyway ;)


Thinkpad kernel module in Ubuntu 9.10 (Karmic)

Posted on Friday, 18 September 2009

The Ubuntu Kernel Team has decided to remove the tp_smapi module from our kernel for 9.10 (Karmic Koala) because the author chooses to remain anonymous and it is therefore impossible to be sure that the code is not based on incorrectly obtained information.

Slightly annoying perhaps, but ultimately a decision that's hard to argue with and fortunately one that's pretty easy to work around. I wish the author would clear things up once and for all because the tp_smapi module is desperately important for Thinkpad owners wishing to protect the life of their laptop battery. Given that my new X301 is tasked with a lifetime of 3 years, I am particularly keen to protect its battery!

The source for the module is still in the archive (and my understanding is that it will stay there, we just don't want to ship it by default) as tp-smapi-source, packaged by Evgeni Golov (a thoroughly decent chap who is the current owner of the Thinkpad X300 I've posted about previously). You can install it with the command:

sudo apt-get install tp-smapi-source

Then run:
sudo module-assistant

and select the tp-smapi module to build and install. You are now just a quick:
sudo modprobe tp_smapi

away from having battery charge control options in /sys/devices/platform/smapi/BAT0/

Woo! If I get a chance I'll try and produce a version of the package which uses DKMS (Dell's Kernel Module management system which makes sure that additional modules like this are rebuilt automagically whenever you get a kernel update).


Brewing Lifesaver 1.1

Posted on Wednesday, 16 September 2009

If you're using the Livesaver PPA I mentioned in my previous post you should shortly get offered a package of 1.1 which I'm almost ready to release.

I want to push it into the PPA first to get a little testing before officially tagging the release. Please file bugs if you spot any!

There are a few little bug fixes, the first external code contribution, and configuration is now done via gconf so you can change the fonts/colours and search keywords. I did also add a source for FriendFeed, but it's currently disabled because I don't want to extend the amount of time it takes to collect data from the web too much.


New project released - Lifesaver

Posted on Tuesday, 15 September 2009

For a few days now I've been hacking away on a new project: Lifesaver.

The idea is really simple - it's a screensaver for GNOME that displays recent posts about Ubuntu from Twitter and Identi.ca. That's it!

The code and bugs and downloads live on Launchpad, and packages for Ubuntu 9.10 (Karmic) are in my PPA.

Please let me know what you think!

Obligatory screenshot:


Update: I had to remove the Jaunty package from the PPA because it turns out that I've been using GooCanvas features that aren't available in the Jaunty version of GooCanvas. Sorry Jaunty users, you'll have to wait until next month when Karmic is released and you upgrade!


Recent FOSS activities

Posted on Tuesday, 10 March 2009

I've not been doing a huge amount of work on Terminator recently - I have to be honest and say that a lot of the fun stuff is already done and behind us, and it's just got a couple of features before it's as complete as it should be, also it's now well over 2 years into the long-slog towards the often-marvelled-about decade it takes for a piece of software to be truly mature.
That's not to say that I haven't still got a burning interest to do good works in the FOSS world. Recently I've been poking around at other things. I've got a patch in GNOME's Bugzilla which fixes the stupid bug where mailto: URIs get mangled when you open them, and yesterday I submitted a branch to the GNOME Do folks which adds a plugin for users of Request Tracker (being a sysadmin, RT rules my working life).

Hopefully both will get merged soon! Now I need to figure out what I should poke at next in 2009 (my rough aim for the year being to produce more and consume less).


Float to the top of the race to the bottom, on wings of quality

Posted on Friday, 23 January 2009

I came across an interesting blog post earlier, in which the creators of some iPhone applications dropped their pricing and asked their users to consider donating money to them if they felt the applications were worth more. This being a reaction to applications on the store generally trending towards $0.99 in a massive war of attrition among the 15,000 applications there. It's a market worth fighting for considering there are 5 million downloads a day, but you need to be compelling, and price is the easiest way to do that. Not necessarily the best though.

I had never heard of their apps, so I went to their site to check them out and they certainly look good - something that a lot of the apps I've tried don't, but I noticed straight away that it was basically just the same idea rehashed in a few different forms. I felt compelled to comment on their forums:

I'm not trying to be dismissive of your work, indeed I've never tried any of your apps (nor would I, since I don't have a car, go to the gym, etc. I'm lazy and I live in a big city).

My natural reaction when I see a selection of apps such as yours is that you've had an idea and you're trying to milk it. Again, I'm not trying to be mean or cruel, it's just my gut reaction.

You appear to have some very pretty and quite flexible technology for logging events, and jotting up costs, so why is there not a single awe-inspiring app for people to do those things in the way *they* want to, not the way *you* want them to.

$5 just to keep track of fuel bills seems a bit bizarre when you can just keep a stack of paper receipts, or search your online credit card statements.
However, a generic tracking tool that lets you create arbitrary tags.... that's a whole new kettle of fish, I could be tagging any work related expenses, things that relate to clothes, or iphone apps, and it's entirely up to me to develop a workflow that I'm comfortable with.

I could also be logging when I go to the gym (if I did), and when I walk to work, which kinda counts as keeping fit, or when my washing machine breaks down so I can complain to my landlord, etc, etc.

I think it's deeply unfortunate that you're finding yourself in a situation where you feel you are undervaluing your hard work, nobody likes that kind of thing, and I don't at all doubt that a significant factor is the general race-to-the-bottom of pricing in the AppStore. At that point it seems like the best way to succeed is to stand out on quality and features. Your screenshots suggest you have the design flair to pull off a quality application, but you're deliberately trying to restrict the features in each app, so people end up buying two apps if they want to track gym and fuel events, but they still can't record their expenditure on flowers for their wife.

Make a free, or $0.99 app that lets people track two or three things, and a $5 app that lets you track as much as you want. I would buy it the $5 version and my gut feeling is that lots of other people would too.

The interesting thing is that I actually asked a Mac developer friend of mine the other day to consider making a generic kind of event logging app, because I do have infrequently recurring events I'd like to keep track off, and I might well be tempted to keep track of arbitrary groupings of expenditure if I had the facility, but I have no reason to buy an application for tracking car fuel costs, or company trip expenses. As it happens I don't have a car, but even if I did I would feel like I should be buying something that can track *anything* and creating a tag for car related expenses, and a second tag for fuel expenses, then creating events with both those tags when I fill the car up.
I guess my fundemental problem is that the iPhone is a general purpose computer, but these apps are very specific in their functionality, and that fundementally annoys me.


Terminator 0.12 released!

Posted on Wednesday, 21 January 2009

I've just uploaded a new version of Terminator!

This time we merry band of elves bring you a delightful mix of bug fixes and new features. Here's the change log:
* Bug fixes
* Simultaneous typing support
* Directional terminal navigation
* Improved search UI
* Graphical Profile Editor
* Bug numbers for launchpad.net are now URLs

Head over to the homepage or Launchpad for the goodness. I've already pushed out source packages to the Ubuntu PPAs, which should be spinning up buildds soon, so expect binary packages shortly. I'm sure we'll filter into Debian, Ubuntu Jaunty, Fedora, FreeBSD and more soon enough :)