Showing posts with label Techie. Show all posts

0

The curious Moto X pricing

Posted on Friday, 2 August 2013

Comparing the Moto X to the Nexus 4 is interesting in one particular respect - the price.

The Nexus 4 (made by LG, sold by Google) had very respectable specs when it was launched, but its price was surprisingly low ($300 off contract). We were told this was because it was being sold very close to cost price.

The Moto X (made by Motorola, which is owned by Google) has mid-range specs, but its price is surprisingly high ($200 up front *and* an expensive two year contract).

Overall Motorola is probably getting something like $400-$600 for each Moto X sold, when you factor in the carrier subsidy.

The inevitable question is why Google is happy to make almost no money off the Nexus 4, but wants to have its Motorola division make a respectable margin on the Moto X.

  • Is it because doing otherwise would undermine the carriers' abilities to sell other phones, so they would refuse to do it?
  • Is it because Google wants the Motorola division to look good in their accounts, which is easier if you are selling mid-range phones for the kind of money that an iPhone sells for?
  • Something else?

0

Moving on from Terminator

Posted on Wednesday, 17 July 2013

Anyone who's been following Terminator knows this post has been a long time coming and should not be surprised by it.


As of a few days ago, I have handed over the reigns of the project to the very capable Stephen J Boddy (a name that will be no stranger to followers of the Terminator changelogs - he has contributed a great deal over the last few years).

We're still working out the actual details of the handover, so for now the website is still here and I am still technically the owner of the Launchpad team that runs the project, but going forward all code/release decisions will come from Stephen and we'll move the various administrivia over to new ownership in due course.

Everyone please grab your bug trackers and your python interpreters and go send Stephen patches and feature requests! :D

0

Some Keyboard Maestro macros

I've started using Keyboard Maestro recently and it is one impressive piece of software.

Here are a couple of the macros I've written that aren't completely tied to my system:

  • Type current Safari URL
    • This will type the URL of the frontmost Safari tab/window into your current application. Handy if you're chatting with someone and want to paste them a hilarious YouTube URL without switching apps, copying the URL to the clipboard and switching back.
    • It does not use the clipboard, it actually types the URL into the current application, so any modifier keys you hold will change what is being typed. I've configured the macro to fire when the specified hotkey is released, to minimise the chances of this happening. 
  • Toggle Caffeine
    • Very simple, just toggles the state of Caffeine with a hotkey.

0

Thoughts on a modular Mac Pro

Posted on Monday, 10 June 2013

There have been some rumours recently that the next iteration of the Mac Pro is going to be modular, but we have had very little information about how this modularity might be expressed.

In some ways the current Mac Pro is already quite modular - at least compared to every other Mac/MacBook. You have easy access to lots of RAM slots, you have multiple standards-compliant disk bays, PCI slots and CPU sockets.

This affords the machine an extremely atypical level of upgradeability and expandability, for a Mac. Normal levels for a PC though.

Even with that modularity in mind, the machine itself is fairly monolithic - if you do need more than 4 disk drives, or more PCI cards than it can take, you have limited or no expansion options. You could burn a PCI slot for a hardware disk controller and attach some disks to it externally, but you are quickly descending into an exploding mess of power supplies, cables and cooling fans.

If Apple decides to proceed along that route, the easiest and most obvious answer is that they slim down the main Pro itself and decree that all expansion shall take place over Thunderbolt (currently 10Gb/s bidirectional, but moving to 20Gb/s bidirectional later this year when the Thunderbolt 2 Falcon Ridge controllers launch). This is a reasonable option, but even though Thunderbolt is essentially an external PCI-Express bus, its available bandwidth is considerably lower than the peak levels found on an internal PCI-E bus (currently around 125Gb/s).

A much better option, it would seem to me, would be to be able to go radically modular and expand the Mac itself, but how could that be possible? How can you just snap on some more PCI slots if you want those, or some more disks if that's what you need?

I will say at this point that I have absolutely no concrete information and I am not an electronic engineer, so what you read below is poorly informed speculation and should be treated as that :)

I think the answer is Intel's QuickPath Interconnect (QPI), a high bandwidth (over 200GB/s), low latency point-to-point communication bus for connecting the main components of an Intel based computer.

If you have any Intel CPU since around 2009, you probably have a QPI bus being used in your computer. Looking at the latest iteration of their CPUs, QPI is always present - on the uniprocessor CPUs it is used on the chip package to connect the CPU core to the elements of the northbridge that have migrated into the CPU package (such as the PCI-Express controller), however, on these chips the QPI bus is not presented externally.

On the multiprocessor-capable chips, it is, and is the normal way to interconnect the CPUs themselves, but it can be used for other point-to-point links, such as additional north bridges providing PCI-Express busses.

So you could buy a central module from Apple that contains 1, 2 or 4 CPUs (assuming Ivy Bridge Xeons) and all of the associated RAM slots, with maybe two minimal disk bays for the core OS to boot from, and a few USB3.0 and Thunderbolt ports. For the very lightest of users, this would likely be a complete computer - you have some disk, some RAM, CPUs and assuming the Xeons carry integrated GPUs, the Thunderbolt ports can output video. It would not be much of a workstation, but it would essentially be a beefed up Mac Mini.

I would then envision two kinds of modules that would stack on to the central module. The simplest kind would be something like a module with a disk controller chip and a load of disk bays and, not needing the raw power of QPI, this would simply connect to the existing PCI-Express bus of the main module.

There would clearly be a limit to how many of these modules you could connect, since there are a limited number of PCI-E lanes provided by any one controller (typically around 40 lanes on current chipsets), but with the second type of module, you could then take the expansion up a considerable number of notches.

That second kind would have a large and dense connector that is a QPI. These modules could then attach whatever they wanted to the system - more CPUs (up to whatever maximum is supported by that generation of Xeon - likely 8 in Ivy Bridge), or very very powerful IO modules. My current working example of this is a module that is tasked with capturing multiple 4K video streams to disk simultaneously.
This module would provide its own PCI-Express controller (linked back to the main module over QPI), internally connected to a number of video capture chips/cards and to one or more disk controller chips/cards which would connect to a number of disk bays. It sounds a lot like what would happen inside a normal PC, just without the CPU/RAM and that's because it's exactly that.
This would allow for all of the video capture to be happening within the module. It would be controlled as normal from the software running in the main module, which would be issuing the same instructions as if the capture hardware was on the main system PCI-E bus, causing the capture cards to use DMA to write their raw video directly to the disk controller exactly as if they were on the main system PCI-E bus. The difference would be that there would be no other hardware on the PCI-E bus, so you would be able to make reasonable promises around latency and bandwidth, knowing that no user is going to have a crazy extra set of cards in PCI slots, competing for bandwidth. Even if you have two of these modules capturing a really silly amount of video simultaneously. It's a model for being able to do vast amounts of IO in parallel in a single computer.

There would almost certainly need to be a fairly low limit on the number of QPI modules that could attach to the system, but being able to snap on even two or three modules would elevate the maximum capabilities of the Pro to levels far beyond almost any other desktop workstation.

As a prospective owner of the new Mac Pro, my two reasonable fears from this are:

  • They go for the Thunderbolt-only route and my desk looks like an awful, noisy mess
  • They go for the radical modularity and I can't afford even the core module
(While I'm throwing around random predictions, I might as well shoot for a name for the radical modularity model. I would stick with the Lightning/Thunderbolt IO names and call it Super Cell)

Edit: I'd like to credit Thomas Hurst for helping to shape some of my thinking about QPI.

2

Alfred 2 clipboard history

Posted on Tuesday, 30 April 2013

The toweringly awesome Alfred 2 app for OS X has a great clipboard history browser. This is how I suggest you configure and use it:


  • Map a hotkey to the viewer (I suggest making it something involving the letter V, since Cmd-V is a normal Paste. I use Cmd-Shift-Option-Ctrl V because I have my Caps Lock key mapped to Cmd-Shift-Option-Ctrl)
  • Turn off the option to show snippets at the top of the Clipboard History, because snippets are a whole different thing and not relevant to pasting history
  • Turn on the option to auto-paste when you hit Enter on a given item
With these options all configured, all you have to do is hit the hotkey, select the old clipboard item you want and hit Enter. It will then be pasted into the active window.

This is also useful to preview the current contents of the clipboard before pasting (which is always a good idea if you're pasting into a sensitive terminal or a work IRC channel and want to avoid spamming some random/harmful nonsense in).

2

Terminator 0.97 released!

The present:

It's been a very long road since Terminator 0.96 back in September 2011, but I'm very happy to announce that Terminator 0.97 was released over breakfast this morning.
There's a reasonable amount of change, but almost all of it is bug fixes and translations.

Here is the changelog:


  • Allow font dimming in inactive terminals
  • Allow URL handler plugins to override label text for URL context menus
  • When copying a URL, run it through the URL handler first so the resulting URL is copied, rather than the original text
  • Allow users to configure a custom URL handler, since the default Gtk library option is failling a lot of users in non-GNOME environments.
  • Allow rotation of a group of terminals (Andre Hilsendeger)
  • Add a keyboard shortcut to insert a terminal's number (Stephen J Boddy)
  • Add a keyboard shortcut to edit the window title (Stephen J Boddy)
  • Add an easy way to balance terminals by double clicking on their separator (Stephen J Boddy)
  • Add a plugin to log the contents of terminals (Sinan Nalkaya)
  • Support configuration of TERM and COLORTERM (John Feuerstein)
  • Support reading configuration from alternate files (Pavel Khlebovich)
  • Allow creation of new tabs in existing Terminator windows, using our DBus API
  • Support the Solarized colour palettes (Juan Francisco Cantero Hutardo)
  • Translation support for the Preferences window
  • Lots of translation updates (from our fantastic translation community)
  • Lots of bug fixes

My sincere thanks to everyone who helped out with making this release happen.

The future:

So. Some of you might be wondering why this release isn't called 1.0, as it was tagged for a while in the development code. The main reason is that I just wanted to get a release out, without blocking on the very few remaining bugs/features targeted for the 1.0 release. I hope we'll get to the real 1.0 before very long (and certainly a lot quicker than the gap between 0.96 and 0.97!)
However, I do think that the Terminator project is running out of steam. Our release cadence has slowed dramatically and I think we should acknowledge that. It's entirely my fault, but it affects all of the userbase.

I am planning on driving Terminator to the 1.0 release, but the inevitable question is what should happen with the project after that.

The fact is that, like the original projects that inspired Terminator (gnome-multi-term, quadkonsole, etc.), technology is moving under our feet and we need to keep up or we will be obsolete and unable to run on modern open source desktops.

There is a very large amount of work required to port Terminator to using both Gtk3 and the GObject Introspection APIs that have replaced PyGtk. Neither of these porting efforts can be done in isolation and to make matters more complicated, this also necessitates porting to Python 3.
I am not sure that I can commit to that level of effort in a project that has, for my personal needs, been complete for about 5 years already.

With that in mind, if you think you are interested in the challenge, and up to the task of taking over the project, please talk to me (email cmsj@tenshu.net or talk to Ng in #terminator on Freenode). My suggestion would be that a direct, feature-complete port to Python3/Gtk3/GObject would immediately bump the version number to 2.0 and then get back to thinking about features, bug fixes and improving what we already have.

0

Some more awesome Alfred 2 workflows

Posted on Thursday, 11 April 2013

I keep finding super handy little things to do with Alfred 2 and so I thought I'd post some more:


  • Alleyoop - updates installed plugins (if the workflow author supports it, which many currently do not).  I hope this will be a temporary workaround until a centralised workflow repository is created.
  • Battery - shows all the vital stats of your MacBook's battery without having to run an app or a Terminal command.
  • Built-in Sharing - lets you share files directly to all the social services that OS X supports.
  • Paste current Safari URL - a workflow I wrote, which pastes the URL of Safari's currently visible webpage, into the application you are using. No need to flip back and forth to copy and paste the URL
  • Symbols - very easy, visual way to search the Unicode tables for a symbol you're looking for (e.g. arrows, hearts, snowmen, biohazard warning signs, etc)
  • TerminalFinder - lets you quickly get a Terminal for the Finder window you're looking at.
I imagine there will be more to come, the total number of workflows is exploding at the moment!

0

Alfred 2 workflows

Posted on Saturday, 16 March 2013

Since I started using OS X as my primary desktop, I've loved Spotlight for launching apps and finding files. I resisted trying any of the replacement apps, for fear of the bottomless pit of customisation that they seemed to offer.

With the very recent release of Alfred 2, I was finally tempted to try it by the previews of their Workflow feature.
The idea is that you can add new commands to Alfred by writing scripts in bash/python/ruby/php and then neatly package them up and share them with others. I was expecting to write a few myself and share them, but the user community has been spinning up so quickly that they've already covered everything I was going to write.

Instead, I decided to use some time to write about the workflows I'm using so far:

  • Google Search - get live results from Google as you type. It's not always what I want when I'm searching, but it's a very quick way to get some insight into the results available.
  • New OmniFocus Inbox Task - Very quick way to create a new task for later triage
  • Open SSH - This collects up all your hosts from SSH's known_host file, config file and local network, then opens terminal windows for you to ssh to the host you choose.
  • Parallels Desktop - Easy way to start/resume your Parallels virtual machines.
  • Rate iTunes Track - does what it sounds like, rate the current iTunes track.
  • Screen Sharing - quickly VNC to the hosts on your network that are advertising it (including iCloud hosts if you have Back To My Mac configured)
  • VPN Toggle - get on/off your corporate network quickly.
Lots more on the Alfred 2 forums. At some point it would be nice to see this unified into some kind of integrated search/download feature of Alfred 2.



Update: (2012-04-12) I've written a second post that covers a few more workflows I've discovered since this one.

0

LCD and a crazy disk chassis

Posted on Wednesday, 6 February 2013

TL;DR, pics and a video, below the jump

If you saw my recent post on some preparatory work I'd been doing for the arrival of an LCD status panel for my HP Microserver, it's probably no surprise that there is now a post talking about its arrival :)

Rather than just waste the 5.25" bay behind the LCD, I wanted to try and put some storage in there, particularly since the Microserver's BIOS can be modified to enable full AHCI on the 5th SATA port.

I recently came across the Icy Box IB-RD2121StS, a hilarious piece of hardware. It's the size and shape of a normal 3.5" SATA disk, but the back opens up to take two 2.5" SATA disks. These disks can then be exposed either individually, or as a combined RAID volume (levels 0 or 1). Since I happen to have a couple of 1TB 2.5" disks going spare, this seemed like the perfect option, as well as being so crazy that I couldn't not buy it!

The LCD is a red-on-black pre-made 5.25" bay insert from LCDModKit. It has an LCD2USB controller, which means it's very well supported by projects like lcd4linux and lcdproc. It comes with an internal USB connector (intended to connect directly to a motherboard LCD port), except the Microserver's internal USB port is a regular external Type A port. Fortunately converters are easy to come by.

Something I hadn't properly accounted for in my earlier simulator work is that the real hardware only has space for 8 user-definable characters and I was using way more than that (three of my own custom icons, but lcd4linux's split bars and hollow graphs use custom characters too). Rather than curtail my own custom icons, I chose to stop using hollow graphs, which seems to have worked.

Pics and a video below the jump.

Read more »

0

Funky lcd4linux python module

Posted on Saturday, 26 January 2013

I've got an LCD on the way, to put in my fileserver and show some status/health info.

Rather than wait for the thing to arrive I've gone ahead and started making the config I want with lcd4linux.

Since the LCD I'm getting is only 20 characters wide and 4 lines tall, there is not very much space, so I've had to get pretty creative with how I'm displaying information.

One thing I wanted was to show the percentage used of the various disks in the machine, but since I have at least 3 mount points, that would either mean scrolling text (ugly) or consuming ¾ of the display (inefficient).

It seemed like a much nicer idea to use a single line to represent the space used as a percentage and simple display each of the mounts in turn, but unfortunately lcd4linux's "Evaluator" syntax is not sufficiently complex to be able to implement this directly, so I faced the challenge of either writing a C plugin or passing the functionality off to a Python module.

I tend to think that this feature ought to be implemented as a C plugin because it makes it easier to use, but I am unlikely to bother with that because I prefer Python, so I went with a Python module :)

The code is on github and the included README.md covers how to use it in an lcd4linux configuration.

At some point soon I'll post my lcd4linux configuration - just as soon as I've figured out what to do with the precious 4th line. In the mean time, here is a video of the rotator plugin operating on the third line (the first line being disk activity and the second line being network activity):


Update: I figured out what to do with the fourth line:


That's another python module, this time a port of Chris Applegate's Daily Mail headline generator from JavaScript to Python. Code is on github.

As promised, the complete lcd4linux config is available (also on github) here.

0

Using Caps Lock as a new modifier key in OS X

Posted on Tuesday, 27 November 2012

I've started using Slate for window management in OS X. It's lovely being able to drive specific operations with the keyboard, but what's not lovely is trying to find a spare keyboard shortcut that doesn't need lots of fingers and doesn't clash with another app.

Taking inspiration from this article and the article it took inspiration from, I've taken to mapping Caps Lock to Shift+Ctrl+Option+Command, which is such a crazy set of modifiers that no app would ever have a default keyboard shortcut based on it. That means I get a single key which acts like all four normal modifier keys combined, so I get a whole blank keyboard of shortcuts all to myself!

The specifics of how I did this:

  1. Open System Preferences, then Keyboard, then Modifier Keys
  2. For each keyboard listed, set Caps Lock to No Action
  3. Install KeyRemap4MacBook and PCKeyboardHack, then restart your Mac
  4. Open System Preferences, then PCKeyboardHack
  5. Tick Change Caps Lock and set the keycode it sends, to 80 (which is really F19. If you have a keyboard which has a real F19 key, you need to change this to a different key you don't have)
  6. Go back to the main System Preferences page (click Show All, or re-open System Preferences) and open KeyRemap4MacBook
  7. Choose the Misc & Uninstall tab and click Open private.xml. This will give you a Finder window with a private.xml visible
  8. Right click private.xml and choose Open With then TextEdit.app
  9. Paste the following between the <root> and </root> tags:
    <item>
        <name>Remap F19 to Hyper</name>
        <appendix>OS X doesn't have a Hyper. This maps F19 to Control + Shift + Option + Command.</appendix>

        <identifier>usercustom.f19_to_hyper</identifier>

        <autogen>
            --KeyToKey--
            KeyCode::F19,

            KeyCode::COMMAND_L,
            ModifierFlag::OPTION_L | ModifierFlag::SHIFT_L | ModifierFlag::CONTROL_L
        </autogen>
    </item>
  10. Save and Quit the private.xml window, close the Finder window
  11. Back in the KeyRemap4MacBook preferences, switch to the Change Key tab and click ReloadXML
  12. You should now see Remap F19 to Hyper at the top of the list. Tick it.
  13. At this point you should be done, but we can easily test to make sure...
  14. Switch to the Misc & Uninstall tab in the KeyRemap4MacBook preferences and Launch EventViewer
  15. Press your Caps Lock key and you should see a series of events appear, each of which adds/removes one of the Shift/Ctrl/Option/Command keys. The middle row should have Shift Ctrl Opt Cmd in the flags column. If you see that, you are done!
  16. Configure lots of shiny new Hyper shortcuts in Slate or whatever other app you like!


0

Photo import workflow

Posted on Sunday, 1 July 2012

Introduction

Since I'm writing about workflows today, I thought I'd also quickly chuck in a guide to how I get the photos and movies that I've taken with my iPhone, onto my laptop and specifically, imported into Aperture.

The Mechanics

This requires a few moving parts to produce a final workflow. The high-level process is:


  1. Plug iPhone into a USB port
  2. Copy photos from the iPhone into a temporary directory, deleting them as they are successfully retrieved
  3. Import the photos into Aperture, ensuring they are copied into its library and deleted from the temporary directory
Simple, right? Well yes and no.

Retrieval from iPhone

This really ought to be easier than it is, but at least it is possible.

Aperture can import photos from devices, but it doesn't seem to offer the ability to delete them from the device after import. That alone makes it not even worth bothering with if you don't want to build up a ton of old photos on your phone.

OS X does ship with a tool that can import photos from camera devices and delete the photos afterwards, a tool called AutoImporter.app, but you won't find it without looking hard. It lives at:

/System/Library/Image Capture/Support/Application/AutoImporter.app
If you run that tool, you will see no window, just a dock icon and some menus. Go into its Preferences and you will be able to choose a directory to import to, and choose whether or not to delete the files:


Easy!

Importing into Aperture


This involves using Automator to build a Folder Action workflow for the directory that AutoImporter is pulling the photos into. All it does is check to see if AutoImporter is still running and if so wait, then launch Aperture and tell it to import everything from that directory into a particular Project, and then delete the source files:


That's it!


Really, that's all there is. Now whenever you plug in your iPhone, all of the pictures and movies you've taken recently, will get imported into Aperture for you to process, archive, touch-up, export or whatever else it is that you do with your photos and movies.

0

Paperless workflow

Introduction


This is going to be quite a long post, but hopefully interesting to a particular crowd of people.
I'm going to tell you all about how I have designed and built a paperless workflow for myself.

Background


This came about some months ago when I needed to find several important documents that were spread through the various organised files that I keep things in. The search took much longer than I would have liked, partly because I am not very efficient at putting paper into the files.
You could suggest that I just get better at doing that, but even if I were to do that, it still only makes me quicker at finding paperwork from the files on my shelf. If I want to really kick things up a gear, the files need to be electronic, accessible from anywhere and powerfully searchable.

The hardware


I started thinking about what I would want. Obviously a scanner was going to be the first pre-requisite of being able to digitise my papers, but what kind to get? After investigating what other people had already said about paperless workflows, it seemed like the ScanSnap range of scanners was a popular choice, but they are quite expensive and it's one more thing on my desk. Instead I decided to go for a multi-function inkjet printer - they have scanners that are good enough, and even though they're bigger than a ScanSnap, I'm also getting a printer in the bargain.
So which one to get? Well that depended on which features were important. My highest priority in this project was that the process of taking a document from paper to my laptop had to be as simple as possible, so in the realms of scanning devices, that means you need one which can automatically scan both sides of the paper.
This turns out to be quite rare in multi-function printers, but after a great deal of research, I found the Epson Stylus Office BX635FWD which has a duplex ADF (Automatic Document Feeder), is very well supported in MacOS X, and is a decent printer (which, for bonus points, supports Apple's AirPrint and Google's Cloud Print standards).

The setup of the Epson was extremely pleasing - it has a little LCD screen and various buttons, which meant that I could power it up and join it to my WiFi network without having to connect it to a computer via USB at all. I then added it as a printer on my laptop (which was easy since the printer was already announcing itself on the WiFi network) and OS X was happy to do both printing and scanning over WiFi.

I then investigated the Epson software for it and found that I didn't have to install a giant heap of drivers and applications, I could pick and choose which things I had. Specifically I was interested in whether I could react to the Scan button being pressed on the printer, even though it was not connected via USB. It turns out that this is indeed possible, via a little application called EEventManager. With that setup to process the scans to my liking (specifically, Colour, 300DPI, assembled into a PDF and saved into a particular temporary directory), the hardware stage of the project was over.

With the ability to turn paper into a PDF with a couple of button presses on the printer itself, I was ready to figure out what to do with it next.

The software


As people with a focus on paperless workflows (such as David Sparks) have rightly pointed out, there are several stages to a paperless workflow - capture, processing and recall. At this point I had the capture stage sorted, so the next one is processing.

When you have a PDF with scanned images inside it, you obviously can't do anything with the text on the pages, it's not computer-readable text, it's a picture, but it turns out that it is possible to tell the PDF what the words are and where they are on the page, which makes the text selectable. So my attention turned to OCR (Optical Character Recognition) software. I didn't engage in a particularly detailed survey because I came across a great deal on Nuance's PDF Converter For Mac product and was so impressed with its trial copy that I snapped up the deal and forged ahead. I hear good things about PDFPen, but I've never tried it.

Automation


Having a directory full of scanned documents and some OCR software is a good place to be, but it's not a great place to be unless you can automate it. Fortunately, OS X has some pretty excellent automation tools.
The magic all happens in a single Automator workflow configured as a Folder Action on the directory that EEventManager is saving the PDFs into:

It will find any PDF files in that temporary folder, then loop over them, opening each one in Nuance PDF Converter, run the OCR function then save the PDF. The file is then moved to an archive directory and renamed to a generic date/time based filename. That's it.

That's it


Like I said, that's it. If you've been paying attention, at this point you'll say "but wait, you said there was a third part of a paperless workflow - you need tools to recall the documents later!". You would be right to say that, but the good news is that OS X solves this problem for you with zero additional effort.
As soon as the PDF is saved with the computer-readable text that the OCR function produces, it is indexed by the system's search system - Spotlight. Now all you need to do is hit Cmd-Space and type some keywords, you'll see all your matching documents and be able to get a preview. You can also open the search into a Finder window and see larger previews, change the sorting, edit the search terms, etc.

Future work


While that is it, there are future things I'd like to do - specifically I don't currently have an easy way to pull in attachments from emails, or downloaded PDFs, I have to go and drag them into the archived folder and optionally rename them. However, if you have your email hooked into the system email client (Mail.app) then it is being indexed by Spotlight, including attachments, so there's no immediate hurry to figure out a solution for that.

I do also like the idea of detecting specific keywords (e.g. company names) in the documents and using those to file the PDFs in subdirectories, but I'm not sure if I actually need/want it, so for now I'm sticking with one huge directory of everything.

2

A sysadmin talks OpenSSH tips and tricks

Posted on Tuesday, 7 February 2012

My take on more advanced SSH usage
I've seen a few articles recently on sites like HackerNews which claimed to cover some advanced SSH techniques/tricks. They were good articles, but for me (as a systems administrator) didn't get into the really powerful guts of OpenSSH.
So, I figured that I ought to pony up and write about some of the more advanced tricks that I have either used or seen others use. These will most likely be relevant to people who manage tens/hundreds of servers via SSH. Some of them are about actual configuration options for OpenSSH, others are recommendations for ways of working with OpenSSH.

Generate your ~/.ssh/config
This isn't strictly an OpenSSH trick, but it's worth noting. If you have other sources of knowledge about your systems, automation can do a lot of the legwork for you in creating an SSH config. A perfect example here would be if you have some kind of database which knows about all your servers - you can use that to produce a fragment of an SSH config, then download it to your workstation and concatenate it with various other fragments into a final config. If you mix this with distributed version control, your entire team can share a broadly identical SSH config, with allowance for each person to have a personal fragment for their own preferences and personal hosts. I can't recommend this sort of collaborative working enough.


Generate your ~/.ssh/known_hosts
This follows on from the previous item. If you have some kind of database of servers, teach it the SSH host key of each (usually something like /etc/ssh/ssh_host_rsa_key.pub) then you can export a file with the keys and hostnames in the correct format to use as a known_hosts file, e.g.:

server1.company.com 10.0.0.101 ssh-rsa BLAHBLAHCRYPTOMUMBO
You can then associate this with all the relevant hosts by including something like this in your ~/.ssh/config:
Host *.mycompany.com
  UserKnownHostsFile ~/.ssh/generated_known_hosts
  StrictHostKeyChecking yes
This brings some serious advantages:
  • Safer - because you have pre-loaded all of the host keys and specified strict host key checking, SSH will prompt you if you connect to a machine and something has changed.
  • Discoverable - if you have tab completion, your shell will let you explore your infrastructure just by prodding the Tab key.
Keep your private keys, private, private
This seems like it ought to be more obvious than it perhaps is... the private halves of your SSH keys are very privileged things. You should treat them with a great deal of respect. Don't put them on multiple machines (SSH keys are cheap to generate and revoke) and don't back them up.


Know your limits
If you're going to write a config snippet that applies to a lot of hosts you can't match with a wildcard, you may end up with a very long Host line in your ssh config. It's worth remembering that there is a limit to the length of lines: 1024 characters. If you're going to need to exceed that, you will have to just have multiple Host sections with the same options.

Set sane global defaults
HashKnownHosts no
Host *
  GSSAPIAuthentication no
  ForwardAgent no
These are very sane global defaults:
  • Known hosts hashing is good for keeping your hostnames secret from people who obtain your known_hosts file, but is also really very inconvenient as you are also unable to get any useful information out of the file yourself (such as tab completion). If you're still feeling paranoid you might consider tightening the permissions on your known_hosts file as it may be readable by other users on your workstation.
  • GSSAPI is very unlikely to be something you need, it's just slowing things down if it's enabled.
  • Agent forwarding can be tremendously dangerous and should, I think, be actively and passionately discouraged. It ought to be a nice feature, but it requires that you trust remote hosts unequivocally as if they had your private keys, because functionally speaking, they do. They don't actually have the private key material, but any sufficiently privileged process on the remote server can connect back to the SSH agent running on your workstation and request it respond to challenges from an SSH server. If you keep your keys unlocked in an SSH agent, this gives any privileged attacker on a server you are logged into, trivial access to any other machine your keys can SSH intoIf you somehow depend on using agent forwarding with Internet facing servers, please re-consider your security model (unless you are able to robustly and accurately argue why your usage is safe, but if that is the case then you don't need to be reading a post like this!)
Notify useful metadata
If you're using a Linux or OSX desktop, you either have something like notify-send(1) or Growl for desktop notifications. You can hook this into your SSH config to display useful metadata to yourself. The easiest way to do this is via the LocalCommand option:
Host *
  PermitLocalCommand yes
  LocalCommand /home/user/bin/ssh-notify.sh %h
This will call the ssh-notify.sh script every time you SSH to a host, passing the hostname you gave, as an argument.  In the script you probably want to ensure you're actually in an interactive terminal and not some kind of backgrounded batch session - this can be done trivially by ensuring that tty -s returns zero. Now the script just needs to go and fetch some metadata about the server you're connecting to (e.g. its physical location, the services that run on it, its hardware specs, etc.) and format them into a command that will display a notification.

Sidestep overzealous key agents
If you have a lot of SSH keys in your ssh-agent (e.g. more than about 5) you may have noticed that SSHing to machines which want a password, or those which you wish to use a specific key that isn't in your agent, can be quite tricky. The reason for this is that OpenSSH currently seems to talk to the agent in preference to obeying command line options (i.e. -i) or config file directives (i.e. IdentityFile or PreferredAuthentications). You can force the behaviour you are asking for with the IdentitiesOnly option, e.g.:
Host server1.company.com
  IdentityFile /some/rarely/used/ssh.key
  IdentitiesOnly yes
(on a command line you would add this with -o IdentitiesOnly=yes)

Match hosts with wildcards
Sometimes you need to talk to a lot of almost identically-named servers. Obviously SSH has a way to make this easier or I wouldn't be mentioning this. For example, if you needed to ssh to a cluster of remote management devices:
Host *.company.com management-rack-??.company.com
  User root
  PreferredAuthentications password
This will match anything ending in .company.com and also anything that starts with management-rack- and then has two characters, followed by .company.com.

Per-host SSH keys
You may have some machines where you have a different key for each machine. By naming them after the fully qualified domain names of the hosts they relate to, you can skip over a more tedious SSH config with something like the following:
Host server-??.company.com
  IdentityFile /some/path/id_rsa-%h
(the %h will be substituted with the FQDN you're SSHing to. The ssh_config man page lists a few other available substitutions.)

Use fake, per-network port forwarding hosts
If you have network management devices which require web access that you normally forward ports for with the -L option, consider constructing a fake host in your SSH config which establishes all of the port forwards you need for that network/datacentre/etc:
Host port-forwards-site1.company.com
  Hostname server1.company.com
  LocalForward 1234 10.0.0.101:1234
This also means that your forwards will be on the same port each time, which makes saving certificates in your browser a reasonable undertaking. All you need to do is ssh port-forwards-site1.company.com (using nifty Tab completion of course!) and you're done. If you don't want it tying up a terminal you can add the options -f and -N to your command line, which will establish the ssh connection in the background.
If you're using programs which support SOCKS (e.g. Firefox and many other desktop Linux apps) you can use the DynamicForward option to send traffic over the SSH connection without having to add LocalForward entries for each port you care about. Used with a browser extension such as FoxyProxy (which lets you configure multiple proxies based on wildcard/regexp URL matches) makes for a very flexible setup.

Use an SSH jump host
Rather than have tens/dozens/hundreds/etc of servers holding their SSH port open to the Internet and being battered with brute force password cracking attempts, you might consider having a single host listening (or a single host per network perhaps), which you can proxy your SSH connections through.
If you do consider something like this, you must resist the temptation to place private keys on the jump host - to do so would utterly defeat the point.
Instead, you can use an old, but very nifty trick that completely hides the jump host from your day-to-day usage:
Host jumphost.company.com
  ProxyCommand none
Host *.company.com
  ProxyCommand ssh jumphost.company.com nc -q0 %h %p
You might wonder what on earth that is doing, but it's really quite simple. The first Host stanza just means we won't use any special commands to connect to the jump host itself. The second Host stanza says that in order to connect to anything ending in .company.com (but excluding jumphost.company.com because it just matched the previous stanza) we will first SSH to the jump host and then use nc(1) (i.e. netcat) to connect to the relevant port (%p) on the host we originally asked for (%h). Your local SSH client now has a session open to the jump host which is acting like it's a socket to the SSH port on the host you wanted to talk to, so it just uses that connection to establish an SSH session with the machine you wanted. Simple!

For those of you lucky enough to be connecting to servers that have OpenSSH 5.4 or newer, you can replace the jump host ProxyCommand with:
ProxyCommand ssh -W %h:%p jumphost.company.com
Re-use existing SSH connections
Some people swear by this trick, but because I'm very close to my servers and have a decent CPU, the setup time for connections doesn't bother me. Folks who are many milliseconds from their servers, or who don't have unquenchable techno-lust for new workstations, may appreciate saving some time when establishing SSH connections.
The idea is that OpenSSH can place connections into the background automatically, and re-use those existing secure channels when you ask for a new ssh(1), scp(1) or sftp(1) connections to hosts you have already spoken to. The configuration I would recommend for this, would be:
Host *
  ControlMaster auto
  ControlPath ~/.ssh/control/%h-%l-%p
  ControlPersist 600
This will do several things:
  • ControlMaster auto will cause OpenSSH to establish the "master" connection sockets as needed, falling back to normal connections if something is wrong.
  • The ControlPath option specifies where the connection sockets will live. Here we are placing them in a directory and giving them filenames that consist of the hostname, login username and port, which ought to be sufficient to uniquely identify each connection. If you need to get more specific, you can place this section near the end of your config and have explicit ControlPath entries in earlier Host stanzas.
  • ControlPersist 600 causes the master connections to die if they are idle for 10 minutes. The default is that they live on as long as your network is connected - if you have hundreds of servers this will add up to an awful lot of ssh(1) processes running on your workstation! Depending on your needs, 10 minutes may not be long enough.
Note: You should make the ~/.ssh/control directory ahead of time and ensure that only your user can access it.

Cope with old/buggy SSH devices
Perhaps you have a bunch of management devices in your infrastructure and some of them are a few years old already. Should you find yourself trying to SSH to them, you might find that your connections don't work very well. Perhaps your SSH client is too new and is offering algorithms their creaky old SSH servers can't abide. You can strip down the long default list of algorithms to this to ones that a particular device supports, e.g.:
Host power-device-1.company.com
  HostkeyAlgorithms ssh-rsa,ssh-dss
That's all folks
Those are the most useful tips and tricks I have for now. Hopefully someone will read this and think "hah! I can do much more advanced stuff than that!" and one-up me :)
Do feel free to comment if you do have something sneaky to add, I'll gladly steal your ideas!

0

Evil shell genius

Posted on Monday, 23 January 2012

Jono Lange was committing acts of great evil in Bash earlier today. I gave him a few pointers and we agreed that it was sufficiently evil that it deserved a blog post.

So, if you find yourself wishing you could get pretty desktop notifications when long-running shell commands complete, see his post here for the details.

0

HP Microserver Remote Access helper

Posted on Friday, 6 January 2012

I've only had the Remote Access card installed in my HP Microserver for a few hours and already I am bored of accessing it by first logging into the web UI, then navigating to the right bit of the UI, then clicking a button to download a .jnlp file and then running that with javaws(1).

Instead, I have written some Python that will login for you, fetch the file and execute javaws. Much better!

You can find the code: here and you'll want to have python-httplib2 installed.

0

HP Microserver Remote Access Card

Posted on Thursday, 5 January 2012

I've been using an HP ProLiant Microserver (N36L) as my fileserver at home, for about a year and it's been a really reliable little workhorse.
Today I gave it a bit of a spruce up with 8GB of RAM and the Remote Access Card option.

Since it came with virtually no documentation, and since I can't find any reference online to anyone else having had the same issue I had, I'm writing this post so Google can help future travellers.

When you are installing the card, check in the BIOS's PCI Express options that you have set it to automatically choose the right graphics card to use. I had hard coded it to use the onboard VGA controller.

The reason for this is that the RAC card is actually a graphics card, so the BIOS needs to be able to activate it as the primary card.

If you don't change this setting, what you will see is the RAC appear to work normally, but its vKVM remote video feature will only ever show you a green screen window, with the words "OUT OF RANGE" in yellow letters.

Annoyingly, I thought this was my 1920x1080 monitor confusing things, so it took me longer to fix this than it should have, but there we go.

0

What is the value of negative feedback on the Internet?

Posted on Tuesday, 11 October 2011

I'm sure we've all been there - you buy something on eBay or from a third party on Amazon, and what you get is either rubbish or not what you asked for.
The correct thing to do is to talk to the seller first to try and resolve your problem, and then when everything is said and done, leave feedback rating the overall experience.

Several times in the last year I have gone through this process and ended up feeling the need to leave negative feedback. The most obvious case was some bluetooth headphones I'd bought from an eBay seller in China that were so obviously fake that it was hilarious he was even trying to convince me I was doing something wrong.
In each of these cases, I have been contacted shortly after the negative feedback to ask if I will remove the feedback in return for a full/partial refund.

This has tickled the curious side of my brain into wanting to know what the value of negative feedback is. The obvious way to find out would be to buy items of various different price and then leave negative feedback and see how far the sellers are prepared to go to preserve their reputations.

The obvious problem here is that this would be an unethical and unfair way to do science. Perhaps it would be possible to crowd-source anecdotes until they count as data?

0

Dear Apple

Posted on Thursday, 6 October 2011

I just woke up here in London and saw the news about Steve Jobs. It's early and, as usual for this time of day, my seven month old son is playing next to me. He has no concept of what my iPhone is, but it holds his fascination like none of his brightly coloured toys do. Only iPad can cause him to abandon his toys and crawl faster.

I'd like to thank you all, including Steve, for your work. You have brought technology to ordinary people in a way that delights them without them having to know why.

Please keep doing that for a very long time