pipeline grumpiness, gate, ditaa

For some reason, a lot of my NLP work for the past 5 or so years has been about building pipelines.

Grumpiness

Here are the kinds of things that induce grumpiness during pipeline construction:

• deciding if the directory structure should be foo/bar/quux or quux/foo/bar or some other permutation
• deciding if we should dump the files into one file per item, or multiple items in a single file with one line per item (various variants) [one file per item may be convenient, but if you're on NFS it's really painful]
  
• munging file names (convert spaces into underscores etc)
• keeping track of all intermediary stages (for debugging)
• remembering how to run bits and pieces of the pipeline (mitigated by scripts)... it's not so easy to say "oh, you should just have a giant 'go' script", because you're always only hacking one piece of the pipeline at a time
• keeping track of multiple filestores: NFS home directory (convenience, backup), local hard disk (speed), local hard disk on the fast machine (speed 2), the global NFS share (space, sharing)

This kind of stuff is not even engineering, much less to say science. It's just dealing with... well, in a word... crap :-)

GATE

Speaking of which, I really really OUGHT to go learn how to use GATE, but I'm partly resisting it because [i] I'm enough of a programming language racist to say "eww, Java" and [ii] it smells like the kind of XML-laden horror that I think grumpy Unix nerds tend to dislike... but I can't say for sure yet.

It's a bad attitude to have. For example, I could be a happy Drupal user by now, but the "eww, PHP" reaction is just strong enough to make me keep wanting to find something else. Come on world... be a little more idealistic and a little less pragmatic ;-)

ditaa

I'd like to endorse a handy little program called ditaa. Ditaa reads in little ASCII-art diagrams such as the following:

+----------+      +------------+   +----------+
| DB       |      | field      |   | foo      |
| CSV      +----->+ extractor  +-->+ json     |
|      {s} |      | & parser   |   |      {d} |
+----------+      +------------+   +-----+----+
                                       |
                      +----------------+
                      |
                      v
+-----------+      +-----+------+   +---------+
| the web   |      | wibble     |   | titles  |
|           +----->+ search     +-->+         +
|      {s}  |      | cRED       |   |     {d} |
+-----------+      +------------+   +---------+
                                       |
                      +----------------+
                      |
                      v
                 +-----+-----+
                 | retrieval |
                 +           +
                 |           |
                 +-----------+ 

and spits out nice graphic files:


This is (so far) a better solution to a problem I previously tried to solve with:

• OmniGraffle: Pro - superb usability. No tutorials, just go. Con - non-standard format by default, Mac only
• Inkscape: Pro - powerful, SVG based, very portable. Con - you have to learn how to use it

Now we have a third solution which is

• not very powerful, but just powerful enough (for now)
• avoids the learning curve - you can just build off pre-existing skills (assuming that you know how to use block-mode editing in your favourite text editor).
• lends itself nicely to a plaintext oriented lifestyle (if you're grumpy enough to insist on using mutt or pine as a mailer instead of something graphical)
  
• produces nice human-readable diffs for version control.

All we need is a graphical editor that can save files in this format and which has the pick-up-and-go usability of Mac software

Now all we need is a friendly GUI program that can spit out these ASCII art diagrams.