makedo still paying off

Another day for confidence building.  I've been benefiting greatly from my build file being written in Haskell.  This the sort of situation I've been able to cope with way way better than my Bash script

problem A: recomputing LLR

1. fragment extraction takes as input: a sentence selection, data selection, and LLR
2. data selection takes as input: sentence selection and LLR
3. NEW: after data selection, we can recompute LLR (call it LLR2) - LLR2 takes data selection as input.  There are two different ways of recomputing LLR
4. What is effect of the LLR method on fragment extraction?

Solution: add a new LLRMethod argument to fragment extraction.  Fragment extraction now takes three arguments.  Furthermore, in computing the LLR, my fragment extraction function just passes in the data selection method it received in its parameters list.

data LlrMethod = LlrNoRecompute
               | LlrRecomputeDoc
               | LlrRecomputeSent

Problem B: decoupling token alignment

1. NEW: we want to change fragment extraction by separating an implicit token alignment step from it. Fragment extraction would no longer take a sentence/data dir, but a token dir
2. Token alignment depends on data selection, and LLR method...
3. There is more than one way to do token alignment

Solution B: add a TokenAlignmentMethod argument to fragment extraction.  Instead of passing in a sentence directory and data directory, I pass in a token aligned directory computed off the arguments passed to fragment extraction.

Here's what fragment extraction looks like now:

fragmentExtraction :: SentSelMethod -> DataSelMethod -> LlrMethod -> TokenAlignmentMethod -> ExactTarget
fragmentExtraction sm dm lm tm = ExactTarget path "" $ \ RedoArgs { rname } -> do
  redoIfChange [ tokenDir, llrDir ]
  createDirectoryIfMissing True rname
  runMyIO "detect-fragments" $ [ tokenDir, llrDir </> "lexicon", rname ] ++ parallelRTS
  recurseStampExt rname [ extEnChoice, extDataPerSentence ]
 where
  path       = d_FRAGMENTS </> showPC sm </> showPC dm </> showPC lm </> showPC tm
  tokenDir   = ePath (tokenAlignment sm dm lm tm)
  llrDir     = llr2Dir sm dm lm

Bonus to Solution B: I had forgotten that after fragment extraction comes a "peers directory" step.  Thanks to the Haskell type checker, I remember that I also need to extend the peers directory step with a token alignment method argument.

Before using my makedo library, I was trying to manage all this argument passing by deconstructing paths using basename and dirname.  Crazy!  Way way better to write your build files in a language that (A) has a notion of sum types or at least Enum types (B) lets you create your own types with minimal fuss (eg. I don't need a separate file per type, just declare them) and (C) statically type checks.  To be honest, the language for this probably does not need to be Haskell.  Even Java could do the job, although I think creating different enumerations like LLRMethod vs TokenAlignmentMethod might be more of a pain.  The advantage to using Haskell over Java (aside from the usual higher order functions, etc) is the HSH library, I think.  Maybe if there was a convenient way to write shell-script-like things in Java it could work.

But Bash or Perl script?  No way!

We missed the deadline for our paper by the way, probably because I was spending so much initial time working on makedo.  But I still think it was the right thing to do.  I would not have been able to understand what I'm working on as well as I could have if I was still wading knee deep in shell script.  Sometimes you just need to step aside and go abstract away some nonsense before you get to the meat of your job.  I hope that if we don't get a paper out of this, I can at least release my makedo library, and that somebody else benefits from it.

Take home message:

Statically typed shell scripts and Makefiles are good for you.

makedo pays off

Today was a good day, a confident day.

For a bit of context, I can be a bit insecure and defensive about the time I spend on "There Must Be a Better Way to Do This", time which I sometimes worry should be spent on "Doing This".  Unlike the smart Haskellers, my "There Must be A Better Way" does not tend to result in neat abstractions or boilerplate scrapping frameworks or what not.  A lot of time is just spent going around in circles, undoing past efforts, or just changing, changing from make to Redo + Bash, to Redo + Bash-in-Haskell to Redo + Haskell, change change change, churn churn churn.

I just spent a month scratching my head over my Complicated Sausage Machine, trying to figure out how not to make it hurt so much, and I think it finally paid off!  All that going around in circles, and I can now write this patch and this other patch.  I can now do the Thing I Was Actually Supposed to Do.  This is either something I would have had to do by hand (as in running commands by hand), or tried to stuff into my Complicated Sausage Machine, horrible fragile little piece of Bash.

So it's done... and I actually understand what the sausage machine is supposed to do.  The end result is much better.  Yesterday I remarked that Haskell version seems just as complex/long as the Bash one; turns out I just needed to bit of refactoring, refactoring which would likely have been harder in the Bash.  Again the beauty contest or Line of Code reduction isn't really the point.  It's about the clarity.  For the first time, I'm feeling confident that all my rewriting efforts were useful. The Haskell rewrite of the sausage machine now does more stuff than the original Bash one (not a superset, mind you) and is way easier to understand and by virtue of factorisation, has way fewer surprises in it, and is now much easier to extend.  OK the downside is that in order to extend it for future papers, you have to know a bit of Haskell.  But just in terms of mistakes avoided for this current paper, I think we have a net win.

Now we'll just have to see what this does for our paper.

makedo simplification - no more Bash

Those of you who have been following may know that for some reason I've been through this process of (1) handwriting a crazy Bash script to encode my processing pipeline for an upcoming paper (2) trying to replace my handwritten Bash script with an auto-generated one in the hopes that writing a program to write my program is easier than writing my program.

It's not.

I think the problem has been me dancing around Haskell because in a sense I know it's unlikely my successor will be able to make direct use of it... so I guess I've been pretending to myself that by generating Bash from Haskell, at least my successor would be able to make use of the latter if the Haskell isn't immediately useful.  But this is stupid.  Nobody is going to read that Bash, and the whole process of shifting back and forth between the two languages (gee, should I store this in a Haskell var, or write a piece of Bash to store in?) and besides, the code that this pipeline is running is in Haskell anyway.

What was I thinking?

I guess another part of me was thinking "gee, I know what my Bash script looks like and I don't know what the equivalent Haskell thing would be, so I'd better generate Bash so I can compare the two"

Foolishness.

Lots of time wasted due to that bit of foolishness.

Now I've dropped all pretence of not-using-Haskell and I'm a much happier code.  Just get to the meat of the issue.  I've also dramatically simplified my goals.  Previously I was trying to put together some kind of half-assed dependency tracking framework in the attempt to Eric-proof.  But as usual, I was solving the wrong problem.  The real pain I was trying to tackle was the combination of dimensionality and encoding parameter information in filenames.  This is stuff that I've resolved by just doing things in Haskell, no need to write a dependency framework.

Now my revised pipeline seems to be feature-compatible with the old one.  I don't have all the pieces yet.  But I'm close now, close to getting caught up with me from several months ago and being able to punch through to what I was really supposed to be working on, and any last minute requirement changes we discover as we flesh this paper out.

That's my excuse for doing all this exploration instead of just sucking it up and writing more bash by hand.  I could have just written more by hand, but sans doute we would have had to introduce some sort of last minute complexification to the pipeline.  Now that I've got all the power of Haskell programming at my disposal, I can handle it.

The new framework is much simpler.  Now that I'm not trying to think about the dependency tree, and I'm not trying to go for any sort of clever abstractions I can actually wrap my head around what I'm trying to do.  The heart of the matter is a simple list of filename to action matchers.  Basically, we take a single filename as input and we compare that filename against each of the possible targets

data Target = Target
  { match  :: FilePath -> Bool
  , description :: (String, String)
  , action :: RedoArgs -> IO ()
  }

For any targets that match the input file name, we invoke that target's action.

The other piece of this puzzle is the wonderful HSH library.  The great thing about HSH is how absurdly easy it is to mix and match pieces of shell script with Haskell.  You can write things like run ("cat foo" -|- "sed -e s/blop/quux/" -|- (take 42 . map toLower)) >>= B.writeFile "out".  Very nice use of Haskell typeclasses.

Much happier camper.  The whole time I've been working on makedo, I've been busy ignoring an increasingly loud/shrill voice in my head just saying "wrong, wrong, wrong, wrong".  Should have listened to it.  Bash-in-Haskell was not a good idea.  If I'm going to do Bash-in-Haskell, I might as well just do Haskell and be done with it.

makedo in anger!

Alright, after declaring that I was giving up on the makedo enterprise, I had a small idea which has allowed me to press on in my foolish quest against this Curse of Dimensionality problem I've been talking about.

So the basic approach has been to try to capture my pipeline in Haskell, but for comparability with the old Bash pipeline, to build up Bash syntax trees in Haskell and pretty print the output.  On top of this, the pipeline uses an obscure build system known as redo.  Rube Goldberg, I know.  All this because I think writing complicated Bash scripts is painful.  The hope is that by taking on this approach, I'll be able to deal with the problem of multiplying out variants of intermediary pipeline stages.  Basically I think I just want the list monad in Bash.  I don't know. I'm not trying to make this convoluted; I'm just violently allergic to the stupidity in my current bash script.  I somehow think that autogenerating said script will be less painful.

Anyway, the basic insight is that the thing I was trying to do was made complicated by the fact that I was trying to keep the bash script compact. Suppose we have variants of a "data selection" process, which in turn use outputs from variants of a "sentence selection" process.  What I think I was doing was contorting myself to ensure that we could have a single data selection target that captures the variants using wildcards in its path (data-selection-dir/*/*).  But I don't actually to do this!  The script is autogenerated; who cares about compactness?  I just keep the Haskell script-generator compact and let it fan out all the combinations of rules.

So that seems to work.  And now for the first time I've used makedo in anger.  Well, actually not-quite-anger because I'm just testing to see if I can replicate part of the old bash process.  But at least I've run my first test on the real server.  The rest of the work should just be trying to translate the rest of my process to Bash-in-Haskell and hope that I'm ultimately right about this making me less crazy.

makedon't

Argh

I ran into a snag trying to implement makedo, and think that in the interest of time I'm going to have write my script the bang-head-against-the-wall way, enumerating all the combinations of dimensions by hand and juggling the various arguments to programs, filenames to look up, etc.  This sort of thing drives me absolutely nuts, but hey... what ya going to do?  Paper has to get written.

I'm getting a slightly less fuzzy idea what makedo is trying to do, though.  Basically, I think there are 3 concepts we're trying to sift through:

• dependency: we have a notion of dependency because asking the human operator (Eric) to remember which bits of the pipeline to rerun is way too risky.  He will inevitably do the wrong thing one step of the way and subtly screw up your results.  So automation is the way to go.  The alternative is to encode the whole pipeline into a straight shot A-Z script, but that's too expensive for incremental development.  You cannot afford to re-run the whole A-Z every time you tinker with something because it would take forever.  You need a dependency-tracking mechanism to let you say "let me rerun step D" which then automatically figures out what you need to recompute as a result of that action
• variation: You are trying to explore different ways of doing steps B, C, D, E.  So this gives you a very small combinatorial explosion.  I guess a combinatorial burp, nothing that would frighten a computer, but the sort of thing that would confuse the heck out of a guy trying to write a script.  Human script author can't deal with your 32 different variants of the process
• iteration: You need to repeat the process the same way on the 100 individual inputs you have.  Iteration looks a  bit like variation.  One difference is that with variation, you have a presumably finite and pre-encoded set of possibilities, whereas with iteration, the set of files is not really encoded in the process but treated as a sort of external knowledge (either a set of files, or the result of running ls).  Also with iteration, you may have the concept of amalgamation.... you have done something to all the files in the directory, and now you want to treat the entire directory as a single object.

I'd gotten to a point where I could encode a good chunk of my pipeline.  It wasn't pretty because it was trying to write Bash in Haskell, but I was hoping that in cases where I had combinations of variations, I would have an easier time than writing the Bash by hand.  It would be an interesting case where composition/abstraction gives you a net win even though you lose out on concision/terseness.  Ie. the blocks are larger because of Bash-in-Haskell, but you can combine them in more powerful ways, leading to ultimately less crazy code.

But I ran into a snag where variants intersect.  I'm not sure I can explain it successfully.  Part of the problem is that because I'm working in a framework where I'm encoding information in filenames, I often have to do some crazy tricks to pass information from one process to another.  Say I have a bunch of sentence selection directories.  I can't pass the full directory names in via the filename unless I do some kind of escaping because I'm using filename components as parameters.  So /foo/bar/baz means "use the foo method for X, the bar method for Y, and the baz method for Z".  This means that I can't make bar itself be a filepath, like /foo/the/path/to/bar/baz unless I automatically munged it, eg doing something like /foo/the_path_to_bar/baz and making sure my munging scheme was reversible and unambiguous.  Also the result would be completely useless to a human being.  The idea I had was that since I was automatically generating the script, I could create a lookup table, something that looks like this:

function plookup {

  case "$1" in
    $'sentence_selection')  case "$2" in
                              BASELINE-s1)  echo \
                                                 "$SENTENCE_SELECTION_DIR"/BASELINE-s1 ;;
                              BASELINE-sA)  echo \
                                                 "$SENTENCE_SELECTION_DIR"/BASELINE-sA ;;
                              gtm)  echo "$SENTENCE_SELECTION_DIR"/gtm ;;
                              t060)  echo "$SENTENCE_SELECTION_DIR"/t060 ;;
                            esac ;;
  esac
}

The idea was that given a path (encoding some arguments to a "function") like /flurb/t060/toto/gargh, I could figure out what the path to the sentence selection directory was by passing the token t060 to the plookup function.   The auto-generated code looks like this:
sdir= $( plookup $'sentence_selection' $( basename $( dirname $( dirname "$1" ) ) ) )
This works more or less reasonably, and allows me to avoid hard-coding the information into the pieces of the my script, but the problem arises when variants combine.  So I now have a way of going from short tokens like "t060" to paths like "$SENTENCE_SELECTION_DIR"/t060, great! Now how do I do the same thing for data selection techniques?  Whoops, now we run into an issue, because the data selection techniques vary on the sentence selection techniques.  In other words, you have a set of data selection methods, and you want to try all the possible pairs of data selection and sentence selection methods.  How do you encode this as a short token, in filepath-friendly ways?

Argh, sorry, that's not very clear.  If you're reading this, you're probably thinking that it's fine and obvious, that you could just do something like representing tuples with some kind of delimiter, so gtm#foo to indicate the combination of the gtm sentence selection technique with the foo data selection technique.

Sigh!

Blather blather blather.  I wish I could be more articulate about what problem I'm trying to solve.  Almost feel like an inmate in a mental asylum, screaming some incoherent gibberish and wondering why nobody believes him.  Need to learn how to describe what I'm even trying to solve in the first place :-(

Anyway, giving up.  Rather than trying to autogenerate this script from a higher-level Haskell representation, I'm going to have to handwrite it instead.  No more time to think.  Just code-monkey around.  I do hope a more clever hacker runs into this sort of dependency/variation/iteration problem and figures out a solution.  Make the Shake build system is what I really want...

a few more steps towards makedo

I've made a bit more progress towards makedo, the build-system-based test harness generator.  Not a whole lot of the stuff because I don't actually know what I'm trying to accomplish.

Basically, all I know is that writing crazy scripts like this drives me crazy.   Right now, I'm trying to replace that crazy script with what I hope will be a less crazy Haskell program, except that the less crazy Haskell program doesn't actually look less crazy because it's trying to generate a Bash script.  Sigh.  Here's the Bash script that hopefully less crazy program generates as output.  You can see here the beginnings of dimensionality, the token alignment method being run off the results of different sentence selection methods.

I don't know what other people doing the same kind of thing I'm doing approach the problem.  Maybe I'm an idiot and everything is doing something obvious; or maybe they just don't care about the same things I do (for example, maybe they are perfectly happy to run a complex N-dimensional process by hand, just writing README files so that they always know what step to execute next, whereas I absolutely insist on automation); or maybe they're just wise enough to treat certain things as non-requirements (for example, maybe they have simpler scripts before they do not use caching of intermediate steps).

In any case, I still expect that I want the right thing but I just don't know how to get it. I think that involves some kind of abstraction.  For me, this means writing something in Haskell.  Right now, I'm reaching for the one abstraction I know, which is a monad, probably the wrong one.

Also, the way I'm currently approaching this is to use Haskell to generate Bash scripts, rather than just make my test harness be in Haskell (and execute IO actions).  This has some downsides, particularly that writing Bash in Haskell can be painful and ugly (gluing bits of syntax tree together)... but I suspect I'll have to go this route for a while longer until I actually have a better sense for what I'm trying to do.  One justification is that I already have a test harness in Bash, so it would be a lot easier for me to compare and see if I'm approaching what I want.

Anyway, so I guess today's work was:

• automatic dependency generation - if you list a target that has some dependencies, then the script will automatically include the targets for the dependencies
• dependency rebuild enforcement - if you pass in a target as input, the only way to get something from it will trigger a redo-ifchange
• rudimentary filename parsing - basically generating a chain of $(basename $(dirname $(...)))

towards makedo : a redo script generator

Not an announcement yet, mostly just a piece of vapourware.  I've long been complaining about the difficulty in writing and maintaining test harnesses for complex systems (where you want to try lots of variations of an experiment that you haven't even finished designing yet).  I call these test harnesses sausage machines.  I suspect that I'm fairly alone in wanting to build a sausage machine where some of my perhaps wiser and more sensible colleagues would just not see the problem of doing things by hand.

My attitude is generally that you should never trust a man to do a machine's job.  Humans are great, but we should be reserving them for things that humans are great at, not in trying to make them reliable executors of a procedure.

Only thing is that making sausage machines can be a very unhappy process.  I'm starting to get a glimpse of an idea for a tool that could ease some of this pain.  Using a bit of Haskell to strip away a lot of the conceptual boilerplate, and hopefully get things for free like "if you try to build off a dependency, you will automatically ensure the dependency is built", and also avoiding a lot of stupid code like building up and deconstructing filenames with basename/dirname when what you really want to do is to pass some arguments to a function.

I'm not entirely clear on what I want yet, I do have a piece of sample code up on darcsden.  If I get this right, I could be a much much happier user of sausage machinery.

sausage machine fabrication

Just jotting down some rough thoughts about this thing I complain about from time to time (starting with the Makefile of Doom blog posts during my PhD).  The kind of work we do requires the occasional construction of "sausage machines" - long multistep pipelines with lots of different components (first you lemmatise, then parse, then extract X, then compute Y then...).

The goal is to eventually nail down what I find difficult about sausage machinery.  The most desirable outcome would be to recognise what exactly I need to do (I suspect "write some Haskell could actually be a reasonable answer here - a lot of my grief comes from trying to use bash, I think).

My notes should also help me shed light on and perhaps even explain to other people why I use a build system (eg. Make, and now for me, redo).  Or maybe the intention is for me to be able to more effectively make the case for Automation (never trust a man to do a machine's job) and the use of sophisticated technology (run than a "straight shot" shell script)

1. repeatability - whatever you do has got to be repeatable 6 months after the paper has been submitted when the whole thing is a hazy memory
2. incremental/exploratory development - we don't know what the sausage machine parts are from A-Z; we're sort of making C, D, E etc up as we go; sometimes changing our minds about the components and wanting to replace/rearrange them
3. caching of slow computations - a lot of times the things we're cranking out take a long time to run.  We don't want to have to run them over and over again because we want to get some results out of it
4. dependencies between steps - if I change the tokenisation scheme, then the parsing results are no longer valid.  One possible solution is to write a "simple" shell script than just makes sure all the steps are run... BUT if we do this we have to think about caching the results from slow computations, and if we do THAT we have to think about what we need to do to make sure the caches are expired.  There's no way I'm trusting a human being to do this.  This is one of the reasons why I've always used build systems for my sausage machines
5. changing requirements (often at the last minute) - hey it's academia, we've got paper deadlines to meet
6. variations in intermediate steps - because there is a slightly scientific bent to this (which works better X Y Z)?, we want to support doing some of the intermediate steps in a different way and collecting all the results.  Note that there is a dimensionality problem here... if you have multiple intermediate steps that vary then you get increased complexity, not necessarily the kind that makes your computer blow up but the kind that makes your programmers sad
7. non-linear sharing of resources - the sausage machine is not a straight pipeline.  If you've got phases A B C D E, it's not always a simple matter of the results from step C flowing to D and then to E.  The wiring is more complex, with E taking as inputs bits of B and C too
8. fine-grained control - this is partly due to incremental development and slow computation  - if we just use a build system and we get it "right" by connecting the software components to the expiry conditions, we'll never get anyway.  The human operator needs to be able to lie.  My lie is NOT connecting the sausage machine dependencies to the source code.  This means that if I change an intermediary bit of software, my sausage machine won't know it needs to rebuild a step.  In theory, I should tell the truth; in practice, having the slightest wibble in my code trigger the whole pipeline would be insane.  I've got to be able to reach in anywhere in the sausage machine and say "just run component C again, please"... BUT if I do this, the sausage machine needs to know automatically to run D-Z as a result of this.
9. parallelism - hey it's 2011, we've got multicore, we've got the desire to get the results NOW NOW NOW.  Without adding much parallelism to the components itself, it can already be helpful for the sausage machine to run components in parallel to the extent that they are daggy and not line-y
10. remote operation - hack on your workstation, run on the workhorse.  I use version control for this; but actually maybe that's not always good because I also have lots of "foo" and "wibble" and "works now?" commits.  So maybe a combination of version control for mental stuff and one-button-mindless-rsync for the works now?" cases

Sausage machine construction tends to make me very grumpy.  I think I'm using the wrong tool for the job in the form of Bash, though.  The problem is that dealing with variation involves a lot of storing of intermediary results in directories (stage1-a/stage2-b/stage3-c), so I spend a lot of time doing a lot of filepath construction and parsing.  Somehow I sense that a lot of what I really want to express could be simplified with some sort of higher order function on contents-of-directories.  The problem with using Haskell is that well, writing things in straight Bash can be convenient... but I kind of suspect that being able to tackle abstraction in some way would make me go a little less crazy.

files in directories

This is a sort of problem you've heard me struggling with several times before.  I think I'm mainly interested in trying to find a succinct characterisation of the problem.  It's a sort of low-level combinatorial explosion... the order is very small and computers are perfectly fine.  The problem is the human.  This low-order combinatorial explosion introduces complexity [in the "OMG! this is so complicated!" sense] which drives the human operator crazy.

• You have a complex process with several stages.
• Each stage takes some input and produces some output
• You have lots of inputs - each input is represented by tuples of files
• You have variants of the stages

How do you lay out a file and directory structure?  For example, I built up this huge infrastructure that has pre-processing, followed by sentence selection, followed by data filtering and then fragment detection.  It actually kind of makes me think there is actually an interesting programming problem hidden in the mundaneness of it all.  You don't have a simple pipeline really because for example, the data filtering step takes output from pre-processing and sentence selection... (so this actually makes me think a bit of designing circuits, and I have the odd desire to whip out some Haskell arrows even though I don't really know what the latter are).

The problem is not the coding of the pipeline (which I'm doing in Bash coupled with redo); but it's the set of implicit and mind-numbingly mundane design decisions (should I name the files foo.bar or bar.foo).

I know one thing that went wrong that I'm cracking my head on.  I've got this setup where I'm storing things in one directory, and successively augmenting this directory with files.  So you've got a big directory called Workspace and in it, you have something like files: foo.{x,y}, bar.{x,y}, baz.{x,y}... and you augment it with a procedure that takes .x and .y files and produces .z files, which you then store in the same directory, giving you: foo.{x,y,z}, bar.{x,y,z}, bax.{x,y,z}.

Being a functional programmer, I find this to be deeply deeply wrong.  Adding files to directories is a sort of mutation, and mutation will always breaks your heart in end.  Ideally the workspace should be one set of files, and if you have something that produces another set of files, well that's a separate directory.

But mutation... it's so convenient.  I knew this could come back to haunt me.  What if the next stage of your pipeline involves taking inputs x and z and producing a new .a output?  That is, going from foo.{x,z} to foo.{x,z,a}.  OK, now what if you want to introduce a variant of the process that produces those .z files?  You're stuck.  You can't dump them in the same directory (lest you clobber the first variants' versions).  You can't put them in a new directory (unless you're willing to write another loop to copy or link the .x files)

So here I am trying to introduce a new sentence selection variant to see how it behaves.  I guess the right thing to do is to break my addiction to the one-big-successively-augmented directory.  But it means digging into each one of my sub-stages and changing them to accept multiple directories as inputs (look in dir1 for your x inputs, look in dir2 for your y inputs); giving them a bunch of logic to break down file names and put them back together etc; and then writing the scripting to give the right directories to the right programs.  But convenience always bites you in the end.

ABC subdirectory permutation

This is the sort of problem I find myself frequently confronted with, stuck on and bored by.  I feel like I've seen this enough times that I should have knocked out a set of standard tools to just get it right so I can move on with the interesting bits of the research, yet despite the handful of variations I've experienced, it always feels like a new problem wanting its own ad-hoc solution.

It's a simple permutation thing, a question of best-practice in laying out your directory.  You have a set of texts which can be characterised by N dimensions:

1. primary method (the thing I'm evaluating, eg. corpus-acquisition technique)
2. secondary method (some other source of variation, eg. generation system)
3. fold
4. evaluation set (training, development, test)
5. subcorpus

The first time I saw this problem, I only had #1, #3 and #4.  Then one day I was introduced to an evaluation which cared about #1, #4 and #5 (intersection between #4 and #5, it's important when dividing your corpus into subsets that you preserve the distribution of files in subcorpus, so if your corpus has 10 As, 40 Bs, and 30 Cs, and you're splitting into tenths, you need to make sure you still have 1 A, 4 Bs, and 3 Cs).  Now I think I'm dealing with most of these (except #4).

My failure to generalise over this problem, the sense that I'm continuously reinventing wheels that I'd previous reinvented, makes me feel stupid.

Some tricks I've learned along the way:

• Store your folds as a list of filenames, not the files themselves (obvious in hindsight!).  
  1. If you're doing repeated random sub-sampling, you save the space wasted by having more than one copy of the same file.  OK maybe you could also have avoided this by using hard-linking, but then what happens when you copy your stuff to another filesystem?  Do the hard-links get preserved? (Maybe, I dunno.  To be clear, I don't mean from filesystem A to filesystem B, but do the links between A/X and A/Y get preserved when in B/{X,Y}?  A simple ls -i can tell).
  2. It reduces the risk of human error that redundancy introduces.  Duplication is the root of all evil.  These arguments don't apply maybe if you're partitioning your data for cross-fold validation (no duplication), but there's still
  3. It makes it easier to communicate your folds.  For example, you do sanity checks like "does my partitioning of the fold into a further training and development set reproduce the original fold" with simple cat and diff.
• Handy tools that somebody else has written. You don't need to knock out your own 5-minute script:
  • split - split a file equal-sized chunks (useful for partitioning)
  • unsort - randomise lines in a file (randomisation)
• Version control randomisation. This goes back to the third argument for why you should store your folds as a list of filenames. You want to make sure you can reproduce every single step of what you do. So if you randomly partition your data for cross-fold validation, save the files containing your list of filenames
• Build systems can be abused as test harnesses.  I've been experimenting with an implementation of DJB redo and am loving it!  

Right now, I'm staring at my monitor dithering over various subdirectory layouts for my test bench.  Do I want my data divided into

• Primary-method/Secondary-method/Fold/Subset (tech-A/gen-I/fold-1/training)
• Fold/Subset/Primary-method/Secondary-method (fold-1/training/tech-A/gen-I)
• Fold/Primary-method/Secondary-method/Subset (fold-1/tech-A/gen-I/training)

Only 24 permutations to consider, I guess.  Which layout is going to maximise sanity-preservation?  The endless struggle.  Go on, you do the science.  I'm still trying to work out how to arrange my lab bench.  I wonder if I'm the only one that has issues with this.  It's just so embarrassingly inconsequential yet it feels like one of my main pre-occupations.

Edit Stepping away from the problem I realised that actually, I once inherited a solution to the first version of the problem using Fold/EvaluationSet/Text.Primary-Method (slightly outside the box -- who says it all has to be subdirectories)?  Still doesn't solve the Secondary method.  In a previous blog post, I wanted to compress the dimensions by merging the Fold and Primary method, but in hindsight, that doesn't make things any easier.

prone to confusion - filename extensions

One of the tensions I face in building up NLP pipelines is extensions for filenames.  Should extensions reflect a sort of format, or a stage of processing?

My code uses the latter, so I get something like:

• en
• en-selected
• en-fragments

And I can tell them apart.  They all start with "en" because they are compatible with each other in a sense.

stage of processing

• can tell the different stages of processing apart at a glance
• can dump all the files into one directory

format

The problem with the stage-of-processing approach, however is that one can get easily confused with the proliferation of extensions (wait, was that .data or .data-subset I wanted), and also it makes it impossible to use directory-wide graphical diff without writing some sort of wrapper process.

It also brings up sorts of side questions like "what if you want to skip stage 2 and pretend the output of stage 1 is the input to stage 3".  This sort of thing makes it much more natural to use format extensions.

Right now the compromise solution is to go with stage-of-processing extensions, and make copies reflecting snapshots of processing using format extensions for easier diffability, but meh

multi-annotation

For the corpus construction work, I'm finding myself in a position where I want to combine different kinds of annotation, say from a

1. domain-specific tokeniser
2. transliteration tool (iconv to strip accents)
3. lemmatiser/pos-tagger
4. named entity recogniser

For the moment, I'm keeping things real simple by just assuming a dumb pipeline (and ignoring NER).  That means making compromises like forcing ASCII text on all components, even those which are sophisticated enough to deal with Unicode, and also working with funny tokens, treating lemmatiser/pos-tagger as input to the corpus construction stuff, giving it tokens like "be_VBZ" instead of "was".

But over time I may find myself needing some sort of multi-track annotation approach because different components require different stages of input.

So I'm guessing here that I should use some sort of standoff-annotation scheme even though my needs don't really have any crossing/overlapping boundaries.  Everything is based on tokens.  I'm mainly attracted to a standoff annotation scheme because

• I want multiple layers of annotation.
• I don't want to decide in advance how many layers of processing I add (a simple column-based text format like tab-delimited would get unwieldy fast)

I'm a bit resistant to just using GATE for this (sorry, Java allergy plus a lot of my components are written in Haskell; and writing little Java stub wrapper programs is boring).  Turns out from a quick Google search that other generic tools exist (Amy Isard's slides [PPT]).  GATE is really neat and all (being able to cook up complex processing pipelines with arbitrary components is great, and I'll admit that having a GUI makes it a whole lot more discoverable), but really I just wanted to know about things like standard formats for standoff style markup.

Having glanced at Amy Isard's slides, I think I'll be trying the LT XML library next, hey maybe even make a Haskell binding to the library if the Unixy command line tools don't do what I want.

Going with the stupid pipeline first, though.

pipeline grumpiness, gate, ditaa

For some reason, a lot of my NLP work for the past 5 or so years has been about building pipelines.

Grumpiness

Here are the kinds of things that induce grumpiness during pipeline construction:

• deciding if the directory structure should be foo/bar/quux or quux/foo/bar or some other permutation
• deciding if we should dump the files into one file per item, or multiple items in a single file with one line per item (various variants) [one file per item may be convenient, but if you're on NFS it's really painful]
  
• munging file names (convert spaces into underscores etc)
• keeping track of all intermediary stages (for debugging)
• remembering how to run bits and pieces of the pipeline (mitigated by scripts)... it's not so easy to say "oh, you should just have a giant 'go' script", because you're always only hacking one piece of the pipeline at a time
• keeping track of multiple filestores: NFS home directory (convenience, backup), local hard disk (speed), local hard disk on the fast machine (speed 2), the global NFS share (space, sharing)

This kind of stuff is not even engineering, much less to say science. It's just dealing with... well, in a word... crap :-)

GATE

Speaking of which, I really really OUGHT to go learn how to use GATE, but I'm partly resisting it because [i] I'm enough of a programming language racist to say "eww, Java" and [ii] it smells like the kind of XML-laden horror that I think grumpy Unix nerds tend to dislike... but I can't say for sure yet.

It's a bad attitude to have. For example, I could be a happy Drupal user by now, but the "eww, PHP" reaction is just strong enough to make me keep wanting to find something else. Come on world... be a little more idealistic and a little less pragmatic ;-)

ditaa

I'd like to endorse a handy little program called ditaa. Ditaa reads in little ASCII-art diagrams such as the following:

+----------+      +------------+   +----------+
| DB       |      | field      |   | foo      |
| CSV      +----->+ extractor  +-->+ json     |
|      {s} |      | & parser   |   |      {d} |
+----------+      +------------+   +-----+----+
                                       |
                      +----------------+
                      |
                      v
+-----------+      +-----+------+   +---------+
| the web   |      | wibble     |   | titles  |
|           +----->+ search     +-->+         +
|      {s}  |      | cRED       |   |     {d} |
+-----------+      +------------+   +---------+
                                       |
                      +----------------+
                      |
                      v
                 +-----+-----+
                 | retrieval |
                 +           +
                 |           |
                 +-----------+ 

and spits out nice graphic files:


This is (so far) a better solution to a problem I previously tried to solve with:

• OmniGraffle: Pro - superb usability. No tutorials, just go. Con - non-standard format by default, Mac only
• Inkscape: Pro - powerful, SVG based, very portable. Con - you have to learn how to use it

Now we have a third solution which is

• not very powerful, but just powerful enough (for now)
• avoids the learning curve - you can just build off pre-existing skills (assuming that you know how to use block-mode editing in your favourite text editor).
• lends itself nicely to a plaintext oriented lifestyle (if you're grumpy enough to insist on using mutt or pine as a mailer instead of something graphical)
  
• produces nice human-readable diffs for version control.

All we need is a graphical editor that can save files in this format and which has the pick-up-and-go usability of Mac software

Now all we need is a friendly GUI program that can spit out these ASCII art diagrams.

naming things

One of the nitty gritty aspects of my work that I struggle with is the occasional need to name things (eg. directories). I worry if this just means I'm stupid or just linguistically (small-l) incompetent.

The challenge is to name things in a way which is (i) evocative - it means something to me right now and is readily accessible and which sort of says "hey, this is important" (ii) durable - which will mean something in 3 months when I've stopped working on this and (iii) accurate/precise.

An example of my current struggle, I have these two directories:

• A set of pairs of database entries with wikipedia articles. The set has gone through a document selection process in which a human being has marked the pairings as good, bad or unsure
• A set of processed pairings which have been split into sentences, filtered, massaged, jiggled and so forth. These pairings are suitable for training/testing by an SMT system and presumably other sort of training-based generation tools.

Right now the former is called "corpus-no-bad", which is evocative [for me] and sort of precise, because I created this by deleting the ''bad" bin from the pairing process, but it's certainly not very durable! The latter is called "corpus-for-smt" because I currently plan to plug it into an SMT system (*), but while the name is fairly durable/evocative, it's not accurate, because I don't intend its use to be limited to SMT. I could call it "corpus-for-training", but then there's the testing data in there too...

Part of the difficulty is that (a) there are actually a lot of things to name, so for durability's sake, you have to come up with names for things that maximise discernibility and (b) things tend to have decreasing relevance as you work, so you have to be careful about making the right things evocative and (c) it's not always clear that it's safe to throw something away.

Corpus-no-bad was perfectly fine when this was all I was working on, it was the big deal, it was THE corpus. Now it's not the corpus. Eh, it's just some input that's used to make this other thing which is now THE corpus, and one day eventually that thing will also fade into irrelevance. But even though it's no longer THE corpus, it still maybe kinda important and anyway you're still using it to build the new THE corpus, so you can't delete it just yet. The result? Proliferation of directories with haphazardly chosen (and increasingly meaningless) names... OR some sort of crazy unnavigable directory structure 26 layers deep. Madness one way or the other.

Anyway, after a good half hour of dithering, I think I'm moving towards renaming "corpus-no-bad" to "selected-doc-pairs" [which I think is more durable, but less evocative] and maybe either just renaming the latter to "corpus" or just not taking things so literally and leaving it as "corpus-for-smt"

(*) Which is an immense source of frustration by the way, the calibration of Rube Goldberg machines. This sort of informatic bureaucracy drives me absolutely up the wall, which is odd because one would think that autistic-spectrum types (the evil of self-diagnosis :-P) like details or something...

the "one-off" directory

Another useful convention when building pipelines or just doing any sort of computer-assisted (corpus driven?) researchy/academic task.

Problem

Sometimes you just need to do one-time transformations on your data. Just this once, you need to grep your data for all the pictures of people with big noses in them. Or just this once, you need to find the ones with funny hats.

Integrating it into your pipeline is kinda silly because you really only just need this thing fast, and like, just this once. It imposes a kind of cognitive overhead that leads to paralysis. Really, this is the sort of thing you just bang out on the command line and be done with it...

Except of course...

It's never really just the once is it?

Proposed solution

Within the project directory (eg. hills-corpus), create a directory called "one-off". The idea is to have a place where you can sweep up all your disposable code. Things you type out on the command line, you should really copy and paste and turn into "one-off/some-foo.sh".

Why I think this works

First: this means you preserve your operation for posterity. I've used some of my supposedly "one-off" scripts (be they in Bash or Haskell) dozens of times. Not so one-off after all.

Second: they reduce the cognitive overhead of trying to figure out what the "right" place to put them into your pipeline (eg. Makefile of Doom) is. You don't even really have to give them sensible names either, because they're really just swept up under the one-off rug, and you don't really expect to see them again anyway (oh but you will, my friend, you will...)

Third: it eliminates clutter. Previously, I was dumping these one-off scripts in my project root (for fear of losing them). The problem with clutter is that you come back to it 3 months later, and you have no idea what you can safely delete, on what you need to hang on to. Now they have some place to go. In the worst case, they go, but that's not so bad.

incremental development + resource constraints = messiness

Say you've got a pipeline that searches for Wikipedia articles, retrieves them and maybe does some post-processing on them.

You later on decide that you need to do some extra work on the results, say parse the disambiguation pages, retrieve the disambiguation results and then do some post-processing on them.

Now, the pipeline takes a long time or a lot of resources to run, so you definitely do not want to run it again. You've already got your WP articles; no point fetching them again. If you had had the foresight to plan for the disambiguation stuff from the beginning, then it would all be part of one nice happy pipeline, but you didn't, tough luck.

So what do you do? You develop a separate post-processing script that operates over the results of your first processing step.

... and?

You've just created a mess. Now you've got yet another script in your pipeline that you need to tell somebody to run (editing your README file), or that you need to augment your pipeline with for future use (which you are not motivated to do because it does not solve any immediate problems). Not only that, you're faced with the dilemma of whether or not you should keep both the original data fetched from the pipeline, plus the additional disambiguation results [for future tracking/reproducibility]; or if you should just mash them all together [for simplicity].

This is the kind of organic incremental growth that leads to a general feeling of sprawl and out of controlness. I don't foresee any solutions.

• One could plan better and try to get things right the first time, but that's mostly like saying one should just be more intelligent, and capable of anticipating changes down the road... "Be more clever" is just not acceptable answer.
• One could use more pilot studies to flesh out as much of the toolchain as possible on small amounts of data (which eliminates the resource constraint problem). This is probably a good idea, but it only goes so far, in the sense that you sometimes need to work with large amounts of data before you can become aware of the problems that lead you to augment your pipeline.
  
• Or maybe one could design processes to easily accommodate these kinds of incremental changes in a tidy manner.

I haven't figured out the latter bit yet, and so I've got yet another Perl script in my burgeoning toolchain that I won't know what to do with three months from now.

more pipeline hacking

There's a constant tension between wanting to make everything work as a single "go" operation and having things be broken down into nice interchangeable modules that you can later isolate. When building the "go" button, you don't always have a good idea what the modules are (well maybe you do; I certainly don't) and you only discover them after the fact. So say your big go button does jobs A B C and D. And later on, you discover that you need to factor out components B and C into a separate standalone utility.

Now what?

Now you're stuck: you need to get the job done NOW so you probably create the module by doing some copy+paste programming. No time to think and do the job right, just get in done, go. But of course you're just creating more misery for down the road because in doing copy+paste programming you've created duplication and we know the lesson from being bitten over and over and over by it:

duplication is the root of all evil
But what else are you going do? Well you could take your time to do things right, but that's not necessarily the right answer either. First, there's always deadlines looming. Second, sometimes the only way to discover what the "doing things right" means involves doing them wrong a few zillion times until your brain gets the idea.

Right, let's get to it...

Then later on we're going to have to address this most heinous duplication between the B+C mini-go operation and the big go operation. I'm sorry I keep writing about the same thing over and over again; it's just that I haven't found the words to say this concisely and precisely yet. I haven't figured out how to drive straight to the heart of the matter.

reproduced results!

Hurrah! I've been able to reproduce the results from our paper last year. That's a good TWO WEEKS of work lost to poor methodology.

OK, so what do we learn from this?

Lay directories out sensibly

I should do the things I mentioned in my series of posts about toolchains, particularly the one about directories. Part of the mess was due to the fact that the paper last year was before I wrote that toolchains post, and I was still doing things in a pretty random way.

Embrace change

This is actually me attempting to defend something which made my life more difficult these two weeks. Part of the difficulty in reproducing things is that I had started shifting to a less random way of doing things, rationalising the names I gave to systems and directories, rearranging tree structures. This made life pretty awful because I couldn't just whip out a graphical diff and say "ah, that bit is different with that other bit"; I had to write a bunch of stupid scripts to rename things from the old way to the new way. So that cost me a lot of time and grief, but then what can you do?

In my opinion, the wrong answer is to say "don't change, because stability is at a premium." The problem with that answer is that when you fail to change things, you propagate unsustainable practices that just cause you increasing amounts of pain down the road. If you're balancing on a slippery rock standing on your big left toe and sticking your other leg in the air, staying vewy still isn't actually going to give you any stability. The way to get the stability is to step off the stupid rock and put both feet on the ground. So if you're doing things a stupid way, you HAVE to change them or suffer the consequences. Change to gain stability. Of course, change has a cost which I've definitely been paying a lot of; but the cost of changing must also be put into perspective against the cost of NOT changing.

Archive and date

I was lucky because I found somewhere a directory labeled delete-after-2009-11/eacl-2008-09-29 that basically contained everything I needed to contain the old stuff.

As I mentioned in my last post: two digits for the conference, four digits for the archival date. It matters.

Use stow; use /usr/local

This is just expanding on sensible directory layout. If you're going to be working with a bunch of third party tools like GIZA++, SRILM, and moses; make sure you install them with stow (whenever possible). The idea is that you can install them into directories like:

• /usr/local/stow/gizapp-1.0.3
• /usr/local/stow/gizapp-1.0.4
• /usr/local/stow/moses-2008-07-11
• /usr/local/stow/moses-2009-02-04

What this lets you do is selectively activate and deactivate different versions of packages. If you do cd /usr/local/stow; stow gizapp-1.0.4, this will dumbly create symbolic links from everything in /usr/local/stow/gizapp-1.0.4 to the equivalent location in /usr/local. If you do then do cd /usr/local/stow; stow -D gizapp-1.0.4, this will delete the symbolic links, making it possible for you to swap a different version in.

Now some tools will not have an installation method, for example, SRILM. I would not bother trying to shoehorn one into them. Just stick them under (for example). /usr/local/SRILM-1.5.5.

This sounds like a stupid detail, but hey, I just wasted two weeks, partly because I didn't have any idea what all the versions of all the crap I was using because I didn't have a consistent methodology for installing such unpackaged dependencies. They were either going in my home directory, in some local space on the hard drive, in /opt. Huge mess. Attention to detail matters.

loria day 1

First day back at the LORIA.

wxHaskell

Last night, I tried again to install wxHaskell using the new recabalised wxcore. This time for no apparent reason, it all magically worked! I can even run sample applications with it. Very encouraging. This means that we are one step closer to cabal install GenI just working with graphical interface and all.

admin

Today I've been doing paperwork and very shyly saying hello to people from the lab... or sometimes not and just hiding behind my screen instead.

integration

I've started rebuilding XMG-GenI integration (the Makefile of Doom) from first principles. I've learned a few things about integrating lots of software components from my Brightonian hacking so I'm hoping that these experiences will make things a lot smoother and more reproducible. I'm pleased to say that I now have an install.sh script which appears to work.

The thing I'm aiming for most is stability: this means that everything is set to a fixed known last-working version which I bundle with the tool chain or constrain with an installer version. With the exception of Mozart and the Haskell platform, I am only installing software in the toolchain's dist directory and not on the machine. We want things to be reproducible and self-contained. I tried hard to make them so in the PhD years, but I failed, partly because I was also trying to do disparate tasks ("experiments"). Now that the experiments pressure is off, I can focus on just a simple task of making the damned thing run. Hopefully with the whole thing rebuilt from scratch we can go back to thinking about how to do "experiments" from there. The integration problem isn't easy. The whole thing is about tradeoffs that you don't know you're making. It takes a lot of experience and burning of fingers to realise what those trade-offs are.

Tradeoff: The experiments tried to be varied + self-contained but at the price of duplication. Duplication is the root of all evil. Because of duplication, my users had the experience of 10 million Makefiles (all self-contained) none of which worked very well. I started trying to explain why things were so bad, but I don't think I was very successful; it sounded more like I was trying to make excuses, which you never want to be doing. Now I'm going for the other side of the trade-off. I want things to be self-contained with minimal duplication, but this means there's going to be less variety. We'll figure the variety out later.

Tradeoff: To save time, the toolchain did all sorts of fancy things like cutting down the metagrammar to just the pieces that we want to recompile. But these fancy things create complexity and a maintenance hassle. Things break because of it. You can't win. Either you have something fast and ugly or you have something slow and nice. Hrmph. Getting grumpy.

installing stuff

I'm squatting a desk with Ubuntu Jaunty. Installing the Haskell Platform on it is no walk in the park sadly. It's super easy on Windows (I believe) and on MacOS X, but on Ubuntu you have to:

• install GHC binary
• install OpenGL stuff (libglu-dev, I think)
• install editline (libedit-dev, I think)
• patch the installer because of bug #84
• and finally install the platform

I'll still require the platform. It's just easier in the long run, when that beautiful future arrives where HP is available on your favourite distribution. We're getting closer and closer. Haskell with batteries included.

toolchain/pipeline - coping with slowness

One aspect of evaluation toolchains/pipelines is that they can often take a long time to run. I suspect this will be true for as long as we'll be writing these things: the faster our computers and algorithms get, the more ambitious we become.

So we need to make sure we get better at managing time. How do we go about doing that?

Option A: One thing we could do is to become better programmers and computer scientists, i.e. write code more intelligently. Right, step 1, become better computer scientist. Check. But then what if you're as dumb as an Eric and pretty much never going to become a better computer scientist?

Option B: learn a bunch of stupid tricks.

set aside the slow stuff

Thanks to my time in Nancy, I've grown to appreciate having one single go.sh script that I can run that does EVERYTHING... except the slow stuff. The slow stuff I set aside in a different fire-and-forget script, for example resolvers-go.sh, because running the resolvers takes a long time.

This can be beneficial because you'll often find yourself with fast little tools that you modify frequently... so what you want to be able to do is just to tweak the tool (say you modified it to print the rows of your table in order of the best score) and just mindlessly rerun go.sh. If you separate out the slow stuff (that doesn't need to be re-run), you are then free to re-run your go script as often as you want.

It's probably also a good idea for your go.sh to detect if you have forgotten to run your slow-thing.sh

Mindlessness is good. The cognitive friction involved in "wait, which script do I need to run again, and which parameters do I need to pass to it" is costly. I don't know why really. Just having to think about that stuff makes you stupider. So make things mindless so you can go spend your time being clever instead of scratching your head at scripts.

use your number cruncher

Somewhere in your lab, somebody ordered a REALLY REALLY fast machine. Don't forget to use it! It sounds obvious, but I didn't think to do it until I was three-quarters through this evaluation exercise. Would have saved me a lot of grief.

That you often need to be running things on more than one machine is why it's a very good idea to make sure your toolchain is as portable and EZ-install as possible (e.g. following the tips in my last post and more). You want to be able to just rsync that crap over and hit go.sh. No sitting around trying to figure out what extra junk to stick into your .bashrc

use screen

The great thing about screen is that it allows you to run something, log out, go home so you can have dinner [leaving the thing crunching away in the background], log back in when you're done with dinner and resume staring at progress bars. Screen effectively removes the requirement to be physically present at our machines all the time from our natural compulsion to stare at progress bars.

Screen is particularly useful in conjunction with your fast machine. You can ssh in from work, detach your screen, log-out, ssh-in from home, reattach your screen.

use rsync

Don't forget that you'll be modifying your toolchain (fixing bugs) and re-running it a whole bunch of times. I guarantee it. You will be running this thing many many times before this evaluation is over.

I have this script I call copy-to-fast-machine.sh in my toolchain directory.

#!/bin/bash
rsync -av `dirname $0` --exclude='*/dist/*' fast-machine:my-toolchain

So I just do

eval `ssh-agent`
ssh-add
cd my-toolchain
./copy-to-fast-machine.sh

Then every time I fix something in the toolchain, I just sigh once, hit the up arrow in bash, and rsync everything over again. Notice in particular that this script tells rsync to exclude the dist directories. This is useful to avoid having the fast machine get confused because you compiled something on your home machine, and then recompile everything out of that confusion.

Pay attention to detail so that you can maximise the mindlessness.

parallelise the heck out of things

Today's happy discovery is a shell script called PPSS. I've always been meaning to figure out how to do clever things like running stuff in parallel (and on multiple machines). Turns out, I don't actually have to be all that clever. The above is a simple shell script that figures everything out for you. You just feed it a list of files, and a command to execute on any one file (e.g. "${SCRIPT_DIR}/opennlp-wrapper/run-opennlp.sh ${INDIR} ${OUTDIR}/openNLP \$ITEM" ($ITEM is a placeholder for the file in question) and BANG, instant maximum resource consumption, all N processors happily crunching away.

conclusion

There's nothing here that counts as rocket science, just stuff that I typically forget to do before it's too late. If you just put these tricks together in your toolchain, you can save yourself a lot of time and open the door to re-running things over and over and over again (which is good if you are like me and prone to screwing things up).

So hopefully, if I ever find myself writing another one of these toolchains, I'll actually remember to do these things from the get go.