second new feature facilitated by the Great Refactor

All this makefile refactoring is starting to pay off. I'm trying out another new feature, introducing a new test where we get rid of some commas. I want to keep the old generation method around for comparison. So just 4 more targets in the makefile and extensions to 2 variables. Relatively painless.

Before the Great Refactor, I would have gotten myself deeper into a tangle of one-off scripts.

I think the lesson here is that while fighting to reduce duplication of information may look foolish when you're deep in the process, it really does pay off even in the short term.

one last refactor? please?

Yes, yes, I did spend yet another day refactoring the makefile. It's an itch you just have to scratch, getting rid of cruft. But now I really think this is it, dropped the footprint on this sucker by homogenising the corpus copying step (each system now uses the same corpus files with the same file names!). This had better be it. This is starting to make me irritable, because every time you fix and simplify something, something else breaks.

Of course, the answer isn't to run away from fixing and simplifying.

The answer is to realise that the reason something else broke is because you designed things badly in the first place so that things were fragile. So you keep doggedly fixing and simplifying, chipping away at the thing until you get to a state where you can add/modify a piece without the whole Jenga tower crumbling down. Only 500 lines of Makefile left...

the Test Harness Problem

I'm sorry to keep yammering on about this very technical and stupid point. I think what I'm saying is that it's not just technical and stupid. There actually are some engineering issues behind this. In principle we are computer scientists and not software engineers. In practice, we can't really get away with trying to do the computer science (in this case the NLG research) without paying some attention to software engineering. In fact, we have been trying to get away with it, and it seems that doing so works, but it only appears that way because you've been offloading the pain to somebody else... pain that really originates from failure to address the software engineering (process engineering?) issues behind setting up a test bench. Let's stop inflicting this kind of pain on our future selves.

desiderata

• Reproducibility - in your ACL 2014 paper you talked about these experiments you ran with such and such system producing such and such data. Given the contemporary software and data, can you actually reproduce the experiments? Can you get the same data back or explain any discrepancies? Why did one of your systems suddenly jump ranking?
• Extensibility - you've got this great ACL 2014 process, but now you want to try some new ideas out for the ACL 2015 paper you are in the middle of writing. Can you extend your shiny new test harness with a new process?
• Granularity - you want to fiddle with your harness a bit, turn a knob, change a weight, add a normalisation step. First of all, is this parameterisability possible? Second of all, can you do it such a way that you don't have to run the WHOLE test process (which at the time of this writing could take hours if not days) again?

pressure points

• Dimensionality - Dimensions multiply. You're in for a combinatorial nightmare. The dimensions I have to contend with:
  1. folds (cross-fold validation)
  2. data set (test vs train vs dev)
  3. methodology (what you're actually testing on)
• Variability - Each point in each dimension brings a different way of doing things. The test data has to be treated different from the training data (for example, you might want to lump the training data into one giant corpus, but oh no, can't do that with the test data; it has to be one file per datum). Each method has its own idiosyncratic toolchain (and requirements for how the data are normalised)

conflicts

reproducibility vs granularity

To improve reproducibility you are tempted to create an automated toolchain... just hit the go button and redo everything. But if you're not careful, you lose out in granularity (because if you change something and just hit go, you re-run the entire toolchain, putting you out of business for a couple of days). A highly reproducible solution would be a single Mother of Shell Scripts... but it's not a very granular one. A highly granular solution is a graphical interface that lets you plug in your data sets, turn some knobs and then hit go... but it's not very reproducible (unless you dump a shell script from the GUI, in which case the shell script is not very granular, because you wouldn't be able to selectively re-run bits).

This is why the various incarnations of Eric were fighting amongst themselves about whether to use make or shell. I've settled on make because it is a fairly decent compromise between reproducibility (make everything) and granularity (make just-that-one-step)

granularity vs extensibility

On the one hand, you want to keep things extensible by reducing duplication of information and breaking things down into nice modular blocks... the idea being that if you want to add something you just recycle some of your modules and add in the difference. On the other hand, highly factored code isn't always very granular. Now if you turn a knob, you are turning the same knob throughout the ENTIRE harness, which may or may not be desired.

The compromise between granularity and extensibility is parameterisation - use more functions instead of variables. Unfortunately, my current understanding of make is that it is not a very natural tool for such parameterisability. I am sorely tempted to go re-write the whole damn test harness in Haskell, using something like Franchise to do the dependency tracking for me. (but of course, then it's less reproducible and extensible in a sense because not everybody hacks Haskell... not everybody hacks makefiles either, but at least there is a chance...)

extensibility vs reproducibility

You want to keep things reproducible by homogenising the process and making everything fit into the same boring template. Basically here I am equating reproducibility with transparency, which may or may not be wise. You want people to be able to just hit the green "go" button, right? But, because of the variability problem, you are tempted to go for quick fixes and local one-off solutions whenever you want to tune your harness, twiddle a knob, add a methodology. You write a one-off shell script for your local problem which reduces your reproducibility by introducing all sort of weird processes that your future self has to learn. Why can't I just have a go button?

You can't help doing it in this short-term way because the deadline is only 3 days away and you need the results NOW. Or even if you weren't facing time pressure, the variability behind the problem obscures the commonalities that would allow you to recycle things. Anyway, life forces you to add yet another shell script to fix just that one problem you're trying to solve. In doing so, you punish your future self by reducing the reproducibility of your harness.

This conflict is mostly due to pressure from variability.

Interestingly, you can also see this as a conflict between short term extensibility (add a shell script now) and long term extensibility (add a framework so that future you doesn't have to wade through a thicket of shell scripts). But here I am focusing purely on reproducibility for somebody else (e.g. Future You) vs short term extensibility.

conclusion

The claim I making in this blog post is that the Test Harness Problem is a difficult one. It's not difficult in a deep and scientific sense, but in an embarrassingly fiddly way. I hope that in writing this, I have made some progress convincing you that spending a fortnight on a stupid test harness is entirely a reasonable thing to do, or at least provided a somewhat more thoughtful characterisation of the fiddliness of the issue.

Note that what I have not tried to do in this article is to convince you that you actually want a test harness... but I think it should be pretty clear that the three desiderata (Reproducibility, Extensibility and Granularity) apply whether or not you just cobble together some ad-hoc shell scripts (or even just rewrite things on the command line), or put together something which you formally deem to be the harness. The ad-hoc solution of not having a test harness is really equivalent to having a super-crappy harness with zero reproducibility... and that I think is harmful to your future selves and to other researchers interested in your work.

So writing a test harness may be hard, but I think it's well worth it. The only danger is that you may find your PhD students or Research Fellows disappearing for weeks as they go off and make Yet Another Refactor to the harness whilst desperately hoping that their efforts are not in vain. But then go write another paper, try to rerun some numbers and you will thank him/her for making that initial investment.

out of makefile refactoring land

Ok, one more half-day of makefile refactoring... just itchy fingers this time, listening to the Evil Voices in my head that tell me "no, it would really be a lot cleaner if you re-arranged the furniture this way". Spent a day moving logic from the outer makefile into the fold-specific makefile... which is good in that now all the intelligent stuff is in one place (less duplication of information; or at least all the same information belongs in the same file even if it's duplicated)... In principle, we can do away with the outer makefile altogether and just replace it with a dumb shell script, say running make on each fold in parallel.

It's pretty frustrating to be doing this kind of work because you feel like you're just running around in your little hamster wheel... and also because it's hard to justify to other people. No, it's not (just) because I'm fastidious. In the past, we have suffered a lot from the non-reproducibility of our results. We had these numbers somewhere in a paper, and had no idea what was the procedure needed to re-generate them. That's what we want to aim for, I want to make it so that for any paper I write, I can somehow go back in time and run exactly the same toolchain that generated the results from that paper. I didn't quite get there with the Mother of Makefiles from my PhD, and I'm certainly not getting there now... but we are slowly approaching this goal little by little. No more fog-of-research mysteries.

doing new stuff!

I'm finally adding new functionality to this makefile! Teaching the makefile how to train MOSES on segments instead of on full forecasts. A bit tedious because you have to create a corpus file, including normalisation, for each segment, test on that segment, de-normalise, re-constitute the forecasts by concatenating the segments. None of these are inherently difficult; it's just dumb Unix commands. But you have to do them.

So I have to work out the boring stuff there... but at the very least, I don't have to machete my way through the mess of quick-and-dirty solutions I had left behind from the EACL paper. It's pretty clear where all the new stuff "plugs in" to the test harness.

Yes, so at the very least 2008-11-27 Eric would like to thank 2008-11-24/2008-11-26 Erics for slogging through and cleaning up the test harness so that adding this new generation technique is slightly less of a pain in the behind.

hooray!

Done! That wasn't so bad; only too half a day (the good thing is that MOSES does the right thing when you have a test file with more than one line). And thanks to the cleaned up makefile, the test harness pretty much does the right thing

still knee-deep in makefiles

Yes yes, he says grumpily, I spent today working on my Makefile. But I think the work is done at last! After patiently scrutinising my make -d -r > /tmp/argh 2>&1 output, I am finally at a point where:

• make everything really does make everything
• make test does not unnecessarily re-do the training steps

Basically it was a whole bunch of bugs in my dependency logic. But this is it! This is the holy grail! This means that now I can actually fire and forget! It means that Future Eric can (presumably) unpack this tarball, run make everything and expect the infrastructure to just work. It probably won't because that's just the way life goes, but now we have struggled struggled struggled so that our future selves can say "yay, that was easy" instead of "HUH? what was Past Me thinking?"

Ok, so here is what I gain by using make instead of straight shell:

1. pattern rules means I don't have to write a whole bunch of crazy for loops or custom functions
2. automatic dependency tracking

And here is the price I have to pay:

1. correctly expressing the dependency tree (and noting down the places where it is broken)

So I've spent three days fixing my dependency tree, mostly because I still didn't know how to use the tool right (and now I'm getting better at it). The benefit is that I now have a nice, cleaner test harness than I did for the EACL paper... which means that adding new functionality should be more straightforward, for example:

• MOSES on segments
• WASP on forecasts (yeah right)
• WASP with a different grammar per segment type
• Hiero?

It's your typical refactor-before-you-add methodology. Invest the time. Do not perpetuate brokeness

identical makefile per fold (best practice?)

So yesterday, I posted about the multi-dimensionality of things being a pain to handle. You can easily expand things along one dimension by using GNU make's pattern rules. You can go up to two dimensions if you combine functions like foreach with the pattern rules (and/or by copying rules). There probably are ways to do things by using some combination of rules and copying, but I basically got dumb after the second dimension and didn't know what to do.

The practice I've settled on is to keep using recursive make, but to curtail the recursivity by having exactly ONE makefile per fold (incidentally, the same makefile). This is really a lot tidier than when I was having a separate makefile per generation method.

I've also tidied up the mother makefile so that instead of using sh for loops, I'm using make's text expasion capabilities.

Rather than write:

foo:
  for i in baz quux; do cd $${i}; blah blah $${i} && argh argh $${i}; cd .. done

I now write:

foo: $(foreach i,baz quux,foo-$(i))

foo-%: workspace/%
  cd $(<F) && blah blah $(<F)
  cd $(<F) && argh argh $(<F)

Finally, one thing I don't yet understand about make is why I have to use static pattern rules... why I sometimes have to write instead of the above:

foo-baz foo-quux: foo-%: workspace/%
   cd $(<F) && blah blah $(<F)
  cd $(<F) && argh argh $(<F)

It's not so bad because you can use foreach to save repetition.

Anyway, I realise these all look like really boring low level technical details with no scientific interest. That's because they are. But paying attention to stupid nitty gritty details like this seems unavoidable if you want to do your work in a methodical and reproducible manner. In fact, it's not even about being methodical... or particularly fastidious. Why do I do all these things? Why do I spend weeks talking about makefiles?

because I'm clumsy and stupid

The holy grail here is a fire-and-forget infrastructure. Without spending all this work on a reproducible test harness (and learning how to use GNU make), I would have wasted far more time tripping over myself, screwing up this tiny detail or another. Better to suffer automation-induced pain than clumsiness-induced one.

makefile mystery

Given this makefile:

% : %.hs
   ghc --make $@

myfile.myext : myfile.myext.in | gencorpus

• make gencorpus works
• make myfile.ext does not work (unless I manually invoke make gencorpus first)

However, given this makefile (the only difference being that the gencorpus rule is no longer implicit):

gencorpus  : % : %.hs
  ghc --make $@

myfile.myext : myfile.myext.in | gencorpus

• make myfile.ext works again
  

why you want a test harness

I've been spending some time debugging my test harness again and trying to clean it up (because we're still using it, and if you're still using it, you should keep investing in making it work right).

Here are the five dimensions my harness is dealing with:

• fold (cross-fold validation)
• data set (full training, vs. training vs. dev vs testing)
• generation methodology
• new variable du jour (could be lumped into generation methodology)
• evaluation methodology
  

It doesn't sound like much, but it's enough to make writing a makefile (or a shell script, whatever) a big pain. Ever try using one of the '%' pattern rules in a four-and-a-half-dimensional space? It's a big pain, but no matter how many times you curse make (actually how many times you curse yourself for writing sloppy makefiles), you know that the alternative (doing your tests by hand) would be far far worse.

Also, the high dimensionality of it all means that I'm too stupid to figure out a nice concise way of expressing things. So I cheat by using recursive make (and pay the price when I do something like copying over a file and unintentionally triggering a re-run of your training process when all you really wanted to do was test). The lessons?

1. order-only prerequisites are your friends
2. barring that, use cp -u to avoid accidentally updating files that haven't changed

Jury's still out on whether I should have used make or a giant shell script. Make still seems to be winning because I can selectively re-run parts of the process. The only bad part is that I have to express all these dependencies and I always get it wrong (the alternative would be systematically re-running everything or commenting out bits and pieces as you go along, which is bang-your-head-on-the-desk-sloppy and bound to lead you to trouble).

oh moses...

General moaning: Sigh, it's really a pain when

1. somewhere in the MOSES tuning scripts, some program expects the language model path in the moses.ini file to be an absolute path
2. somewhere in the MOSES training scripts, relative paths are generated, even if you pass them in as absolute paths

So I either have to integrate a dumb sed script into my makefile (calling MOSES) or patch the scripts so that they are smarter about paths.

MOSES is big, scary, and hard to use. A million scripts with a million switches. You probably can't blame the MOSES team because maybe the problem is inherently big and scary. But trying to use this software is certainly giving me a good experience in learning how to understand a new program from the user's perspective. (And I'm sure this is exactly what GenI feels like to a newcomer) . For an MT newbie like me it's really annoying to wade through this thicket of scripts (some of which have minor filepath bugs, it seems). Can't blame them. Writing software is hard. Integrating a bunch of disparate components into a single tool is hard. At least it's got a nice friendly tutorial and a helpful community to compensate, though.

Maybe a tarball showing each of the steps being run on an example might be useful.

The fact that they use centralised revision control, of course, makes it harder to send them patches (I should see if git-svn is any help here).

Here's one issue
Ah-hah. One thing is that their enhanced cmert script assumes an environment variable
${SCRIPTS_ROOTDIR} is set. The wrapper scripts use the Findbin module and set it automatically, as in
SCRIPTS_ROOTDIR=$Bin
But on my system, no-workee. So I have to edit the training scripts and hard-code SCRIPTS_ROOTDIR (I guess I could also do something like dirname $0, but I gather that's what FindBin is supposed to do).

Why does this silently fail, though? What coding practices could we employ to avoid these kind of silent failures? Ok, in Haskell we can use the Maybe type to explicitly denote things that fail or succeed, return Nothing if you fail to find an entry in the dictionary. This helps because it puts more burden on the programmer to get it right the first time (lest the type-checker complain) instead of making code that only looks like it works. But even that is not enough.

That's the trouble with programming, that's the thing that's so hard to communicate with people: relatively speaking, it's very easy to make something that looks like it works. But when something silently fails to work, the end-user gets strange grelims and doesn't know who to blame. It's hard to program robust things. I wish I knew how to do it.

more scfgs

Have not been working much lately (focusing on darcs a bit, but trying to steer things back into work-work a bit more)

No this blog isn't really back yet, but I guess I could use it jot random stuff down.

Been rereading Chiang 2005 : A Hierarchical Phrase-Based Model for Statistical Machine Translation, and trying to make sure I understood it better. Also want to revisit the WASP stuff and see what they say about training.