redo meh

I still like the idea of redo (and its new search-parent-dir feature, while adding some disturbing complexity does allow for building simpler looking systems of build files, no more crazy symlinks).  But now I'm struggling with things being unnecessarily rebuilt and it's annoying me.  Hopefully not a full day of build-system debugging again.  Hopefully just some minor easily fixable error on my part

more djbredo'ing ahead?

The Makefiles of Doom have only been partly transitioned over.  I'm a bit reluctant to continue my experiment transitioning to redo because I don't know what my corpora look like (although it seems perfectly reasonable to use METEO as a stand-in).

Maybe what I really need to do is start thinking about the hard stuff again, the corpus generation.  Seem to be putting it off by taking on little tasks like the redo-isation of the makefiles (all in the name of preserving the sanity of future Eric)

redo as an alternative to makefile of doom

I basically spent today dismantling a big confusing Makefile, experimenting with this new system called redo.  You should visit that page and spend 1 minute looking at the hello world example to see how it works.

One thing that makes different from Yet Another Make Replacement is that instead of forcing you and your collaborators to learn another build language,  redo subtracts a language.  The whole system is based on 2 main programs you have to learn how to call:

• redo - running redo is like running make
• redo-ifchanged - tells redo that the arguments are a dependency of the current target

And that's it.  So wait, if there isn't a build language, how do you automate builds?  You do it the old-fashioned intuitive way, by writing a script!  You can use any scripting language you want.  Redo doesn't care, although if you don't specify a shebang line on top of your script, it'll assume you want the most POSIX-y sh implementation your system can provide, with the -e flag set so the script aborts on the first error.

I find this really elegant.  The only potential downside is the current assumption that you want one script per extension (or particular file) and subdirectory.  This is problematic for my work because I sometimes want to repeat a process across different subdirectories.

current setup

For the corpus stuff, I have this crazy setup involving 2 makefiles and a bunch of shell scripts.  The thing is that in this situation, I don't so much want a build system but a test harness and my use of make is slightly abusive.  The reasons I'm using a build system is so that I can

• have dependency analysis for free
• pick out parts of the process to re-run (make training-data)

The reason I have two makefiles and not one is that the tests we're doing are multi-dimensional, and I have not yet found a way of coping effectively with these dimensions.  We're evaluating:

• different corpora
• ... with cross-fold validation
• different generation methods

For now I'm assuming I can collapse the different-corpora and the cross-fold validation into one dimension.  But we're still stuck with 2 to deal with.

So the setup right now uses an outer makefile to deal with the validation dimension and the inner makefile to deal with generation-methods dimension.  In cross-fold validation, we cut up our data say 10 different ways and run the same test on each of the cut-ups to make sure we didn't just get (un)lucky with our data partitioning.  As far as your grumpy research-infrastructure-coder is concerned (I find myself doing a lot of this), cross-fold validation basically means you have to repeat the same process in N nearly-identical places.  The outer makefile then has the role of creating the folds (lie: it runs a shell script that does the work), copying the corpora and test code into subdirectories; and the inner Makefile runs the tests.

The inner makefile is confusing because of all the generation methods that are packed in there... and I just got tired so I'm stopping here.

Age-old battle between clarity and duplication-avoidance (need a nice English word for that as counterpart to clarity)

march is semi-stuck

Well, I think I've worked out what triggers the negative array size exception in Joshua (although I don't know the underlying cause): rare tokens.  The METEO corpus has lots of wind speeds (GUSTS 30-35) and if I replace all the numbers by zero (GUSTS 00-00), everything is fine.  I can start trickling numbers back in, for example, leaving the 3s alone (GUSTS 30-30), but if let in too many numbers, Joshua blows up.  So we'll have to wait till we have a more reasonable looking corpus via the British hills stuff (or until we finish training on the example data).

In the meantime, I'm trying to build up some kind of corpus testing infrastructure.  I'm quite tempted to try this with redo a simple-looking and fairly elegant alternative to make.  Right now I have a mish-mash of two makefiles (yes, yes, recursive make considered harmful) and some shell scripts, and I'm wondering if pushing everything into scripts will simplify matters at all.

avoiding duplication in evaluation sites

I've been working on evaluation campaigns this past few years, and juggling a bunch of variants/combinations of 3 evaluations. This year, I think I have something like 6 evaluation sites to manage, cut up into two projects which each use one of the 3 each. It's kind of crazy, especially since the sites are all basically the same but have some subtle difference in functionality.

It's been a constant battle against code duplication. Over the years, I've been shuffling things out of scripts and into standalone libraries, little tiny piece at a time because it's not always self-evident that code actually can be refactored, [particularly under deadline pressure] or if I need to tolerate a bit of duplication just a little longer to make it easier to slip in some subtle functionality difference. But slowly slowly, I'm finally getting to a point where at least as far as the CGI scripts are concerned, the duplication is down to a tolerable minimum.

Now we've got a problem with Makefiles. Today I finally got fed up with problems debugging my site on the live server, problems that I'd solved several times in the past, but lost the solution to due to code duplication. The solution was always buried away in one of the 6 sites, one of the makefiles. Oh you just need to copy library foo over. Oh, you just need to make sure you chmod that file. The reason why I had all this duplication was basically lack of mastery over GNU make. Didn't know how to make my rules parameterisable enough, and even if some Past Eric did, I didn't have the time/courage to go recover that knowledge to just get the job done. Eternal conflict between doing things now, or doing them right so they won't be so horrible in the future... a future you're even sure will arrive (because you might just dump this research prototype and never see it again anyway).

But now one of my server-related troubles finally annoyed me enough to roll up my sleeves and choose the side of Doing Things Right. No more makefile duplication!

Basic layout

1. project-root/makefile.lib: the rules refer to variables which are undefined, but that's OK - this is where our parameterisation comes in
2. project-root/site/Makefile : defines the variables that makefile.lib wants and has this block of code

   # die, duplication, die!
   include makefile.lib
   
   # complain if makefile.lib hasn't been symlinked in (see below)
   makefile.lib:
   $(error Looks like you need to run ./configure first)
   
   # use of :: instead of : means I can define all in makefile.lib
   # and just append a new pre-requisite to it (the | means order-only)
   all :: | makefile.lib

3. project-root/foo/configure is a stupid little bash script that just does ln -s ../makefile.lib .

So that's fairly tidy. The key trick here isn't so much that you can include bits of makefile in other makefiles, as that you should remember that variables in makefiles are only expanded on demand. This makes it possible to leave all sorts of holes in your common makefile that you instantiate in the outer file. Sort of thing you know, but fail to connect with your goals until you really sit down and set your mind to refactoring.

Aside about version control

This can partly be blamed on technical limitations with Darcs (incidentally a job which consists just in working on Darcs is potentially bad because it would deprive me of grist for the mill -- doing some sort of coding as work-work gives me a tiny bit more knowledge about my target audience, although then again, the temptation is to think that all programmers are like me, which is far from the truth!):

• [no repository nesting] I don't know how to set up some structure version control structure which reflects the fact that the 6 sites have similar content, while grouping them, say 3 at a time into distinct projects. I *could* treat each of them as individual repositories, and share patches between them. I've done this with other sites before and it works fairly well. In this particular context, though I'd really like my 3 sites to co-exist within a single container project.
• [no symlinks] My version control software does not manage symlinks

reaping my makefile rewards

Had another idea for improving MOSES results. Really really glad I spent that week improving the makefile. I was worried when doing it that it was just me being really stupid and finicky, but sometimes being stupid and finicky pays off. Adding new generation methods is a lot easier now. Debugging things is much cleaner and more predictable too. I still have a small problem with the re-generation of scores, but I hope a little rule tweaking will fix that.

second new feature facilitated by the Great Refactor

All this makefile refactoring is starting to pay off. I'm trying out another new feature, introducing a new test where we get rid of some commas. I want to keep the old generation method around for comparison. So just 4 more targets in the makefile and extensions to 2 variables. Relatively painless.

Before the Great Refactor, I would have gotten myself deeper into a tangle of one-off scripts.

I think the lesson here is that while fighting to reduce duplication of information may look foolish when you're deep in the process, it really does pay off even in the short term.

one last refactor? please?

Yes, yes, I did spend yet another day refactoring the makefile. It's an itch you just have to scratch, getting rid of cruft. But now I really think this is it, dropped the footprint on this sucker by homogenising the corpus copying step (each system now uses the same corpus files with the same file names!). This had better be it. This is starting to make me irritable, because every time you fix and simplify something, something else breaks.

Of course, the answer isn't to run away from fixing and simplifying.

The answer is to realise that the reason something else broke is because you designed things badly in the first place so that things were fragile. So you keep doggedly fixing and simplifying, chipping away at the thing until you get to a state where you can add/modify a piece without the whole Jenga tower crumbling down. Only 500 lines of Makefile left...

the Test Harness Problem

I'm sorry to keep yammering on about this very technical and stupid point. I think what I'm saying is that it's not just technical and stupid. There actually are some engineering issues behind this. In principle we are computer scientists and not software engineers. In practice, we can't really get away with trying to do the computer science (in this case the NLG research) without paying some attention to software engineering. In fact, we have been trying to get away with it, and it seems that doing so works, but it only appears that way because you've been offloading the pain to somebody else... pain that really originates from failure to address the software engineering (process engineering?) issues behind setting up a test bench. Let's stop inflicting this kind of pain on our future selves.

desiderata

• Reproducibility - in your ACL 2014 paper you talked about these experiments you ran with such and such system producing such and such data. Given the contemporary software and data, can you actually reproduce the experiments? Can you get the same data back or explain any discrepancies? Why did one of your systems suddenly jump ranking?
• Extensibility - you've got this great ACL 2014 process, but now you want to try some new ideas out for the ACL 2015 paper you are in the middle of writing. Can you extend your shiny new test harness with a new process?
• Granularity - you want to fiddle with your harness a bit, turn a knob, change a weight, add a normalisation step. First of all, is this parameterisability possible? Second of all, can you do it such a way that you don't have to run the WHOLE test process (which at the time of this writing could take hours if not days) again?

pressure points

• Dimensionality - Dimensions multiply. You're in for a combinatorial nightmare. The dimensions I have to contend with:
  1. folds (cross-fold validation)
  2. data set (test vs train vs dev)
  3. methodology (what you're actually testing on)
• Variability - Each point in each dimension brings a different way of doing things. The test data has to be treated different from the training data (for example, you might want to lump the training data into one giant corpus, but oh no, can't do that with the test data; it has to be one file per datum). Each method has its own idiosyncratic toolchain (and requirements for how the data are normalised)

conflicts

reproducibility vs granularity

To improve reproducibility you are tempted to create an automated toolchain... just hit the go button and redo everything. But if you're not careful, you lose out in granularity (because if you change something and just hit go, you re-run the entire toolchain, putting you out of business for a couple of days). A highly reproducible solution would be a single Mother of Shell Scripts... but it's not a very granular one. A highly granular solution is a graphical interface that lets you plug in your data sets, turn some knobs and then hit go... but it's not very reproducible (unless you dump a shell script from the GUI, in which case the shell script is not very granular, because you wouldn't be able to selectively re-run bits).

This is why the various incarnations of Eric were fighting amongst themselves about whether to use make or shell. I've settled on make because it is a fairly decent compromise between reproducibility (make everything) and granularity (make just-that-one-step)

granularity vs extensibility

On the one hand, you want to keep things extensible by reducing duplication of information and breaking things down into nice modular blocks... the idea being that if you want to add something you just recycle some of your modules and add in the difference. On the other hand, highly factored code isn't always very granular. Now if you turn a knob, you are turning the same knob throughout the ENTIRE harness, which may or may not be desired.

The compromise between granularity and extensibility is parameterisation - use more functions instead of variables. Unfortunately, my current understanding of make is that it is not a very natural tool for such parameterisability. I am sorely tempted to go re-write the whole damn test harness in Haskell, using something like Franchise to do the dependency tracking for me. (but of course, then it's less reproducible and extensible in a sense because not everybody hacks Haskell... not everybody hacks makefiles either, but at least there is a chance...)

extensibility vs reproducibility

You want to keep things reproducible by homogenising the process and making everything fit into the same boring template. Basically here I am equating reproducibility with transparency, which may or may not be wise. You want people to be able to just hit the green "go" button, right? But, because of the variability problem, you are tempted to go for quick fixes and local one-off solutions whenever you want to tune your harness, twiddle a knob, add a methodology. You write a one-off shell script for your local problem which reduces your reproducibility by introducing all sort of weird processes that your future self has to learn. Why can't I just have a go button?

You can't help doing it in this short-term way because the deadline is only 3 days away and you need the results NOW. Or even if you weren't facing time pressure, the variability behind the problem obscures the commonalities that would allow you to recycle things. Anyway, life forces you to add yet another shell script to fix just that one problem you're trying to solve. In doing so, you punish your future self by reducing the reproducibility of your harness.

This conflict is mostly due to pressure from variability.

Interestingly, you can also see this as a conflict between short term extensibility (add a shell script now) and long term extensibility (add a framework so that future you doesn't have to wade through a thicket of shell scripts). But here I am focusing purely on reproducibility for somebody else (e.g. Future You) vs short term extensibility.

conclusion

The claim I making in this blog post is that the Test Harness Problem is a difficult one. It's not difficult in a deep and scientific sense, but in an embarrassingly fiddly way. I hope that in writing this, I have made some progress convincing you that spending a fortnight on a stupid test harness is entirely a reasonable thing to do, or at least provided a somewhat more thoughtful characterisation of the fiddliness of the issue.

Note that what I have not tried to do in this article is to convince you that you actually want a test harness... but I think it should be pretty clear that the three desiderata (Reproducibility, Extensibility and Granularity) apply whether or not you just cobble together some ad-hoc shell scripts (or even just rewrite things on the command line), or put together something which you formally deem to be the harness. The ad-hoc solution of not having a test harness is really equivalent to having a super-crappy harness with zero reproducibility... and that I think is harmful to your future selves and to other researchers interested in your work.

So writing a test harness may be hard, but I think it's well worth it. The only danger is that you may find your PhD students or Research Fellows disappearing for weeks as they go off and make Yet Another Refactor to the harness whilst desperately hoping that their efforts are not in vain. But then go write another paper, try to rerun some numbers and you will thank him/her for making that initial investment.

out of makefile refactoring land

Ok, one more half-day of makefile refactoring... just itchy fingers this time, listening to the Evil Voices in my head that tell me "no, it would really be a lot cleaner if you re-arranged the furniture this way". Spent a day moving logic from the outer makefile into the fold-specific makefile... which is good in that now all the intelligent stuff is in one place (less duplication of information; or at least all the same information belongs in the same file even if it's duplicated)... In principle, we can do away with the outer makefile altogether and just replace it with a dumb shell script, say running make on each fold in parallel.

It's pretty frustrating to be doing this kind of work because you feel like you're just running around in your little hamster wheel... and also because it's hard to justify to other people. No, it's not (just) because I'm fastidious. In the past, we have suffered a lot from the non-reproducibility of our results. We had these numbers somewhere in a paper, and had no idea what was the procedure needed to re-generate them. That's what we want to aim for, I want to make it so that for any paper I write, I can somehow go back in time and run exactly the same toolchain that generated the results from that paper. I didn't quite get there with the Mother of Makefiles from my PhD, and I'm certainly not getting there now... but we are slowly approaching this goal little by little. No more fog-of-research mysteries.

doing new stuff!

I'm finally adding new functionality to this makefile! Teaching the makefile how to train MOSES on segments instead of on full forecasts. A bit tedious because you have to create a corpus file, including normalisation, for each segment, test on that segment, de-normalise, re-constitute the forecasts by concatenating the segments. None of these are inherently difficult; it's just dumb Unix commands. But you have to do them.

So I have to work out the boring stuff there... but at the very least, I don't have to machete my way through the mess of quick-and-dirty solutions I had left behind from the EACL paper. It's pretty clear where all the new stuff "plugs in" to the test harness.

Yes, so at the very least 2008-11-27 Eric would like to thank 2008-11-24/2008-11-26 Erics for slogging through and cleaning up the test harness so that adding this new generation technique is slightly less of a pain in the behind.

hooray!

Done! That wasn't so bad; only too half a day (the good thing is that MOSES does the right thing when you have a test file with more than one line). And thanks to the cleaned up makefile, the test harness pretty much does the right thing

still knee-deep in makefiles

Yes yes, he says grumpily, I spent today working on my Makefile. But I think the work is done at last! After patiently scrutinising my make -d -r > /tmp/argh 2>&1 output, I am finally at a point where:

• make everything really does make everything
• make test does not unnecessarily re-do the training steps

Basically it was a whole bunch of bugs in my dependency logic. But this is it! This is the holy grail! This means that now I can actually fire and forget! It means that Future Eric can (presumably) unpack this tarball, run make everything and expect the infrastructure to just work. It probably won't because that's just the way life goes, but now we have struggled struggled struggled so that our future selves can say "yay, that was easy" instead of "HUH? what was Past Me thinking?"

Ok, so here is what I gain by using make instead of straight shell:

1. pattern rules means I don't have to write a whole bunch of crazy for loops or custom functions
2. automatic dependency tracking

And here is the price I have to pay:

1. correctly expressing the dependency tree (and noting down the places where it is broken)

So I've spent three days fixing my dependency tree, mostly because I still didn't know how to use the tool right (and now I'm getting better at it). The benefit is that I now have a nice, cleaner test harness than I did for the EACL paper... which means that adding new functionality should be more straightforward, for example:

• MOSES on segments
• WASP on forecasts (yeah right)
• WASP with a different grammar per segment type
• Hiero?

It's your typical refactor-before-you-add methodology. Invest the time. Do not perpetuate brokeness

identical makefile per fold (best practice?)

So yesterday, I posted about the multi-dimensionality of things being a pain to handle. You can easily expand things along one dimension by using GNU make's pattern rules. You can go up to two dimensions if you combine functions like foreach with the pattern rules (and/or by copying rules). There probably are ways to do things by using some combination of rules and copying, but I basically got dumb after the second dimension and didn't know what to do.

The practice I've settled on is to keep using recursive make, but to curtail the recursivity by having exactly ONE makefile per fold (incidentally, the same makefile). This is really a lot tidier than when I was having a separate makefile per generation method.

I've also tidied up the mother makefile so that instead of using sh for loops, I'm using make's text expasion capabilities.

Rather than write:

foo:
  for i in baz quux; do cd $${i}; blah blah $${i} && argh argh $${i}; cd .. done

I now write:

foo: $(foreach i,baz quux,foo-$(i))

foo-%: workspace/%
  cd $(<F) && blah blah $(<F)
  cd $(<F) && argh argh $(<F)

Finally, one thing I don't yet understand about make is why I have to use static pattern rules... why I sometimes have to write instead of the above:

foo-baz foo-quux: foo-%: workspace/%
   cd $(<F) && blah blah $(<F)
  cd $(<F) && argh argh $(<F)

It's not so bad because you can use foreach to save repetition.

Anyway, I realise these all look like really boring low level technical details with no scientific interest. That's because they are. But paying attention to stupid nitty gritty details like this seems unavoidable if you want to do your work in a methodical and reproducible manner. In fact, it's not even about being methodical... or particularly fastidious. Why do I do all these things? Why do I spend weeks talking about makefiles?

because I'm clumsy and stupid

The holy grail here is a fire-and-forget infrastructure. Without spending all this work on a reproducible test harness (and learning how to use GNU make), I would have wasted far more time tripping over myself, screwing up this tiny detail or another. Better to suffer automation-induced pain than clumsiness-induced one.

makefile mystery

Given this makefile:

% : %.hs
   ghc --make $@

myfile.myext : myfile.myext.in | gencorpus

• make gencorpus works
• make myfile.ext does not work (unless I manually invoke make gencorpus first)

However, given this makefile (the only difference being that the gencorpus rule is no longer implicit):

gencorpus  : % : %.hs
  ghc --make $@

myfile.myext : myfile.myext.in | gencorpus

• make myfile.ext works again
  

braaaaaains

That would be zombie eric talking.

Hacking up the Makefile of Destitution (that sucker keeps getting bigger and bigger... now we're trying to support batch directories and all that jazz). I just realised that I had done something silly in the Makefile which makes it completely incapable of handling the case where the user is running several things in parallel <-- which happens. It shouldn't be hard to fix; there's just lots of jackhammering to do, dust flying everywhere. Makefile, Prolog, functional programming; like all things declarative, it's love, lots of love and a little bit of hate. The hate comes from the fact that sometimes you actually have to reason about things, whereas the love comes from knowing that you pretty much don't have any other choice. Get the feeling that this stuff blowing up is just it not hiding the ugly truth from you, but what do I know?

That Makefile hacking took a surprising amount time. I had actually found myself switching from one of my jobs to another, completely unrelated one. Thank darcs for cherry picking! It wasn't super-easy because some lines I didn't want to record were mixed in with stuff I wanted. Some spontaneous branching took care of that, though. Anyway, now I've got a pretty big bug in there that I want to fix, that crazy parallelism stuff... but have to do some digging.

I can't remember what I've been doing; it doesn't seem like actualyl that much, but I'm not sure. Maybe just inefficient. Spent quite some time working out the logic for clumping a bunch of data points together at increments. Feel dumb... seems like it should be easy, but I end up getting something that looks like this:

>   rawPoints = someListOfNumbers -- [ 1, 7, 8, 9, 4, 4, 13, 5, 8, .. ]
>   xTics  = takeWhile (<= cutoff) $ iterate (+ incr) 0 -- [ 0, 5, 10, 15, .. ]

>   points = pointsFor rawPoints xTics -- [ (0,3), (5,4), (10, 1) ]
>   pointsFor  ps ts     = reverse $ pointsForH ps (reverse ts)
>   pointsForH ps []     = [(0, length ps)]
>   pointsForH ps (t:ts) = point : pointsForH smaller ts
>     where point = (t, length bigger)
>           (bigger, smaller) = partition (> t) ps

misc

make run-batch SUITE=all

Added a new feature to the Makefile-of-Deprivation: the ability to run a directory-full of test suites. This is one thing semconst has been able to do for a long time, but which I never bothered to implement. But now that we have a clear need for it...

1. put all dyalog-output in suite/semconst-output
2. make run-validator SUITE=foo
       (re)runs the validator for a specific suite. Do this for each suite. Optional, but probably more comfortable this way.
   
   
3. make run-batch SUITE=all
       validates and geni-converts all semconst suites; combines the resulting geni suites into one file (*.geni-input => all.geni-input) and works from that one file. By virtue of Makefile dependency tracking, this detects if you have already run the validator, so don't worry about any redundant steps taking place. Also, if you ever make a mistake in validation, you can just re-run the step above to correct a specific file and then re-run this step.

Well, it's a bit of a hack, and it isn't smart enough, say, to use the all grammar when you're just trying to run a single suite. I could that functionality in too, but only if it's really useful, because it's just more intelligence/complexity to throw into the mix.

yang1991ffs

Trying to read this article now. Should hopefully give us more details on the systemic TAG stuff.

active projects

• Papers with C - 23 jan
• paraphrase selection chapter - 15 feb
• applying for a job - 20 feb
• Adjunction polarities implementation

adjunction polarities probably play nice with intervals

I was talking to the good folk at Clgrm yesterday. They say that they use a similar idea to our adj-as-multiplication for dealing with coordination, which is interesting, except that I don't really know what they mean. In other news, Clgrm's doing some fun stuff with their polarities for parsing. I'm not sure if they've written up about it yet, so I guess I should keep the blogging on a leash.

Also, I was a bit worried about whether it would be practical to implement the multiplication scheme given intervals (atomic disjunction)... but then it occurred to me after coffee that intervals don't really matter. Look, the intervals you're going to get for adjunction polarities are ALWAYS going to be zero to one. So who cares? When you multiply them together, the worst that can happen is that you go from a zero-to-one interval to... a zero-to-one interval.

yesterday

In our discussion, it came up that I should consider looking into ACG and see if there is any connection with the systemic stuff (eh?!). Thanks for the thought!

what next

Tidied up and pushed in all my changes on the polarity and the trace filter stuff. It should just work, but I'll have to run it chez C to be sure.

I'm really tempted to work on those adjunction polarities right now. I'm a bit worried because there's two layers of polarities to deal with (abstract-data-type, anybody?). And I'm not convinced that we have our indices in the right place, but that's a seperate issue. But now that the intervals thing is taken care of, I feel like we're ready to go ahead.

writing your dissertation...

I'm really impressed by our Centre de Documentation: they always manage to find what I need, fast and friendly for that matter. They've also completed my suggestion to purchase Writing Your Dissertation in Fifteen Minutes a Day by Joan Bolker, now in my grubby little hands. I don't know how useful it's going to be. Maybe it's going to be cheesy self-help. Or maybe it's going to be The GTD of PhD students. Will have to see.