refex generation

It often happens for me that trying to implement something subtle/tricky is way more effective a learning process than going cross-eyed over a paper

refexes:
  within rx:
    b1 (a box): [OK]
    b1 a1 (a box and an ant): [OK]
    b1 b2 (two boxes): [OK]
    b1 a1 b2 (two boxes and an ant): [OK]
    a1 b1 b2 (an ant and two boxes): [OK]
  across rx:
    a1|a1 (an ant|the same ant): [OK]
    a1|a2 (an ant|another ant): [OK]
    a1 a2|a1 (two ants|one of the ants): [OK]
    a1 a2|a2 (two ants|one of the ants): [OK]
    a1 a2|a2 a1 (two ants|the same ants): [OK]
    a1 a2|a2|a1 (two ants|one of the ants|the other ant): [OK]

morph testing

Hello from beyond the academic grave! Just reporting that morphological realisation (and some simple aggregation) seem to lend themselves quite nicely to unit testing, at least for spotting simple blunders.

(My bug was just a flipped bit, meant "Plural" when I typed "Singular")

verbs:
  inflection:
    walk (walks, walked): [OK]
    push (pushes, pushed): [OK]
    pass (passes, passed): [OK]
    abuse (abuses, abused): [OK]
    banjo (banjoes, banjoed): [OK]
    play (plays, played): [OK]
    cry (cries, cried): [OK]
  expansion:
    walk:
      walks: [OK]
      walk: [OK]
      walked: [OK]
    call_in:
      calls in: [OK]
      call in: [OK]
      called in: [OK]
    take_in (cooperation w lexicon):
      takes in: [OK]
      take in: [OK]
      taken in: [OK]
concepts:
  star (stars): [OK]
  egg (eggs): [OK]
  patch (patches): [OK]
  boy (boys): [OK]
  spy (spies): [OK]
  thesis (theses): [OK]
  elf (elves): [OK]
  ace (aces): [OK]
refexes:
  box (a box): [OK]
  box ant (a box and an ant): [OK]
  box box (two boxes): [Failed]
Failed: expected: ["two boxes"]
 but got: ["two box"]
  box ant box (two boxes and an ant): [Failed]
Failed: expected: ["two boxes and an ant"]
 but got: ["two box and an ant"]
  ant box box (an ant and two boxes): [Failed]
Failed: expected: ["an ant and two boxes"]
 but got: ["an ant and two box"]

blast from the past

Last week, I cleared out my office at the University of Brighton as I'm starting my scary new life as a freelance programmer.

I found this note that I brought with me from Nancy

deb packaging

I've been working on trying to make some of my work from last year easier to install.  So far I have a script to create a massive tarball (or copy everything to a root tree).  Now have some moderately tweaked default Debian package creator files with configuration files (like apache site) meant for a VM.  If I do this right, it should be possible for somebody to just

1. install Debian on a fresh VM
2. copy the deb file to the VM
3. use dpkg --info to list the package dependencies; install with apt-get
4. dpkp -i the-deb-file

I'd initially resisted doing this, opting the for the write-a-big-README file approach.  But really automation is the way to go.  Hopefully the readme will be useful as backup if something goes wrong.

Resulting deb file is going to be pretty massive.  Has GATE, RASP, Scala, FlexBuilder.... I suppose the right way would be to go down and make sepearate debs for each one of these dependencies.   But no.

the Scala rewrite

This week I've switched over to one of the projects that's been funding my time at Brighton, 3D-Coform. Took a while to dust off my old work from last summer, but basically slowly starting to get into a groove.

One of my goals is to improve the packaging of some complex pipeline I'd built. It involves Flash (not my choice!), PHP (I'm innocent!), Java (for GATE), Scala and Haskell. The Scala and Haskell parts are my fault, mostly a refusal to go back to writing Java. What's funny about the pipeline though is that it's essentially Java, with a thin bit of Scala which is really just calling this one dinky component written in Haskell. Making people install a Haskell compiler just for that dinky component is stupid. In my defence, writing said dinky component in Haskell was probably the right thing to do, in other words, use the tools you know best to get the job done and out the door. But now that we have a little bit of time to put into packaging it up nicely, I'm trying to replace that little dinky with of Haskell with Scala. This way all the backend bits run on JVM and we have simpler dependencies.

It's also a good way for me to learn the language a little bit better.  I guess I've reached a stopping point for the week.  I've reimplemented the shallow part of my code which defines a bunch of data types, parses from XML and pretty prints the output.  If you want to take a look, the Haskell and Scala code are not identical (the XML parsing may be a bit better in Scala version, due to reimplementation?) but I think they are roughly comparable.

The deep part is the bit that does graph traversals, basically simplifying RMRS semantics into subject-object-predicate triples.  That bit I'll have to try and implement next week.

Things I've learned so far:

• always use case classes, never use new: I was wondering why for some classes, like Option, I could just call the constructor (eg. Some), but for others I had to write new Foo(...).  Turns out that it was just because I wasn't declaring my classes to be case classes.  Using case classes means I get to avoid all the confusing stuff that mutability gives you, for example, having to worry about the distinction between equality and identity.
• I may not be using traits right, but I think I like Haskell type classes better simply because with type classes, I can separate the instance declaration/definition from the class definition
• subtypes make for easy heterogeneous lists: in Haskell I'd wrap the things I want in some existential type E and then make a list of E.  In Scala, I make my types all extend the trait E.  For some reason, I find the latter a bit "easier", either more memorable/clear or just more convenient
• to get methods associated with classes rather than objects, just create a singleton object (that extends the desired trait)
• Scala lets you pattern match XML: it's kind of neat that you can interleave XML in your code and do things like pattern matching on it.

I get the impression that you basically have two kinds of Haskeller.  The first is the kind that uses Haskell because they can take on a lot of abstraction and use it to do amazing things, the smart Haskeller.  I think I fall more into the dumb category of Haskeller who tends to use it because I find the consequences of subtyping (Liskov substitution) and mutability to be too confusing.  And maybe there are also smart-dumb Haskellers... the kind who fall into both camps simultaneously.

makedo still paying off

Another day for confidence building.  I've been benefiting greatly from my build file being written in Haskell.  This the sort of situation I've been able to cope with way way better than my Bash script

problem A: recomputing LLR

1. fragment extraction takes as input: a sentence selection, data selection, and LLR
2. data selection takes as input: sentence selection and LLR
3. NEW: after data selection, we can recompute LLR (call it LLR2) - LLR2 takes data selection as input.  There are two different ways of recomputing LLR
4. What is effect of the LLR method on fragment extraction?

Solution: add a new LLRMethod argument to fragment extraction.  Fragment extraction now takes three arguments.  Furthermore, in computing the LLR, my fragment extraction function just passes in the data selection method it received in its parameters list.

data LlrMethod = LlrNoRecompute
               | LlrRecomputeDoc
               | LlrRecomputeSent

Problem B: decoupling token alignment

1. NEW: we want to change fragment extraction by separating an implicit token alignment step from it. Fragment extraction would no longer take a sentence/data dir, but a token dir
2. Token alignment depends on data selection, and LLR method...
3. There is more than one way to do token alignment

Solution B: add a TokenAlignmentMethod argument to fragment extraction.  Instead of passing in a sentence directory and data directory, I pass in a token aligned directory computed off the arguments passed to fragment extraction.

Here's what fragment extraction looks like now:

fragmentExtraction :: SentSelMethod -> DataSelMethod -> LlrMethod -> TokenAlignmentMethod -> ExactTarget
fragmentExtraction sm dm lm tm = ExactTarget path "" $ \ RedoArgs { rname } -> do
  redoIfChange [ tokenDir, llrDir ]
  createDirectoryIfMissing True rname
  runMyIO "detect-fragments" $ [ tokenDir, llrDir </> "lexicon", rname ] ++ parallelRTS
  recurseStampExt rname [ extEnChoice, extDataPerSentence ]
 where
  path       = d_FRAGMENTS </> showPC sm </> showPC dm </> showPC lm </> showPC tm
  tokenDir   = ePath (tokenAlignment sm dm lm tm)
  llrDir     = llr2Dir sm dm lm

Bonus to Solution B: I had forgotten that after fragment extraction comes a "peers directory" step.  Thanks to the Haskell type checker, I remember that I also need to extend the peers directory step with a token alignment method argument.

Before using my makedo library, I was trying to manage all this argument passing by deconstructing paths using basename and dirname.  Crazy!  Way way better to write your build files in a language that (A) has a notion of sum types or at least Enum types (B) lets you create your own types with minimal fuss (eg. I don't need a separate file per type, just declare them) and (C) statically type checks.  To be honest, the language for this probably does not need to be Haskell.  Even Java could do the job, although I think creating different enumerations like LLRMethod vs TokenAlignmentMethod might be more of a pain.  The advantage to using Haskell over Java (aside from the usual higher order functions, etc) is the HSH library, I think.  Maybe if there was a convenient way to write shell-script-like things in Java it could work.

But Bash or Perl script?  No way!

We missed the deadline for our paper by the way, probably because I was spending so much initial time working on makedo.  But I still think it was the right thing to do.  I would not have been able to understand what I'm working on as well as I could have if I was still wading knee deep in shell script.  Sometimes you just need to step aside and go abstract away some nonsense before you get to the meat of your job.  I hope that if we don't get a paper out of this, I can at least release my makedo library, and that somebody else benefits from it.

Take home message:

Statically typed shell scripts and Makefiles are good for you.

makedo pays off

Today was a good day, a confident day.

For a bit of context, I can be a bit insecure and defensive about the time I spend on "There Must Be a Better Way to Do This", time which I sometimes worry should be spent on "Doing This".  Unlike the smart Haskellers, my "There Must be A Better Way" does not tend to result in neat abstractions or boilerplate scrapping frameworks or what not.  A lot of time is just spent going around in circles, undoing past efforts, or just changing, changing from make to Redo + Bash, to Redo + Bash-in-Haskell to Redo + Haskell, change change change, churn churn churn.

I just spent a month scratching my head over my Complicated Sausage Machine, trying to figure out how not to make it hurt so much, and I think it finally paid off!  All that going around in circles, and I can now write this patch and this other patch.  I can now do the Thing I Was Actually Supposed to Do.  This is either something I would have had to do by hand (as in running commands by hand), or tried to stuff into my Complicated Sausage Machine, horrible fragile little piece of Bash.

So it's done... and I actually understand what the sausage machine is supposed to do.  The end result is much better.  Yesterday I remarked that Haskell version seems just as complex/long as the Bash one; turns out I just needed to bit of refactoring, refactoring which would likely have been harder in the Bash.  Again the beauty contest or Line of Code reduction isn't really the point.  It's about the clarity.  For the first time, I'm feeling confident that all my rewriting efforts were useful. The Haskell rewrite of the sausage machine now does more stuff than the original Bash one (not a superset, mind you) and is way easier to understand and by virtue of factorisation, has way fewer surprises in it, and is now much easier to extend.  OK the downside is that in order to extend it for future papers, you have to know a bit of Haskell.  But just in terms of mistakes avoided for this current paper, I think we have a net win.

Now we'll just have to see what this does for our paper.

the makedo yak looking pretty bald

Bald - is that what you'd call a yak when you've shaven off all its fur?  I guess I can say that the makedo exercise was been a month-long yak shave.

I had something to do (under what is now deadline pressure), but I was feeling so annoyed/daunted by the prospect of doing it my old painful way (Bash) that I spent what turns out to be a month looking for a Better Way of Doing Things.  85% of that time was spent barking up the wrong tree (Bash in Haskell), and the remaining 15% was these two days scrambling to rewrite my pipeline in pure Haskell.

If you want to compare the results, have a look at the not-quite-equivalent Bash and Haskell scripts.  Interestingly, the Haskell is just about as verbose than the Bash, maybe more because I've not got all the same functionality.  But I still have a sense that it's a lot simpler in a way, and a lot easier for me to understand/extend.  Or maybe I'm just using the Haskell wrong.  Part of the difficulty in comparing the two is that of course I wrote the Bash pipeline first, which means that the Haskell version reflects not just the language difference but the benefit of hindsight.  Hmm :-/

Anyway, not trying to win a beauty contest here.  Just trying to make it so that I understand what I'm doing well enough that I can do what I was initially supposed to do.  Before all my efforts were obscured by a lot of nonsense involved in deconstructing and reconstructing filenames.  Now that I've got that offloaded to function parameters, and now that I've got type signatures reminding me what kinds of inputs I use (SentSelMethod, DataSelMethod), I have a much easier time.

This is one those double-edged sides of my working style.  I have a strong enough pain aversion that I'll (A) insist on automating things and (B) often spend a lot of time on technological wild goose chases (bad) instead of what I'm really "supposed" to be doing and I'll invest effort into promising bits of technology (like redo as a build system) even if they are not widely popular yet... but as a result of all this investment, I'll often be able to knock things out faster or more solidly... or at least so goes the theory.  An Eric on permanent self-retraining is one which does Future Things faster and faster at the expense of Present Things.  The hope being that my Present things reflect on a past experience of investing in Future Things.  All about striking a balance.  Not sure if I've got it right.

makedo simplification - no more Bash

Those of you who have been following may know that for some reason I've been through this process of (1) handwriting a crazy Bash script to encode my processing pipeline for an upcoming paper (2) trying to replace my handwritten Bash script with an auto-generated one in the hopes that writing a program to write my program is easier than writing my program.

It's not.

I think the problem has been me dancing around Haskell because in a sense I know it's unlikely my successor will be able to make direct use of it... so I guess I've been pretending to myself that by generating Bash from Haskell, at least my successor would be able to make use of the latter if the Haskell isn't immediately useful.  But this is stupid.  Nobody is going to read that Bash, and the whole process of shifting back and forth between the two languages (gee, should I store this in a Haskell var, or write a piece of Bash to store in?) and besides, the code that this pipeline is running is in Haskell anyway.

What was I thinking?

I guess another part of me was thinking "gee, I know what my Bash script looks like and I don't know what the equivalent Haskell thing would be, so I'd better generate Bash so I can compare the two"

Foolishness.

Lots of time wasted due to that bit of foolishness.

Now I've dropped all pretence of not-using-Haskell and I'm a much happier code.  Just get to the meat of the issue.  I've also dramatically simplified my goals.  Previously I was trying to put together some kind of half-assed dependency tracking framework in the attempt to Eric-proof.  But as usual, I was solving the wrong problem.  The real pain I was trying to tackle was the combination of dimensionality and encoding parameter information in filenames.  This is stuff that I've resolved by just doing things in Haskell, no need to write a dependency framework.

Now my revised pipeline seems to be feature-compatible with the old one.  I don't have all the pieces yet.  But I'm close now, close to getting caught up with me from several months ago and being able to punch through to what I was really supposed to be working on, and any last minute requirement changes we discover as we flesh this paper out.

That's my excuse for doing all this exploration instead of just sucking it up and writing more bash by hand.  I could have just written more by hand, but sans doute we would have had to introduce some sort of last minute complexification to the pipeline.  Now that I've got all the power of Haskell programming at my disposal, I can handle it.

The new framework is much simpler.  Now that I'm not trying to think about the dependency tree, and I'm not trying to go for any sort of clever abstractions I can actually wrap my head around what I'm trying to do.  The heart of the matter is a simple list of filename to action matchers.  Basically, we take a single filename as input and we compare that filename against each of the possible targets

data Target = Target
  { match  :: FilePath -> Bool
  , description :: (String, String)
  , action :: RedoArgs -> IO ()
  }

For any targets that match the input file name, we invoke that target's action.

The other piece of this puzzle is the wonderful HSH library.  The great thing about HSH is how absurdly easy it is to mix and match pieces of shell script with Haskell.  You can write things like run ("cat foo" -|- "sed -e s/blop/quux/" -|- (take 42 . map toLower)) >>= B.writeFile "out".  Very nice use of Haskell typeclasses.

Much happier camper.  The whole time I've been working on makedo, I've been busy ignoring an increasingly loud/shrill voice in my head just saying "wrong, wrong, wrong, wrong".  Should have listened to it.  Bash-in-Haskell was not a good idea.  If I'm going to do Bash-in-Haskell, I might as well just do Haskell and be done with it.

makedo in anger!

Alright, after declaring that I was giving up on the makedo enterprise, I had a small idea which has allowed me to press on in my foolish quest against this Curse of Dimensionality problem I've been talking about.

So the basic approach has been to try to capture my pipeline in Haskell, but for comparability with the old Bash pipeline, to build up Bash syntax trees in Haskell and pretty print the output.  On top of this, the pipeline uses an obscure build system known as redo.  Rube Goldberg, I know.  All this because I think writing complicated Bash scripts is painful.  The hope is that by taking on this approach, I'll be able to deal with the problem of multiplying out variants of intermediary pipeline stages.  Basically I think I just want the list monad in Bash.  I don't know. I'm not trying to make this convoluted; I'm just violently allergic to the stupidity in my current bash script.  I somehow think that autogenerating said script will be less painful.

Anyway, the basic insight is that the thing I was trying to do was made complicated by the fact that I was trying to keep the bash script compact. Suppose we have variants of a "data selection" process, which in turn use outputs from variants of a "sentence selection" process.  What I think I was doing was contorting myself to ensure that we could have a single data selection target that captures the variants using wildcards in its path (data-selection-dir/*/*).  But I don't actually to do this!  The script is autogenerated; who cares about compactness?  I just keep the Haskell script-generator compact and let it fan out all the combinations of rules.

So that seems to work.  And now for the first time I've used makedo in anger.  Well, actually not-quite-anger because I'm just testing to see if I can replicate part of the old bash process.  But at least I've run my first test on the real server.  The rest of the work should just be trying to translate the rest of my process to Bash-in-Haskell and hope that I'm ultimately right about this making me less crazy.

makedon't

Argh

I ran into a snag trying to implement makedo, and think that in the interest of time I'm going to have write my script the bang-head-against-the-wall way, enumerating all the combinations of dimensions by hand and juggling the various arguments to programs, filenames to look up, etc.  This sort of thing drives me absolutely nuts, but hey... what ya going to do?  Paper has to get written.

I'm getting a slightly less fuzzy idea what makedo is trying to do, though.  Basically, I think there are 3 concepts we're trying to sift through:

• dependency: we have a notion of dependency because asking the human operator (Eric) to remember which bits of the pipeline to rerun is way too risky.  He will inevitably do the wrong thing one step of the way and subtly screw up your results.  So automation is the way to go.  The alternative is to encode the whole pipeline into a straight shot A-Z script, but that's too expensive for incremental development.  You cannot afford to re-run the whole A-Z every time you tinker with something because it would take forever.  You need a dependency-tracking mechanism to let you say "let me rerun step D" which then automatically figures out what you need to recompute as a result of that action
• variation: You are trying to explore different ways of doing steps B, C, D, E.  So this gives you a very small combinatorial explosion.  I guess a combinatorial burp, nothing that would frighten a computer, but the sort of thing that would confuse the heck out of a guy trying to write a script.  Human script author can't deal with your 32 different variants of the process
• iteration: You need to repeat the process the same way on the 100 individual inputs you have.  Iteration looks a  bit like variation.  One difference is that with variation, you have a presumably finite and pre-encoded set of possibilities, whereas with iteration, the set of files is not really encoded in the process but treated as a sort of external knowledge (either a set of files, or the result of running ls).  Also with iteration, you may have the concept of amalgamation.... you have done something to all the files in the directory, and now you want to treat the entire directory as a single object.

I'd gotten to a point where I could encode a good chunk of my pipeline.  It wasn't pretty because it was trying to write Bash in Haskell, but I was hoping that in cases where I had combinations of variations, I would have an easier time than writing the Bash by hand.  It would be an interesting case where composition/abstraction gives you a net win even though you lose out on concision/terseness.  Ie. the blocks are larger because of Bash-in-Haskell, but you can combine them in more powerful ways, leading to ultimately less crazy code.

But I ran into a snag where variants intersect.  I'm not sure I can explain it successfully.  Part of the problem is that because I'm working in a framework where I'm encoding information in filenames, I often have to do some crazy tricks to pass information from one process to another.  Say I have a bunch of sentence selection directories.  I can't pass the full directory names in via the filename unless I do some kind of escaping because I'm using filename components as parameters.  So /foo/bar/baz means "use the foo method for X, the bar method for Y, and the baz method for Z".  This means that I can't make bar itself be a filepath, like /foo/the/path/to/bar/baz unless I automatically munged it, eg doing something like /foo/the_path_to_bar/baz and making sure my munging scheme was reversible and unambiguous.  Also the result would be completely useless to a human being.  The idea I had was that since I was automatically generating the script, I could create a lookup table, something that looks like this:

function plookup {

  case "$1" in
    $'sentence_selection')  case "$2" in
                              BASELINE-s1)  echo \
                                                 "$SENTENCE_SELECTION_DIR"/BASELINE-s1 ;;
                              BASELINE-sA)  echo \
                                                 "$SENTENCE_SELECTION_DIR"/BASELINE-sA ;;
                              gtm)  echo "$SENTENCE_SELECTION_DIR"/gtm ;;
                              t060)  echo "$SENTENCE_SELECTION_DIR"/t060 ;;
                            esac ;;
  esac
}

The idea was that given a path (encoding some arguments to a "function") like /flurb/t060/toto/gargh, I could figure out what the path to the sentence selection directory was by passing the token t060 to the plookup function.   The auto-generated code looks like this:
sdir= $( plookup $'sentence_selection' $( basename $( dirname $( dirname "$1" ) ) ) )
This works more or less reasonably, and allows me to avoid hard-coding the information into the pieces of the my script, but the problem arises when variants combine.  So I now have a way of going from short tokens like "t060" to paths like "$SENTENCE_SELECTION_DIR"/t060, great! Now how do I do the same thing for data selection techniques?  Whoops, now we run into an issue, because the data selection techniques vary on the sentence selection techniques.  In other words, you have a set of data selection methods, and you want to try all the possible pairs of data selection and sentence selection methods.  How do you encode this as a short token, in filepath-friendly ways?

Argh, sorry, that's not very clear.  If you're reading this, you're probably thinking that it's fine and obvious, that you could just do something like representing tuples with some kind of delimiter, so gtm#foo to indicate the combination of the gtm sentence selection technique with the foo data selection technique.

Sigh!

Blather blather blather.  I wish I could be more articulate about what problem I'm trying to solve.  Almost feel like an inmate in a mental asylum, screaming some incoherent gibberish and wondering why nobody believes him.  Need to learn how to describe what I'm even trying to solve in the first place :-(

Anyway, giving up.  Rather than trying to autogenerate this script from a higher-level Haskell representation, I'm going to have to handwrite it instead.  No more time to think.  Just code-monkey around.  I do hope a more clever hacker runs into this sort of dependency/variation/iteration problem and figures out a solution.  Make the Shake build system is what I really want...

GenI file loading cleanups

OK, so I think the GenI code for loading in input files is a bit better now.  The code is actually longer, not a great sign, but it seems a lot more uniform, composable and easy to modify.

Recalling the problem, I have something like 8 different kinds of input files that I read into GenI and my code was a bit of a mess.  It was heavily factored, but somehow in the wrong way.  Something very fragile and useless feeling about it.  Some things that vary:

• most files read lists of things, but some deal with more exotic objects
• most files are text based, but some have the option of some binary encoding
• most files are with parsed with Parsec, some others with simple line processing, or with JSON
• some files are "mandatory" - if we try to load them and you don't supply them as input, something bad should happen; some files are optional - trying to load them should be a no-op if you don't

change 1: use a typeclass

I haven't changed anything fundamental about how the code works so much as restructure it.  The first big difference is that I now have a typeclass that captures the 3 main things GenI does when it loads some file:

class Loadable x where
  lParse       :: FilePath -- ^ source (optional)
               -> String -> Either ParseError x
  lSet         :: x -> ProgState -> ProgState
  lSummarise   :: x -> String

Having this typeclass has made my code a lot more uniform.  The old approach was to have a generic higher order load function that took lots of other functions and various bits and bobs as input (so it was factored but incomprehensible).  Now I only need to specify that the format I'm loading in is Loadable, and it's implicit that I have access to these three functions.

The main function here is lParse.  There are some implicit decisions here. First, parsing should be treated as something that's done on strings, and not files. I'm not sure if I'll come to regret this. In the future, I may update this decision so that parsing is done on some sort of binary blob instead, but with the core idea that this excludes any of the IO behind it. Second, parsing a file and massaging it is treated as a single cohesive black box. My various parsers used to scatter this in different places, mainly in the bit that updates my program state.

The other functions, lSet and lSummarise are not so important.  They update the program state and provide a count string respectively, something along the lines of "3 elementary trees".

change 2: more composable helpers

Instead of having one big load helper function, I've broken down some of the choices into standalone functions that you can then mix and match.

I now have

• withFlagOrIgnore and withFlagOrDie: extracts the value of some flag and respectively silents ignores if it's not there, or just exits the program
• withLoadStatus: takes a file parsing function and decorates it with "now loading X" and "...n items loaded" output (using the lSummarise function)
• throwOnParseError: takes a parse output and either throws an exception if there is a parse error or returns the value
• lParseFile and lSetState: IO bound wrappers around Loadable

Having these functions separated from each other makes it easier for me to modify behaviour, and also helps me to avoid redundancy when I have subtly different requirements (like the binary file thing).   I can just mix and match them in slightly different ways.  One change that this facilitated was for me to replace a dieOnParseError with throwOnParseError, just a straight shot darcs replace and a tweak on the function.

So, GenI file loading now a lot better.  Not going to win any awards or be a Functional Pearl, but at least it's less horrible than before.

a few more steps towards makedo

I've made a bit more progress towards makedo, the build-system-based test harness generator.  Not a whole lot of the stuff because I don't actually know what I'm trying to accomplish.

Basically, all I know is that writing crazy scripts like this drives me crazy.   Right now, I'm trying to replace that crazy script with what I hope will be a less crazy Haskell program, except that the less crazy Haskell program doesn't actually look less crazy because it's trying to generate a Bash script.  Sigh.  Here's the Bash script that hopefully less crazy program generates as output.  You can see here the beginnings of dimensionality, the token alignment method being run off the results of different sentence selection methods.

I don't know what other people doing the same kind of thing I'm doing approach the problem.  Maybe I'm an idiot and everything is doing something obvious; or maybe they just don't care about the same things I do (for example, maybe they are perfectly happy to run a complex N-dimensional process by hand, just writing README files so that they always know what step to execute next, whereas I absolutely insist on automation); or maybe they're just wise enough to treat certain things as non-requirements (for example, maybe they have simpler scripts before they do not use caching of intermediate steps).

In any case, I still expect that I want the right thing but I just don't know how to get it. I think that involves some kind of abstraction.  For me, this means writing something in Haskell.  Right now, I'm reaching for the one abstraction I know, which is a monad, probably the wrong one.

Also, the way I'm currently approaching this is to use Haskell to generate Bash scripts, rather than just make my test harness be in Haskell (and execute IO actions).  This has some downsides, particularly that writing Bash in Haskell can be painful and ugly (gluing bits of syntax tree together)... but I suspect I'll have to go this route for a while longer until I actually have a better sense for what I'm trying to do.  One justification is that I already have a test harness in Bash, so it would be a lot easier for me to compare and see if I'm approaching what I want.

Anyway, so I guess today's work was:

• automatic dependency generation - if you list a target that has some dependencies, then the script will automatically include the targets for the dependencies
• dependency rebuild enforcement - if you pass in a target as input, the only way to get something from it will trigger a redo-ifchange
• rudimentary filename parsing - basically generating a chain of $(basename $(dirname $(...)))

towards makedo : a redo script generator

Not an announcement yet, mostly just a piece of vapourware.  I've long been complaining about the difficulty in writing and maintaining test harnesses for complex systems (where you want to try lots of variations of an experiment that you haven't even finished designing yet).  I call these test harnesses sausage machines.  I suspect that I'm fairly alone in wanting to build a sausage machine where some of my perhaps wiser and more sensible colleagues would just not see the problem of doing things by hand.

My attitude is generally that you should never trust a man to do a machine's job.  Humans are great, but we should be reserving them for things that humans are great at, not in trying to make them reliable executors of a procedure.

Only thing is that making sausage machines can be a very unhappy process.  I'm starting to get a glimpse of an idea for a tool that could ease some of this pain.  Using a bit of Haskell to strip away a lot of the conceptual boilerplate, and hopefully get things for free like "if you try to build off a dependency, you will automatically ensure the dependency is built", and also avoiding a lot of stupid code like building up and deconstructing filenames with basename/dirname when what you really want to do is to pass some arguments to a function.

I'm not entirely clear on what I want yet, I do have a piece of sample code up on darcsden.  If I get this right, I could be a much much happier user of sausage machinery.

GenI parser cleanups and error handling

I have a piece of code which gives me a nagging feeling of wrongness.   There aren't going to be any answers in this blog post, just an exploration of the problem space.

GenI uses a handful of different input files, a tree schemata file, a lexicon and a test suite.

There are others, but we can focus on that for a moment.  I have this common pattern which I use to work with such files:

1. read configuration setting from program state (IORef): this corresponds to flags passed in from the command line
2. read the file
3. parse the file with Parsec
4. post-process the parsed entries (for example, if it's a list of items, weed out the duplicates)
5. update the program state with the post-processed entries

This is neither rocket science, nor computational linguistics, just your basic IO and dance.  Here is the code that handles this bureaucracy.  The type signatures may be a bit confusing, what with the use of existential types to try to support an heterogeneous flag set (probably severely misguided, and now potentially obsoleted by the really nice magical libraries like cmdargs and cmdlib), but I think you can see the common pattern lurking through this.

Some some reason I've spent the whole time working on GenI walking around with this nagging feeling of Wrong-Wrong-Wrong, the feeling that this should be a lot simpler and tidier than it is; but also that I had far more important things to be doing that worrying about a minor itch.  Unfortunately, the itch has just turned into a rash.  The issue is that the error handling is not useful outside the context of an academic software prototype, a prototype which can afford to crash autistically because its programmer has high priority items to hack on.  If you want to use GenI as a library, crashing on bad input is not going to fly.

Notice that the problem is not the code isn't factored.  In fact, it's probably excessively factored, but just somehow wrongly factored, like I've started barking up the wrong tree. I kind of feel like I need to redistribute the pieces and try again.  Part of this is the natural, organic growth of an academic software prototype.  You just sort of glom on features as your needs change.   The problem now is how to tease apart the necessary and unnecessary bits of complexity:

• error class: missing command line arguments -- notice I have a loadFileOrIgnore for some reason, looks like there are cases where you do want to support a command line argument being missing, but not always
• error class: bad file
• error class: missing file, etc [not really handled]
• use cases: reading from GUI support (if the user supplies bad input from the GUI, pop a dialog box instead of just crashing -- this isn't visible in the snippet but it does do that for some inputs), reading from command line and reading via a library call
• status messages (printing "now loading foo..." to stdout
• wanting to avoid code duplication
• binary file detection - in the case of macros files, we have two file formats, a fast-loading binary file format (probably a bad idea?) and slow-loading text file format.  We want to support the behaviour where we first try to read the file as binary, and if that fails, read it as text.  It's worth thinking about dropping this future, but we don't want to break compatibility with old grammars in the process

There are basically a couple of other issues I'm trying to wrap my head around

1. Can we eliminate the use of a global program state to pass things around in?  It feels so horrible, but every time I make a stab at it I somehow burn my fingers and give up
2. What kind of error handling do I want?  We have a lot of tools at our disposal in Haskell.  For example, parse errors can be differentiated from successful parses in the type system.  Unfortunately, one tool I've been using a lot (for convenience) is fail, a function whose presence in the Monad type class is commonly held to be a mistake in Haskell.
3. From the maintenance perspective, how do I divide the code?  What lines do you factor it along?  This is probably one of those tricky questions it would take years of maturation to answer properly.  What do you make easier at the expense of others?
4. From the usability/API perspective, how I divide the interface?  What should I provide as discrete functions, and what should I ball up into do-everything functions?  

Error handling is a subtle topic with tradeoffs, and I think it is very very easy to get the wrong idea about what to do with errors. For example, Freshman Eric tried to take to heart the idea that code should be defensive and robust with respect to errors but ended up writing a lot of code which just "gracefully" caught errors only to hide them (and yes, before you weigh in, I know that you can catch exceptions selectively).  I'm just saying it's more subtle than one may think... Maybe a more mature Haskeller or a LISPer can come in and smack my head around a bit.  For now, I'm working on this sort of rough basis:

1. expected unusual cases - not actually errors (for example, find me the first entry in the list which satisfies condition foo should return Nothing or Just the item)
2. my errors as a programmer - this-should-be-impossible style error should either be cunningly made inexpressible with the type system (this is the sort of thing I admire in clever Haskellers), or greeted with a CRASH.  The crash is something Freshman Eric would not have understood, "wouldn't you want to be robust?"
3. bad input from other programmers using my code - ??? if I'm division, what do I do if the programmer fails to heed the API  and passes me a zero denominator?  If I'm the head function, what should I do if the programmer gives me an empty list?  The Haskell stdlib crashes in these cases (you should know better), but an alternative option would be to be express the errors with something like Maybe.  But does the Maybe really make sense here?  I can see how you would want to express the idea "this dictionary does not have the entry you're looking for", but saying that 4 / 0 is Nothing where as 4 / 2 is Just 2 seems to imply that 4 / 0 is legitimate.
4. user errors - be as graceful as humanly possible, detect errors at the earliest possible stage
5. completely unanticapted errors - ??? nobody is perfect... you can't predict everything that could possibly go wrong (perhaps unless you're one of these Formal Methods types?)

I don't want to get too general about this, not trying to write a programming book here.  Just saying, this sort of thing can be a bit subtle.  In my case, I have a user error (trying to parse a file that's not in the expected format), and I think actually my code does recognise it as such.  The Parsec library recognises it as an expected usual case (which makes sense), and my code simply takes it out with pattern matching (Left/Right) and blithely exits on parse error.  In a sense I'm doing the right class of things (knowing it's a user error), but I'm doing it in the worst possible way; exiting when I should be giving the user some recourse to action.

That should be a straightforward fix to make... just throw an exception, but I still think this code needs a lot of cleaning.

back to GenI

I have to returned to GenI!

Starting this week, I'll be working 2 days a week with Computational Linguistics Ltd (London) and SRI to develop new features, improve scalability of the generator (hopefully by exploring the notion of equivalence class based packing) and integrated it into an interactive tutoring application.

I'm still working at UoB 2 days a week and at Well-Typed 1 day a week for the moment, so my schedule (and brain) are a bit packed at the moment.   In any case, I am very excited to get a chance to get back to GenI.  Ever since I've left Nancy, I have had this feeling of "unfinished business".  It was slightly relieved when I had a chance two years ago to work in LORIA for a month tidying up GenI (for example, JSONifying a lot of the ad hoc file formats), but still.... there's work to do!

Now I have a somewhat more substantial chance to get back to this unfinished business.  I know GenI could be much much better.  I have slight suspicion that being exposed to a bunch of smart hackers in the open source community (my Darcs hobby) may have equipped me a little bit better to tackle these improvements.

The three areas I'm prioritising at the moment:

1. integration - our current client would like to integrate GenI with their software; anything I can do to smooth this process comes first
2. performance - I want to start working towards a notion of equivalence class based packing, really basic standard stuff in GenI but for the longest time, kind of hard for me to reconcile with a tree-as-basic-unit-of-processing world view.  Now I think I have a workable algorithm, and if I understand correctly, one that is also used elsewhere
3. cleanup - in freshman year of undergrad, I had this lecturer (was it Mitch Marcus?) that said that basically every programmer walks around with a slight feeling of embarrassment about his code.  The cleanup will be essentially motivated by integration and performance needs.  Using GenI as a component in other pieces of complex software means that we're going to have to start thinking a bit more carefully about getting a reasonable API

Naturally, I'll be wanting to blog about the progress as I go along, writing being a thinking tool after all.  See you in future posts!

sausage machine fabrication

Just jotting down some rough thoughts about this thing I complain about from time to time (starting with the Makefile of Doom blog posts during my PhD).  The kind of work we do requires the occasional construction of "sausage machines" - long multistep pipelines with lots of different components (first you lemmatise, then parse, then extract X, then compute Y then...).

The goal is to eventually nail down what I find difficult about sausage machinery.  The most desirable outcome would be to recognise what exactly I need to do (I suspect "write some Haskell could actually be a reasonable answer here - a lot of my grief comes from trying to use bash, I think).

My notes should also help me shed light on and perhaps even explain to other people why I use a build system (eg. Make, and now for me, redo).  Or maybe the intention is for me to be able to more effectively make the case for Automation (never trust a man to do a machine's job) and the use of sophisticated technology (run than a "straight shot" shell script)

1. repeatability - whatever you do has got to be repeatable 6 months after the paper has been submitted when the whole thing is a hazy memory
2. incremental/exploratory development - we don't know what the sausage machine parts are from A-Z; we're sort of making C, D, E etc up as we go; sometimes changing our minds about the components and wanting to replace/rearrange them
3. caching of slow computations - a lot of times the things we're cranking out take a long time to run.  We don't want to have to run them over and over again because we want to get some results out of it
4. dependencies between steps - if I change the tokenisation scheme, then the parsing results are no longer valid.  One possible solution is to write a "simple" shell script than just makes sure all the steps are run... BUT if we do this we have to think about caching the results from slow computations, and if we do THAT we have to think about what we need to do to make sure the caches are expired.  There's no way I'm trusting a human being to do this.  This is one of the reasons why I've always used build systems for my sausage machines
5. changing requirements (often at the last minute) - hey it's academia, we've got paper deadlines to meet
6. variations in intermediate steps - because there is a slightly scientific bent to this (which works better X Y Z)?, we want to support doing some of the intermediate steps in a different way and collecting all the results.  Note that there is a dimensionality problem here... if you have multiple intermediate steps that vary then you get increased complexity, not necessarily the kind that makes your computer blow up but the kind that makes your programmers sad
7. non-linear sharing of resources - the sausage machine is not a straight pipeline.  If you've got phases A B C D E, it's not always a simple matter of the results from step C flowing to D and then to E.  The wiring is more complex, with E taking as inputs bits of B and C too
8. fine-grained control - this is partly due to incremental development and slow computation  - if we just use a build system and we get it "right" by connecting the software components to the expiry conditions, we'll never get anyway.  The human operator needs to be able to lie.  My lie is NOT connecting the sausage machine dependencies to the source code.  This means that if I change an intermediary bit of software, my sausage machine won't know it needs to rebuild a step.  In theory, I should tell the truth; in practice, having the slightest wibble in my code trigger the whole pipeline would be insane.  I've got to be able to reach in anywhere in the sausage machine and say "just run component C again, please"... BUT if I do this, the sausage machine needs to know automatically to run D-Z as a result of this.
9. parallelism - hey it's 2011, we've got multicore, we've got the desire to get the results NOW NOW NOW.  Without adding much parallelism to the components itself, it can already be helpful for the sausage machine to run components in parallel to the extent that they are daggy and not line-y
10. remote operation - hack on your workstation, run on the workhorse.  I use version control for this; but actually maybe that's not always good because I also have lots of "foo" and "wibble" and "works now?" commits.  So maybe a combination of version control for mental stuff and one-button-mindless-rsync for the works now?" cases

Sausage machine construction tends to make me very grumpy.  I think I'm using the wrong tool for the job in the form of Bash, though.  The problem is that dealing with variation involves a lot of storing of intermediary results in directories (stage1-a/stage2-b/stage3-c), so I spend a lot of time doing a lot of filepath construction and parsing.  Somehow I sense that a lot of what I really want to express could be simplified with some sort of higher order function on contents-of-directories.  The problem with using Haskell is that well, writing things in straight Bash can be convenient... but I kind of suspect that being able to tackle abstraction in some way would make me go a little less crazy.

RASP is now LGPL!

Hooray!  No more custom license for RASP!  The latest version is under the LGPL.  Now I wish this sort of thing would happen with more academic software.  Whatever the terms, converge on a small set of known and well understood licenses.

files in directories

This is a sort of problem you've heard me struggling with several times before.  I think I'm mainly interested in trying to find a succinct characterisation of the problem.  It's a sort of low-level combinatorial explosion... the order is very small and computers are perfectly fine.  The problem is the human.  This low-order combinatorial explosion introduces complexity [in the "OMG! this is so complicated!" sense] which drives the human operator crazy.

• You have a complex process with several stages.
• Each stage takes some input and produces some output
• You have lots of inputs - each input is represented by tuples of files
• You have variants of the stages

How do you lay out a file and directory structure?  For example, I built up this huge infrastructure that has pre-processing, followed by sentence selection, followed by data filtering and then fragment detection.  It actually kind of makes me think there is actually an interesting programming problem hidden in the mundaneness of it all.  You don't have a simple pipeline really because for example, the data filtering step takes output from pre-processing and sentence selection... (so this actually makes me think a bit of designing circuits, and I have the odd desire to whip out some Haskell arrows even though I don't really know what the latter are).

The problem is not the coding of the pipeline (which I'm doing in Bash coupled with redo); but it's the set of implicit and mind-numbingly mundane design decisions (should I name the files foo.bar or bar.foo).

I know one thing that went wrong that I'm cracking my head on.  I've got this setup where I'm storing things in one directory, and successively augmenting this directory with files.  So you've got a big directory called Workspace and in it, you have something like files: foo.{x,y}, bar.{x,y}, baz.{x,y}... and you augment it with a procedure that takes .x and .y files and produces .z files, which you then store in the same directory, giving you: foo.{x,y,z}, bar.{x,y,z}, bax.{x,y,z}.

Being a functional programmer, I find this to be deeply deeply wrong.  Adding files to directories is a sort of mutation, and mutation will always breaks your heart in end.  Ideally the workspace should be one set of files, and if you have something that produces another set of files, well that's a separate directory.

But mutation... it's so convenient.  I knew this could come back to haunt me.  What if the next stage of your pipeline involves taking inputs x and z and producing a new .a output?  That is, going from foo.{x,z} to foo.{x,z,a}.  OK, now what if you want to introduce a variant of the process that produces those .z files?  You're stuck.  You can't dump them in the same directory (lest you clobber the first variants' versions).  You can't put them in a new directory (unless you're willing to write another loop to copy or link the .x files)

So here I am trying to introduce a new sentence selection variant to see how it behaves.  I guess the right thing to do is to break my addiction to the one-big-successively-augmented directory.  But it means digging into each one of my sub-stages and changing them to accept multiple directories as inputs (look in dir1 for your x inputs, look in dir2 for your y inputs); giving them a bunch of logic to break down file names and put them back together etc; and then writing the scripting to give the right directories to the right programs.  But convenience always bites you in the end.

read wider

Three things I constantly forget to do:

1. read
2. review my action/waiting lists
3. check my calendar

I get so caught up in trying to put things together that I never get around to these feeding activities.  I guess there are three basic kinds of reading that I'm aware of:

• Depth - this is the sort of unspooling-the-ball-of-yarn reading I did in the first part of the PhD, a lot of "OK, I want to learn about X... wait I don't understand X, maybe I need to read background Y and Z... oh wait, I don't understand those either... and so forth"
• Breadth - somebody had to teach me this (thanks B!): you need to build up a sort of awareness, not only about what's going on right now but just more generally how the whole field fits together.  You're not reading to understand things here but (A) to know that they exist (B) to understand what things are connected to what other things.  Benefits to this are not being stuck in your sub-sub-sub-sub-specialisation (being a bit more open to cross-fertilisation) and also having an easier time when you're reading for depth (I find that it takes me several passes to understand something and having some reading-for-breadth awareness that X is a software for Y makes it easier to understand X)
• Hell of it - this is the sort of reading I do on No Screen Time Sunday.  Things like Language and Society undergrad sociolinguistics textbook I unearthed during a trip home, or Evidence Based Teaching (although I don't lecture), Designing for the Digital Age, just whatever strikes your fancy.  It's kind of depth-ish because you're going in book-deep, but not really trying to unspool any balls of yarn.  Just letting the author take you on a journey.  It's kind of breadth-ish because you want to cover ground that's outside your discipline.  But unlike the first two it has a kind of purity of being not-so-goal-directed.

So I guess I used to do reading for depth, never much reading for breadth.  Now I tend to do a lot more reading for the hell of it, but I've totally neglected the first two.  Sitting here waiting for my program to crunch through the first 10% of Europarl, I just realised that I have a chance to do a little reading for breadth.  You know, look at the ACL anthology.  Read the titles, read the abstract, flip through the paper and get that glimpse of awareness.  This guy is connected to that guy.  This algorithm builds off that one. 

Kind of like twitter for research.

equivalence-class based packing for GenI - node counting idea

Here's an idea that came to me when I was quite literally sitting around doing nothing.  Ah yes, the whole business of getting ideas for things you are not working on.

GenI is a chart generator.  It has a notion of agenda/chart-based control to order operations and sub-tree sharing (basically for free by using Haskell), but what it lacks, crucially is a notion of equivalence-class based packing.  Packing is extremely important because it's what allows you to control the combinatorial explosion of intermediary chart items as you explore the search space.  In particular, in GenI, we're interested in exploring the whole space (enumerating all possible parses), so we're not particularly after any sort of pruning.  Really, the chart is all about making sure we don't do a lot of stupid redundant work.

What you want your equivalence classes to do is partition your set of possible chart items in such a way that lets you compare the subsets with each other rather than the individual items within.  So to illustrate example, imagine you're starting out with 8 items (in classes A, B, C) :

• A: 2 items
• B: 3 items
• C: 2 items

If you were just doing straight parsing without any sort of equivalence classes, you're going to be deriving 2x3x2=12 items.  But if you use the equivalence classes, you only need to do 1x1x1=1.  At the end of the day, you're still going to have 12 items when you expand the set, so what have you gained?  Well, I'm still a bit fuzzy on that to be honest.

Part of my answer is that it saves you work when you have chart items that don't lead to any derivations. If you have say another 4 items that can combine with these 12, that gives something like 48 items to sift through (before you work out that you can only keep 12 of them).  With equivalence-class packing at least, you avoid this sort of explosion.

I'm not sure if that's the whole story.

Anyhow...

There's one way I knew to get equivalence-class based packing for GenI, and that would be to implement a CKY/Earley parser for TAGs.  I tried that during my 2nd year of the PhD, but the whole thing sort of fell apart because my coding sucked (I think now that I know about the Zipper, I can have a much better chance).  Nevermind, it gave me a chance to refactor the hell out of the code.  I still think it'd be good to get back to it... but what if there was a simpler way?

the plan

First, let's make some simplifying assumptions:

• no substitution - we're in an adjuction only grammar (which is easy because GenI does 2-phase generation only)
• auxiliary trees all have the same root node category (we can relax this later on)

The problem I was struggling which trying to work out how packing would work is that it's not just good enough to partition your set of chart items.  What you to do is to partition them in such a way that the result of combining two equivalence classes can be derived from the classes themselves and not the items within them.  Coming up with any old scheme for putting your chart items into bins is the easy part.  Now what happens when you plug two of those bins into each other?  And if you do come up with such a scheme, is it going to chunk your inputs so that you get that factorisation thing going on, or are you just going to partition into a bunch of singleton sets?

And here's the scheme I just thought of:

Two items are in the same equivalence class if they have the same semantics, and the same number of nodes  (of a given category X) which can receive adjunction

If you do adjunction with this scheme, all you have to do is take the union of the semantics (à la Martin Kay) and the sum of the numbers of the nodes (minus 1 or something like that).  And if you want to deal with more than one category, just do it in phases.  First focus on one category, derive everything you have.  Then pick a new category and re-pack everything into a different chart.

I hope that this scheme is broad enough that it actually divides our chart into non-singleton sets. 

Will it work? Will it work? Will it work?

I can't wait to find out.

Too bad I don't actually have time for GenI hacking.

folds and subsamples

Ah, one bit of clumsy language (I always get terminology subtly wrong like this).  I noted that it's a good idea to store your folds as text files and not as a directory tree of copies of files.  Actually what I really meant is that it's a good idea to store the subsamples, ie. the subsets of your partition.  You could also store the folds, but duplication root of all evil.

ABC subdirectory permutation

This is the sort of problem I find myself frequently confronted with, stuck on and bored by.  I feel like I've seen this enough times that I should have knocked out a set of standard tools to just get it right so I can move on with the interesting bits of the research, yet despite the handful of variations I've experienced, it always feels like a new problem wanting its own ad-hoc solution.

It's a simple permutation thing, a question of best-practice in laying out your directory.  You have a set of texts which can be characterised by N dimensions:

1. primary method (the thing I'm evaluating, eg. corpus-acquisition technique)
2. secondary method (some other source of variation, eg. generation system)
3. fold
4. evaluation set (training, development, test)
5. subcorpus

The first time I saw this problem, I only had #1, #3 and #4.  Then one day I was introduced to an evaluation which cared about #1, #4 and #5 (intersection between #4 and #5, it's important when dividing your corpus into subsets that you preserve the distribution of files in subcorpus, so if your corpus has 10 As, 40 Bs, and 30 Cs, and you're splitting into tenths, you need to make sure you still have 1 A, 4 Bs, and 3 Cs).  Now I think I'm dealing with most of these (except #4).

My failure to generalise over this problem, the sense that I'm continuously reinventing wheels that I'd previous reinvented, makes me feel stupid.

Some tricks I've learned along the way:

• Store your folds as a list of filenames, not the files themselves (obvious in hindsight!).  
  1. If you're doing repeated random sub-sampling, you save the space wasted by having more than one copy of the same file.  OK maybe you could also have avoided this by using hard-linking, but then what happens when you copy your stuff to another filesystem?  Do the hard-links get preserved? (Maybe, I dunno.  To be clear, I don't mean from filesystem A to filesystem B, but do the links between A/X and A/Y get preserved when in B/{X,Y}?  A simple ls -i can tell).
  2. It reduces the risk of human error that redundancy introduces.  Duplication is the root of all evil.  These arguments don't apply maybe if you're partitioning your data for cross-fold validation (no duplication), but there's still
  3. It makes it easier to communicate your folds.  For example, you do sanity checks like "does my partitioning of the fold into a further training and development set reproduce the original fold" with simple cat and diff.
• Handy tools that somebody else has written. You don't need to knock out your own 5-minute script:
  • split - split a file equal-sized chunks (useful for partitioning)
  • unsort - randomise lines in a file (randomisation)
• Version control randomisation. This goes back to the third argument for why you should store your folds as a list of filenames. You want to make sure you can reproduce every single step of what you do. So if you randomly partition your data for cross-fold validation, save the files containing your list of filenames
• Build systems can be abused as test harnesses.  I've been experimenting with an implementation of DJB redo and am loving it!  

Right now, I'm staring at my monitor dithering over various subdirectory layouts for my test bench.  Do I want my data divided into

• Primary-method/Secondary-method/Fold/Subset (tech-A/gen-I/fold-1/training)
• Fold/Subset/Primary-method/Secondary-method (fold-1/training/tech-A/gen-I)
• Fold/Primary-method/Secondary-method/Subset (fold-1/tech-A/gen-I/training)

Only 24 permutations to consider, I guess.  Which layout is going to maximise sanity-preservation?  The endless struggle.  Go on, you do the science.  I'm still trying to work out how to arrange my lab bench.  I wonder if I'm the only one that has issues with this.  It's just so embarrassingly inconsequential yet it feels like one of my main pre-occupations.

Edit Stepping away from the problem I realised that actually, I once inherited a solution to the first version of the problem using Fold/EvaluationSet/Text.Primary-Method (slightly outside the box -- who says it all has to be subdirectories)?  Still doesn't solve the Secondary method.  In a previous blog post, I wanted to compress the dimensions by merging the Fold and Primary method, but in hindsight, that doesn't make things any easier.

double/parallel symlinking

Here's a little pattern that some people might find useful.

Scenario

You are primarily working on a shared file system plus a personal hard drive.  You have source code and data/crunchy-chrunchy-work.

• The shared file system is backed up and also makes for easy accessibility for colleagues. 
• The disk drive is way faster (lower latency), which makes a difference if you're going crunchy-crunchy over a lot of files.  It also has more space (in 9 years working in some lab/university or another, we've never had enough space on the file server)

Basic setup

So how you do you store things?  The pattern I'm starting to take is: 

• /shared-space/my-project-dir (on shared)
  • src
  • etc
  • README
  • ...
  • symlinks to /hardisk/my-project-dir/{data,tmp}
• /harddisk/my-project-dir
• data
• tmp
• symlinks to /shared-space/my-project-dir/{src,etc,README}

I wouldn't go so far as to call this a best practice.  The key here is for both directories to mirror each other  (which is an important detail that I key forgetting) in terms of tree depth.  The reason for this is to allow you to preserve the illusion (for your own sanity) that this is all one tree.  For example, you want to be able to be in the src directory and access the tmp directory using a relative path like "../tmp/foo/bar".  Likewise, you also want to be able to do the reverse from the tmp directory.

If I'm not thinking carefully about it, my links will tend to be assymetrical.  Rather than creating a parallel my-project-dir under /harddisk, I'll just like /shared-space/my-project-dir/tmp to something like /harddisk/tmp-for-my-project-dir... which breaks the minute I try to round-trip from src to tmp and back with relative paths.

prone to confusion - filename extensions

One of the tensions I face in building up NLP pipelines is extensions for filenames.  Should extensions reflect a sort of format, or a stage of processing?

My code uses the latter, so I get something like:

• en
• en-selected
• en-fragments

And I can tell them apart.  They all start with "en" because they are compatible with each other in a sense.

stage of processing

• can tell the different stages of processing apart at a glance
• can dump all the files into one directory

format

The problem with the stage-of-processing approach, however is that one can get easily confused with the proliferation of extensions (wait, was that .data or .data-subset I wanted), and also it makes it impossible to use directory-wide graphical diff without writing some sort of wrapper process.

It also brings up sorts of side questions like "what if you want to skip stage 2 and pretend the output of stage 1 is the input to stage 3".  This sort of thing makes it much more natural to use format extensions.

Right now the compromise solution is to go with stage-of-processing extensions, and make copies reflecting snapshots of processing using format extensions for easier diffability, but meh

multi-annotation

For the corpus construction work, I'm finding myself in a position where I want to combine different kinds of annotation, say from a

1. domain-specific tokeniser
2. transliteration tool (iconv to strip accents)
3. lemmatiser/pos-tagger
4. named entity recogniser

For the moment, I'm keeping things real simple by just assuming a dumb pipeline (and ignoring NER).  That means making compromises like forcing ASCII text on all components, even those which are sophisticated enough to deal with Unicode, and also working with funny tokens, treating lemmatiser/pos-tagger as input to the corpus construction stuff, giving it tokens like "be_VBZ" instead of "was".

But over time I may find myself needing some sort of multi-track annotation approach because different components require different stages of input.

So I'm guessing here that I should use some sort of standoff-annotation scheme even though my needs don't really have any crossing/overlapping boundaries.  Everything is based on tokens.  I'm mainly attracted to a standoff annotation scheme because

• I want multiple layers of annotation.
• I don't want to decide in advance how many layers of processing I add (a simple column-based text format like tab-delimited would get unwieldy fast)

I'm a bit resistant to just using GATE for this (sorry, Java allergy plus a lot of my components are written in Haskell; and writing little Java stub wrapper programs is boring).  Turns out from a quick Google search that other generic tools exist (Amy Isard's slides [PPT]).  GATE is really neat and all (being able to cook up complex processing pipelines with arbitrary components is great, and I'll admit that having a GUI makes it a whole lot more discoverable), but really I just wanted to know about things like standard formats for standoff style markup.

Having glanced at Amy Isard's slides, I think I'll be trying the LT XML library next, hey maybe even make a Haskell binding to the library if the Unixy command line tools don't do what I want.

Going with the stupid pipeline first, though.

summing logs

Got a bit of hacking to do today for the corpus acquisition stuff.  Trying to work out why our sentence extraction (or deletion) depending on your POV works so badly.

Trying to evaluate results (informally) is a bit of a tricky enterprise for me.  I tend to just rely on your basic Unix text processing tools and graphical diff.  So if example, I want to see what our tool, which is supposed to delete boring sentences from a text actually does, what I'll do is to run cd before; wc -l -w *.txt and the same with cd after.  This gives me the number of lines and tokens in each set of texts.  Assuming we're working on the same files, I can then view the two word count files, finding out for example that there are no differences (which means we haven't been deleting anything), or that there are systematically zero tokens/lines in the after end... which means something has gone wrong.  In my case, both are true...

I think I've worked out what it is.  Either I was using some kind of crude method before (deliberately), or I just misunderstood some idea somewhere (likely), but in any case I've got some logarithms to sum.  Sure wish I had maths more firmly anchored in my brain.  I can think thoughts like "oh, summing logs of probabilities is like multiplying them, which makes sense if you've got a reasonable independence assumption and you want to see the joint probabilites"... but then if you push my brain just a little further "so what does it mean when take the mean of these sums" it explodes.  Puny Erician brain...

paging Past Eric...

Yesterday was unhealthy.  Taking on anti-pattern after anti-pattern.  I've discovered a kind of unthinking foot-dragging that I can be prone to, the kind of thing I need to target+kill.  It goes like this:

In my hillwalking package have something like 15 scripts, all written using the cmdargs 0.1 library.  Cmdargs-0.6 was recently released and to avoid even greater pain in the future, it'd make sense to upgrade all my scripts to work with it (smaller jumps than bigger leaps).  I discover the incompatibility after hitting cabal build to try and compile all 15 of my scripts and seeing the first one fail.  After figuring out how to update my script, I hit compile again and move on to the same source of failure in the second script... and so and so forth until script 15.  Can you see how amazingly inefficient that is?  To make matters worse, because my code is compiling and it takes a little while and I don't have the patience to sit there, I shift to another desktop and go do something else, but not something productive because it's only a little while to wait anyway.  Of course, while I'm stuck there reading some Wikipedia article on random topic, the compilation has completed and script 7 is waiting to be corrected.  HUGE time sink, could have been easily avoided.

What I should have done:

• If I absolutely have to go through this repeated tweak-and-compile loop, then at the very least, I should be using my easy notification script.  So instead of running "cabal build", I should have run "notify-long cabal build", in which case I would have gotten a little notification telling me that the job was completed and if there was an error.  That would have avoided that inefficiency from the build having been long completed while I sat there absorbing some totally useless trivia (and not even in the positive Russell-defence-worthy sense of useless)
• ... or I could have avoided the loop altogether by stepping back and thinking: wait.  I seem to be doing this a lot.  Eric, did you use cmdargs in all 15 of these scripts?  Maybe you should open them all up in your text editor and try to make the same change everywhere.  Cabal build in one go.

Gah! This sort of mental sleepiness (maybe due to actual sleepiness), makes me really angry at myself.  Funny how I only seem to recognise these things long after the fact...

past eric

Now that I've gotten over the stupid hump, and I've rebuilt all my 15 scripts from the hillwalking code, I can actually get to the work I was meaning to do this week, which was to understand what exactly I was doing.  This entails re-reading some emails, my slides maybe and staring really hard at my code again.

archiving fail - confXX-bodyofwork

OK, just got bitten by an archiving anti-pattern of mine.  What happens when you have an ARCHIVE/2010-brighton that looks like this?

• ...bunch of stuff...
• coling10-hillwalking
• ... bunch of other stuff
• inlg10-hillwalking
• ... bunch of other stuff

Now you want to revisit the hillwalking work.  So you're too clear-headed about it because the midafternoon drowsies are sinking their hooks into you and you only notice one of the things marked hillwalking.

... and you spend the next week working on a piece of code that's at least one conference paper old until some voice in your head says "huh! I thought I made myself a diagram of this stuff.  Where's my diagram?!"  (and five minutes later "DOH!").

Good thing I was using darcs and could easily merge my work back together.

Revised scheme

Basically I think that Big-Endian is the way to go, and that when it comes to work that's somehow conference-related, the conference is not the Big End.  The Big End is the thematic work which may be related to several conferences.

Maybe the scheme I should have used is

• ARCHIVE/2010-brighton/hillwalking-A-inlg10
• ARCHIVE/2010-brighton/hillwalking-B-coling10
• PROJECTS/hillwalking-C

And if I later on discover that this needs to be pegged to conference and I'm done with it, I can save it as ARCHIVE/2011-brighton/hillwalking-C-foo11.

Note that I'm using 2-digit years to denote a conference naming year and not the calendar year.  So I'd say 2010 to refer to the year 2010, but INLG 10 to refer to the INLG conference that takes place in 2010.   So you might see me with things like ARCHIVE/2009-brighton/project-A-acl10

The amount of thought I put into this must be a sign of mental illness.