march is semi-stuck

Well, I think I've worked out what triggers the negative array size exception in Joshua (although I don't know the underlying cause): rare tokens.  The METEO corpus has lots of wind speeds (GUSTS 30-35) and if I replace all the numbers by zero (GUSTS 00-00), everything is fine.  I can start trickling numbers back in, for example, leaving the 3s alone (GUSTS 30-30), but if let in too many numbers, Joshua blows up.  So we'll have to wait till we have a more reasonable looking corpus via the British hills stuff (or until we finish training on the example data).

In the meantime, I'm trying to build up some kind of corpus testing infrastructure.  I'm quite tempted to try this with redo a simple-looking and fairly elegant alternative to make.  Right now I have a mish-mash of two makefiles (yes, yes, recursive make considered harmful) and some shell scripts, and I'm wondering if pushing everything into scripts will simplify matters at all.

joshua march resumes

Well, I've trained something or another in Joshua (just following the excellent instructions and example), which took something like 19 hours on our fast machine [I failed to heed the advice to start with a subset of the data, this is also partly due to copy and pasting example commands which are based on the full data].  Now basically the mission is to replicate the rest of the example all the way to running the decoder on some  test inputs.

Once I have that working, I probably want to see if I can make it work on some of our own data.  I'm a bit worried about this NegativeArraySize exception that got thrown the last time I tried (thought it may be related to a bug I saw on the list, but upgrading to SVN version didn't help).  Possibly the "best" outcome of the data would be some kind of minimal example I can send to the list.