Somehow between darcs and work, I just can't muster the energy to do koweynlg on a regular basis anymore. Maybe it's a good thing, and maybe it's served its purpose anyway (helping to figure out 'now what the heck have i been doing all these months'?)
Lately, been working on the evaluation website... tightening up some Perl scripts, getting the hang of working with taint mode. Almost there. Next we need to generate some new forecasts, which is scary because it's murky.
I'll be doing a lunchtime seminar in December for my GenI work, which I hope I have not completely forgotten
Thursday, 25 September 2008
Thursday, 11 September 2008
methodological blunder!
I'm an idiot. Say it three times.
Solving the date problem made me think that I should a little sanity check : all of the gold texts for each generation method should be the same (duh, right). A little graphical diff later and my heart sinks. Of course they're not. Argh! I've been comparing apples and oranges the whole time (now I hear the voice of CG in my head, "I don't take too much stock in numbers").
Of course, if I wasn't an idiot, I would have realised that the silly dates problem (preserving the id of each test case across corpus construction) was actually a really big deal, because it means you can compare each system against the same set of gold standard texts (instead of different copies of what you assume is the same texts, but whoops, you had a little post processing step in the middle that you never noticed which inserted whitespace between punctuation... not the same gold standard text after all!)
/me kicks himself
This should have been fricking obvious. I think this means that our recent scores are inflated because we're getting rewarded for easy-breezy trigrams whereas the other systems were not (maybe I'll be lucky and find out that actually our bleu scripts also do some pre-processing of their own, and it all cancels out).
Looking at this graphical diff also helped me catch a bunch of bugs in my scripts. For some reason, one of my helper scripts was dropping the last line of its corpus files. This is no big deal when you have a thousand entries or so... but when you're breaking things down into itty bitty widdle subcorpora with maybe three lines each, heheh... oops?
The other bug is the generic sumtime segment detection script which I extracted from the pCRU code. Only I didn't extract it right, because I left the text pre-processing regexps in the code (not generic!), which means that our segments do not look like forecasts. Now I've got this separated again.
The only good news in this is that automating all this crap has just paid off handsomely. If I had to go through all this by hand again, I'd seriously be in tears right now. But all I have to do is fix my scripts, tweak my Makefile, hit the big green button and go home. Except not tonight because I probably won't be done in time. Lessons we learn from this debacle:
It makes you wonder about this whole process though. How many reported results are actually due to bugs in people's test harnesses? Hopefully everybody is just more thorough and careful than I am. Seems like we should do something about this though.
Solving the date problem made me think that I should a little sanity check : all of the gold texts for each generation method should be the same (duh, right). A little graphical diff later and my heart sinks. Of course they're not. Argh! I've been comparing apples and oranges the whole time (now I hear the voice of CG in my head, "I don't take too much stock in numbers").
Of course, if I wasn't an idiot, I would have realised that the silly dates problem (preserving the id of each test case across corpus construction) was actually a really big deal, because it means you can compare each system against the same set of gold standard texts (instead of different copies of what you assume is the same texts, but whoops, you had a little post processing step in the middle that you never noticed which inserted whitespace between punctuation... not the same gold standard text after all!)
/me kicks himself
This should have been fricking obvious. I think this means that our recent scores are inflated because we're getting rewarded for easy-breezy trigrams whereas the other systems were not (maybe I'll be lucky and find out that actually our bleu scripts also do some pre-processing of their own, and it all cancels out).
Looking at this graphical diff also helped me catch a bunch of bugs in my scripts. For some reason, one of my helper scripts was dropping the last line of its corpus files. This is no big deal when you have a thousand entries or so... but when you're breaking things down into itty bitty widdle subcorpora with maybe three lines each, heheh... oops?
The other bug is the generic sumtime segment detection script which I extracted from the pCRU code. Only I didn't extract it right, because I left the text pre-processing regexps in the code (not generic!), which means that our segments do not look like forecasts. Now I've got this separated again.
The only good news in this is that automating all this crap has just paid off handsomely. If I had to go through all this by hand again, I'd seriously be in tears right now. But all I have to do is fix my scripts, tweak my Makefile, hit the big green button and go home. Except not tonight because I probably won't be done in time. Lessons we learn from this debacle:
- Automate from A to Z: don't let yourself write sh loops on the command prompt or throwaway scripts... you want the Official Experimental Test Harness, and you want to make it so that you can just hit go to rerun the WHOLE experiment from scratch. The harness should be pretty stateless, should be able to go from starting point to the very end. This is the one thing I did right here (only I didn't really do it right because I only decided to get my act together and automate it after producing the numbers the first time)
- Audit your code. I was copying helper code from other experiments that I had been running, stuff with pCRU and WASP... but then my helper code had all sorts of experiment-specific pre-processing (like this remove-the-last-line crap)
- Sanity check your results. Use graphical diff tools to sanity check (but use non-graphical diff to automated sanity checking; a good check is "there should be no differences between these files")
It makes you wonder about this whole process though. How many reported results are actually due to bugs in people's test harnesses? Hopefully everybody is just more thorough and careful than I am. Seems like we should do something about this though.
Wednesday, 10 September 2008
slight scripting
Today: work on darcs stuff from 09:30 to 14:00 (excessive, but I note that today I also did a bit of infrastructure work, for example fiddling with mutt/gmail to figure out how to access my Sent Mail, and also to enable the header cache... that stuff is one-off)
14:00-18:00 work on work-work (although not giving it as much of my attention as possible). Did a small script modification for the genchal stuff (using the Text::Levenshtein module from CPAN), and also the Makefile. Getting closer! I hope I can have numbers by the end of the week.
I'm tempted to skip Aikido practice tonight because I really want to get work-work done, but I guess I need to keep doing eric-repair in order to survive process.
I'm just going to have to accept that my work week consists of 6 or 7 half-days. I didn't come in last Sunday, which was bad [but probably needed]. Soon I will figure out how to make the darcs stuff more sustainable. Two hours at the office every day is ideal. Will probably need to get more disciplined about allocating fixed blocks of time for this. Right now, I'm trying to get over a few humps though.
14:00-18:00 work on work-work (although not giving it as much of my attention as possible). Did a small script modification for the genchal stuff (using the Text::Levenshtein module from CPAN), and also the Makefile. Getting closer! I hope I can have numbers by the end of the week.
I'm tempted to skip Aikido practice tonight because I really want to get work-work done, but I guess I need to keep doing eric-repair in order to survive process.
I'm just going to have to accept that my work week consists of 6 or 7 half-days. I didn't come in last Sunday, which was bad [but probably needed]. Soon I will figure out how to make the darcs stuff more sustainable. Two hours at the office every day is ideal. Will probably need to get more disciplined about allocating fixed blocks of time for this. Right now, I'm trying to get over a few humps though.
Tuesday, 9 September 2008
silly dates problem
Haven't done much work-work early this week (yes darcs stuff, but it is settling down!).
Got my Further Leave to Remain, so will be here till end of Dec at least.
Been working on the silly dates problem: which is that the tools I'm using don't allow me to associate corpus data with arbitrary identifiers. For WASP, for example, I could use an XML idref, but not an arbitrary string. This means that naming the individual data points is tricky (the names are some alphanumeric sequence, actually an English date like "25Mar04"). This is just a silly pain in the butt. I'm sure there is a nice clean solution out there.
My horrible solution is just to generate a separate test corpus for every input file instead of lumping them into one giant corpus. This means I've been spending the past week extending my Makefile with all sort of pattern rules and patsubsts.
Almost there. I think I've got it licked for WASP now.
Got my Further Leave to Remain, so will be here till end of Dec at least.
Been working on the silly dates problem: which is that the tools I'm using don't allow me to associate corpus data with arbitrary identifiers. For WASP, for example, I could use an XML idref, but not an arbitrary string. This means that naming the individual data points is tricky (the names are some alphanumeric sequence, actually an English date like "25Mar04"). This is just a silly pain in the butt. I'm sure there is a nice clean solution out there.
My horrible solution is just to generate a separate test corpus for every input file instead of lumping them into one giant corpus. This means I've been spending the past week extending my Makefile with all sort of pattern rules and patsubsts.
Almost there. I think I've got it licked for WASP now.
Thursday, 4 September 2008
test harness almost complete
Phew. Another day of Makefile writing. I now pretty much have my whole test infrastructure in place so that you can just hit 'make everything' and rebuild the whole test.
Now I need to deal with that silly dates problem.
Unfortunately, I suspect that some of the software (WASP?) isn't really written with parallelisation in mind. So I can't take advantage of anything like make -j to run a whole bunch of stuff in parallel (unless I start using different machines for each job... hmm)
Caught up on my sleep debt and managed to get in before 14:00 today. Making progress! Hopefully this means the darcs stuff is starting to become manageable
Now I need to deal with that silly dates problem.
Unfortunately, I suspect that some of the software (WASP?) isn't really written with parallelisation in mind. So I can't take advantage of anything like make -j to run a whole bunch of stuff in parallel (unless I start using different machines for each job... hmm)
Caught up on my sleep debt and managed to get in before 14:00 today. Making progress! Hopefully this means the darcs stuff is starting to become manageable
Wednesday, 3 September 2008
more makefile
Another one of those days where I spend the whole morning working on darcs stuff. I could have gotten into to work earlier, but I was also really tired from stupid-insomnia and took many nap-like things.
The darcs stuff looks like it's settling down a bit, though. A lot of pre-work has been done just putting infrastructure in place. So we may be getting to a point where darcs can become part of a sane and comfortable daily routine!
Anyway, today I extended the Makefile, so now I know I can reproduce the whole moses train and test cycle. Tomorrow I work on WASP, and then on dealing with the dates issue the best I can.
The darcs stuff looks like it's settling down a bit, though. A lot of pre-work has been done just putting infrastructure in place. So we may be getting to a point where darcs can become part of a sane and comfortable daily routine!
Anyway, today I extended the Makefile, so now I know I can reproduce the whole moses train and test cycle. Tomorrow I work on WASP, and then on dealing with the dates issue the best I can.
Tuesday, 2 September 2008
argh... koweynlg is breaking down
Darcs stuff is keeping me very busy.
Got in this afternoon sometime after lunch. Did some work on the reg stuff, also on the EACL paper (no actual writing). Then couldn't really stop my mind from wandering. Going to be a long week.
Sent a mail to the moses people to get advice. Got some nice comments. Thanks!
Ok, cool, did a little bit of work tonight (although I made the mistake of staying late... which I know from past experience is ALWAYS a bad idea). Starting setting up a test harness for the EACL paper. I know from the PhD that this is definitely a good idea. In fact, if I had done it in the first place, I would have a bit less work cut out for me this week; just tweak a few scripts here and there and hit the big 'go' button. You always want things to be canned and reproducible. You will thank yourself a few months later when you have no idea what's going on and for some reason or another you need to repeat an experiment.
You won't be able to make everything self contained (disk space being a constraint), but at the very least, all your little homegrown processes and scripts should be canned.
Got in this afternoon sometime after lunch. Did some work on the reg stuff, also on the EACL paper (no actual writing). Then couldn't really stop my mind from wandering. Going to be a long week.
Sent a mail to the moses people to get advice. Got some nice comments. Thanks!
Ok, cool, did a little bit of work tonight (although I made the mistake of staying late... which I know from past experience is ALWAYS a bad idea). Starting setting up a test harness for the EACL paper. I know from the PhD that this is definitely a good idea. In fact, if I had done it in the first place, I would have a bit less work cut out for me this week; just tweak a few scripts here and there and hit the big 'go' button. You always want things to be canned and reproducible. You will thank yourself a few months later when you have no idea what's going on and for some reason or another you need to repeat an experiment.
You won't be able to make everything self contained (disk space being a constraint), but at the very least, all your little homegrown processes and scripts should be canned.
Subscribe to:
Posts (Atom)
