state of the thesis 2006-06-24

Monthly report deux, a little early because of the upcoming vacation.

The first few days of June were exteremely hectic, much like the last days of May, because I was finishing up a trio of papers with Claire, Yannick and Carlos. Things slowed down a little bit afterwards, and I concentrated my energy on finishing off the posters that went with those papers, and summarising our recent paper-writing activity into a Claire report. Said report consisted essentially of copy and pasting content from both T8 papers and trying to massage them together, though not with very much effort, as well as performing a little bit more analysis on some of the open questions. It is my hope that this report could eventually become a thesis chapter, maybe with heavy rewriting and reworking, but at the very least, with little morsels of content on somewhere on paper. All of these bits and pieces of writing made up for half the month.

The second half was spent phd-working in earnest: I couple of days trying to find SFG bib material, trying to install KPML (with some success, no graphical interface though) and hacking together a quick little SFG system browser. Dear future eric: you should consider running the non-gui KPML generator on those Nigel grammar examples... see what comes out. After discussing this for a bit, me and Claire decided to focus instead on the paraphrase selection stuff. Wrote her a short report, and spent the rest of the month making improvements to GenI, gtester, and the makefile-of-desire. My improvements were motivated by two short term aims:

1. make the generation-end of semtag easier to install and use, more critically, easy enough that Claire could take advantage of all my recent engineering work in GenI (I think I have accomplished what I set to in that department)
2. implement the new selection stuff (done)

I'm not sure what I'm going to do with July. I have some ideas for improving the user interface, and think I can implement them pretty quickly. It would make GenI less confusing to use when you have different sessions, and unify some graphical loose ends (debugger vs generator vs main window). The two ideas are that each debugger session should include a copy of the main window as a tab -- although this copy only affects the current session instead of spawning a new one, and that completing a debug operation should send you into a results tab. Other than that, I except that there will be more discussions with Claire about selection, test suite construction, and a general accumulation of ideas to pursue while she's gone the month after.

In any case, here is the slightly amended thesis graph with this month's changes:

holiday till 5 july

Didn't really feel like coming in to work today. Sort of straggled in around lunchtime, a bit like in the pre-overdrive days (except I still got up early). Meh. Didn't accomplish very much either.

Had some coffee.
Tidied up some stuff.

Learned a little bit about so6 (it's a pun, pronounced saucisse, as in sausage), homegrown revision control system that is patch-based very much like darcs. Spent some time chatting on #revctrl about this, and wrote a mail to PM the "father", hoping that he would get in touch with the darcs folk and exchange ideas.

Worked on a couple of shell scripts for the trac server (putting together a nightly dist script, and a tarball cleanup tool).

Learned all about Alex's reference resolution stuff so that I can present his poster for him in Sydney (not sure it all fit into my head though). Reference domain. H, H*, H+. Salience, focus, alternation. Check. Any questions, please send mail to those guys.

Anyway, off on vacation I go.

geni and gtester hacking

Some finishing touches on GenI and gtester before handing them off to Claire. Made another fix/refactor of the gui code after realising that I had screwed up something fundamental. Looking at the changelogs, I get the impression of not really having done very much. A handful of gtester improvements, I guess, but I wonder where all the time went today. Let's see

07:00 - mail
11:00 - made some photocopies
13:00 - showing new GenI and gtester to Claire (eep! I let my satisfaction with myself slip... bad eric! Hubris alert BEEP BEEP BEEP! Pride is a dangerous thing)
15:00 - attended a talk by a friend (and ashamed to admit i fell asleep during)

And now leaving early. I blame the Fête de la Musique... amateur band playing the same Soundgarden cover OVER and OVER and OVER again right outside my apartment. Ugh.

buncha changes to geni

07:00 - 09:00: correcting some random lab-friend's paper (English). Graphics stuff, nothing to do with NLP. Interesting to see something completely different and well nigh incomprehensible. Have refined my paper correcting technique - page o general comments, add a page 0 to use as comments for page 1, all comments on back of previous page, number 'em.

09:00 - 19:00: geni-hacking with random assorted breaks (lunch, post lunch walkies, etc). Got the ViewTAG button fixed again (and worked in a very nice refactor, which put me in a very good mood). Added a mode for ignoring the literal constraints. Cleaned up some random GUI code (because I tried to add a checkbox, screwed it up and decided to refactor some more so that future additions to the GUI would involve less screwing up). Bref... that GUI code's looking a good bit better/more maintainable after this round of geni-hacking.

Thought a bit about adding soft timeouts, but kinda went "ugh", because I have to extend my monadic stack (eh... which isn't hard, I don't think) and count iterations and all that. Makes me tempted to go do something different, like implement the pickler combinators paper so that we can pickle the grammar. Why? Cuz it'll beat waiting 3 hours waiting for the grammar to compile, and it makes it possible to have a bunch of separate GenI processes instead of one big batch which could go kaput at any time. Grr... SelectTAG not completely obsolete, I guess, but I'm still happier with that code gone.

While looking for a standard pickler library, I stumbled upon this tree drawing paper, showing a nice tree drawing algorithm in functional style (using ML). Very nice and elegant... which might be good, because I'm starting to realise that Blobs doesn't do layout stuff by itself (does it?) and knowing that tree-drawing isn't actually all THAT hard is a relief.

gui / m-o-d / haskell meeting

Morning: lots of hacking on the Makefile-of-devastation. Actually was something tout bête, but I had added a stupid dependency somewhere and spent the whole morning trying to figure out why it was recompiling the grammar all of a sudden. Silly eric.

Early afternoon: meeting with Daniel on Haskell - I showed off the trac server and he taught me lots of great stuff about Haskell, namely phantom types, forall and way to avoid doing a whole bunch of (++) by doing the whole function composition thing. Sweet. It was fun to be able to really talk Haskell with somebody.

Afternoon: (actually bits of morning and early afternoon thrown in, but I'm simplifying here) Improvements to the GenI gui. Now we have the abilty to show XMG trace information and save them to a kind of clipboard. The idea is that Claire can then use this clipboard to paste new stuff into the input semantics. Was fighting a bit with layout, garr, and wasn't too happy to be doing GUI stuff. The ideal toolkit would just figure all the layout stuff by itself and default to something sane, but I imagine that is mad difficult to do:



Evening: helping Claire with a darcs issue (whew! false alarm, just scalability problems), and refactoring some GUI code to put ViewTAG functionality back in. Oh and "simple" has now be rechristened "simple-2p", for, like, clarity and shit.

discipline slipping!

Uh-oh! Had a couple of incidents of letting my discipline slip today.

Spent a little too much time on the #darcs channel discussing why people did not like darcs (new model hard to get used to, not widespread, buggy) and trying to promote the general idealistic viewpoints that other revision control systems like monotone are our friends, not our competitors. Also in the afternoon, made some revisions to the wikibook on Computer Programming, well, added some stuff to the discussion page anyway. Ugh, what's that? Two hours lost? Naughty eric, no more wikibooks at work for you!

As for the productive stuff, I implemented the literal constraints (didn't really get a chance to discuss them) over bits of Sunday and today. Showed that stuff to Claire.

Also fixed a really annoying GUI behaviour where I was sorting the semantics, which means that all the user's careful use of whitespace is thrown away. Now I just insert the raw test case contents, which makes things a lot more readable. Mostly an issue of grouping: for example, now you can have

like(e1)  agent(e1 x) patient(e1 e2) 
dance(e2) agent(e2 x)

instead of the slightly less comprehensible

agent(e1 x) agent(e2 x) like(e1) dance(e2) patient(e1 e2)

You'll just have to believe me on this one: me sorting your semantics gets very annoying when they are long

Otherwise, fixed a few more things in GenI (and also dabbled a bit in LexConverter). Sort of to underscore the usefulness of monads, like what we were saying in the Haskell workshop paper, I managed to change the error reporting behaviour by switching from the Either String monad to Either LexCombineError, where LexCombineError is just a datatype I made up. Much more readable. It's amazing how much difference a little summarisation makes.

starting selection

Just broke ground on the new literal-constraint selection code. First, upgraded the HaXmL code to use version 1.15 (the darcs version to be more precise), because Blobs wants it and I figure if I'm going to touch the XML stuff, I might as well get it upgraded while I'm at it. Wrote two miniscule patches to parse/produce the new "trace" field, which captures the XMG traces for use by GenI. Implementing the selection itself shouldn't be too hard. It's just a matter of checking, for each lexically selected item, if all the elements in that literal's constraints are members of the item's trace. But need more time. Will also need to hack the GUI so that it doesn't sort the semantic input (that is so confusing) and so that it preserves the original whitespace.

Oh and found a small bug in geniconvert (was calling writeFile over and over again instead of appendFile). Make my heart stop... all of a sudden everything stopped working. Whew! Good thing it was just something stupid like that.

gtester / geni / m-o-d hacking

Much more hacking today on gtester, geni and the Makefile of Devastation. Got the batch processing stuff working in GenI and integrated into gtester (yet another Python object) as well as the makefile. Now, when you run make run-gtester SUITE=C_Complementation, it takes care of everything: filtering the grammar, compiling it, translating it from XML to GenI, running the test suite. It even knows the difference between grammar hacking and benchmarking, the Makefile does.

gtester stuff

Gtester is also now a bit user friendlier still in that now you have one file that you edit to change all the little parameters: gtesterrc.py. Beats having to dig through all that source code. It also now has the ability to run without a baseline geni, which is useful when you're just doing grammar hacking and all you want to know is if things pass or if they fail. Made my first use of dynamic typing today: to test if a variable is set, I just use it as a boolean, something like:

if baseline:
 foo = baseline.attribute

There is a joyful messiness to this, knowing that someday, somewhere that piece of code is going to blow up in my face (actually, it did blow up in my face several times. I had to run gtester like 6 or 7 times before I caught all the run time errors), but eh... I still think I like the strongly typed way better.

case mbaseline of
 Nothing -> return ()
 Just baseline -> do stuffWithBaseline

Whatever... I don't really know what I'm saying. Point is that, this is just gtester we're talking about, so I don't care if it blows up. But no way am I letting dynamic typing near GenI. I'm much too sloppy a programmer for that. If I was some kind of programming super-genius or just REALLY REALLY REALLY REALLY REALLY rigourous in my coding, maybe, but the way I code, no.

geni stuff

Now that we have a working batching processing mode, I have finally removed all traces of the old XMG SelectTAG based code in GenI. This also forced me to get rid of one of my hacks for filtering the XMG grammars. Now have a much cleaner solution where I have a sort of mini-GenI whose only job is to lexical selection and print out all the families used. No more calling outside shell scripts making horrible abuses of regular expressions. I am telling you, this shit is SOOO much cleaner. I feel immensely relieved, 300 very embarassing lines of Haskell, shell, Perl, Makefile gone gone gone. And it's not just raw code count either, it's the fact that is was some of the worst code in the SemTAG toolkit (all my fault), one of those one-hack-leads-to-another situations.

The generation half of SemTAG is becoming more and more presentable by the day. Less and less horrible, in other words.

gtester and the makefile of doom

Spent this morning making gtester much user-friendlier and fleshing out the Makefile of doom to take over some of the stuff I was doing with messy shell scripts. The main improvement to gtester is that I now use the Python optparse module to get command line switches instead of ordered arguments. Makes it much easier for users to know what's going on, and more importantly, makes it easy for me to extend. Kinda glad for object-orientedness of Python, I guess. It wasn't very hard to add other kinds of GenI, like the basic GenI, and eventually batch GenI.

Some extensions to the Makefile of doom. Now it runs the haskell script that splits TAGML files into little itty bits and handles the conversion into GenI format. I wish I had this when Ulrike and Claire were working on GenI: that would have saved them a lot of trouble, saved me a lot of GUI hacking (which ended up being useless). Anyway... this too is much cleaner. You can feel it when after a few tries you finally got things right, and I suspect that the Makefile of doom is much much closer in the direction of getting things right than all my previous attempts to make GenI linguist-friendly (at least, on a day-to-day grammar-debugging cycle). The real test is what happens to Claire when I'm holiday, I guess... can she grammar-hack or does GenI slow her down?

Also tried to make geniserver work, but I realised that this solution is really fragile and not neccesary for the grammar-debugging cycle. What I really need is to reintroduce batch processing into the GenI code. Now I had taken that out once, but that was because I was trying to do a really fancy kind of batch processing with configuration file processing and all that. Naw, this time, it's just going to a simple straight-shot mapM on the entire test suite. Extra benefit of this is that I've finally gotten around to cleaning up the console code. Console code is not done yet, but should be ready by end of tomorrow morning.

Attended longish talk by JMP on the TLFI and all the neat things that the folks at ATILF are doing. I pestered him about making the TLFI file freely downloadable, but it turns out they can't because they have a contract signed with the previous editor from like way back when, and said copyright thingy will be still around for 60 or 70 years to come...

short day, report

It was a short day and I took a break in the early morning that I probably shouldn't have. Not that it's bad to take breaks of course, certainly important to switch tracks for a while and also to socialise a little bit, but the problem was taking it when I really didn't need one and when I was sorta on a roll. Nah, no breaks before 09:00 from now on, I think. Take advantage of the two most productive hours of my day.

Had a meeting with Carlos and Daniel. Carlos had the nice idea of getting us, two Haskell programmers working on completely different projects (but both with C contact) chatting with each other. Nice idea, lots of stuff for me to learn! Maybe after talking to him, I can actually understand fancy uses of the type system and be able to just roll stuff off my tongue like "phantom existential witness type" and "rank 2 polymorphism", because that stuff looks useful.

Wrote a small report for Claire on the paraphrase work that we are going to do. She said don't spend more than an hour on it, and I tried to listen to her, but it ended up taking me until 15:00. Not really that I spent the whole time working on it of course. There was the chat with Carlos and Daniel, a longish lunch in between, and sleepiness in the afternoon (does this mean eating less?) to slow me down. Nevertheless, I did get carried away and tried talking about stuff that it wasn't yet time to talk about. Ended up just deleting huge chunks of what I wrote and replacing it with a skeleton. Anyway, sent, done. Hope it's to the point.

Now working on making geni and gtester easier to install/run in the grammar-hacking cycle. I'm glad with the change I made this afternoon. Now gtester itself should be easier to install (just drop it into place and stick it in your path) and all the intelligence has been centralised to the Makefile-of-destiny in the TSNLP testbed directory. Much cleaner this way. Got to get geniserver working properly now. No way am I going to make Claire sit through 3 hours of grammar compilation!

sfgbrowser / selection

Spent today cooking up a quick and dirty browser for SFG networks, building off yesterday's hacking. Got some rough edges, but it more or less works. The idea is that you select the networks you want to see, and it shows you how they relate to each other. If you decide that you've selected too much, you can also choose to remove some of the ones you had chosen. You could select all of them, but that probably wouldn't be a good idea because the graph would be completely unreadable. Here's the screenshot:



This would be wxhaskell and graphviz at work.

Other than that, had a meeting with Claire. We're going to push in a slightly different direction for the selection stuff, something I'm much more comfortable with, techniques for restricting the lexical selection via input. Got a small little report to write up for her and some GenI hacking to do. And fast too because I'm on vacation starting end of next week.

quiet monday

I spent the whole day staring at SFG stuff. Couldn't really find any technical report stuff. There might a useful doc in the 1991 HPSG workshop proceedings somewhere if I could just dig that up, and also a book which I hope the interlibrary loan people can find. Otherwise, the rest of it is just KPML documentation. Guess it's a fair start. Have I done my job then? We'll see. Right now, I'm just trying to help us understand the resources a little better. Grepped all the SPL files to see what kind of target sentences we have, what kind of things we expect this Nigel grammar to be able to generate. Also started working on a small Haskell script to parse the systems, maybe plug it into Blobs one day. Might be easier to install than KPML... oh and speaking of which, am now in touch with JB, the primary author of it (i think). Yay for helpful people! Maybe we'll even get this sorted out.

one day wasted

Meh. I think this experience has dampened my enthusiasm for Lisp. It's not really Lisp's fault, I guess, but man is it ever a pain in the butt into install stuff. Lisp has this thing called ASDF, have no idea what it does wrt Cabal, but seems pretty nice and handy. You just symlink your stuff in and go. So that's ok, but the hard part is that everyone seems to write for Allegro (and say that they do ANSI Common Lisp, but just haven't tested it everywhere), and things just go all sorts of wrong when you try to install them with free stuff like Clisp and SBCL.

Spent a bit of time in the morning fighting just to get McClim... after an afternoon of prodding and tinkering, I seem to have gotten
1) nowhere with CLISP (i suspect that the CLX that the darwinports version comes with is no good for it)
2) address book and listener examples to display with SBLC, OpenMCL (i think) [don't forget to set DISPLAY in your shell), and Allegro (you have to fiddle with the authentication settings in X11.app)

Installing Loom and KPML is a whole other nightmare. Loom really does *not* like SBLC. Something or another not-Loom does not like OpenMCL, but I don't remember what, and now I'm having trouble getting KPML to compile on Allegro Personal Edition 8 (and allegro seems to be the "preferred" variant.... I am randomly commenting out bits of code like "finally" to make things go, without knowing why, and just sort of hoping it'll work)

I really should *not* be doing this. Claire said to go look for tech reporty stuff so that we can more easily make sense of the Nigel grammar we downloaded, not "go install KPML". I guess I figured it would be useful if we could actually browse the thing, but honestly, eric, grep and emacs probably do just a good a job for our needs as KPML would. Sort of an evil mix of procrastination and determination. Don't really feel like going bib-spelunking, and can't put down this stupid installation because I refuse to be defeated. Stupid stupid behaviour on my part.

Maybe this is one of those breaks my brain sometimes forces on me.

posters done

Heh... took me a week to do the T8-generation poster, and half a day to do the toolbox one. Of course, then again, I only did half the toolbox and am waiting on Yannick to write his bit (take your time, you have redaction to do!). But goes to show that I am starting to make my peace with Scribus. Being able to use the scrollwheel to control the spinboxes is nice. Also noticed that zooming in 1:1 really helps for the speed issues, which seems a little silly to me, but I guess I don't understand the problem for what's worth. Still some things which infuriate me about the user interface, especially when trying to manipulate fonts, but I guess after all, it is free software and I do tend to hold free software UIs to a lower standard than commercial ones. UI is very very hard work and not something that people working on their free time are likely to be hacking on.

So the conclusion me and Claire had come to from the T8-generation stuff is that we need more data! So this means holding off on the evaluation stuff until we can find a way to get data, ahem, such as by using a parser. Oh yeah, handed off my benchmarking report to Claire, so I guess with the posters done and the report written, I should be... uh... happy now? Less stressed? Feel like I have a little bit of that coveted freedom back?

Claire has me working on the selection stuff. We're going to be playing with the SFG stuff at last! Doing a literature review, trying to find out more about the English grammar we hadd downloaded from the generation bank last December. Looking for documentation, tech report type stuff. On the side, I am also trying to install KPML. Written in Common Lisp, fun! I've always had an ignorant admiration for LISP and LISP programmers. I say ignorant because my only exposure to LISP was Freshman Scheme, but Freshman Scheme was an experience which very strongly marked me and made me want to stick around as a programmer, hence the fondness for LISP. I suspect it would make a better sloppy language than Python, but this is probably just the ignorance talking.

Other activity is that I need to make gtester more portable and make sure the whole compilable-grammar thing runs on Claire's machine, and fast enough for a grammar-hacking-generation cycle. My vague idea is that I want to kill the proliferation of directories. I want everything to be run from the data directory, not gtester. Be much tighter that way, imo, no weird circular loops.

benchmarking, poster bis

Finished writing up benchmarking stuff in the morning and making some more analysis of the data. Not thinking too clearly about all this though. Think I need Claire's help to wade through all the numbers and figure out how to use them properly.

Killed some time in the afternoon getting set up with the thloria style file for theses at Loria. It's not an official style or anything, but I figure, eh, if the guy already did all the work figuring stuff out for me, why replicate it? Changed a bit of tex so that it would use utf8 instead of latin1. Seems to work ok. Never too early to start, I guess. I mean, I don't officially start writing until 2006-12, but (1) Claire says I shouldn't think in terms of a big Period-o-Suck-style write fest but really work in small bits, sort of a write early and write often thing (2) I kinda like the idea of me starting to write up... like... now. I mean, I clearly am not ready to do so, as I have no real results yet, and still have my head up my butt as far as the literature goes, but it sorta helps dampen the constant anxiety if I say to myself, here I am already starting. The reality though is that I'm just still doing reports for Claire, nothing more.

Did a bit more work on the poster. Am grateful for Latex Equation Editor's flexibilty. Just paste in arbitrary latex, like a tabular, and you're good to go. If only I could drag+drop onto Scribus on a Mac, though... that would simplify matters.

benchmarking / poster

Two main activities today:

benchmarking report

Still trying to synthesise the results of our T8 writing spree, basically merge the two articles together to get a coherent story both on the properties of our test suite (ambiguity, etc), and on the performance results (detailed stuff, lots of tables). Copy, paste, rearrange, rewrite. I shouldn't really work too hard on this because in any case, it's going to be completely shredded up and rearranged, or maybe even obsoleted, but somehow overall just trying to write this in terms of how this would fit into an actual thesis. For example, in the article, we tend to have something like Tech-Results, Tech-Results, Tech-Results layout, which makes sense because we don't have much room and no background. But presumably, in a thesis, I want to explain all the techniques, fit them all in, have something like Tech, Tech, Tech, Results, Results, Results. And of course doing that just screws up all the transitions and crap (not to mention all the second-guessing in rearranging all this, is this really a good idea? am I just wasting time?)

root node filter

Part of the benchmarking stuff. I'm starting to get interested in this question of root node filtering. Want to poke a little bit deeper into this question of why root node filter seems to help proportionally less when the problems get bigger (although, absolutely more). Our two hypotheses are basically that (1) the root node filter only wipes away things that are sentences (by definition) whereas in reality, those only make up a very small subset of the lexical items. This is very test suite and grammar dependent of course, so we need to be careful, and at least, prepare a basic test that would let us confirm this (2) the root node filter is redundant with the subst node filter. Stuff that would be filtered by the root node stuff should already be washed away by the subst node stuff.

Since many performance phenomena are contingent to the grammar, it's pretty part to make super-firm conclusions about anything, or have explanations that really hold water. Maybe a good methodology when putting these reports together would be to write some simple recipes for testing hypotheses out, along with guidelines on how to interpret the data, i.e. experiment plans, but super-easy to run and interpret. Eric, do X, with parameters Foo and Bar. If Z goes up, be happy. If Z goes down, be puzzled. That kind of thing. Sort of regression testing your science. Every time you change the grammar, you re-run your experiments to make sure your explanations still hold water.

the poster...

Sigh. One trick I'm starting to use now is to remove all temptations to format things. Just have plain white text boxes, simplify the crap out of it all, and only once I have my content, and have a rough notion of places and sizes, do I worry about putting frames around things.

GenI on Dapper Drake

Oh, and got GenI installed on the latest Ubuntu today. It's a bit annoying in that you have to pass in a linker flag to grab an OpenGL version of the wxgtk library (I don't even use OpenGL!), but it works. Also updated the wiki with a simplified installation procedure standardising on my Unicode patch.

a slow slow day

Maybe this was my brain telling me I needed a rest, but I don't think I got very much done today. Took many breaks, chatted with people (like real life people) a bit.

Generate some numbers for Claire: comparing the number of foot nodes (nodes, and trees) before and after polarity filtering. Trying to synthesise all our T8 stuff into my benchmarking report for Wednesday.

Improved gtester's reports functionality some more so that we can now perform comparisons along multiple X axes. This means that we can crank out more information without having to keep editing the Python code all the time. I really need to sit down one day and figure out how to make gtester usable, I mean, something that somebody other than eric can run and tweak. Perhaps in this we find the wisdom of the Yannickian semantic construction graphical interface. Well, I'm not willing to go quite that far yet, but it would be nice to be able to do this with configuration files instead of Python hacking, for example.

Ahh Python... I really can't tell if I enjoy the experience or not. I'm definitely more comfortable in the pure functional world, I like my type checking and everything... but sometimes programming in a sloppy language is fun too. Just generally being naughty. Slapping things together without caring if they are correct, just as long as they do something that... eh... looks right. Probably costs more in the long run, though.

Starting working on the T8-generation poster. Man, working on posters SUCKS. I can't describe how boring it is, nor can I explain it, but there is something stupidly painstaking about the process, which doesn't make sense, right, because what it really boils down to to is thinking real hard about how to convey the information in a easy-to-digest yet sufficiently thorough way. It is a thinking man's exercise, not one I really have the right to call boring.

Part of it is that Scribus was dog-slow. I miss Omnigraffle. I mean, rah rah free software and everything, but night and day. Imagine software for which you don't really need a manual. You just grab it and go... but not in the crippled Microsofty-wizardy way but really use the software to express exactly what you had in mind. Omnigraffle is like that. Brain, mouse, diagram. Scribus, eh... scribus is getting there, but it suffers from the typical airplane cockit user interface of free and open source software. Buttons, lots of them! All shiny! And stupid little annoyances that might not be Scribus's fault, like the text window that stays in the background even though you select edit text, so that you have to use your manager to bring it back up. Oh well. Scribus it is, though. Working with people and all that. Plus part of me knows that a lot of things is just a matter of getting used to the software. But *whine*, that's the difference! With Mac software, you don't "get used" to the software, you sort of grow into it, discover it over time. I am really looking forward to the day we in the free software community get this kind of stuff right

t8 done

Ouf! Both t8 papers done. Claire did most of the heavy lifting in the generation one, of course. I was just her numbers man. Rewrote some stuff in the toolbox one because I had made an abrupt non-transition.

Back to normal, I guess. Time to focus on the real thesis.

yet another flurry of data

One good thing about today is that I'm starting to learn a couple of tricks for not getting myself confused under the profusion of test cases and comparisons. It really helps to write yourself a set of orders, and give short names to things that make sense for you, things like DS-1 DS-2, etc (which have nothing to do with each other). Also did a bit of hackery to save time, some shell scripts to merge test cases together, a wee ifdef to disable XMG recompilation when clearly c'est pas la peine. I just wish a lot of what I did was more automated. It just means gtester isn't configurable enough yet. Things I need are

• gtesterrc type files that I can use to save configurations, probably containing arbitrary Python code (sometimes using a sloppy language like LISP or Python can be fun!)
• some mechanism for associating a configuration with a test suite, so that I can just re-run stuff

I'm really really glad for Carlos or Carlos's students' idea of pickling all the test configurations so that you could just rerun the report generations part without doing the tests all over again. That really made my life a lot easier.

Oh, and uh, hacked up gtester a bit more so that the latex is easier to copy-and-paste into our final reports. We're getting closer. Hectic hectic.

T8 blur

It's kinda worrisome how the days and hours are starting to blur together.

Edit 22:03Just had a useful engineering idea while walking home from the tram stop. When GenI timesout, we kill the thread and lose all statistical data. How about this instead: how about a simple iterations limit? Then we can do this in a purely functional way, and dump out all the statistics we have plus any results before politely commiting suicide (the process, not the hacker)

one-phase bugs

This morning I sat down with the one-phase algorithm until I figured out what had gone wrong. Sure enough, it was something to do with top and bottom unification, namely that when you're doing the regular old one-phase algorithm, you cannot do top and bottom unification on the fly. You have to do it at the very end, because in one-phase land, you don't have any guarantees really about when a node is going to be seen. Uhh... something like that. Look, I made a nice clean schema showing how we manage to block "mechant le chat noir" and not "le mechant chat noir" using the old top and bottom minus trick. (Dear future Eric: please note that this is filed under GenI in the cabinet and will be mad useful for debugging adjunction) And the thing about this is that if you do your top and bottom unification on the fly, that mechanism breaks. Shit, I can't explain it any longer, but the point is that in two-phase, on-the-fly adjunction is equivalent to doing it at the very end, but it doesn't work that way in one-phase.

Anyway, good thing I had geni under revision control. Just unpulled the relevant patches in a mirror repository and copied my tb unification code back in. Am also VERY glad we have the dispatch filter combinators, because it was really easy to just to reinsert the top/bot unification code back into the filter sequence. The new top/bot code is somewhat simplified since I don't try to extract the results (all I want to know is if tb unification suceeds). I also had to think a bit harder about it because of the fancy new behaviour where we only propagate node names in the derived trees. Store a list of TagSite with pending top/bot unification.

T8+generation test suites

I don't really know what I did all afternoon. I vaguely recall that it was work, and it had something to do with test suites, but I don't know why it took such a long time (well, running the suites took a long time, fair enough, but what was I doing in parallel?) In my haze, I see me writing some Haskell code, trying to think up ways to speed up XML processing (I thought maybe HaXmL might have a space leak, so I tried putting together some small script to extract blocks of N entries at a time).

Also rewrote my performance stuff on the T8-toolbox paper, and I think it is really a lot less confusing now. Hope Claire and Yannick will like it.

Data. Just cranking out a lot of data.

I am le tired.

new desk and hw done

hw done

The Haskell Workshop paper is sent! Carlos spent some more time (this is really something isn't it, FIVE iterations, who has time to sit through five iterations with you?) cleaning up my English. Simple rules that I tell all the francophones whose English I proofread: Make the sentences short and simple. English speakers can't handle big sentences with complicated sentence structure. We like things in nice straight lines, thank-you very much. No nesting levels of N, no weird twister pointers, just get your point straight across. Should practice what I preach, eh? Anyway, Haskell workshop paper is now sent, so I guess I should feel somewhat relieved.

new desk

I got me a new desk! How exciting. I took advantage of our office-wide shuffle -- everyone's moving except little old b224 -- to ask for a new table. My old table had this really evil jaggedy metal bar that gave me a poke everytime I move my knee or something. Plus it was really unstable and shaky, which really sucks when you have one of those fancy imac things where the hard drive is right there, packed behind your monitor, whirring away, and not safely on the floor. Very pleased. No more pokes in the leg. Maybe this will improve the lifespan of my trousers as well.

bug in one phase

While trying to produce data for the T8 generation paper, I found out that my profiling madness had introduced a bug into the one phase algorithm. I was kinda expecting something like this would happen. I bet it has something to do with top and bottom unification. I keep flipping between undergeneration and overgeneration trying to fix it. Tweak tweak, bang bang.