eventful day

Man, what an eventful day.

arrows

Made some progress on the arrows wikibook last night. I understand what the Swierstra and Duponcheel parser does now. Kinda funny, I keep calling it S+W when I should be saying S+D. Think I must be getting confused by the 'w' in Swierstra.

the Useful Thought

I had a Useful Thought while eating my Weetabix this morning: if I understand correctly, which I probably don't, the usefulness of arrows come from the fact that they helpful avoid to needlessly expect sequence conditions. That is, when two calculations do not depend on each other, arrows help you to avoid fooling your computer into thinking they do. As a result of this, you get to avoid all sorts of nasty space and time leaks. Or maybe I'm completely missing the point here, maybe it's just because you can do tricks like the Swierstra and Duponcheel static parser thingy... ugh... I hate this hazy understanding of stuff.

Anyway, the Useful Thought is that if avoiding stupid sequencing is important, then the way for me to exploit arrows would consist of finding computations that are inherently parallel... but ugh... what? This is bottom up parsing, items depend on other items being generated. There is one place where we can have some parallelism, though: everytime you select an agenda item, you try to combine it with all of the items in the chart. Each of these comparisons can be done in parallel, yet in the GenI code, they are sequenced by a mapM. Could I do something useful like the StaticParser trick to avoid a bunch of useless computation, or to help my computer know when it can garbage collect something? I dunno. What's kind of amusing about this, again if I understand correctly, is that arrows help your program behave like you'd naively expect it do - they help you get down to the theoretical complexity.

Anyway, be careful with the paragraph above. I might also be full of shit.

the mammoth patch

Early this afternoon, I finished off my mammoth patch from yesterday. I guess I'm coming to terms with the added complexity. Part of it is that I found a less ugly way of doing something I was trying to do last night. The part is that I realised that some of this complicatedness can not be helped, because in some places, I'm adding more sophisticated behaviour such as using bit vectors to represent the semantics.

dependency graph

Look at the difference between these dependency graphs.
Before:

After:

See what a difference a little transitive reduction makes?

kowey-generic

Made some updates to my kowey-generic scripts because my one user (besides myself) found some bugs and undesirable behaviours. Namely, the improved Makefile was stopping whenever BiBTeX fails. For me, this behaviour is desirable, but Yannick managed to convince me that it might not be for everybody else. So my solution now is to make it parameterisable. By default, if BiBTeX fails, we move on and continue latex'ing, but if you flip a switch in the Makefile, it dies like it "should". I will migrate them to the trac server one day, as soon as I upgrade to Lele's nice new plugin.

trac vs msnbot

My poor little trac server got visited by msnbot today. I was wondering why the machine was so slow, so I did a top, and holy crap, look at all those darcs processes owned by www-data! Did a netstat -a and and nslookup and understood. I had to take apache down just to be able to use the machine. Hopefully, this robots file will make life easier. Didn't get much else done. Was trying to administer the trac box, but I think it's being firewalled in, so I can't do something like apt-get update. Grr.

holiday eric!

Anyway, I won't be in until Wednesday. Going to FOSDEM this weekend! Plus hanging around Brussels an extra day and then resting up one day in Nancy. Hopefully my day-o-rest will consist of me being good and doing all those chores I've been telling myself to do for the past two months.

boooorrring

This is the kind of cleaning up that I hate to do, because I'm never sure that I'm really doing the right thing. I'm trying to push a lot of functionality out of my representation of TAG trees and into my representation of chart items, which makes my code more modular, but also makes it larger and more complicated. It's usually a symptom of something being wrong when you clean things up by making them more complicated. Sigh. Spent the afternoon pushing code around, making things break all over the place. Good thing I've got the type-checker watching my back... or else I'd be sitting here recompiling GenI, testing it, and watching it fall apart. Yeah, yeah, so I rely on type-checking because I'm too sloppy to think of everything at once, and because I've not got a million unit tests protecting my code. Sue me. Garr... all of this is making me grouchy. Hmm... got here around 14h30 today, so leaving now would make me... what do you call it... a slacker?

I dunno. I was spending this morning writing the wikibook on Haskell arrows. I had figured out a visual representation of arrows, so was just uploading diagrams and explanations online. Does that count as work? I mean, I'm still somewhere trying to figure out if there's any use of arrows to sequence stuff in ways my state monad can't so that I avoid doing stupid computation, things which make things longer than would be naively expected. So learning about arrows is useful. But is writing the wikibook really necessary? Do you really have to make diagrams for them? I don't know. I usually don't understand things until I actually try to explain them myself, so that's sort of work, right?

not so productive today either

I spent today working on some pretty unimportant stuff: tidying and rearranging the simple builder stuff, fixing the latex (bad references, etc). Big patches that don't really do very much.

It doesn't really seem kosher to be making patches for trivial stuff like stripping off the trailing line endings, but I guess it doesn't really matter that much, as I'm the only developer. I just got sick of seeing those ugly [_$_] in the darcs whatsnew output to indicate a newline following some trailing whitespace. The way I figure, it's probably useful for the lines of my source code to be in some canonical form so that when other people make changes... eh... I don't know, I actually didn't think this through that far. I just didn't like the [_$_], and no, telling darcs not to print them would just be 'cheating'. Argh. Random obsessiveness.

Anyway, this was a good time to do this, I figured, since I was already moving lots of code around. Ideally, the diff algorithm would be magical enough to understand when you're just moving hunks around instead of adding/removing them, but we're not quite there yet. I hope the result of all this moving stuff around is something that reads like a document and not just an incomprehensible pile of code.

haskell stuff

Spent a very sizable chunk of the afternoon learning about the difference between strict and lazy pattern matching. Just a friend who was working on something vaguely related had asked me a question, and I didn't have the slightest clue how to respond. Learned a lot in the process, so I'm quite pleased with that. Consider the following code:

> foo x =
>   case x of
>   ~Nothing -> "n" 
>   Just y  ->  "j " ++ y

What happens when you run foo (Just "hello")? You get "n"! It's because we demanded lazy matching on the Nothing, so it automatically suceeded without looking into the value.

Another thing I learned from this is that as-patterns in haskell are just syntactic sugar. Very useful sugar, mind you. The following two fibinonacci are equivalent:

> fib@(1:tfib)    = 1 : 1 : [ a+b | (a,b) <- zip fib tfib ]
> fib2 = let (1:tfib) = 1 : 1 : [ a+b | (a,b) <- zip (1:tfib) tfib ] in (1:tfib)

Yay, random bursts of learning! Oh, and I'm slowly starting to get a better grasp of arrows... not all the way there, but at least now I understand what makes S+W's fancy new parser work. A very useful trick is to think about the implementation for arrows for functions (->).

fantastically unproductive

Meh... installation pains with the trac server. It turns out that the machine can't talk to the name server from the DMZ, so all ssh connections have a 30 second delay as the machine tries to look something up and times out. I suppose I could ask sshd not to verify something or another, seeing as to how ssh seems to default to not caring if it fails anyway.

Spent an hour or so throwing together some generic documentation for using darcs with the trac server. Guess the test project is useful for something after all - nice sandbox for playing with the wiki. Eventually though, I want to integrate this with my adminDT repository, make it so that when people follow the TracOnDarcs howto on the darcs wiki, they also get this default documentation for free.

Also spent some time trying puzzling with Yannick over some repository issues he was having. Turns out to be some corruption introduced $deity-knows-when, probably by means of unison, COMPOUNDED with a simple darcs conflic COMPOUNDED with the slowness of debugging because of all this ssh nonsense.

Another thing that ate a bit of time: shapr mentioned on #haskell an arrows tutorial he wrote for the monad reader, so I jumped on the opportunity to ask him permission to import it into the wikibooks module. Spent a good chunk of time merging it in. Still a big mess, though.

Other than that, Research, research... what research? I did most of my research today lying in bed this morning not wanting to get up, and staring out the window in the tram. I was thinking that I'm going to need to make sure I'm comparing the same stuff between simple and CKY. This week, I need to import some of the nice CKY features back into Simple and do a little bit of factorisation. For example, one thing CKY does and Simple doesn't (but could) is use a bit vector representation for the semantics. Simple is just stupid and does set operations using the Haskell List library. Another thing I was thinking is that it would be nice to introduce a data structure like SimpleChartItem instead of just TagElem. This way, I can clean TagElem up, make everything nice and tidy.

Oh, mini-idea I had: one useful thing to count would be what percentage of chart items (in each) actually gets used in results. Need to think very hard about how I'm doing all this. I don't want my statistical stuff to interfere with Haskell's laziness. I don't want to be forcibly inspecting things which don't even have the business being created in the first place. But having measure of the overall usefulness of the chart items might be a good measure of efficiency.

Slowly, very slowly starting to understand the arrows stuff. Not there yet, still mostly puzzling over it, still made up largely of confusion, but at least now I'm starting to understand this idea of static computation, independent of input. Could this be useful for GenI?

ugh...

I'm so not keeping a healthy schedule here, need to shift back to reasonable schedule.

Right, today, we finally moved our trac server into the DMZ... it's open! it's open! You can see it at http://trac.loria.fr. For example, you should now be able to see the GenI homepage. I think i'm not going to sweat the whole ssh from the outside deal. I figure that darcs send is good enough for that. We'll have to see how it goes.

Ahh, other news: I fixed some more bugs in the promettre grammar, bugs that were causing it to overgenerate. Not really a linguistically satistifying fix, but good enough for me. Also fixed a CKY issue which was causing its own brand of overgeneration. Heh, now I can just show you guys the changeset instead of talking about it.

stupid eric tricks

Sigh... this is why I will never be a decent sysadmin: Eric (logged in via ssh): oh, looks like I need to restart this ssh daemon.

sudo killall sshd
Connection terminated

Stupid stupid stupid! Furthermore, the machine was locked away in a server room, and nobody was around to give me the key for it. Doh... and this is even talking about the time I tried to change the root username. I thought that's what the sys group wanted me to do, just sorta hazy like that sometimes. That turns out to be a fantastically bad idea. So much stuff depends on the 'root' user by name. Well... actually, it probably wasn't such a big deal. It's just that the new root user's name no longer matches the stuff in /etc/shadow, which means, surprise! You can't sudo and su anymore! Yay! I had to spend some time with a paperclip and the Debian installer to get my old /etc/passwd file back. Sigh...

Other than that, did a bit of refactoring between the CKY and the Simple generators. Was really just trying to add polarities to CKY. Turns out I had already done it, and just forgotten about it. Don't remember why, but I also wanted to simplify some stupid stuff I did in the Polarities during my DEA. Haskell is lazy, right? Everything works with pointers and thunks and all that, so it doesn't matter how big the data structure you're passing around is, because (1) it's just a pointer anyway and (2) it may not even exist for all you know. To deal with some performance issues, I had previously tried to copy my data into "lightweight" data structures so that they took less space in memory. But that's just stupid, just a bunch of pointless overhead, and it introduces complexity into the code. So, took a moment to tidy all that away.

Also cleaned up the handwritten grammar. There are a lot of stupid little copy and pasty bugs in the toy grammars. Sort of underscores the need for a good metagrammar system. Writing grammars by hand is bad news! Anyway, this gave me a chance to revive the old nl and grep trick... ahh nl...

nl -b a foo | grep type:subst | sed some horrible regexp | sort 

Nice way to search for crap in a file.

cky automaton merging

That a took a surprising amount of rewriting, the new automaton merging code, mostly to figure out what the heck I'm supposed to do exactly. But there was also a lot of copy and pasting, just transfering the old code over and reworking it to fit the new construct-the-automaton-after-derivation paradigm. It was one of those tasks where you spent a lot of time trying to figure out what the types are, but once you do that, and you compile it, it just works the first time. Well, fine, the second time. I had to invert a little piece of logic. But always a nice feeling when the thing actually does what it's supposed to. So here is my méchant chat with the automata merged like they should be.



So what's up next. Got a whole butt-load of work to do.

1. Review papers for student session (i so am not smart enough for *that* job)
2. Implement timeouts so that I can "let" GenI infinite loop while running the full TSNLP test suite
3. Figure out the best way to evaluate this stuff.
4. Implement automaton minimisation to squish my unioned automata back to size.

my first Idea?

Oh my! I think I just had my first Idea! Maybe I'm starting to do my job of researcher-in-training? All thanks to my shower, too!

So here's the deal: lately, I've been interested in learning about arrows, a generalisation of monads. As with the monad transformers, I am trying to learn by teaching, that is, by building the wikibooks module on arrows. Now, I still have no idea whatsoever what an arrow is, but while I was reading John Hughes's paper Generalising Monads to Arrows on it (JH is the guy who invented arrows), he says that the reason he came up with arrows is because

1. Monadic combinator parsing libraries kick ass
2. But they have an enormous space leak in certain cases, some things which can't be garbage collected, but should
3. Swierstra and Duponcheel came up with a really neat way to fix it (I still don't understand the neat way)
4. But you can't represent the neat way in monads

JH thus invented arrows to do things the Swierstra and Duponcheel way, and still keep all nice advantages of monads (it keeps your code tidy, among other things). Alarm bell! Alarm bell! GenI has an important space leak. GenI uses a State monad, like parser combinator libraries do. Is there anything about GenI's leak in common with parsing libaries? Could arrows help you to build natural language parsers and generators in a space-efficient way, whilst keeping all the things that make functional programming fun?

BTW, you could also say that Carlos gave me this idea. Why? It's because of the whole idea of using monad transformers to cleanly insert Statistics into Hylores without having to rewrite a whole bunch of code. It's because of the way he had to implement monads himself for his own use when Haskell was still young and things were only just being published. It's this whole recurring idea of using what the functional programming community comes up with to solve natural language engineering problems. Simply put, if learning about monad transformers turns out to be super useful for building HyLoRes and GenI, maybe learning about arrows would be as well.

I think I have a research direction. It might lead nowhere, but it'll be fun, and I'll learn stuff. Ok, Self-doubt time. This just sounds like more programming. Oooh, so now you apply arrows to do parsing, like it's not been done before. So what? Shrug. We'll have to see.

Almost there! (... define almost)

Ended up staying late last night.

CKY GenI is very close to working now! Last night I fixed the bug that was making perfectly good results go away when you extend the grammar. Was a simple, stupid issue: I was letting items merge even if they didn't belong to the same tree.

And with that fix, GenI produces results for all the eyeball regressions... but not the right stuff. I was getting stuff like (le mechant chat chasser le souris) noir instead of (le mechant chat noir) chasser le souris) (parentheses added by me to clarify why this is wrong). It turns out I was building my automata all wrong. For example, I was propagating automata only when I was directly above the foot node, and not on the spine as I should have. Pretty much fixed that last night. This afternoon, I fixed another issue where I was not taking into account which side of the spine I perform operations on. For example, if you substitute into the right of the spine, you want to propogate into the automaton that comes after the hole and not before it.

And now I get the rigt results! I think. That is, for each toy grammar I have, it seems to proudce the right output. But I'll need to do more extensive testing on it though. I plan to borrow the test suite from darcs source code to wring GenI out a little better. Good thing it's GPL, like GenI.

This afternoon, I also implemented the dumb version of automaton merging, dumb in that it makes zero attempt to minimise the automata, so you end up getting a whole bunch of redundant paths. I'll implement minimisation, later, but now a more pressing task is to fix a fundamental automaton construction bug: Say you got two items K1 and K2 (kids 1 and 2). They combine into parent P1 with a nice automaton. Fine, but what happens if
you later produce a new item K3 which is equivalent to K2? P1 doesn't know about the new item and its automaton is incomplete! It's not automaton construction itself, but the code around it. Stuff like this makes me glad I'm doing functional programming, though. It seems like the fix is not to return automata, but functions that construct automata. It will be a bit like the way I construct derivation trees. You don't actually want to construct the automaton for P1, because you don't know what the sub-automata for K1 and K2 are until the very end. If you return an automaton constructing function (a continuation?) instead of the automaton itself, then you can access the very latest version of K2.

But that's tomorrow's fight. Oh yeah, the other little thing I did was to clean up the literate Haskell in my CKY builder. It's a bit more readable now, I think. The basic model is to use literate Haskell to talk about Why and How issues, and use Haddock-style comments to document API-like stuff. There's a few places where I was overusing literate programming in the past, and I'll need to work on eliminating those. But overall, I still believe that that is is a very useful way approach to documenting your code. You just have to figure out what it's best suited for.

Anyway, here's the little pseudo-UML thingy I threw into the literate source code to explain the fancy new architecture:

huh?

I had to wake up at normal-people time today. Meeting at 9h30 to discuss the new livret d'accuiel de nouveau doctorant, sort of a way to participate in the "life" of this lab. My contribution was broken-record-like... just to suggest that we use the wiki to build our document. People actually agreed to it. Huh.

I'm hoping the early wake-up time means I can start drifting back into a reasonable schedule. This going to bed at 3 thing is not a very good idea.

Got one stupid little thing done: a wee results gui for CKY mode, so that you're not forced to use the debugger all the time. Minor detail, but it's nice to have this checked of the list. Other than that, still puzzling over my CKY problem. Basic issue is that you just have the n0Vn1 tree, then it generates fine. But if you add other trees, without removing n0Vn1, then it stops finding results.

Don't know why yet, but fixed a subsitution bug along way: I was allowing things to substitute even if they weren't no-adjunction. That was creating lots and lots of derivations for a stupid little thing. Wouldn't have been able to figure this out without the help of the graphical debugger, I don't think.

There is a second, outstanding issue though... I notice that for some reason, certain adjunctions are being permitted, which really really should not fly. Why is it adjoining "le/def(a) into souris/mouse(b)"? It also adjoins the right thing "le/def(b)", but that first adjunction there... that's bad news.

Oh yeah, and I've got a handy little visualisation trick for the debugger. Because we are now working with nodes instead of entire trees, there is a whole messload of chart items. To help me pick out the most interesting ones, I now prefix the label for the ones which correspond to root nodes where adjunction is complete with ">". Makes them easier to pick out in the item selection gui. The other guys just get boring old gorn addresses:



Edit 18h15: Fixed the weirdo adjunction bug. It actually had nothing to do with adjunction; it was just the kidsToParentRule which wasn't filtering strictly enough. It was using idname to filter trees, which is stupid because "le/det(a)" and "le/det(b)" are both "det" trees. Switching to tidnum makes this unique. I should probably just use (==) and abstract away from this. Yay! Graphical debugger wins again!

more test harness tweaks / cky again

Made the gtester harness a little bit more robust. I still haven't figured out why GenI crashes randomly, (c.f this last post) , but the least I can do is handle it a little more gracefully. So now, if GenI dies, the gtester just sort of shrugs and says so in the report.

I'm also starting to dabble with CKY again. I figure that since we're giving up the COLING stuff, I might as well try to get this working so that the next papers won't be talking about such ancient non-tabulation technology. As if cky was state-of-the-art or something. Sigh. Progress in small steps. I fixed a stupid adjunction bug where I was set the bottom feature of the adjunction site to the unified foot bottom + site bottom. Solution was to set the bottom feature to empty list, and now, with that little fix:

le méchant chat noir chasser le souris!



IT PARSES! IT PARSES! MWUHAHAHAH!

But... but.... I don't get it. I had to simplify the grammar. How come when i put those other trees back, it no longer parses? They don't interact! What's the deal with this?

burst of euphoria

Get the feeling that all this stuff I've been trying to do in parallel is finally converging. We are so close to having a full-fledged test harness together. Today I got gtester to make use of the statistical data outputted by GenI. I also managed to get the timing code to work. So now we can use gtester to measure some serious stuff. We're about to get empirical on somebody's butt. One other gratifying thing is that I think I was right about making the metrics code flexible, using strings instead of hard coded types. Extending GenI to keep track of a substitutions and adjunctions metrics was super easy. Things seem to be going along smoothly then. Now we'll need to sit down and figure out what the right stuff to measure is.

Other news: Jorge is a Brazillian PhD student visiting our team, also working on generation. Looks like we'll be talking to each other once a week, sharing the work we're doing. Very informal, undirected. Quite happy about this. Low pressure scientific communication.

Attended nice little talk about making better use of language model in stastical speech recognition. Basic idea is to distinguish between stuff which is simply unknown in the corpus, and stuff which is outright impossible.

Completed the upgrade of our local wiki. That was a pain in the butt. I had to insert some php include paths into index.php before it would work. And then there were a couple of things which had changed from 1.4 to 1.5, so it took a while to figure all that out. Also had to copy my images over from the old wiki, and remember to make the directory writable. Glad that's over with. Overall, not too painful experience. Had to ask sysadmins for some rights here and there so that the installer could create some new tables, but overall, not too bad.

Doesn't look like we'll have time for a COLING paper. But I've got enough stuff on my plate. Now need to bust ass, start measuring some shit, get something useful to talk about in April.

python duh!

Been working on integrating Geni's statistical output into gtester. Mostly done. It now parses the statistical stuff, and the only thing I need to do now is make it print it out in the report. Must find a nice way to do it though. Not a very productive day. One thing I discovered is the reason I had been having so much trouble with the Python: I had bookmarked the standard library, but had completely neglected the Python tutorial. Was trying to learn the language by reading its library. Silly me! The python tutorial even has a big link that says "Start here!" Anyway, so now I'm hoping gtester work will progress a lot faster. No matter thinking up the right thing to type into google to find a good reference. Guess it's the difference between functional languages and the others (this is Seb thinking). In a functional language, there is practically syntax. Learning the language consists mostly of learning its library. With something like Perl or Python, what you need is an actual tutorial.

Other than that, been working on upgrading the wiki at work, cuz one of the users wants to receive mail notification when his favourite pages change. Slow going. Think we're not running the right version of PHP chez nous. Gah.

Ok, I'm having trouble focusing my eyes. I think this means it's time to go home.

GenI now outputs statistical data again! I had removed all of this functionality in the Great Rearchitecturing of November 2005. Now that I have Carlos's monad transformer stuff, I can put it all back in with much much cleaner code. I guess this is my insight for the day: monads let you do absurdly complicated things in relatively simple ways. Well, not so much simple, but clean. No erician spaghetti. Or less of it. One thing I'm doing differently than Carlos and his thésards, though, is that I'm representing the statistics with string keys, instead of making seperate constructors for them. This means I get less type-checking, but maybe maybe just a little more flexibility and a cleaner-seperation of code. The Statistics stuff tries to be general and application agnostic.

Right, so I've got graphical output. Would put a screenshot up, but blogger is not responding. Also got text output into a file or to stdout, depending on what options you pass to GenI. Next thing I need to do is make use of this for gtester, and to print out timing data. Then we're set to evaluate performance!

One thing I'm dreading: Claire thinks we should be thinking of papers to submit to TAG+8 and INLG. I agree with her, but there's this tiny little problem. I don't have crap to say. Not even intelligent questions to ask people who've been thinking about stuff longer than I have (i.e. the big people), nothing to share with my fellow students. Nothing. Uhm. My name is Eric. I like Haskell. You can use monad transformers to cleanly separate your performance data tracking from the business end of your code. Implementing CKY was hard, but fun in a perverse kind of way.

No, that's not going to cut it. I don't think I'd even have enough stuff to say for a Haskell applications workshop, for example. Sure, I'm sure she'll have ideas, directions, etc. But dangit, I'm supposed to be working on this stuff too. So neglient of me. So this week, part of my job is to stop for a moment and think really hard about papers. Put another way, I need to actually work on the research part of my PhD and stop coding so much.

StatisticsState

I think I've now more or less got a grip on the monad transformer stuff, so I've started integrating Carlos's StatisticsState into my code. Globally, the amount of changes I have to make to my code is small. For example, I had to augment the common architecture stuff to track the stastistics in addition to the regular state ticket, as well as to return stuff in the combined monad:

Before:

{ init     :: Input -> pa -> st
, step     :: State st ()
, stepAll  :: State st ()
, run      :: Input -> pa -> st

After:

{ init     :: Input -> pa -> (st, Statistics)
, step     :: BuilderState st ()
, stepAll  :: BuilderState st ()
, run      :: Input -> pa -> (st, Statistics)
}

type BuilderState s a = StateT s (State Statistics) a

There are also four or five places that I had to slightly uglify the initialisation and wrapper code for the builder:

Before:

-- in run
evalState stepAll $ init input2 config

-- in initBuilder
in execState (mapM dispatchNew cands) initS

After:

-- in run
let (iSt, iStats) = init input2 config
     in runState (execStateT stepAll iSt) iStats

-- in initBuilder
in runState (execStateT (mapM dispatchNew cands) initS) emptyStats

But globally, it's a drop-in affair.

monad transformers

I'm still working on understanding monad transformers. Had an important insight in the shower today, kind of obvious if you know what monad transformers are, but not if you're completely lost like I was. This insight was that monad transformers are just cousins of their non-transformed variants. Monad transformers are monads too. Nothing simpler. As I go along, I'm trying to write a wikibook module which is perilously blurring the line between work and leisure. I think pretty much the last three things I need to get before I can work with them are

1. how the StateT transformer works
2. what the essence of monad transformer is - basically just threading your inner monad through
3. what lift is for and how to use it

Also, my diagram from last night was a little bit wrong. Here's an updated version:

ugh...

Feeling a little under the weather here. Been this way for a week now.

Tried to integrate Carlos's StatisticalState monad into my code, but I then I realised to my dismay that it's not just a drop-in replacement for the State monad. I'll have to do some monad transformer stuff to get what I want. Which means I need to bite the bullet. Sit down and try to work out what the heck monad transformers are and how I go about using them. Just a simple question of combining two monads together.

So most of the day spent trying to figure this out. I'm using the wikibooks talk page to jot my notes down, hopefully build this into a tutorial one day. Concentrate on understanding MaybeT. Sure it'll all click and make sense eventually. Right now, just sort of wading around. I think the astronauts in space suits metaphor is not going to do the job here, because with monad transformers, it seems like you've got three space suits to deal with: the wrapper newtype space suit which just encapsulates all the magic, the inner monad and the innermost monad which you are trying to thread through the inner monad. Need to find a way to make this less confusing. Make this easier to digest.

Edit 19h50: after spending an hour or so playing with Omnigraffle, I think I finally understand the ListT transformer! Or at least the bind operator for that monad. Yippee!

sematiser

Some days are best described by a ChangeLog (email elided):

$ darcs changes --match='date today' --reverse 
Wed Feb  1 14:43:20 CET 2006  eric
  * sematiser: nicer comment-detection code?

Wed Feb  1 14:44:55 CET 2006  eric
  * sematiser: recognise 'en' and 'y' as clitic (cl)
  only when TreeTagger assigns them PRO:PER, though!
  

Wed Feb  1 14:45:32 CET 2006  eric
  * sematiser: recognise du/PRP:det as TSNLP de/p

Wed Feb  1 14:47:07 CET 2006  eric
  * sematiser: use macro type to govern semantic output behaviour
  
  Also: 
  - make code a little more modular - first we identify all the
    ambiguities, then we worry about output.  This will allow us to some
    nice tricks later on like ignoring spurious ambiguities (ambiguities
    which give you the same exact semantics)
  - be more explicit when we don't know what something is
  - handle quantifiers a little more robustly
  

Wed Feb  1 14:53:35 CET 2006  eric
  * sematiser: recognise toi/PRO:PER as TSNLP toi/cl

Wed Feb  1 14:54:12 CET 2006  eric
  * sematiser: remove trailing spaces from code

Wed Feb  1 15:09:51 CET 2006  eric
  * sematiser: corrected reading of ambiguous tsnlp lexicon
  
  only one entry per lem/cat tuple was being read!
  

Wed Feb  1 15:41:58 CET 2006  eric
  * sematiser: ignore ambiguities which produce the same results

Wed Feb  1 16:14:33 CET 2006  eric
  * sematiser: fix bug reading last lexical entry

Wed Feb  1 16:36:20 CET 2006  eric
  * sematiser: Treat ne/ADV as TSNLP ne/cl.

Wed Feb  1 16:36:54 CET 2006  eric
  * sematiser: Improved error messages.

Wed Feb  1 16:37:14 CET 2006  eric
  * sematiser: Corrected handling of NP quantifier.

Wed Feb  1 16:42:34 CET 2006  eric
  * Fixed bug in call to tree tagger 
  (was removing last letter of sentences without punctuation)
  

Wed Feb  1 16:51:10 CET 2006  eric
  * sematiser: nicer ambiguous lemma output

Wed Feb  1 17:12:12 CET 2006  eric
  * sematiser: more tree-tagger -> tsnlp coercions
    pour/KON        to TSNLP pour/p
    aujourd'hui/ADV to TSNLP aujourdhui/adv
    que, lequel, quoi (either PRO:REL or PRO) to TSNLP category n