morphology semi-done

pre-terminals

I was trying to (re)-implement the morphology stuff, basically modify GenI so that we read feature structures off the pre-terminal nodes instead of the terminals. One problem is that I had always assumed (and so had CA) that you just replace/unify the lexeme with the anchor node. Only much later (bugs later) did I learn from YP that in fact, you do not unify/replace the anchor, but you attach something on to it.

The first worry was what exactly we mean by pre-terminal. A node above a terminal node, right? What if it's above TWO terminal nodes? (Maybe you only notice these things if you're actually trying to write the pattern-matching code?) What if it's above a terminal node and a non-terminal one? You could argue, I guess that such cases never occur in TAG, but (1) i don't really know that (2) even if it didn't... yeck! who wants to have walk around with ticking timebombs in their code? Anyway, this turned out to be not so hard to resolve. You just associate the same pre-terminal node with each one of its children (maybe I'm going to regret this):

> -- | Return pairs of (parent, terminal)
> preTerminals :: Tree a -> [(a,a)]
> preTerminals (Node _ []) = []
> preTerminals (Node x ks) =
>  [ (x,y) | (Node y ys) <- ks, null ys ]
>  ++ concatMap preTerminals ks

Now the "fun" part. The next obstacle is that I tried to make the "simple" builder faster/simpler by only using node names for building the derived trees. You don't actually need the feature structures for that; it's a separate layer of control. When you're unifying the feature structures, you don't actually care about the structure. Problem is what happens when you want to return the results. I solve this by putting in a structure that maps node names to back to lexemes and to features structures, but only for the leaves. But that's wrong! That's wrong because the features you want to return for morphology are not leaf features but pre-terminal features (aaaaaaghhh!). Ok, at first I thought you had to modify adjunction and rename any preterminals that received adjunction. But that's just wrong; splicing is saner than that. So no big deal: just tell the builder code to track preterminals instead of leaves. Took me a while to figure out, though.

morphology stuff

Anyway, now that the pre-terminals stuff is taken care of, la morphologie remarche... it's working again. It used to be broken because the morphological post-processors were always getting empty fs on input, but that was because of the old assumption that we were using terminal nodes.

I have also implemented a small in-house morph post-processor. It is dead stupid. It just finds the first thing that the lemma and fs unifies with. But it should be good enough for our needs. Implementing this stuff was super-easy wrt to worrying about pre-terminals and all those gut things. Most of the time was spent refactoring the parser code, and trying to document things as I go along.

The result?

Eh, not so hot. Edit: to be more precise, it sucks. Mucho ambiguity I spent a couple minutes puzzling over the thing, trying to figure out why we didn't have any results. Turns out it was because the morph lexicon calls part of speech "pos", whereas we call it "cat". There might be a technical reason for this. I don't care.

Anyway, it works (sorta). No more "jean aimer marie", but bona fide "jean aime marie". I've got it returning all matches from the morph lexicon, so you get a little combinatorial burp. This exercise makes it pretty obvious why you want a proper morph generator instead of just looking things up in a table.

doc/manual

I know I've already said this, but I'm feeling slightly more commited to the idea now. I think I'm eventually going to push for a darcs-style manual, where the source code comments are the user's manual. This means re-writing a good bit of the comments so that they actually make sense to somebody who's not looking at the code, but going in this direction should improve both the user doc and the code doc. Anyway, what the user gets in the end is a PDF, so s/he shouldn't even have to know that they are source code comments.