eric reads

Oh man. Vijay-Shanker and Weir 93, page 595... that paragraph talking about why we don't need to carry the whole stack around all the time? That took some serious wading. Not that I'm criticising their writing or anything -- ideas like this are inherently hard to communicate both clearly and correctly (i.e. watertight) and no way in hell am I claiming I can write this more clearly than them -- but I sometimes wonder if we need to be thinking harder about the user interface of our scientific articles. Surely there must be some way to get the point across more effectively? Now here I am trying to figure out where the heck A_t(βγ_t) came from. Ah... ok, so it's a direct descendant of B(βγ₁...γ_k... γ_m)... ok fine, but why are starting to talk about this? Oh, ok, I see... the whole point is that if you're building up from A_t all the way up to A, you only need to store the bits of stack that you add in each derivation, plus a pointer to where you came from. Just like with CKY derivations pointing only to the root, just like the article says.

Gah no! The next paragraph is even more confusing! Wait, what's γ_t again? Mnyahh!

I guess this is one of my problems reading articles like this. I spend too much time poring over the details, wading, stumbling over each sentence, going "huh?" when I should really be trying to just, i dunno, get the point. Here I am worrying about all these symbols, when there's a whole chunk of explanation around the corner concerning terminators, which is what I'm really trying to understand, which for that matter isn't really sure that I need to understand this, but seems like the most efficient way to understand their inference rules for TAG.

You know another thing? After all this years having random bits of formal being thrown at me, I still haven't sat down and learned my greek letters right. Can you tell the difference between little gamma (γ) and big upsilon (Υ)? It's a stupid minor detail, but one which really hampers my formula reading ability. When I see a pair of symbols, my brain goes "funky greek thing and other funky greek thing" instead of the instant-recognition, "phi and psi". It's just one more little bit friction preventing the ideas from sliding right into my brain like they should. It's certainly not the primary reason that I wade and struggle, but it is a barrier and one which should be relatively simple to overcome. It's high time I sat myself down and learned me the Greek alphabet. Maybe I'll make flashcards.

cky adjunction

While perusing Djamé's former home directory today (on his advice), I stumbled upon Parsing Some Constrained Grammar Formalisms, Vijay-Shanker and Weir 1993. More required-reading kind of stuff... how do people find these things? I always get them accidentally like today, but smart people active seek out this kind of stuff, seem to know what they're looking for. Bit intimidating looking, but I'm sure if I sit down and work it out slowly, it'll be fine. Reading pipeline's getting pretty big though.

Flipping through the thing, I realised that I needed to implement a spine-tracking mechanism in my CKY. Or to put it better, I need to keep track of where a node is going to be adjoined into some other tree just in case we have this situation where you adjoin one auxiliary tree into another.
Maybe that's what Djamé, you were talking about, in your comment. Not just the bottom-top phase stuff but the where-does-this-node-go.

Other than that. Bof... can't really concentrate today. Been doodling around in my code. I don't know if this is one of those sophomore haskeller pathologies (where you discover a new way to do things and apply it indiscriminately without stopping to think that maybe it isn't so wise after all), but I've lately been using these idioms a lot:

• Using Maybe instead of boolean success flags - safer because you enforce more properties with the type-checker, but more annoying because it forces you to account for things. For example, what happens if unification fails? I think this is ok. Safe is better than convenient... which is generally why I like static type checking. Type checking is for the weak-minded. I am weak-minded.
• This one is iffy: using the Maybe as a monad, instead of having case... of statements.

When you have a huge chain of case-of statements, the maybe-monad idiom makes sense because it makes the code a lot clearer. But if you only have one or two, maybe it is wiser after all to say

case foo of 
Nothing -> Nothing
Just bar -> Just $ someFunctionOn bar

Instead of

do bar <- foo
   return $ someFunctionOn bar

Or even "worse"

someFunctionOn `liftM` foo

Now, my natural tendency is to prefer the second or the third form to the first one. Short and sweet, especially if you think of liftM as just being monadic functional application. But one disadvantage is that it leaves the code less accesible to novice haskellers. People who aren't too comfortable with the whole monad thing, not to mention stuff like liftM or MonadPlus. I err on the side of short, sweet and high learning curve. But maybe that's not the best way to go.

Dunno. Way i figure is that when you make things really really concise, new properties of your code tend to become so apparant which help you make the code even more concise. Conciseness begets conciseness, and this is where the code really begins to sing. That is, concise code is something which directly communicates what you mean without a bunch of syntactic clutter. Maintainability decreases because haskell newbies don't grok the code right away, but it also increases because the resulting code means exactly what you think and nothing more.

Besides, if making the code readable by newbies was really the important thing, I'd be writing GenI in Java (cringe!) or C. I could compromise by writing it in long haskell, but I'm going to take the bet that the gain in clarity far offsets the price of the newbie learning curve. But it's just a bet, nothing more.

100th post

One hundred days of the erician PhD! Not the first hundred, since I started this blog only at the end of my first year, but hey wow, that's a milestone, right? So in not-quite 100 days (because of the rare multi-post days), what have I achieved?

*chirp* *chrip* *chrip*

Shut up, crickets.

Maybe the next 100 will be more productive.

Anyway, so CKY implementation is moving along smoothly. Got a pretty good framework for printing out tracing and debugging info. The thing seems do the context-free stuff correctly on a toy grammar. I've only got the initialisation rules (though not in inf rule form), the kids to parents rules, the non-adjunction rule and the subsituttion rule running. What's left to do?

• adjunction
• extracting the output (will have to build an automaton into my items)
• dealing with unification on the kids to parent rule
• top to bottom unification on the kids to parents rule
• visualisation of all this in the graphical debugger
• re-enabling equivalence classes and implementing merging
• and dealing with all sorts of unexpected things that come up

Yeeehaw.

At lunch, Bertrand reassured me that whilst CKY is conceptually really simple to understand, yes, implementing it is a bitch. That makes me feel better. It also makes me worry that behind the corner is a huge frickin buttload of difficulties to encounter. Positive attitude, eric, coding is fun, remember?

I also have an idea for dealing with unification a little more efficiently: maybe I can use that collect typeclass. In the beginning, I collect all the uninstantiated variables in my TAG trees. Then when I'm doing the derivation stuff, I can use this list of values (+ the replace typeclass) to propogate only the "final" values for these variables. It should be equivalent to keeping around a big honking list of variable substitutions to perform but always reducing said list to its final form. (So if you got A -> B; B -> C, the only thing to keep around is A -> C). Don't know if I need to deal with foreign variables being introduced though, i.e. through subsitution or adjunction. Might throw a kink into all this.

Anyway, here's some trace output to celebrate day 100 of koweynlg.

-> agenda               0       _b_er   _b (na)
-> agenda               0       _b_er   c (s)
-> agenda               0       _c_san  c:_c (na)
-> agenda               0       _a_yi   _a (na)
-> agenda               0       _a_yi   b (s)
kidsToParents           0       _a_yi   b (s)
 relevant chart: []
 routing info: (["Left"],a,True)
subst                   0       _a_yi   b (s)
 subst variant
nonAdj                  0       _a_yi   b (s)

kidsToParents           1       _a_yi   _a (na)
 relevant chart: []
 routing info: (["Right"],a,True)
subst                   1       _a_yi   _a (na)
nonAdj                  1       _a_yi   _a (na)

kidsToParents           2       _c_san  c:_c (na)
subst                   2       _c_san  c:_c (na)
 root variant
nonAdj                  2       _c_san  c:_c (na)

kidsToParents           3       _b_er   c (s)
 relevant chart: []
 routing info: (["Left"],b,True)
subst                   3       _b_er   c (s)
 subst variant
nonAdj                  3       _b_er   c (s)

<- subst                3       _b_er   c (s)
-> agenda               0       _b_er   c:_c (na)
kidsToParents           4       _b_er   c:_c (na)
 relevant chart: []
 routing info: (["Left"],b,True)
subst                   4       _b_er   c:_c (na)
nonAdj                  4       _b_er   c:_c (na)

kidsToParents           5       _b_er   _b (na)
 relevant chart: [c:_c (na)]
 routing info: (["Right"],b,True)
 matches: ["[c:_c (na)]"]
subst                   5       _b_er   _b (na)
nonAdj                  5       _b_er   _b (na)

<- kidsToParents        5       _b_er   _b (na)
-> agenda               0       _b_er   b
kidsToParents           6       _b_er   b
subst                   6       _b_er   b
 root variant
nonAdj                  6       _b_er   b

<- subst                6       _b_er   b
<- nonAdj               6       _b_er   b
-> agenda               0       _a_yi   b
-> agenda               0       _b_er   b (na)
kidsToParents           7       _b_er   b (na)
subst                   7       _b_er   b (na)
 root variant
nonAdj                  7       _b_er   b (na)

<- subst                7       _b_er   b (na)
-> agenda               0       _a_yi   b (na)
kidsToParents           8       _a_yi   b (na)
 relevant chart: [_a (na)]
 routing info: (["Left"],a,True)
 matches: ["[_a (na)]"]
subst                   8       _a_yi   b (na)
nonAdj                  8       _a_yi   b (na)

<- kidsToParents        8       _a_yi   b (na)
-> agenda               0       _a_yi   a
kidsToParents           9       _a_yi   a
subst                   9       _a_yi   a
 root variant
nonAdj                  9       _a_yi   a

<- nonAdj               9       _a_yi   a
-> agenda               0       _a_yi   a (na)
kidsToParents           10      _a_yi   a (na)
subst                   10      _a_yi   a (na)
 root variant
nonAdj                  10      _a_yi   a (na)

kidsToParents           11      _a_yi   b
 relevant chart: [a (na),b (na),_a (na)]
 routing info: (["Left"],a,True)
 matches: ["[_a (na)]"]
subst                   11      _a_yi   b
nonAdj                  11      _a_yi   b

<- kidsToParents        11      _a_yi   b
<- nonAdj               11      _a_yi   b
-> agenda               0       _a_yi   a
-> agenda               0       _a_yi   b (na)
kidsToParents           12      _a_yi   b (na)
 relevant chart: [a (na),b (na),_a (na)]
 routing info: (["Left"],a,True)
 matches: ["[_a (na)]"]
subst                   12      _a_yi   b (na)
nonAdj                  12      _a_yi   b (na)

<- kidsToParents        12      _a_yi   b (na)
-> agenda               0       _a_yi   a
kidsToParents           13      _a_yi   a
subst                   13      _a_yi   a
 root variant
nonAdj                  13      _a_yi   a

<- nonAdj               13      _a_yi   a
-> agenda               0       _a_yi   a (na)
kidsToParents           14      _a_yi   a (na)
subst                   14      _a_yi   a (na)
 root variant
nonAdj                  14      _a_yi   a (na)

kidsToParents           15      _a_yi   a
subst                   15      _a_yi   a
 root variant
nonAdj                  15      _a_yi   a

<- nonAdj               15      _a_yi   a
-> agenda               0       _a_yi   a (na)
kidsToParents           16      _a_yi   a (na)
subst                   16      _a_yi   a (na)
 root variant
nonAdj                  16      _a_yi   a (na)

All that for a grammar of three trees, and a grand total of 7 nodes. Maybe I should simplify my example some more.

change everything!

Here's me and my messy way of coding, never knowing what the final product is going to look like, always throwing cans of paint around. I realised that there are a couple of things I had not accounted for in my implementation from yesterday

• unification
• the actual non-disposability of rules, i.e, the kazillion different things that could make a rule fire

So back to the drawing board.

Alright first of all, the inference rules are just regular old functions. We're not going to create a new function for every tree or any nonsense like that. Second of all, I got over my hangup with associating nodes with information about who their sisters are. Third, I'm putting an agenda control mechanism back in. One simple little bit of technology. I was trying to simplify matters by getting rid of it (and put it back when I better understood what was going on), but turns out that having rules based on the "current item" is just as simple and does wonders for assuaging my combinatorial worries...

Now in the initialisation phase, we extract from every tree something I call a routing map (stupid name) which points the last node of every to its sisters and to its parent. So the way this works is that all rules are fired all the time. In the case of the kidToParent rule, we only fire when we have some routing information at the node at hand, which would only be the case when we hit the last item of any row... I hope that's right, any reason why I would want it to fire upon any other item? After all, having all the child items is a precondition of the rule working. Hmm... anyway, my code's a tiny bit cleaner, no more records and weird stuff all over the place, there is a little bit more progress, but who knows? Maybe by next week it'll be completely different again.

chugging along on the CKY

My code is a huge mess with very minimalistic comments right now... but I've pretty much got the basic inference rule architecture set up. Now I have little bits and pieces of CKY running: the initialisation rules and the bottom-up (child to parent) rules. Huge chunks of self-doubt running in the background here. I think I do kind of like the general approach I'm taking though: inference rules are functions, functions are values, so just manipulate the inference rules like you would any other value. This is a functional programming language after all.

The benefit is that now you can implement any kind of parser/generator you want just by defining your chart items and your inference rules accordingly. Don't know how generic it is, but somewhere down there I'm thinking that this treat-everything-as-rules approach is correct. Besides, it looks very much like the algorithm as it's described to you in the papers, so it can't be that wrong.

Here's the code for the two rules I have so far

initRule :: TagElem -> GNode -> CKY_InferenceRule
initRule te node = Rule 
  { showRule = ckyShow "init" (idname te) ++ " " ++ show node
  , retrieve = \_ -> Just [] -- always suceeds
  , infer    = \_ -> [nodeToItem te node] }

kidsToParentRule :: TagElem -> GNode -> [GNode] -> CKY_InferenceRule
kidsToParentRule te parent kids = Rule 
  { showRule = showFn
  , retrieve = retrieveFn
  , infer    = \_ -> [nodeToItem te parent] } 
  where 
    source = idname te
    showFn = ckyShow "kidsToParents" source ++ " p: " ++ show parent ++ " k: " ++ show kids     
    retrieveFn chart = 
      let isMatch n c =  
            (gaconstr.ciNode) c && ciSourceTree c == source &&
            (gnname.ciNode) c == gnname n
          hasMatch n = any (isMatch n) chart
      in  if all hasMatch kids then Just [] else Nothing

Note that these functions aren't inference rules themselves, but functions which create the rules. I'm stretching the definition of inference rule here, I guess. By inf rule, what I really mean is application of an inference rule to a particular case.

Here's the output so far

applying inf rule init (a_one) a (na)
applying inf rule init (a_one) b (s)
applying inf rule kidsToParents (a_one) p: a k: [a (na),b (s)]
applying inf rule init (c_three) c:c (na)
applying inf rule init (b_two) b (na)
applying inf rule init (b_two) c (s)
applying inf rule kidsToParents (b_two) p: b k: [b (na),c (s)]

Oh, and remember that cryptic code I was talking about last night? I was just being stupid. 'liftM' clearly does not do what I thought it did yesterday. Here's the corrected version:

query :: (BuilderStatus -> a) -> BState a
query fn = fn `liftM` get 

I'm a lot less bothered by my implementation of alpha conversion from last night now. One benefit, I realised is that now I can write a generic alpha conversion function that works on any object I want! In fact, this is the same function as last night, except that I had to define one tiny little typeclass more (Idable, which returns a unique id for an object).

alphaConvert :: (Collectable a, Replacable a, Idable a) => a -> a
alphaConvert x =
  let vars   = Set.elems $ collect x Set.empty
      suffix = "-" ++ (show $ idOf x)
      convert v = GVar (v ++ suffix)
      --
      subst = map (\v -> (v, convert v)) vars 
  in replace subst x 

Now that I've got all my other distractions out of the way, I'm focusing on getting that CKY implementation done. Oh goodness, how terribly embarassing, I'm actually having trouble implementing CKY. Nothing could be conceptually simpler than this, but...

I guess a thing that's bothering me is how get things working efficiently. But that's really not something I ought to caring about now. Besides, Haskell's great for writing inefficient-looking code that actually turns out to be perfectly fine because of lazy evaluation. Just make it nice and simple is all I need. Worry about efficiency later. I guess one thing that gets me is how to implement the daughter-father rule: how do I identify if all the sisters of a node are in the chart... I supppose I could map something like isInChart sisters. How do I crawl back up to the mother, do I associate each node with its mum?

Should I maybe introduce a notion of disposable inference rules, just as an implementation point of view, rules that stop getting triggered once they succesfully fire. Toying around with this notion of treating each instance of a rule as an individual object. So you don't have one daughter-mother rule, but a daughter-mother rule for each branch point of each tree. Then you just loop through blindly, fire all the rules that you can; remove the rules that succesfully fire.

Argh. CKY. Nothing can be simpler. Why am I making this difficult for myself?

Edit: I swear, the more Haskell I learn, the more cryptic my code gets. I can't help it. "Small is beautiful" is stronger than me; i am pathalogically incapable of writing things less concisely than I know how. The "you could do this with less" or the "hey, this really could be refactored" just keeps gnawing at me:

query :: (BuilderStatus -> a) -> BState a
query fn = get >>= liftM fn 

mrmh, slow and half awake

Tinkering around with GenI, today. Corrected an obscure alpha conversion bug that only popped up when I started accepting variable semantics (note, however that variable semantic instantiation is not implemented). The basic issue was that variables in the semantics were not being alpha converted, normale because up to now, lexically selected items were completely instantiated and had constant semantics.

I'm not sure if my approach here is right, maybe a little too enthusiastic on the type classes here. What makes me worry is that my new method just seems so overkilly. Before I had a simple thing that appended a suffix to every variable. Now, I collect all the variables in the structure, create replacements for them (i.e. "replace ?A with ?A-1"), and then use the Replacable typeclass to propagate the values. No longer nice and simple, and introduces many many more lines of code, but eh, the new code is more general -- I have already managed to find a completely unrelated use for it -- and it's less vulnerable to me screwing things up. Here's the type instances:

class Collectable a where
  collect :: a -> Set.Set String -> Set.Set String

instance (Collectable a => Collectable [a]) where
  collect l s = foldr collect s l

instance (Collectable a => Collectable (Tree a)) where
  collect = collect.flatten

-- Pred is what I had in mind here 
instance ((Collectable a, Collectable b, Collectable c) 
           => Collectable (a,b,c)) where
  collect (a,b,c) = collect a . collect b . collect c 

instance Collectable GeniVal where
  collect (GVar v) s = Set.insert v s
  collect _ s = s

instance Collectable (String,GeniVal) where
  collect (_,b) = collect b

instance Collectable GNode where
  collect n = (collect $ gdown n) . (collect $ gup n)

instance Collectable TagElem where
  collect t = (collect $ tinterface t) . (collect $ ttree t) 
            . (collect $ tsemantics t)

Other than that, I have tracked down a systemic grammar for English in the University of Bremen generation bank... so I guess this will be interesting to look at and play with. Maybe should see if there are any generation systems downloadable.

Fixed the weird runInteractiveProcess issue: the basic problem is that you have to read from the subprocess's stderr, or else it's just going to sit there and block, probably waiting for the file handle to be available. So that's a relief after all these bugs of this annoying bug, all this time keeping that ugly old code around because I couldn't get selector to come back, all that time wondering what the heck was going on in process invocation land. Ahh... so nice when things actually work like they should.

Other than, I've got the semantic test suite generator to handle ambiguity from TreeTagger. Showed this and recent advances in GenI/SelectTAG integration to Claire. One other relief for me is that the lexicon is not supposed to have any null semantic items, so that means less work for me trying to figure out how to handle them gracefully, for now anyway.

So maybe now that I've gotten all this random crap out of the way, I can finally sit down and start working on CKY again.

Oh and this blog is now open for anonymous comments. We'll see if this keeps the spam out.

Edit: Yay regression testing! I have some pretty primitive eyeball regression tests, a simple shell script that runs GenI over a bunch of inputs. No smarts, just eric to inspect the output and see if it "looks" right. But even this simple chunk of testing has spotted an interesting error from one of the new features I introduced into GenI. GenI now refuses to generate if there is an empty semantic item and IgnoreSemantics is unset... but it should only do this on the builder level. For example, having empty semantic pronouns is fine as long as you've got the polarity optimisation covering back and inserting the semantics for you... Yet for some reason, GenI still complains. To investigate. Hmm...

Edit: Fixed! Was a pretty minor bug actually; I was pointlessly initialising a state with data that made GenI complain. And now my eyeball tests run again. Whew! Regression testing saves the day.

HPSG, subprocesses and arrows

Attended the second half of Jesse's HPSG course. Today we learn how to combine signs into phrases. So now I know that the ways bits can combine are controlled by a selection of schemas and principles. You pick a schema and apply all the principles with nothing more complicated than unification and the non-destructive list manipulation (you say stuff like "the valence of the phrase is defined as that of its head daughter minus the synsems of its non-head daughters"... i'm just reading the diagram here, so I might sound like an idiot).

I'm in kind of an odd position, being a programmer with mild exposure to these random bits of formal linguistics. All I know how to do is code. Have zero linguistic knowledge or experience, and so whenever somebody throws a new formalism at me, I just sort of shrug and say "that seems like a good way to do things". I've had this same mindlessly uncritical reaction to TAG, CCG and now HPSG (although I do admit that CCG does especially appeal to my programmers' notion of simplicity and elegance, has thus a special place in my heart)

As for the programming bits, well, I managed to shake myself awake this morning with the realisation that I need to get back to my agile programming outlook: damnit eric, make it work NOW and only then worry about making it work right. So thanks to the magic of darcs unrecord and darcs revert, I reinstated my old ugly fork stuff and have my calls to the Selector working again. And strangely enough, curious as all heck, the subprocess calls to the grammar cutting tools are magically working again. Really don't understand what's going on here. Oh well, if it happens again, I have a workaround in mind: don't use the tree selection mechanism. Instead, introduce a "grammar cutting" mode which does lexical selection, prints out the list of lexically selected items and quits. No subprocess calls. Do the grammar cutting in the Makefile.

Djamé sent me a nice mail (coucou) explaining Shieber's restrictions to me. He also mentioned some potentially useful reading: Principles and Implementation of Deductive Parsing, also by Shieber. From my superficial birds-eye view, this looks like it should be required reading for anybody who wants to work on parsers or generators. I dunno. We'll see when I actually get around to reading it (numéro deux in my pipeline).

For bedtime reading last night, I've been reading Manuel Ohlendorf's masters thesis, where (i think) he adds an Arrows interface to the Haskell XML Toolbox, which is one of the two major (?) XML parsing libraries for Haskell. It's gotten me interested in learning more about arrows... looks like it has the potential to be useful. It's a bit over my head right now. I mean, if you look at the example below

    addA :: Arrow a => a b Int -> a b Int -> a b Int
    addA f g = proc x -> do
                    y <- f -< x
                    z <- g -< x
                    returnA -< y + z

one of the useful things you can do with arrows is perform the computations f and g independently of each other and then return a result which is dependent on both their results. But part of me wonders "so what?" couldn't you just simulate that with monadic sequencing? What's the extra bonus that arrows buy you?

  
    addM f g x = do
                    y <- f x
                    z <- g x
                    return (y + z)

There must be something I'm overlooking. I guess with arrows, you have some kind of guarantee that f and g really are completely independant. If f modifies the monadic state (for example), then g simply does not see that modification. Something like that? So much more to learn...

annoying wednesday

I'm feeling somewhat grumpy about this: having some trouble calling subprocesses in Haskell. Had some old clunky code that forked and did random weird crap with stdin/stdout handles, and for some reason it was spitting out executeFile errors, but like, not consistently so, only with certain inputs. Fine, so I got rid of my old clunky code and used the System.Process runInteractiveProcess like i've been wanted to do so for like next to forever, and everything seems to work great until I try using it to run SelectTAG, and there I run into my old nemesis, the not-coming-back problem. Emulator.exe just sits there chewing up CPU time and doing nothing. I've ktraced it, and it seems to complain about some broken pipe somewhere:

 10560 emulator.exe RET   write 1
 10560 emulator.exe CALL  sigprocmask(0x3,0xbffff1d0,0xbffff1d4)
 10560 emulator.exe RET   sigprocmask 0
 10560 emulator.exe CALL  select(0x100,0xc5064,0xc50e4,0,0xbffff070)
 10560 emulator.exe RET   select 1
 10560 emulator.exe CALL  sigprocmask(0x3,0xbffff220,0)
 10560 emulator.exe RET   sigprocmask 0
 10560 emulator.exe CALL  write(0x2,0x1819400,0x22f)
 10560 emulator.exe RET   write -1 errno 32 Broken pipe
 10560 emulator.exe PSIG  SIGPIPE caught handler=0x1bbbc mask=0x0 code=0x0
 10560 emulator.exe CALL  sigprocmask(0x3,0xbfffe970,0xbfffe974)
 10560 emulator.exe RET   sigprocmask 0
 10560 emulator.exe CALL  sigprocmask(0x3,0xbfffe978,0)
 10560 emulator.exe RET   sigprocmask 0
 10560 emulator.exe CALL  sigreturn(0xbfffeaa8,0x14)
 10560 emulator.exe RET   sigreturn JUSTRETURN
 10560 emulator.exe CALL  write(0x1,0x92348,0x1)
 10560 emulator.exe GIO   fd 1 wrote 1 byte

But where, wha? Dangit... I wish I knew how to work these computer thingies.

Sigh. Another thing I've been thinking about is not using Yannick's SelectTAG utility and letting GenI do its own large scale anchoring. Way I see it, SelectTAG has two functions:

1. provide an efficient way to query a large grammar for a set of trees
2. factorise the process so that it can be reused by other tools

For the efficient querying of trees, I think I have a simple way of splitting up the grammar (just write a file for each family, and by virtue of lazy IO, only read the families you want) and for the factorisation... well... in all likelihood, GenI will probably be SelectTAG's only client. I dunno. Maybe I'm not being farsighted enough. I'm tempted to go run off for a couple of days and do my own anchoring mechanism. But then so what? No more subprocess crap -- but that's just a minor implementation issue. There is also another advantage maybe of not having to bug Yannick for each tiny little change I want to make in the interaction... there is a slight simplification of code... there is also the fact that I get to manipulate the raw grammar, which means maybe I do all the weird stuff I want without having to hack up the Selector for it. I dunno. Keep busying myself with these stupid software engineering questions.

Anyway, so I started looking into this, trying to dust off my old XML parsing code. The HaXmL API has changed in the latest versions, which is kind of annoying. Maybe I should start playing with the Haskell XML toolbox instead. Seems like a much more serious toolkit. Maybe it also won't have the mega space leak thing (the space leak could also just be me being stupid). Got a brain like mud today. Can't decide anything useful. Everything is broken.

quiet monday

Beats me. Can't remember what I did on Monday. It is now Wednesday. I vaguely recall helping Claire get everything set up on her computer. Was fiddling around with my TSNLP-semantics makefile, making it run nicely.

peaceful day of perl

Been improving my semi-automated test suite generator. Mostly consists of refining the correspondance between TreeTagger
categories and those in the TSNLP test suite.

Events are conspiring against me ever getting this CKY stuff done. Always something else to do.

I'm starting to wonder, though, what i'm doing still writing my scripts in Perl. It seems that anything past the level of simple shell scripts, I might be better off doing them in haskell, using something like #!/usr/bin/env runhaskell to make them go. Smaller, more beautiful code? I mean, on the one hand, you want to be a polyglot, but on the other hand, you want to get the job done as cleanly and quickly as possible. And you also want your stuff to be usable, extendable by other people. Hmm.

evaluation stuff slowly falling into place

This morning, attended a three hour course by Jesse on HPSG. Nice simple introduction, a couple of exercices. That feels better -- don't feel quite as ignorant as before. Always had the vague understanding that HPSG was doing everything with recursive feature structures, and now have more of a handle of other details like linear precedence being expressed as a set of constraints on the side, on how lists work, on the whole notion of typed feature structures, the multiple inheritance stuff.

Afternoon was spent finishing up the import of the lexicon repositories into darcs, especially working to Claire to get her very latest stuff pushed into the repository. Looks like things are pretty clean now. Hope they stay that way. Also got the sematiser working again with the new semantic macros stuff, but sorta botched that job and will have to look into it some more tomorrow.
Same thing for runXMGfilter. There's some kind of error, and for some bizarre reason, it's killing Claire's grammar (thank goodness for darcs revert!). Claire also pointed me to a new article I might want to be reading. More on that Monday.

hmm... not so productive today

Took me next to forever, (hour and a half with lots of ADD pacing around), but I finished reading Systemic Grammar in Computation: The NIGEL Case by Christian M.I.M. Matthiessen. Pretty nice article actually, in that it provides a for-dummies intro to systemic linguistics. The guy speaks my (coder's) language when he says stuff like "the specification of structural presence and the specification of constituency relations are separate from ordering specifications." SFG is all about factoring, and hey, me being a programmer, I like factoring!

Another thought is that this sounds a lot like what the metagrammar system is for: writing bits of grammar in a highly factorised manner. Would be interesting to explore the relationships between this and the SFG stuff: could we write the metagrammar in a "systemic" way and somehow propogate that information out into the real grammar so that we can exploit in for generation? How about the systemic parsers (Winograd 72)? How the heck do those work? Could we do something interesting with that and TAG? I dunno... I hope these are useful questions to ask. I guess I'll start digging deeper into them after I get the CKY stuff done. Probably should talk to Claire about this, see what she thinks; she's probably a few steps ahead of me on this one.

Other than that, I haven't been much productive. Showed Yannick some darcs and Unix stuff, got the folli list administration set up, found another interesting case that the darcs conflictors stuff should fix (aka: probably a bug).

rappport d'activité

Been working these past couple of days cleaning up my team's Rapport d'activité. Fun with Vi(m) and with regular expressions. Two useful things:

1. :help jump in vim one day; ctrl-o is a very useful command for going back to where you last were
2. \_ before an atom allows you match over multiple lines

Probably the hairest vim command I ever executed to date:

'<,'>s/\\item \(\_.\{-}\) *^M*(\(\_[^\(\)]\{-}\)). *$/\\item \2: \1./cg

Which takes lines like
\item blah blah blah (Bubba Jones)

and translates them to
\item Bubba Jones: blah blah blah

Other than, been discussing with Claire on how to organise the semfrag repositories, as well as with Yannick and Claire on how he sets aside parameters from normal attributes in XMG.

Tried to read a systemic grammars thing, but couldn't really concentrate. Although I did take a second look at Shieber and restriction, and it now makes a good bit more sense now that I'm less bleary eyed.

Looks like tomorrow, i'll have some repository hacking to do, and then maybe I can work on my CKY implementation. But things are looking good. Darcs rollback is my friend.

modularisation complete

Got the graphical interface fully modularised now. Have GenI set up with a configuration flag -b=simple|cky which lets us choose what builder (maybe should call it engine) GenI uses.

More discussion with Bertrand about the generation algorithm he's been cooking up this past month. Well... après tout, he's worked his way back to a simple CKY for TAG, except defining his items to contain automata for output, but really, nothing could be more standard than what he's got. Suits me. Much easier for me to understand. Now that the modularisation is complete, I just need to get this dang thing implemented as soon as possible so that we can have some packing, a little factorisation like all the big boys and start working on something new.

Read Shieber's Using Restriction to Extend Parsing Algorithms for Complex-Feature-Based Formalisms, but I was pretty bleary-eyed this morning, so my basic reaction to this paper was something along the lines of "wha?". If I had known this paper existed, I wouldn't have used to word "restrictor" in GenI for our paraphrase selection stuff. His usage and my usage are clearly different and I wouldn't want people to think I was talking about the Shieber thing and get confused. This paper and the bag of tricks one are useful for understanding the Carroll and Oepen paper better.

modularisation: starting to fall into place

It compiles! It runs! It does regression tests! It has a Gui! ... and it does pretty much the same stuff that it had been doing for the past couple of years, but now GenI is a whole lot tidier and has a higher chance of being understandable. I have almost completed the process of tearing GenI down into its component parts and putting it back together. The difference of course, being where I put the parts.

Instead of sing GeniFn -- the confusing callback function approach -- i now decompose the generation process into four steps; either you call the four steps individually from the graphical debugger, or runGeni calls all of them for you

This is what my builder data type looks like for now

data Builder st it pa = Builder
  { init     :: Input -> pa -> st
  --
  , step     :: State st ()
  , stepAll  :: State st ()
  , run      :: Input -> pa -> st 
  --
  , finished :: st -> Bool
  , stats    :: st -> Gstats
  , setStats :: Gstats -> st -> st 
  , unpack   :: st -> [UninflectedSentence] }

And this is what runGeni looks like:

runGeni :: ProgStateRef -> B.Builder st it Params -> IO ([String], st)
runGeni pstRef builder = 
  do let run    = B.run builder
         unpack = B.unpack builder
     -- step 1
     initStuff <- initGeni pstRef
     --
     pst <- readIORef pstRef
     let config  = pa pst
         -- step 2 
         finalSt = run initStuff config
         -- step 3
         uninflected = unpack finalSt
     -- step 4
     sentences <- finaliseResults pstRef uninflected 
     return (sentences, finalSt)

This was probably one of the most harrowing refactors I've ever been through, and it's not done yet, but I think I'm pretty happy with the result. There is a much clearer seperation of tasks now.

Other than that, in the past couple of days, I've read Carroll and Oepen's High Efficiency Realization for Wide-Coverage Grammar, which is kind of neat because they've found something even better than their 2-phase realisation approach - something that I'll need to look into for GenI. Following the bib of this article, I also read Kiefer et. al A Bag of Useful Techniques for Efficient and Robust Parsing. I mostly just wanted to know what the heck this quickcheck filter was that Carroll were talking about. I really liked this second paper. Collection of simple, correct techniques that help you avoid doing lots of unification. Not sure I get all of it, but looks like a good bit of this is going to be useful. Was wondering if it would be simpler, instead of making a seperate quick check filter, to just use failure frequency as a sorting criterium, but I guess that's just an implementation detail. So yeah, let that be a lesson to me. Lots of interesting little things you can play around with to make things go faster. But for the moment, tabulation is my life, my life is tabulation (what Carroll calls packing).

darcs+unison do not mix!

I was in the middle of setting up the central grammars repository (Yannick's idea) when I came across an inconsitency in Claire's darcs repo. Scary stuff. But I think I know where the error comes from : the basic problem is that she's got a patch that modifies some file which has previously been removed. Normally when you darcs remove a file, darcs stops keeping track of it and further modifications are not taken into account. If however, (1) you unison after your darcs remove and (2) unison gets confused and puts the file back into your _darcs/current then shit happens. Darcs doesn't know that the file no longer exists. Another case where this could go wrong is this sequence:

machine A - darcs remove the file
machine B - edit the file 
unison
darcs record

Same problem as above. You or unison get confused and you basically copy back a file which darcs should no longer be keeping track of, and tada! inconsistent patch. So it's not really darcs's fault, but the global issue that combining unison with version control systems (CVS, Subversion, etc) is really not a good idea.

The solution I'm going with, I guess, is not to keep a seperate directory for my darcs projects. On my laptop, I just darcs pull from my workstation, without any unisoning at all. Only thing is that I have to do a lot more darcs record on the workstation end, even with stuff which is plain broken, or which doesn't compile, but it's a much safer bet than letting unison get in the way.

Oh and other than that, still working on modularisation of GenI. Getting ever closer to something that can compile.

modularisation

Been MIA monday and tuesday... was doing a first aid course.

Still trying to refactor and modularise GenI so that we can run it with different engines. Bit of a pain trying to figure out what bits go where and how to fit them together in a nice clean way. Making some sacrifices to simplify the problem a bit: no more trying to time GenI for instance, especially since I know it can be done from the outside (though timing the lexicon loading stuff may be an annoyance).

Had a discussion with Claire and Yannick about what the mid term future of our PhDs are. Seems that one direction for me to follow is the set of fun things we can do with evaluation:

• thinking in terms of how different types of input affect performance (not just different types of optimisations
• seeing how random input formulas would be useful
• systematic testing of different optimisations (like in my Masters, but larger scale)

So plate is full as ever: got to get this modularisation stuff done, get some tabulation implementations going.

Some minor wxhaskell patch stuff. One of the Japanese users had patched my patch to make it compile under Windows with Visual C++. So I had to incorporate the patch into mine. But I screwed this one up though. Will have to make a corrected patch tomorrow.

functional software engineering

Still working on generalising my code. Now my only goal is to get the old surface realisation algorithm up and running, but in the cleaner, more general way.

Also talked a bit more with Bertrand on his automaton ideas. It seems that this time, I actually managed to make a useful remark. Yay! Unfortunately, the useful remark was sort of along the lines of "but... erm... that doesn't work."

I'm trying to work through this thing I call "Le Big Ugly Mess"... rough problem is how to get it so that you can do your surface realisation with different algorithms. Thing which makes this more complicated is that each surface realisation algorithm interacts differently with stuff like optimisations and graphical debugging tools and so forth. I'm not trying to make things hard on myself; for example, I've got my makefile set to ignore the Gui code, but there is a lot of craziness brewing and I'd like to see how I can get it under control.

Had a nice conversation with Cale on #haskell today. He basically reassured me that thanks to laziness, the idea of returning all your intermediate results in a list is perfectly OK. If you don't observe them (i.e. pass them through the graphical debugger), laziness means that they get thrown away instead of being list constructed. And I was worried that I'd have to figure out some kind of crazy monadic threading stuff to make things cleaner.

Also had some ideas about roughly how to make things work. I do my best software engineering in the shower. Vague notion that every algorithm has its own gui and its own Builder and each of these depend respectively on the generic gui and builders. Trying to work out how to get it all nice simple and maintainable.

Other than that, skimmed through John Hughes's Generalising Monads to Arrows. Maybe it will be handy for dealing with le big ugly mess?

and more tabulation

Lots of rearranging going on here. Slowly starting to smooth out bits and pieces of GenI that had assumed we'd be pushing derived trees around, a little generalisation here and little generalisation there. It's a pretty big pain in the butt, but it'll be worth it.

I'm using a data type I call Builder to provide a uniform interface to all surface realisers. For now, the Builder data type looks like this, but it keeps changing and shifting as I discover new needs and constraints:

data Builder st it pa = Builder
  { init     :: Sem -> [TagElem] -> pa -> st
  , step     :: State st [it] 
  , finished :: st -> Bool
  , stats    :: st -> Gstats
  , setStats :: Gstats -> st -> st 
  , unpack   :: [it] -> [UninflectedSentence] }

Another thing I've been doing on the side is cleaning up some of the current code. One major change I've made is to observe that the idea of propagating changes from a unification is something that occurs very frequently in GenI. So rather than having a special function for each type, something like substFlist, substGNode, substTree, substHelper, substTagElem and so on and so forth, I've created a new type class Replacable that looks like this:

class Replacable a where
  replace :: Subst -> a -> a 

  replaceOne :: (String,GeniVal) -> a -> a
  replaceOne s = replace [s]

The simplest implementation looks like this:

instance Replacable GeniVal where
  replace sl v = foldl' (flip replaceOne) v sl
  replaceOne (s1,s2) v = if (v == GVar s1) then s2 else v

All of this leads to much cleaner looking code. You just have one function to call, replace. As an added bonus, many of my previous substFoo functions magically vanish because they are obsoloted by a more generic polymorphic equivalent, such as the implementation for lists:

instance (Replacable a => Replacable [a]) where
  replace s    = map (replace s) 
  replaceOne s = map (replaceOne s)

So yeah, it looks like this is one place where type classes are most useful.

tabulation

Still trying to code a very vanilla tabulation up -- even more vanilla than the stuff I had thought up yesterday. Realised that it was problematic and that I was trying to run before I can walk. So I'm going to (for the moment) ignore adjunction and focus on getting basic Günter Neummann style equivalence classes, using passive completion and active completion to stitch things together from both directions. Very classic, very GenI, and with the possibility that we have a gazillion singleton equivalence classes. But most likely the gazillion singleton class problem is just stupid FUD that my mind generates to keep me frozen in place.

Taking advantage of this moment to clean up a lot of the GenI core stuff. Now that I understand this monad stuff a little better, the use of functions like liftM and filterM seem a little more self evident. This allows me to clean the code up a bit. My function for dispatching new results is a lot more readable, and lot more "linear" than previously, feels a lot more like a first do X, then do Y kind of thing.

Finished reading Jerry Hobbs's Ontological Promiscuity. No definite use of it in GenI, but it was nice to see all the thinking that goes behind putting a formalism together, and I was pretty much curious to see what the SPUD people where talking about when they mentioned this term.

Also checked out a book from the library, Optimisation combinatoire 1 (edited by Vangelis Th. Paschos). Doesn't actually mean I'll read it, but hey, the title was intriguing enough. Combinatoric? Optimisation? Anything to do with GenI perhaps?

Talked a bit with Bertrand about his automaton stuff. He can guarantee me that the sentence automaton will be loopless, which is handy because then I can just reuse the automaton walking code that I have.

I thought I had worked out a pretty standard, pretty primitive approach to chart generation, building derivation trees, so I set out to start coding the damn thing up, start generalising the current GenI code for my needs. But as a result of my coding attempts, I realised that my definition of chart and agenda item might need to be rewor... eh... no in fact, in writing this blog entry, maybe not. Maybe it could work after all. I just have to transfer the new node to the list of open adjunction nodes. Right, so nothing special. I'm not going to try to push the dots around, but since I'm carrying the list of next available nodes, i'm sure it all works out to being the same thing. The idea is be somewhat conservative, do things a GenI way, but get a nice chunk of factorisation in.

My definition of items so far looks something like this

Agenda Items: (id, onS, onA, op, back, node)

1. onS - open substitutions nodes; nodes on which we can perform a substitution
2. onA - nodes on which we can perform an adjunction
3. op - the operation used to connect it to some other item (via an id)
4. back - the item this item is connected to
5. the node of the item that...

Argh. This is still a mess. I think I still need to work this one out.

Today, I set myself to thinking about the chart generation stuff. Mind a complete blank. Immensely perturbed by not having some concrete physical action to perform like "code Foo up".

If I am going to get a hang of this research thing, I need to learn to structure my thought process so that it always feels like I'm doing concrete physical actions. I need to learn to decompose problems so that instead of having this annoying blank mind feeling, I can just think to myself, ok, "find a solution to subproblem Bar". When you're coding, the decomposition of problems comes so naturally; you don't even think about it. You look at the problem and it all unfolds in your head, all the unknowns just branch out beautifully. You don't neccesarily know how to solve all the problems, but you build a rough mental map of where all the tasks and subtasks are, and set about them one at a time. Like aikido and multiple opponents. Solve this, solve that, dispatch dispatch, whoosh, whoosh. I don't know why it shouldn't be same thing for research. I don't know why I can't deal with researchy stuff in the same manner, why the task doesn't just magically unfold into a rough mental map like they do when I program.

Anyway, one useful (programmer) thought did occur to me: no matter what path I choose, we're going to need to represent the set of output sentences as an automaton. So whilst continuing to think about it, I could maybe push things forward by implementing sentence automata, albeit in a very naive manner where each tree gets its own automata.

Took an afternoon of coding. I factored my automaton code out of the Polarity stuff, had to fiddle a little bit with it to make it more general (i'm using Maybe to model null transitions), but for the most part the reuse is pretty good. For example, I didn't have to reimplement my automaton walking stuff. Just recycle!

What's nice about this is now I've solved that annoying problem of how to output sentences when some of the tree leaves have atomic disjunction in them. Just a transition for each lexical entry in the disjunction. This is pretty good timing, now that I'm using atomic disjunction to make synonymous trees together.

Only thing is that I slightly hacked up the GUI to ignore all this stuff. So will have to do some more coding later to use sentence automata in the graphical interface. But the core stuff is working. Yay regression testing! That's the third time today that regression testing (poor man's testing; just a shell script which runs GenI on a bunch of examples) has saved me today.

Well, using my thoughts from Thursday night and Friday afternoon, I have written a tutorial on monads with the space station and astronaut analogy as a reccurrent theme. It's not research, hey, but I've done really the bulk of it on my free time, the exception being the couple of hours I spent deepening my understanding of monads -- which could be useful for GenI anyway -- and testing my metaphor on Dmitry. I dunno. Hope the tutorial winds up being useful to somebody. Add another different perspective on things.

On than that, I tried to run Yannick's new lexical converter on the current tsnlp lexicon. But I ran into a bit of trouble and sort of got distracted debugging the thing, instead of thinking about something more important. Fixed now.

Ok, ok, so that's not what I'm supposed to be doing either, but hey, it's Sunday anyway...

More discussion with Bertrand about his "parse the polarity automaton" idea. I understand it better now that I've asked him to be super-explicit about how he constructs his associated state info and where. Much clearer, and obvious now that I understand better what he's saying. Now, parsing the automaton as he is suggesting allows us to produce sort of a derivation grammar which can then be read into a output automaton. If you want to really squish everything together, you can have a process which parses the polarity automaton and constructs the output automaton on the fly, but I'm not really sure you're gaining anything out this process. Way I figure, if you think of it as two seperate processes: producing a derivation grammar and unfolding it, then I think the conventional approach to producing the d-grammar probably works just as well as what he's suggesting, assuming the chart sharing stuff from GenI. But hey, lets try to implement it and see.

Sigh... the thing that's depressing about these conversations is never really feeling like you're adding anything useful. Just always struggling to catch up with yesterday's ideas.

I think now I understand how G.N. defines his equivalence classes (the trick is to understand that an item for him is essentially a root node and a list of substitution nodes, and that as the substitution nodes get plugged, the list gets smaller and smaller). So it's actually nothing fancy at all, probably the simplest definition you can get. But I wonder how on earth he avoids having a kazillion singleton equivalence classes during generation? Am I just being stupid?

Another thought: my definition of equivalence classes during adjunction is likely to be stupid because the whole point is to be able to use a backwards pointer to refer to a class (as opposed to an individual item), and that is clearly not something I'm accomplishing. Back to drawing board. Or I just kill this line of thinking and go do what Bertrand is suggesting, and maybe think about Schema TAG on the side; or maybe just do the simple thing and accept that we're going to have a kazillion singleton classes, which wouldn't be any worse than that we have now.

Oh yeah, and I beta tested my explanation of monads on Dmitry. Very cooperative. We've extended the space suit metaphor: the space suit in a State monad is a special suit that has some kind of card reader. You stick the card into the suit and it opens up, revealing an astronaut and a receipt (which could be used as a card for some other suit). What makes the metaphor awkward is that sometimes, astronauts can be cards, and cards can be suits (or something), but it was a useful way to anchor the thinking down in visual terms.

Edit 20h: I implemented my idea of merging synonyms into a single lexical entry using atomic disjunction. Seems to work. Only took an hour of hacking.

monads!

This is pretty much unrelated to what I'm supposed to be doing. But I've had so many friends complain either that they don't understand monads or that they don't understand them either. That set me to trying to figure out how exactly one goes about explaining monads, until I realised that I didn't understand them either. Now after thinking about things a little harder, I've come to a much clearer picture of things, and might have a fighting chance of explaining monads, cleanly and simply.

Problem is that I'm never clean and simple in my thinking. For starters, I so do not cope well with abstraction. Or rather, with unfamiliar concepts. I tend to rely very often on metaphor and mental image, something nice and concrete. So for me, what is useful is to think of things in terms of space stations and astronauts. I'll get to that later.

insights

1. Your space stations already pack their astronauts into space suits before sending them out into space. All functions being manipulated are of the form (\a -> suit) and the type (a -> mb). return for Maybe does \x->Just x, return for List does \x->[x], etc)
2. for the State monad: the space suit consists of a FUNCTION. That's what makes it all work! There is zero magic in all of this. The space suit is a function, and removing the space suit consists of calling the function on something.)
   
   The implementation of State's get probably looks something like this:
   

   get = \_ -> \st -> (st,st)

3. To understand the IO monad, it is useful just to focus on one aspect, say Output.
4. the Output monad is just a state monad where the state is a list. Putting something to the output consists only of appending something to the list
   
   The implementation of Output's putStr (if you forget about putChar) probably looks something like this:

   putStr x = \_ -> \st -> (st ++ x)

curriculum

I've also worked out a curiculum for understanding monads:

1. $
2. $ in "reverse" (i call it €)
3. parameterisable types
4. bind
5. the Maybe and List monad
6. return (up to here, you know everything about how it works, just not what it's for)
7. the State monad
8. the IO monad as a type of State monad

Final note: explaining the State monad's implementation of bind is a lot clearer if you use where instead of let:

bind suit fn = \st -> suit2 st2
  where (st2, a) = suit st -- here we remove the space suit by applying it!
        suit2 = fn a

amazing feats of nonproductivity

Finished reading that article from yesterday that had the Schema TAG in them. They guy gives a simple algorithm for translating a preexisting TAG into a Schema TAG, so it would be interesting to implement the thing and give it a try. Yes? No? Maybe. There's also some nice thoughts on NLG system architectures, basically a unified mechanism that allows you to represent any architecture you want, from the traditional pipeline to something a little more blackboardy. But that architecture stuff isn't really directly related to the job at hand, so skip.

Skimmed through a second article, Vijay-Shanker, Weir 1987. Would have taken too much effort to understand it and it wasn't really 100% neccesary at the moment, looking a lot more formal and big picture than what I need to be looking at right now. But there's also something very nice and pretty there, some generic mechanism for characterising a bunch of different formalisms and predicting what you can do with them. Or, err... something.

Had a thought: if synonyms is all I'm really worried about, then I don't really need tabulation. All I have to do is treat synonyms as an atomic disjunction on the lexeme. Voilà, there's your factorisation. Of course, tabulation is interesting to have itself because it could factorise other stuff, but if you only really care about the synonyms, you can pretty much get them for free.

Rest of the day down the toilet. What did I do? Fiddled around with the trac server because we were having some problems with permissions. Seems to be fixed. Made a rudimentary web page for GenI. Completely useless activity.

Looked at the G.N. stuff for all of 5 minutes.

Man where did all that time go?

ow! ow!

Ok, so I've been "helping" Bertrand think, which mostly means wasting his time by saying completely irrelevant stuff. But he's pushing forward, discovered some criteria for deciding -- based on the polarity automata -- if two bits of derivation grammar should be treated as a disjunction, or if they should be treated as two seperate rules with renaming to make sure they don't overlap. I could not keep up with him. Ow... ow... head hurting. The man seems to be on a roll, though. I'm not sure I see exactly what he's talking about yet. Wonder if it's just equivalent to this chart sharing stuff we already do. Anyway, so the holy grail there is "parse" the polarity automaton and produce directly the output sentence automaton without reconstructing lots of trees. Whatever's going on here, it's just highly highly factorised.

Other than that, I've started reading Integrated Natural Language Generation with Schema-Tree Adjoining Grammars. Schema-TAG seems like a neat idea. Each SU-TAG tree is a factorised representation of a little forest: one big messy tree, using at each node, regular expressions on Gorn addresses, to select which child nodes you want and which grandchildren. Starting talking with Yannick to see if it's the kind of thing that XMG could produce directly. It'd be sort of a third solution -- rather than use a chart, why not do the factorisation in the trees themselves?

Other than that, still looking at the Günter Neumann stuff...

short day

Read Schabes's An Earley-type parsing algorithm for Tree Adjoining Grammars just to make myself feel a little less uneasy. Thing is, I already (think I) understand Earley, maybe not very deeply, but enough to have made these slides a couple of years ago. So I dunno... I figured maybe I could see if stuff like feature structures needed anything special, but the paper seemed to sort of gloss over it. That or I'm just being illiterate. Set of unification equations on the DAGs. K. No need to do fancy stuff with your states, just do the feature stuff on the side.

Other than that. Bof. Admin crap.

Looked a little bit at Günter Neumann's PhD thesis trying to see what he does for equivalence classes. Slow going.

Did a smidgen of cleaning in the GenI source code, now using ifdefs to merge two files together.

Looked into some darcs stuff that Yannick had, no real trouble there. One false alarm, but turns out to be completely reasonable behaviour on darcs' part.

With my actions list completely dry, I have no choice but to turn to actual research. Well, I still haven't the faintest idea what that means, but here we go anyway:

Read Yasuharu Den's Generalized Chart Algorithm. Not sure what I got out of it. Vague notion that "abduction" is a form of educated guessing and that cost based abduction is educated guessing with weights. And also that head driven generation has several instances: left-corner parsing, semantic-head driven generation, and head-corner parsing. Oh and his parsing strategy is bottom-up, as opposed to Earley top-down.

Also skimmed through three other articles about paraphrase and reversible grammar. Didn't really understand them much, but don't think I really needed to.

Worked out a basic bottom-up algorithm which builds derivation trees and uses equivalence classes. I define my items differently according to phase and type, that is

1. Substitution phase
   • active edges - next subst node
   • passive edges - root node
2. Adjunction phase
   • active edges - next adjoinable node
   • passive edges - root+foot node

The rough idea is that every active edge has a payload. For each pairing between an active edge and a passive edge, we look at each payload item, and populate the agenda with a new set of active edges.

What's interesting about this is that in the substitution phase, most of the factorisation happens in the passive edges (eh... maybe), whereas in the adjunction phases, most of it happens in the active ones. And why go through all this trouble to define the items differently? Well, I figure that this reduces greatly the number of equivalence classes that end up getting generated. It, uh, increases factorisation. Or something.

Ran this through Bertrand. He thinks it's really the same thing as his automaton building approach. I dunno. He's given me an Nth explanation of the idea, and it's slowly starting to sink in. But all this explicit tree walking makes me nervous. All this state creation business. I get the feeling we'll going to be producing lots and lots of items. But the idea is cool and I definitely want to try it out. But I think i'll be doing two branches of GenI: a more conservative one where I try my idea (which is basically business as usual, but with derivation and elementary trees), and a whole new world one where I try out Bertrand's idea. Even if he thinks that it's more or less the same thing. Sort of a devil in the details thing.

not productive

Still sick! Morning running errands, seeing doctor.

Some minor admin bits.

Talked with Bertrand a very short while.

Wrote "automatic detection of semantic polarities" on a sheet of paper and stared at it for a while.

Helped Yannick in various bits and pieces.

Ran my trac+darcs admin scripts, reimporting GenI from scratch, just to keep things tidy. Also took the plunge, asked the sysadmins to open the server to the world. Eek.

day of coding

Implemented the handling of atomic disjunction in my polarity automata. It wasn't too painful. I just had to define an Interval type and use that to replace my Ints in the automaton code and elsewhere.

Yes, yes, I know I should be coding less and thinking more. But I figured, "hey, just one more task!" It was on my action list... promise, it was the last item!

Anyway, here's what my Interval code looks like. I hope it's not too retarded. Maybe i should define it as a datatype and make it instance Num and Show instead of just defining this weird stuff. But then... intervals aren't really Nums and trying implement some of the Num stuff seems silly. Show at least, then?

type Interval = (Int,Int)

(!+!) :: Interval -> Interval -> Interval
(!+!) (a1,a2) (b1,b2) = (a1+b1, a2+b2)

ival :: Int -> Interval
ival i = (i,i)

showInterval :: Interval -> String
showInterval (x,y) =
 let sign 0 = ""
     sign x = if x > 0 then "+" else "-"
     --
 in if (x==y) 
    then (sign x) ++ (show x) 
    else show (x,y)

Anyway, it seems to work, on my toy example, at least. Kind of annoys me though. I think this weakens the filter too much. That is, whenever we detect an atomic disjunction, say in a subst node, for every node, we declare the polarity to be an interval [-1,0]. This way we're saying "either we're asking for a FOO node, or we're not". Problem is that the atomic disjunction means "either we're asking for a FOO node or a BAR node" whereas I'm implementing "either we're asking for a FOO node, and either we're asking for a BAR node" independantly. It'd be nicer to have some kind of mutual exclusion or something. But then again, I doubt this atomic disjunction thing is going to play a big role in the grammar (who knows), so I'll accept this cruddy intervals implementation for the time being.

N.B. Now that I have intervals, it might be a good idea to revisit this page i threw together to the wiki and think about how we can exploit them:

Got to work in time for a late lunch. Helped Sébastien with some Unix trouble (looks like a dead hard drive). Did some administrative crap for a couple of hours.

Spent the later part of the afternoon talking with Bertrand about his idea of building an automaton from a packed derivation forest. He keeps improving his idea. Coming to a point where we might be able to do this directly from a set of elementary trees, or a packed set of sets using the polarity automaton chart-sharing idea to avoid combining certain items. One reassuring thing which made the whole bit easier to swallow is that the definition of an item in his construction algorithm consists of an entire elementary tree (features and all!), plus a node, plus the automaton states. He's going off on various ideas on how we can nudge and pack things in tighter ways.

Bertrand has also discovered a bunch of useful chart generation references on this survey of Human Language Technology. For a moment, my heart sank because there seemed to be all this stuff that I had not thought of myself to hunt down and read, but no, actually most of it is stuff that I've gone through before (at Claire's behest, though). Still got a few interesting articles to look into. Should try to really explore this whole chart generation thing fully.

Hmm... learned how to use getInput and setInput today. Yannick was trying to implement include directives, so it became useful to pick this up.

Didn't get much done today. A bit under the weather. Hacked a bit on gtester, trying to make the output a little more useful.

Bertrand has some of his derivation tree automaton outputting stuff implemented. I'm going to need to think about splitting the construction of derived and derivation trees (GenI produces derived trees with derivation trees being an afterthought) and your standard chart factorisations. Thinkthinkthink.

Encountered some bizarre bug in GenI, and haven't the slightest clue what to about it. If you use the -o flag to output sentences to one of the randomly generated gtester paths, GenI sometimes complains with something like this... I can't figure out what triggered this either... doesn't seem related to changes in grammar or gtester. The simple workaround is not to use the -o flag, but do not like this state of not having a clue what the hell is happening.

geni: runXMGselector.sh: executeFile: unsupported operation (Operation not supported)

geni: : commitAndReleaseBuffer: resource vanished (Broken pipe)

ton of minor things

Checked off a bunch of minor things I've been meaning to get around to. Code cleanupy deals, various bits and pieces I've been wanting to get around to for a while. Thanks to the simplified parsing stuff, I now I have a cleaner version of my code dependency graph:


Other than that... nothing. Now that I've got this code closet tidied out, I'll need to starting behaving like a real PhD student and a not a random coder monkey. Sigh. I code to forget. To forget that I really don't have any ideas. And by coding all the time, of course, I don't think up any fun new ideas, and so... and so I code to forget. :-)

parsec migration complete

Yeah, so now that my mental model of Parsec has been tweaked, I'm all good and happy with it again.
So the migration's all done, and I'm feeling rather relieved. Pleased with the result too.

A brief comparison of the code size (with most comments stripped)

Old code: 600 loc
New code: 330 loc

And that's how I like my code. Nice and tidy. Now, the new parser *is* subtly different from the old one, which makes this an ever so slightly unfair test, but I still think the rough savings of 270 lines of uncommented code is pretty telling.

Other than that, I'm still talking with Bertrand about his packed forest idea. He says he has a way of producing an automaton which represents all the realisations, that is, all of the sentences we output. I'll have to work on producing output in the form of a derivation grammar, and his module (that he says I'll likely want to recode anyway) will walk the chart/tree and turn it into an automaton, that is, a set of output sentences. Cool stuff. Won't have time to work on it for at least a month, but I really really feel stupid for not thinking to ask him about charts and generation earlier. Should have been head thumpingly obvious how under my nose he was. Good thing he spotted my lack of chartedness.

Fought a bit with Parsec today, but that was mostly because I hadn't fully absorbed the whole lookahead of 1 unless you use "try" thing. And using "try"... man, that's tricky! Embrace the tiny lookahead... and use 'try' sparingly. Starting to get an intuition for things now at least.

But yeah, all is not well in Parsec-using land. Not so much Parsec's fault, but me trying to break free of treating stuff as reserved words. Reserved words are a real pain when you're not trying to make a programming language but some simple natural lanugage stuff. Wouldn't that suck if you couldn't generate a sentence with the word "sentence" or "semantics" in it? Only problem is that once you try to break free of that prison, all hell breaks loose and you have to reintroduce some reserved-wordish behaviour, but in a more limited way. I guess it's just the price of the extra flexibility. Hopefully will get in solved before lunch tomorrow.

Talked some more with Bertrand (at his initiative). He'd been thinking some more about his highly factorised chart generation idea. Pretty cool: he's worked out a way to factorise not just the derivation tree stuff, but some means of producing all the output sentences without creating every single derived tree. A bit over my head, but I think once we start whipping through some code, it'll be a lot clearer for me. I'm not touching this stuff for at least a couple months, though. Got some polarities to take care of and some tidying I need to do in the code first. Not to mention all this TSNLP evaluation. Good stuff.

Only dissapointment is that this discussion did not come from me going and asking him about how to factorise the chart, but him observing that our chart stuff was not factorised. Need to get into this habit of picking people's brains about stuff they know more than me about. The whole advice taken and not given thing.

Edit: Oh yeah, forgot to mention: was playing around with Haddock today. Some simple makefile stuff to unlit all my source files and haddock it into a nice API. Problem is that unlitting my literal haskell code does not preserve comments, so I spent some time putting together a Perl script that would pass at least some of the comments through. Not done yet though.

Mostly taking the national holiday off, but for a little impulsive late night coding. Had manually converted all example lexicons and macros to use the new "?" notation for variables. Put together a Parsec implementation of the macros parser. As usual, much tidier than the Alex/Happy/helper code contraption I had beforehand. Except it does compile yet. But it's the sleepy time now (1h15 on 2 nov)

Talked to Bertrand this morning to see if he had an idea for using polarities during parse time instead of just lexical selection. Got a bit more than I bargained for. He didn't really have one -- just a variant of the filter I'm already using, but a little cruder and faster; however, he did notice that our chart generation algorithm was manipulating entire derived trees and pretty much on the spot worked out a more traditional push-the-dots-around algorithm to make better use of our chart. So we pretty much spent all day discussing that idea. I noted down bits of an example on the wiki, but we're going to have to document the inference rules later when he has some time. Now he also wants to prove that the algorithm is correct. This is pretty traditional stuff though, so we likely just have to trawl through the literature and find more or less the same thing. Maybe something like Shieber's Uniform Architecture deal.

Other than that, spent a few moments working on the Parsec migration. Realised that sometimes declaring something optional doesn't cut it. You have to use the try combinator to forget you parsed something, sort of a lookahead thing. Makes me wonder what good option is by itself then... Hmm...

awright!

With Claire's help laying things down and planning them out, we're finally moving forward on the ParI implementation. Our first job: get lexical selection working. Well, actually not so much forward, in that I've just given a short tour of the Geni.lhs module and Yannick's going to be off working on the lexical converter for some time. But I think things are just about ready to take off.

Other than that, I've been running errands today, so not especially productive. Started work migrating the GenI lexicon parser to Parsec. Pretty painless. And a good bit tidier.

gtester is operational

Man that was annoying. I gave up on trying to preserve some of the object oriented structure in the original code, smooshed everything into one object, no more trying to flit back and forth trying to be compatible with interfaces and crap lack that. Object oriented programming on an interpreted language is a big pain in the butt. But I'm not really a Python hacker, so what do I know?

Anyway, gtester is operational, in that it runs GenI, reads the results back in and then spits out a latex summary which it then compiles. I even added colours so that test cases which have at least one failure are highlighted in red.

Other than that, attended the weekly Seminaire TAL - the Calligramme folks talking about Leopar and the polarity stuff they're working on. Listening to them gives me a question to think about, though: is there any way we could make use of these polarities in the surface realisation phase proper... in TAG? That is, not just as a lexical selection filter, but as some means of guiding the tree combination loop? Apparantly, Bertrand has some ideas similar to this, so I'll have to go talk to him about it and see what comes out (and with the Calligramme people too, naturally)

unproductive as all hell today.

Helped Yannick install his shiny new iBook.

Fiddled around some more with Gtester. Got the new test case extractor thing running from the platform. Now working on parsing the results and printing everything into a nice LaTeX summary. Almost there, but oh, so BORED!

Part of the problem is that I'm really not a python programmer. Don't get me wrong, I still think Python has a nicer syntax than Perl, but I so much more used to the latter, and know my way around the combination of Perl quick reference guide on the web and the various Perl man pages. I still don't know the Python doc well enough enough to get that fingertip lookup feeling. So I'm all gummed up and handicapped, spending most of my time trying to find stuff.

Got to work in time for lunch today -- had an eye exam in the morning, after four years of never getting around to it :-)

With Dmitry's help, I brought Hélène's old iMac down from storage. Then I set Linux up on it and did an install of the trac+darcs server. This was because Djamé might be wanting to borrow the first Mac I had used, so I wanted to see how viable the iMac bubble would be as a server. Seems to work pretty well. It only has 7G of storage, so one possibility is to swap its disk out with puddle. Be interesting to see how much Linux likes being transplanted to some whole new computer. After all, I don't have any customised kernels or anything. Should work, right? Anyway, since I was doing the reinstallation, I was glad to have written my trac+darcs howto. Was just step by step, one command followed by mindless command. It also have be a chance to correct bugs in the howto. I'll still have to get around to building that project creation script though. Someday, when I'm really bored out of my wits.

Other than that, I've been hacking away at Gtester. I'm thinking of bringing back some of Carlos's original code that generated test cases randomly with hgen... Now, i'm not ambitious enough yet to think of this random generation stuff, I'm just going to preserve this structure of running some dedicated process to produce a nice test suite in which each test case is a self-contained subdirectory.

I was going to extend the test case extractor to do all this stuff, but suddenly I realised that previously, I had taken something out of my Happy code, and these missing bits were preventing me from getting the job done. Now... I could have spent like 15 minutes to rectify the situation, but instead, I decided "screw it".

I'd been yearning so long to throw out my messy Happy+Alex+helper code pile of BLEAH... so here was the chance! I took a three hour detour to reimplement the test suite parser in Parsec. 3 hours for that dinky little thing? Heck, yeah! I had to reimplement EVERYTHING: variables, feature structures, atomic disjunction, semantics. But I'm glad this is done. Much cleaner now. Moving the rest of the parsers over to Parsec should be finger in ze nose from here on because all the common stuff is done. This was also a good opportunity for me to switch to new convention. Before variables started with capital letters, and constants with lowercase. This was clearly getting impractical, so now, we ask for variables to be prefixed with a "?". Anyway, so that's all done.

The extractor now behaves as advertised: give it a test suite and a directory and it creates a subdirectory for every test case. Can't wait to polish off gtester, and then kill off the rest of my Happy/Alex/helper mess to celebrate.

working on gtester

Gtester is my name for the GenI tester forked off Carlos's utility for testing HyLoRes. The G currently stands for "GenI", but maybe it'll one day stand for "Generic" instead. Stripping lots of code away, took out the functionality that randomly creates test suites, and am now looking into having something which systematically runs the generator on all test cases.

I feel kinda silly doing this, but I made a small, standalone Haskell program using the GenI modules to just parse the test suite and spit out the names of the test cases. This way I don't have to write a parser.

It's a pretty boring job, so to keep myself entertained, I take breaks by writing shell scripts for the experimental darcs+trac server. Trying to build something which will let me create new projects at the snap of a finger. But i'll need to toss in some parameterisability so that people can either give me a URL to a pre-existing darcs repository, or just ask me to initialise one for them.

trac+darcs

Been working on getting a trac+darcs server up.

Here are the steps I took to get everything set up:
Edit 24 oct: Moved to the darcs wiki.

We're not completely done yet: still need to figure out if supporting multiple projects will be hard. I also installed the WebAdmin and AccountManager plugins, but i'm too tired to post the details right now.

I guess alll I need for GenI is to get my collaborator(s) to mail me their ssh public keys and I can get them to log on as the darcs user or something. But from a community perspective, it might be nice I pushed ahead and got some kind of sourceforge like environment running. Will think about this.

it works! it works!

Mac (Tiger) users can now download an unstable version of GenI as a .dmg at this temporary location.

I had to go crazy with otool, copying over all of graphviz's libraries and changing all the library name things to go with it, and now it works! Tested on both Dmitry and Guillaume's computer.

There's a little bit of ugly hard-coded crap that I'm going to need to finesse out, but now at least all the annoying bits of packaging GenI on Mac are solved.

Edit 18h10: And now... on to something useful:

Slightly modified my use of the XMG filter - before, I required the user to pass in the includes.mg stub file as an argument, but the problem is that this creates confusion (without filterXMG you pass in the .rec file, with filterXMG you pass in a .mg). So to make things a little more uniform you now pass in the name of the .rec file you would like to produce. This way, no matter what happens, you are passing in a .rec argument. This should make this a bit easier to use.

I am now working on building a testbed for GenI. Carlos has kindly donated the testbed that he uses for HyloRes, nice little chunk of Python code that produces latex reports and octave graphs with minimal hoops for the user to jump through. I'm going to strip his testbed down to only the bits that GenI needs. When we build back up to something more sophisticated, maybe I'll start thinking about turning it into something generic, but for now, I focus exclusively on making it useful for GenI.

more stuff I shouldn't be working on

Well i've found a workaround to the menu problem. I stick in a little bit of code to hide the window and the show it again:

       windowHide f
       windowShow f
       return ()

Also figured out how to really get the icons used. You have to call /Developer/Tools/SetFile -a C.

Other than that, I've had to do some ugly C and Haskell stuff in the process calling code just so that I can invoke graphviz from the application bundle. The idea is that Geni runs from geni.app/Contents/MacOS/ but graphviz lives in geni.app/Contents/Resources/bin. So I had to make my own version of getProgName which returns the full path used to launch GenI.

And so now it works!

Almost.

I tried it on Dmitry's computer, and for some reason it has trouble running graphviz... will have to look into this later. Agh, bad eric! Stop doing useless crap!

I know... I know...

I really ought to be working on something more important... it's a really bad habit, this sort of pointless tenacity. I've been spending all day working on making GenI distributable as a standard Mac application: open the disk image, drag the icon on to your hard drive and double click. Things I've "accomplished":

1. Made an icon for GenI:
2. Modified the Info.plist stuff from wxhaskell to have nicer values
3. Hacked up my own copy of wxhaskell's macosx-app stuff to copy over my icons and Info.plist
4. Created a script to copy dynamically-linked libraries over into a private framework (this QT help page was very useful!)
5. Created a script to create the .dmg file and copy the relevant bits over
6. Figured out how to make Finder use display the icon instead of the standard application stuff. It has to be called wxmac.icns; not even synchronising the name in your Info.plist helps)

The only thing left to do on this front is to fix this annoying menu problem, where when you first call the application, there are no menus whatsoever until you Cmd-Tab out and back. It seems to be the same issue as these guys (i also have the same symptom that running it out of the bundle is just fine), but none of my attempts to copy .rsrc files seem to yield any results.

config GUI done!

Yippee! Back to square one! :-)

So the thing I was fighting on Saturday was not Haskell lazy evaluation's fault. It was just a simple... uhm... race condition? I'm such a bad computer scientist. I can never keep my terminology straight. The reason why the main window was not getting updated is actually pretty simple and stupid: all the config window is throw some widgets on the screen and update an IORef whenever the user clicks Load. Nowhere, nowhere, nowhere do I say that the config window should return only when the user has clicked Load. So how do we solve this? Simple, pass a function to onLoad so that the params window updates the main window when the user clicks load. Simple callback.

Here's what the config GUI looks like now:

And the advanced stuff:

Another thing I've done is to strip away all the fancy test suite code, rearranged some of my putStrLn and tada! A much Unix-friendlier GenI that sends only sentences to stdout, without any of that superfluous crap, like status messages (all that to stderr).

So now to celebrate, I'm working on something completely useless, trying to make a GenI application bundle. It'd be nice to distribute GenI in .dmg for Mac people and give them something nice and double-clickable. Don't worry. I'll stop working on that soon enough... like tonight, even if not finished.

Edit: Whoops: I almost forgot: Yannick says that (1) producing m4 macros is a pain in the butt and (2) unordered parameters are really a bad idea. Since I'm making a conscious decision to trust his UI instincts more than mine, I'm going with it. I suggested instead a more explicit format where the user suppplies the semantics as a simple feature structure... which means now GenI will need to learn how to propagate variable values into semantic predicates. Shouldn't be too hard, should it? Famous last words?

Edit: Another thing I almost forgot: I've removed almost all mention of Common Grammar Manifesto from the GenI code and user interface. I've been thinking about this since Friday, the idea is that if I replace this with XMG, we can avoid a proliferation of excess terminology and acronyms, without really losing accuracy about what we're doing. Besides, telling the user he is doing XMG stuff is a lot more concrete/direct than refering to some obscure Manifesto. So I guess this is another reason I'm celebrating... one more annoyance removed.

Still working on that GUI. Getting really really sick of this by now. Trying to get the functionality implemented, but I seem to be getting bitten by the whole Haskell laziness thing. The idea is that I've got a main window and some staticTexts which tell the user what files he is using. Whenever you pop open this configGui I've been talking about these past few days and click load, what happens is that the staticTexts are supposed to be refreshed with whatever new information you put in. But they are not. The weird thing is that once you pop open the config thingy again, then the main window decides to refresh, which makes me think that there is somekind of bizarre laziness thing going on here.

Yannick pointed out last night that it's really not a good idea to have two file loading modes (the Common Grammar Manifesto stuff and the regular GenI stuff), and that if I really need to have these two mechanisms (which I still think I do), then instead of having a checkbox, I should have two separate GenIs. I stubbornly refused this last night, today, finally, I have given in, since after all:

1. the second GenI is really just a shell script or alias to geni --cgmformat
2. I can still keep the checkbox (just bury it in the advanced config stuff)

Shrug. I may be a Mac user, but that doesn't mean I have the slightest clue how to make a usable interface. We're slowly getting there, plodding our way into releasability.

Finally got a first draft of the file loading interface working. Writing this kind of code by hand sucks. It would be nice if I could plug a tool like InterfaceBuilder into wxHaskell.

The functionality isn't there yet, but at least the widgets are all in the right place. Here's what the basic options are:


And the advanced ones:


New keyboard shortcut I learned... when you do Command-Shift-4 to make a screenshot, hitting spacebar toggles the screenshot of window mode, which is much handier than selecting the thing with your mouse.

I actually shouldn't really be working on this right now... but it's been bugging me for a long time, and I'd really like to get rid of it. Plus we're in a relatively slow period, so I'm loosening the discipline a bit and doing these non-essentials even though I really ought to be hacking up test suite stuff instead.

Continuation of the discussion with Yannick. I'm starting to agree that yes, it should be possible to produce the macros using the XMG tools (here's the interesting bit: produce them directly from the metagrammar without writing any new stuff), but there is a big wide gaping issue of how to deal elegantly with multi-literal semantics, or how to compose two macros together in such a way that we get the sharing of stuff right.

Slightly improved the GenI debugger today. Now it tells you whenever it succeeds in lexically selecting an item but fails in anchoring said item to a tree. Before, it would just have looked like there was no lexical selection whatsoever for said item, and you'd puzzle over why it might be the case. I guess you'd still be puzzling, but it cuts the time in half.

Discovered also that our semantic representation for the Common Grammar Manifesto was not rich enough at all. Spent the morning and early afternoon with Claire working out a new way to represent the semantics. She had this idea of using macros, which we eventually hashed out into this proposal.

Another debate with Yannick: my thinking is that we could most easily implement this by manually writing the lexicon with the help of m4, but Yannick believes it's possible to automate even further and avoid having to hand-write the macros. I'm not really sure. The hunch is that we don't have enough information to work with, but I will concede that the tree semantics and lexical semantics do look a lot alike (in fact you want to instantiate the tree semantics from the lexical semantics, so of course they should look alike), so maybe there is some way to get this for free.