choo-choo / puff puff

Tiny bit of progress today. I have sort of settled into a stuck-state with the meow grammar (still seems a bit sensitive to order that I present the flat literals in).

Then I realised that my caledonian-light grammar doesn't actually use logical variables, so I went and implemented that in WASP. Seems to work, ok. I even have the hinted grammar working and trained.

generating paired input/sentences

I came to my senses about trying to use pCRU to generate the set of possible MRs (i was doing something really hacky and complicated). So now instead of trying to generate English and MR texts independently and then try to align them by grammar-hacking, I have a simple Haskell script that dumps out the list of possible pairs via list comprehensions (probably would be just as easy to write this in Python). Having this made life a lot easier, especially when I wanted to generate both hinted and unhinted sentence/input pairs:

combos = map withHint cs ++ map sansHint cs
 where
  cs = [ C lse (":leaves(" ++ ts ++ "," ++ ds ++ ")") (unwords [te,lse,de])
       | lse <- [ "leaves" , "departs" ]
       , (ds,de) <- departures
       , (ts,te) <- trains
       ]
  sansHint (C _ s e) = (s,e)
  withHint (C h s e) = ("lex(leaves,"++h++")"++s,e)

Yeah it's probably not very readable, but that's what you get in quickie get-the-job-done scripts.

more notes

SCFG init rules are really useful for preventing the lexical acquisition module from learning the wrong thing. Before, it was learning things like:

TIME -> in the morning and six ; 1800 

because it only saw examples like "leaves at 10 in the morning and 6 in the evening", i.e. where the first time was in the morning. If you gave it an input like times(1300, 1800), you would get the wrong output ("leaves at 1 in the morning and 6 in the evening" instead of the expected "leaves at 1 in the evening and 6 in the morning" [yeah, should be afternoon, but nevermind that for now]). This was pretty easy to fix. You just feed an initial rule like

TIME -> one in the evening ; 1300

to guide it towards a more reasonable result. In fact, I had to do something like this just to prevent the system from learning completely nonsensical things like:

TIME -> london ; 1800

Hmm. Maybe just having MOAR DATA would fix it too.

Another thought is that if I recall correctly, everything has to be aligned to something. You can soften this a bit by putting in zero-fertility constraints on the rules. But you occasionally want to just elide a whole bunch of stuff (but not always). Seems like this could be tricky.

closer inspection?

I kinda have a suspicion that stuff like this might be sneaking through in NLG evaluations... but I dunno. Maybe people are just really careful to ensure that systems are really producing output that communicates the correct things (and not just sounds like English / is readable). In the above example, 1 in the morning and 1 in the evening are, err... quite different. Another interesting thing you can say about this is that this is something you can definitely not fix with type-checking the parse output (if you were doing parsing and not generation)

sumtime

I've started working on generating weather reports. I've got a toy MRL grammar that can be trained on a single pair (and several variations). Next I'm going to do a little bit of scripting to convert the data from AB's scripts into the format that WASP (and my grammar) wants. Should be pretty straightforward. No logical variables here, still very exploratory stuff. I'm still generating segments here, and not full reports. Start small.

another problem/solution pair

Problem:

Exception in thread "main" java.lang.RuntimeException
 at wasp.nl.SRINgramModel.exec(SRINgramModel.java:184)
 at wasp.nl.SRINgramModel.train(SRINgramModel.java:128)

Solution: not sure. I simplified my MRL grammar an it went away

another sadurday

Well, I thought I might come in this morning to make up for Monday's sleepy non-work. But I seem to be rebelling against the idea. Hmm. Took me till noon to start working. Well, I'm gonna call it a day. That's 2 measly hours, whoopee, but I really really do not feel like being here. Plus there is the weekly grocery shopping to do.

I'm trying to use the associative-commutative feature, but trying to use it gives me a null pointer exception.

Exception in thread "main" java.lang.NullPointerException
 at wasp.scfg.RuleExtractor.edgeWeight(RuleExtractor.java:483)
 at wasp.scfg.RuleExtractor.transformAC(RuleExtractor.java:401)
 at wasp.scfg.RuleExtractor.transformAC(RuleExtractor.java:352)

Not using it gives me the "interesting" results I mentioned yesterday.

After studying the geo-prolog grammar a little closer, the solution seems to be to use some sort of start wrapper. Instead of (def(X),cat(X),black(X)) you need to write start(X,(def(X),cat(X),black(X)). Will have to ask YWW why.

Other thoughts:

• custom tokeniser probably not used for parsing the mrl-grammar [same tokeniser for that], but for the mrl expressions in the corpus (duh)
• you can write (A,B,C,D) for the ac rules, even though the syntax would lead you to believe you have to write (A,(B,(C,D)))... of course, the 1st way is nicer especially since we're talking about associativity here

Another thing I'm worried about is that I can generate from black(X), cat(X) (+ start) but not from cat(X), black(X). Hmmph. Don't know if I just need a bigger corpus here or if I'm doing something wrong...

less flat

For some reason, I'm thinking we can unflatten the input a little bit. Not sure why I feel it may be a good idea. Thinking we can have something like meow(E,X,(cat(X),meow(X),def(X),black(X))

Interesting thing about commutativity problem above. It can't generate from cat(X), black(X), but it can generate from meow(X,cat(X),black(X)). Huh

more playing with WASP

The next most obvious thing to do with my meow grammar, before proceeding to the Caledonian domain and eventually the weather reporting one is to see if I can make use of logical variables

I'm stunned by how often I fall into the trap of starting too big whenever I'm trying out some new software. Stop trying to generate big(x) cat(x) if you don't know how to generate cat(x) yet, sheesh!

Also, weird today, somewhere around 17:00, I just a stopping point. No more real work possible. So, emptied out by box-o-actions, and did some more reading of SH thesis draft.

more notes

1. todo: would like for relative paths in config files to be treated as relative to the config file; this is how GenI did it back in the day and it really simplified having self-contained grammar packages (you can do cp -R)
2. You can get null pointer exceptions in the SRI n gram stuff if you have less than three examples

3. Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 3
    at wasp.mrl.MRLGrammar.addProduction(MRLGrammar.java:221)
    at wasp.mrl.MRLGrammar.read(MRLGrammar.java:372)

   Solution:Was not using the right grammar format in my config file
4. No rules learned
   Solution:Was not using the right tokeniser (had copied and pasted code from a different grammar)
5. I don't know how to get the ac modifier used in the geo-prolog grammar working for meow. It is probably related to the issues below (I'm guessing ac stands for associative-commutative)

slight flatness

There is something I never figured out how to do without flatness, which is to refer to things more than once. For example, if I want to say "the big white cat", the only obvious way that comes to mind is to have a logical variable x which I can point to with both white and big. Now, we may not need to have these, but it looks like whatever input representation we come up with is going to want at least a little flatness in it. I'm thinking a simple discourse tree structure, but within each clause you get a list of literals.

grammar design

This is related to the chop-the-onion problem I was slightly worried about. Given the current alignment algorithm, the MRL grammar needs to be designed with a little bit of care. You don't want rules like

*n:Statement -> ({ *n:Creature })
*n:Statement -> ({ *n:Action })
*n:Statement -> ({ *n:Def  })
*n:Statement -> ({ *n:Modifier })
*n:Statement -> ({ *n:Statement, *n:Statement })

where all the recursion is packed into one rule. The problem is that the SCFG grammar doesn't have any scaffolding to build a notion of English syntax on top. It can't understand what the relative position of modifiers wrt creatures is because when it learns about the realisation of two adjacent literals in the input, it doesn't actually know anything about the literals.

The result is that if you give it inputs like def(Y),white(Y),chicken(Y),meow(Y), you get garbage like "white the chicken meows". A better way to write the grammar is distribute the recursive bits over the results:

*n:Statement -> ({ *n:Creature , *n:Statement })
*n:Statement -> ({ *n:Action , *n:Statement })
*n:Statement -> ({ *n:Def , *n:Statement  })
*n:Statement -> ({ *n:Modifier , *n:Statement })
*n:Statement -> ({ stop })

Come to think of it, I am no longer 100% certain I understand why this works better. Need to remember that the chart generation algorithm knows how to re-order inputs, so in principle, the order that I write things in the test corpus should not make a difference. How I write them in the training corpus may be another story. Will need to refer to the alignment algorithm again.

"interesting"

I've been training the thing on single sentences. For laughs, I decided to see what would happen if I gave it some odd input: bark(Y),dog(Y),meow(X),cat(X),stop (the odd thing here being that I'm trying to say two sentences at once). The result of this is something like: the dog the cat meows barks. Actually hmm... if I write that input in another order, I get a different output (which should not be the case, no?): the cat the dog meows barks. Anyway, you can't really blame the system for coming up with this (in a way, it's better than nothing, right?) since I never showed it anything like it. But it was amusing to see what would happen.

this weekend

I think it's time for me to have a meeting about this, get debriefed. Otherwise, this weekend [if I'm a good boy] I might start working on the Caledonian light grammar. If I get some results out of there, maybe start throwing in some hints to see what we can do. Need to also keep in mind that this is very exploratory, no real results yet, because I'm just working with toy grammars and toy data. Mainly doing this so that I can learn to use the tools, not so much to understand what their true capabilities are.

Oh and before I do that, I should probably look into that ac modifier some more

hypertagging

I forgot I had finished the CCG hypertagging paper on Wednesday... and now have forgotten what it said. Heh. They use hypertagging to mean supertagging from the logical form (which I guess does not mean the same thing as what the TAG people say). Use some templates to slap logical forms on to CCGBank parses. Use the context of LF (i.e. fancier than ngrams in parsing) to figure out what the elementary tree should be. Faster and better results (or better results, because the right tree bubbles up top and faster because you can prune with more confidence). I hope I got the gist anyway.

fun with wasp

Back from London, where I saw some nice museums and a LondonHUG talk [ideas from this; building your own little type system for the DSEL]. Slow and late start today. Tricky to get started with WASP despite the very high quality documentation. Basic problem is a general lack of robustness wrt dumb users, i.e. NullPointerExceptions instead of something more informative. Hey, that's research. Get the job done and worry about the niceties later. Another annoyance was that you have to write code if you want to have your own application domain. For now, I just copied the Robocup CLANG grammar java stuff.

boring stuff

Embarassed to admit that I was hampered by a lack of a build system. At first I tried mmake to generate a Makefile for me... but mmake does not appear to support src dirs. So I dusted off kowey-generic and installed myself an Ant template. Still sorta works. One major flaw, I realise now that it's been long enough for me to have to re-learn my own tools is that I never made it clear how to parameterise a kowey-generic Java project (you have to edit etc/make/{version,config}). I guess I must have done it so many times in my Java days [5 years ago!] that it became natural. Yeah, so kowey-generic/Java is still revelant to my needs after all. Will just need to get rid of the Ant jar files and figure out how to make JunitDoclet work again someday.

semi-boring stuff

• mrl-grammar parser does not like whitespace (null pointer exception)
• you do not really need an scfg-init-rules (blank one will do), afaik
• you do not need types
  

fun stuff!

Even the Caledonian grammar was too complicated for an initial run with WASP. So I backed off to an even simpler domain, which I call "meow" [this is based off a CRU example].

Keep in mind that as far as the corpus is concerned, there is no difference between English and Engwish. It's just an eric-distinction. Also, I'm not really trying to accomplish anything here, other than get a "feel" for WASP.

meow	English	Engwish
meow(cat)	the cat meows	meowth the cat
meow(dog)	the dog meows	meowth the dog
meow(chicken)	the chicken meows	meowth the chicken
meow(cow)	the cow meows	meowth the cow
bark(cat)
bark(dog)	the dog barks	barketh the dog
bark(chicken)		barketh the chicken
bark(cow)
cluck(cat)		clucketh the cat
cluck(dog)
cluck(chicken)	the chicken clucks	clucketh the chicken
cluck(cow)	the cow clucks	clucketh the cow
moo(cat)	the cat moos	mooth the cat
moo(dog)	the dog moos	mooth the dog
moo(chicken)
moo(cow)	the cow moos

Put another way:

	cat	dog	chicken	cow
meow	l w	l w	l w	l w
bark		l w	w
cluck	w		l w	l w
moo	l w	l w		l

Hmm... if I had sat down and thought about this, I might have made this prettier. Oh well. Note that there is one more Engwish sentence here than English one. After training WASP on this, I then ran the generator on some of the squares to see what it returns (notably the red ones). The results:

• meow(cat) [+lish, +wish] meowth the cat
• moo(cow) [+lish, -wish] the cow moos
• bark(chicken) [-lish, +wish] barketh the chicken
• cluck(dog) [-lish, -wish] clucketh the dog

Well, I don't know what to make of this, but it was amusing enough for me of three year old mind. Oh and also

• moo(chicken) [more lish] mooeth the chicken
• bark(cat) [more wish] barketh the cat

My guess for why you get 'mooeth the chicken' even though there's more English moo is that there are more Engwish chickens.

scfg rules

Oh and here are the rules it learns from this:

*n:Action -> ({ barketh })({ bark }) phr-prob-e|f -0.40546510810816444 lex-weight-e|f -0.40546510810816444
*n:Action -> ({ barks })({ bark }) phr-prob-e|f -1.0986122886681098 lex-weight-e|f -1.0986122886681098
*n:Action -> ({ clucketh })({ cluck }) phr-prob-e|f -0.5108256237659907 lex-weight-e|f -0.5108256237659907
*n:Action -> ({ clucks })({ cluck }) phr-prob-e|f -0.916290731874155 lex-weight-e|f -0.916290731874155
*n:Action -> ({ meows })({ meow }) phr-prob-e|f -0.6931471805599453 lex-weight-e|f -0.6931471805599453
*n:Action -> ({ meowth })({ meow }) phr-prob-e|f -0.6931471805599453 lex-weight-e|f -0.6931471805599453
*n:Action -> ({ moos })({ moo }) phr-prob-e|f -0.5108256237659907 lex-weight-e|f -0.5108256237659907
*n:Action -> ({ mooth })({ moo }) phr-prob-e|f -0.916290731874155 lex-weight-e|f -0.916290731874155
*n:Creature -> ({ cat })({ cat })
*n:Creature -> ({ chicken })({ chicken })
*n:Creature -> ({ cow })({ cow })
*n:Creature -> ({ dog })({ dog })
*n:Statement -> ({ *t:Bound barketh the *n:Creature#1 *t:Bound })({ bark ( *n:Creature#1 ) }) phr-prob-e|f -0.40546510810816444 lex-weight-e|f -2.315007612992603
*n:Statement -> ({ *t:Bound clucketh the *n:Creature#1 *t:Bound })({ cluck ( *n:Creature#1 ) }) phr-prob-e|f -0.5108256237659907 lex-weight-e|f -2.4203681286504293
*n:Statement -> ({ *t:Bound meowth the *n:Creature#1 *t:Bound })({ meow ( *n:Creature#1 ) }) phr-prob-e|f -0.6931471805599453 lex-weight-e|f -2.6026896854443837
*n:Statement -> ({ *t:Bound mooth the *n:Creature#1 *t:Bound })({ moo ( *n:Creature#1 ) }) phr-prob-e|f -0.916290731874155 lex-weight-e|f -2.825833236758594
*n:Statement -> ({ *t:Bound *n:Action#1 the cat *t:Bound })({ *n:Action#1 ( cat ) }) phr-prob-e|f -0.5108256237659907 lex-weight-e|f -1.9095425048844386
*n:Statement -> ({ *t:Bound *n:Action#1 the chicken *t:Bound })({ *n:Action#1 ( chicken ) }) phr-prob-e|f -0.5108256237659907 lex-weight-e|f -1.9095425048844386
*n:Statement -> ({ *t:Bound *n:Action#1 the cow *t:Bound })({ *n:Action#1 ( cow ) }) phr-prob-e|f -0.916290731874155 lex-weight-e|f -1.9095425048844386
*n:Statement -> ({ *t:Bound *n:Action#1 the dog *t:Bound })({ *n:Action#1 ( dog ) }) phr-prob-e|f -0.6931471805599453 lex-weight-e|f -1.9095425048844386
*n:Statement -> ({ *t:Bound *n:Action#1 the *n:Creature#2 *t:Bound })({ *n:Action#1 ( *n:Creature#2 ) }) phr-prob-e|f -0.6466271649250525 lex-weight-e|f -1.9095425048844386
*n:Statement -> ({ *t:Bound the cat *n:Action#1 *t:Bound })({ *n:Action#1 ( cat ) }) phr-prob-e|f -0.916290731874155 lex-weight-e|f -1.9095425048844386
*n:Statement -> ({ *t:Bound the chicken *n:Action#1 *t:Bound })({ *n:Action#1 ( chicken ) }) phr-prob-e|f -0.916290731874155 lex-weight-e|f -1.9095425048844386
*n:Statement -> ({ *t:Bound the cow *n:Action#1 *t:Bound })({ *n:Action#1 ( cow ) }) phr-prob-e|f -0.5108256237659907 lex-weight-e|f -1.9095425048844386
*n:Statement -> ({ *t:Bound the dog *n:Action#1 *t:Bound })({ *n:Action#1 ( dog ) }) phr-prob-e|f -0.6931471805599453 lex-weight-e|f -1.9095425048844386
*n:Statement -> ({ *t:Bound the *n:Creature#1 barks *t:Bound })({ bark ( *n:Creature#1 ) }) phr-prob-e|f -1.0986122886681098 lex-weight-e|f -3.0081547935525483
*n:Statement -> ({ *t:Bound the *n:Creature#1 clucks *t:Bound })({ cluck ( *n:Creature#1 ) }) phr-prob-e|f -0.916290731874155 lex-weight-e|f -2.8258332367585934
*n:Statement -> ({ *t:Bound the *n:Creature#1 meows *t:Bound })({ meow ( *n:Creature#1 ) }) phr-prob-e|f -0.6931471805599453 lex-weight-e|f -2.6026896854443837
*n:Statement -> ({ *t:Bound the *n:Creature#1 moos *t:Bound })({ moo ( *n:Creature#1 ) }) phr-prob-e|f -0.5108256237659907 lex-weight-e|f -2.4203681286504293
*n:Statement -> ({ *t:Bound the *n:Creature#2 *n:Action#1 *t:Bound })({ *n:Action#1 ( *n:Creature#2 ) }) phr-prob-e|f -0.7419373447293773 lex-weight-e|f -1.9095425048844386

tupak

Still sort of slow and sleepy (although feeling much better). Heading to London tomorrow for Haskell user's group.

This morning, after corresponding, I got WASP generation working, with JW's help. Yay!

After a morning and early afternoon of just... being...stupid (and not even procrastinating! I have seriously no idea where the time went), I finally worked out that I was supposed to be looking at the my caledonian light grammar.

Took a while to tweak the CRU rules so that the inputs came out in the right order (using the dumb erician alignment algorithm called "don't even try"). Now facing some WASP NullPointerExceptions parsing my grammar file.

Sigh...

At least it's sort of clear what I'm doing Thursday, then. Hint to self: figure out what is wrong with my minimal WASP mrl-grammar. If I get that going, then I can start experimenting with my hinted grammar.

kaput

Just too sleepy today to get anything done. I'm going to go into town and just do nothing for a while. Taking Wednesday off too, so not a productive week. Will have to make up for it this weekend.

Finished the disjunctive input over the weekend and started reading the "hypertagging" one. The mechanism for tracking alternatives in the chart is the same as the one we use for chart sharing over polarity automaton paths [which kinda makes sense in a way]. This reminded me that if you wanted to keep polarity filtering working with disjunctive inputs... well... you might want to think of a way to do it. Should be doable, just gotta sit down and work it out for a week.

churn churn

Well, I haven't produced much this week... but I think I've tried hard enough that I can take the whole weekend off.

researcher dimensions

Idea I've been kicking around are key parameters for a researcher

personality

attitudes, tastes, temperament

ability

intelligence, creativity, aptitudes (classes of things you're likely to be good at)

knowledge

skills (e.g. coding), breadth (needed for cross fertilisation), depth (needed for follow-through), self-knowledge

methodology

reading,publishing,"reacting",talking with people

The basic point is that you can more easily control the latter two than the first two.

Put another way, personality and ability are irrelevant... sure maybe they will help to distinguish the grunts from the superstars. But hey, look, I don't want to be a superstar. I just want to figure out how to do my job, (a) to make sure I continue to have a [useful and non-boring] job and (b) to reassure myself that my keep is being earnt. Since I am merely trying to figure out how to be your standard run-of-the-mill research grunt, what I need to be focusing on are knowledge and methodology.

No amount of dual n-back is going to fix Eric-is-dumb, so don't worry about it. What you have to do is (i) make sure he knows as much different shit as humanly possible and knows it well (ii) change the way he does things. An example of poor methodology is sitting around writing blog posts, or sitting around worrying that you're doing it all wrong instead of doing it (even if all wrong) [to be fair, the "doing it all wrong" consists of sitting around a lot]

hinted caledonian

I've been working on my hinted Caledonian MR grammar today. It occurred to me just this minute that I don't actually need logical variables here (so maybe I could go play with WASP 1.0 for a while). There may be some other domain where they come in handy, though. Making a little bit of progress

for monday

finish off CRU port of -light grammar. Make an aligned corpus. Try training WASP 1.0 on light grammar and see what happens.

parsing model / language model

Another entry for the Handbook of Simple Answers: what's the difference between a parsing model and a language model?

Lifted straight from p 415 of M+S (*)

parsing model

P(t|s,G) where SUM_t(P(t|s,G)=1

language model

P(t|G) where SUM_t(P(t|G)) = 1

The parsing model allows you to discriminate between different parses for a given sentence. The language model tells you only what the likelihood of any given tree is in the whole grammar. Conversion from language model to parsing model is easy [divide joint prob by prob of sentence], but not the other way around.

(*) This is a case of not keep enough things consistent across comparisons. In M+S they drop the (|G) for the language model [why, just to keep things looking nicer?], but that makes it hard for the dumb reader to understand the relationship between parsing and language model. Maybe there is a reason to do this, but it seems like this kind of inconsistent simplification is needlessly confusing.

M+S

Finished up the parsing chapter (ch12.2 up), although I don't think I absorbed anything from it [again, me being thick]. My reading of it was along the lines of blah blah blah model blah blah blah blah dependency blah blah blah Collins [woof!]. So, state of the art a decade ago was Collins 1997, dependency based parser with bits of linguistics creeping in, under 90% precision and recall... I wonder how the parsing community has been doing since. Note the

Qualitative breakthroughs may well require semantically richer lexical resources and probabilistic models

Seems also like the question of how to evaluate parsers should be poked at some more. Dependency grammar also looks like a worthwhile thing to explore (huh, is what that the recent flurry is about? here's me living under my rock)... Oh and the bigger picture of why we want to do parsing in the first place, presumably so that we can build meaning representations.

λ-SCFG

Argh! I'm confused again.

Another thing is that they claim they really are doing the lambda calculus, and not just a custom lambda-inspired variable binding mechanism... which throws up all my discomfirts about renaming free variables. Ok, fine, whatever, we just gotta trust people; besides I don't know nothing about lambdas [I'm just a user]

The confusion comes from actually trying to write a MR grammar. Need to remember that we don't use the lambdas here; it's just used for propagating logical variables correctly in the synchronous stuff. Just not sure about where the lambdas come from though. Also tend to mix up the actual tree I'm trying to build, with the derivation tree of the tree.

Oh, and here's one kind of thing I can get easily confused by:

answer(x1,
(λx1.smallest(x2,(
     (λx1.state(x1))(x1),
     (λx1.λx2.area(x1,x2))(x1,x2)
)))(x1)
)

Let's rewrite this so that somebody like me can understand it

answer(x1,
(λa1.smallest(x2,(
     (λb1.state(b1))(a1),
     (λb1.λb2.area(b1,b2))(a1,x2)
)))(x1)
)

Yeah I can understand deliberately shadowing that x1 to communicate the intention (it's the same x1 in the MR), but that just trips up people like me. Back to the original question. I understand that the MR is lambda-less... but how do we decide to attach lambdas (and arguments) to non-terminals when we're building up the SCFG? The answer is right there in the thesis (and highlighted even). A variable must be bound if it appears both inside the rule and outside of it. This way we can pass it down from the outside... make sure we get really the same variable. In that light, the only "free" variables are local ones, i.e., in the sense that given the current MR parse tree, you will never see that variable again.

... but when you actually try to use the SCFG, you may combine that rule with other rules that use the same variable name. Uh-oh. I guess then this is where you have this free-variable renaming thing. Because the rule might be used to construct whole new parse trees, and you really do have the possibility of wanting to substitute one of your bound variables with something which just so happens to be the same as your free variable.

I shouldn't be thinking of those free variables as globals, but as local lets. I wonder if there is a word for this not-really-free-variables. In my new hazy understanding of it, the lambda terms in an SCFG rule really are closures after all (if you think of those free variables like let..in). Seems like you'd want to introduce a new binding thingy, ι (iota) just to keep such variables unique. You don't do anything with ιs except alpha-convert when the need arises. At the end of the day, you just drop any ιs in your expression. But then again, I guess this kind of machinery really just amounts to their "you must rename your free variables" rule, so why bother? Only reason I'm resisting it is instinct, a spluttering protest, "but, but, you can't do that!" that only arises because you've never seen it done.

The core problem is that I don't actually know about lambda calculus. I've seen it used for natural language semantics, so I have a non-rigorous mechanical view of things (and now the alligator eats this family, and the egg hatches), and I've done some functional programming (which is why I talk in terms of globals and lets). But I don't actually know this stuff. Anyway, it is purely an angels on pinheads discussions. Not practical. Doesn't help me use the stuff.

fear in a high chair

maybe I just suck at this

I'm doing this all wrong. I keep telling myself that I'm doing this all wrong. I feel like I should doing more stuff, picking a direction and running with it. Instead, I just try to read things (which I don't understand, which just makes me fall asleep), or I spend a lot of time doodling, slash pacing the hall. Doodling and "thinking" is fine, as long as you got something to think with. All fire and no fuel.

It's not that I'm not trying. I'm just doing it all wrong. There is a difference.

At least I'm starting to put this stuff down in computer [in mindmap form; harness the chaos], so that I go around in circles less (less likely to forget what I just came across two days ago).

Furthermore, the Eric-is-dumb problem is not the point. Whatever my limitations are, I should be learning to work around them. If your brain is a lemon, make lemonaaaaaaaaaaade. It's all methodology, custom methodology. Yep.

underspecification for NLG

It occurred to me that I should go see what other people how other people use this notion of underspecification in NLG. Found a couple of papers, one on incremental generation (it uses this thing called Constraint Language for Lambda Structures), and then another thing by some folks in Italy that says something about mixing template based and NLG-based generation in a more principled fashion. I couldn't really read the articles very well, brain too foggy, rate of information absorption too low.

M+S

learning more about parsing (ch12) 12.1 is nice in a way, short bursts of info-nuggets. Was interested to learn about the PARSEVAL metric (which reminds me of some objections we had to n-gram based evaluation metrics; could something like this be applied? watch for easiness-to-fool though... or is that PTB related?). Grateful to be nudged into not thinking that PCFGs are the only way to add probabilities to a CFG (for example, p left-corner grammars [good thing I made the effort to understand that left-corner parsing stuff a couple of years ago]). Had a brief taste of dep parsing, very brief.

rtg

When searching for the above, I accidentally stumbled on some very fresh seminar slides (23 may 2008) on Regular tree grammars as an underspecification formalism [A. Koller]. I wasn't really paying attention (again more of this inability to absorb information quickly), but I sort of get this idea of eliminating redundant/identical unpackings from a packed structure (i.e. before you actually do the unpacking)... and saw some impressive looking performance estimates (e.g. something which should take billions of years would only take 75 seconds). Sounds like a pretty cool idea, another variant of, eliminating combinatorial blow-up via exact [non-lossy] methods by getting rid of the stupid stuff. [which is makes the polarities stuff cool; thanks, G!]. Or maybe I didn't actually understand the slides. But it may be good to look into this. Downloaded a Graehl and Knight paper called Training Tree Transducers which looks pretty incomprehensible to me (in the Eric-is-dumb sense)

underspecification bis

Still banging my head against the wall about this one. What the heck can I say about this? It occurred to me maybe that I am thinking about it completely the wrong way (no shit). Instead of saying "how do we introduce a notion of underspecification into our semantic inputs [which would work with the SCFG stuff] so as to permit saying things in more than one way", I should instead be working backwards, saying "what is the absolute minimum that goes into a message, and then how would we embellish it to guide the generator?"

For the trains time table, and just the simple examples we've seen, it looks like we can't say anything less than:

leave(train_1, city_1, times(t1,t2,t3))

Oh oops. I've implicitly aggregated the times (what if you want to print multiple slices, say tell about different cities). Would it be more something like

leave(train_1,city_1,t1), leave(train_1,city_1,t2), leave(train_1,city_1,t3)

? Hmm, ok, confusing myself. Maybe we should just make the assumption that pre-aggregated units are inviolable and that is that. In other words, if the input specification lumps something together, you can't split it up! (On the other hand, you could lump together other stuff if you wanted)
Idea then would be that you could add hints to bits of the message with extra information to guide the generator:

leave:[lex=depart](train_1:[ref=pronoun],city_1,times(t1,t2,t3))

Still not sure what to do about multi-sentential generation. We should really just have a philosophy of keeping things simple and stupid [because we don't understand all the literature out there]. Assume discourse is a tree, the leaves of which correspond to individual messages. Discourse relations are binary. No textual order implied [learned from context]. Still have no idea about what an appropriate inventory of relations should be, so I propose the make-it-up-as-we-go-along strategy (frantically flipping through related work to see what exists). Discourse relations can also be annotated in the same fashion with guiding information.

It's conceivable that this [the hints idea] could be made to work with the SCFG stuff. In its lexical acquisition phase, we would pick up the link between hints and specific connectives. Would also perform acquisition over un-hinted inputs, paired with same texts.

I'm also worried about the referring expressions. If we've got some entity to refer to, and it has some attributes (say, the big red onion), how do we make repeated references to it without having to duplicate all the attributes [baroque solution: go ahead and duplicate the attributes]. To be clear, we do not necessarily have to think about generating distinguishing descriptions [that would require more fancy tech], but we do want to be able to refer to it more than one way. Well, actually, what do we want really? We could just be happy with being able to discriminate between pronoun, common-noun phrase (the onion), and proper-name. We could treat the fully specified thing (the big red onion) the same as a proper name. We could maybe have a separate lookup table that maps from, say common-noun-phrase(x1) to "the onion" and pronoun(x1) to "it". We could then say that it's not the job of the probabilistic component to learn these. Just plug in templates like _COMMON-NOUN(x1)_ and let them be filled in with post-processing. All we do then is learn the distinction between types of expressions.

I have no idea what I'm saying. No justification for this, no prior research, no specific questions which lead me to think that this makes any sort of sense. Just making this shit up. It sort of sounds like an idea [likely a rehash of something I saw somewhere else]. Question is: how can I run with this? Could I write a MR grammar for a hinted input?

fear returns! (maybe)

Another day of not actually doing very much.

I understand vaguely that "thinking about stuff" may not be the most productive way to think about stuff, and that smarter people know to think about stuff by "doing things" that guide the thinking process, for example by submitting papers to conferences so as to force oneself to familiarise yourself with one bit of problem X. But I'm still stuck in that useless useless mode of furrowing my brow, pacing around a lot and just going huh?

Yeah. I spent today thinking [*]. What good is that?
* oh, and upgrading GHC, and getting the HTTP module to work with my proxy server, and making cabal-install deal with those changes

freemind

This morning, it occurred to me that I should try some mind-mapping software for Linux. Turns out that there is this thing called Freemind (a Java application, actually). The thing works surprisingly well. There is one annoying bug (?) where it stops responding to keyboard input periodically, but I've found a workaround for that (just open the help window, tada, keyboard comes back!).

Mind mapping is hard to do on paper for me. Too messy. But computer is much easier.

Not sure if I buy into the technique (or understand it), but I do know that one of the core inefficiencies about my thinking is that I lose track of what I'm trying to do. I have loads of sheets of paper that I have scribbled thoughts on, but then it all just sorta gets lost. Shards of disconnected thought, can't fit the whole picture in my head. Part of why I stay confused. Mind mapping might help, especially computer assisted, because it allows me to have unstructured shards of thought (world 1), but connect them with a global big picture and structure them (and fluidly too so that the structure isn't too strongly coercing the flow). Dunno if that makes sense or not.

Basically, I can't fit everything in my head. Having a mindmap helps because it lets me see the whole picture, but then drill down to that little bit I'm thinking about, and also make links explicit (and remember them).

Here's a screenshot of my prodigy mindmap:



Maybe this is the kind of thing that foolish people do (jump on fads, and read self-improvement books instead of actually doing the job). Maybe. But I do feel like it has the potential to help me. (in the same way that koweynlg helps me; it's self documentation so that I don't keep losing pieces and having to pick up again... only problem with koweynlg is that pieces get lost in the stew)

CCG chart realization from disjunctive inputs

Am reading and enjoying Michael White's INLG paper from 2006. Not sure I understand all of it, but I do sorta kinda get the idea. Noting down HLDS for representing discourse structure (they seem to mean in the DRT sense). Nice to get back a little bit into realisation. Seems to be achieving what underspecification is after, want to say the same stuff in different ways (assuming that's what we mean by that). Looking forward to reading the 2008 paper Hypertagging: Supertagging for surface realisation with CCG

constraints

I just remembered this morning that if I like the idea of constraints (un-ordered interactions across different pieces of language), I should learn more about stuff like XDG [which I remember having a vague fondness for, albeit with little actual understanding]. Was triggered by the CCG paper talking about CCG as a constraint-based formalism.

log-linear model

Web search + Wikipedia tells me that log linear models are a form of regression analysis. Don't fully grasp what that means. Vague web-skim idea is that given a bunch of data, you want to create some function that will tell you what, given values for some independent variables (age, stress at work), the predicted value would be for some dependent variable. You can have linear regression (simple weights) or a Poisson regression (that's what log-linear redirects to in wikipedia). But to be honest, I don't really get it all. Still, maybe this gives me a wooden stump to stand on.

less fear

pcfg

Today M+S taught me about PCFGs. There really isn't much to it, is there? So each Foo -> ... gets a probability. Universe is the set of things that Foo could expand to. You want to

• figure out the probability of a sentence given a PCFG? - that's your inside and outside probabilities (like your forward and backwards)
• find the most likely parse for a sentence - do something Viterbi like
• assign weights to the rules - do EM, just like like HMM stuff you saw in second to last chapter

Oh believe me, I still don't understand all the math (sorry), and I'm still too much of a chickenshit to sit down and crank out a toy implementation as an exercise [there's your wheat and your chaff right there... wheat rolls up her sleeves, does this easy thing, and learns something from it]... But I do sorta see how the pieces fit together. I don't really understand how to transpose these ideas into what people do with other formalisms, but maybe that will come later. ch 12 looks like it might be kind of a fun read.

Also, I need to get around to understanding what these log-linear models are that everyone keeps talking about. Just get an idea so it doesn't phase me when I see the word. Barrel down that checklist.

trains

Tried putting the trains grammar together. First observation is that unless I'm willing to shift to a variable-free formalism, I'm going to need WASP 2.0, but then I'm WF for some tech support. Second observation is that we can't use the LHS to pass down logical variables (although, there is the lambda-SCFG stuff which we need to pay a little closer attention to). This isn't CRU; shift your thinking, curb your dogs. It all boils down to what we're trying to do with this grammar. Simple, it's just a grammar that tells you what all the possible MR strings are, right? Well, do we them to be fixed strings (with an order), or should we try to be flat-flat (literals can come in any order). I guess flat-flat is easier for users. But then again, fixed order does not imply any need for linguistic knowledge, just of the MR grammar. I'm going flat-flat for now because it's easier to fit into my text editor window, and writing the type annotations otherwise would be a pain in the butt. But it's not a good reason.

building applied natural language generation systems

Easy read. Noticed it from the Prodigy case for support (my Nth reading). I've seen this bit before, but I feel like I need a reminder

six tasks

1. content determination :: Data -> Set Message
   
2. discourse planning :: Set Message -> [Message]
   
3. sentence aggregation :: [Message] -> [[Message]]
   
4. lexicalisation
   
5. referring expression generation
   
6. linguistic realisation

a typical architecture

1. text planner [cd, d-plan, sent-agg]
2. sentence planner [lex, reg]
3. realisation

No, it's not parole de $DIEU but it's a pretty good overview, especially for somebody who has been realisation-realisation-realisation from 2003-2007. And of course, my type signatures are grossly oversimplifying things (for example, discourse planning probably throws in some extra information so that you know what's holding the list together), but for a dumb programmer like me, it helps.

Odd. It seems to be sticking better, this time, don't feel as lost. A lot of this is just repeating the blue book (as far as I can tell), but somehow I find that I "feel" this more, whereas before it would be just sorta rote. Guess it's just a feature of Erician Learning: frequent repetition from different sources (and angles) or else it don't stick (or make sense). Either that or the particular article I'm reading has just the right mix of high-level and comprehensiveness (good survey). Or maybe it's at the right time after me trying to wade through all this discourse-related literature, and then think about the semantic underspecification stuff. Or maybe I'm wrong and tomorrow I'll just be discouraged again.

Anyway, now the Caledonian example makes more sense. Now when I see a rule that expands into leave/departs, I get the idea that it's a about lexicalisation being modeled in the representational space. Kinda wish I noticed this paper way way earlier. It would have helped me understand what this project that I've been working on for the past 6 months is about.

occupational health

Had a workstation assessment from the occupational health folks. Good she came by. She thinks it's ok that I continue the vertical keyboard (since I like it). Spent an hour and a half looking at what I do. She got me to scoot my keyboard closer to me and raise my chair way up so that my arms were in a more natural position, raise my monitor (now got three monitor stands stacked below) and get a foot rest. Basically, either the keyboard goes down (impossible, since the desk is fixed), or I go up. She's also going to recommend that I get a vertical mouse.

Yeah, it seems obvious now that she mentions it. Really is much more natural to have this keyboard closer and my arms at its height. Only problem now is that if I raise everything up, working on paper becomes trickier (because desk is too far). So I have to lower my chair to deal with paper, and then raise it again to deal with computer. Not fluid. That, or find some stuff to effectively raise my desk, maybe big stacks of stuff to write on. Also, with the chair at a good height, my legs get squished by the desk. Sigh.

But yeah, it's good to have an actual person -- a pro -- do this, because she can help you figure out what all the constraints are and how they fit together. Double-good that the pro actually sees your situation so that she can tailor the advice to your needs.

constraints

It suddenly occurred to me that I am quite interested in constraints. Not necessarily in doing constraint programming, but in the idea that different generation tasks interact with each other in a non-pipeline fashion. How do we make systems that can deal with that? Or maybe I'm not as interested as I thought. Dunno. would be fun to build a truly non-directional generation. A bit like what ILG is doing with her rows and columns, but somehow using a more traditional (from the SE standpoint) notion of modules. Sure, sure, blackboard architecture's been done before and doesn't give so nice results in practice (yet). And yes, there is the SPUD stuff, which I have always liked. But maybe we can do more. Or less! Maybe a more modest version of this idea. Argh, nothing new in this little brain, but it's a good note to Future Eric.

Maybe Future Eric wants to work on this. Or maybe Future Eric wants to work on incremental generation. Or maybe Future Eric decides he really does suck at research after all [inner research is nowhere to be found], and wants to shift to teaching [which he might actually suck at too], or research-hacking [which would only be fun if working for the right person, i.e. with the right mix of Vision and realism], or (yuck) the Industry.

But this is weird. Why have I suddenly stopped feeling scared of the future, or not being good enough, or just Not Getting It? How long will it last? And can I turn it into something useful?

today

I haven't actually done anything today. I sat down and tried to think about underspecification. But then I got confused about why we want this in the first place. Then I tried reading our case of support, which lead me to the applied NLG paper, which suddenly made me feel calm. But I didn't actually do anything today. [hey, maybe we should use the WASP stuff for just realisation, and go back to trying to model all the hard stuff with CRU]

But stick a happy face tag anyway, because it's been a while.

tomorrow

what do we do tomorrow? More reading? More "thinking" about underspecification?

and then they fight

Sorta decently behaved today. Doing a decent job respecting my darcs-ban (although maybe I should see if I can get an MSR internship to go hack on it for 3 months... wonder if I could convince them to let me do that).

Still not producing very much though.

playing with WASP

Getting a bit of tech support on this. Guess we want the 2.0 version (which does generation and knows about lambdas [the wasp ones, not lambda lambdas].

bit more M+S

Breezed through ch10 of M+S (i.e. not really paying attention, just trying to pick up key words). Did not so much read, but daydream as the words passed by, thinking of possible tagging strategies.

One thing I'm wondering is if it would be any use to train a sort of meta-tagger than can classify words according to [I tend to get this one right] and [I tend to make mistakes on this one]. You could imagine that if the meta-tagger was fairly accurate, it could then do something like change, with a high probability, the tag it would pick for things that it tends to get wrong.

Another vague thought was that you can have a bunch of one-tag taggers, you could imagine taggers that are either very good at picking out NNs (or maybe a bit over-zealous about picking out NNs). They either would just tag everything else as X, or maybe do proper tagging, but somehow more sloppily. Then you make them fight. "Well, I'm a VBG tagger myself. I think that's a JJ right there [I don't actually know what these tags mean], but mister NN says he knows an NN when he sees one, so, like, whatever, man"

Sorry, nothing precise enough to be useful, and probably been done anyway.

wasp chart generation

I think I now have a better idea how the chart generation component of WASP works. Before, I vaguely understood that somewhere lurking beneath it was the standard Kay chart generation idea (non-overlapping bit vectors). But what these bit vectors corresponded to was a bit trickier. As usual, the answer is actually pretty simple.

1. Each bit corresponds to a production in the original MR grammar (I guess, note here that the MR part of the SCFG does not generate the same language as the original MR grammar). You have to (?) parse the MR using the original MR grammar.
2. Your initial chart items are the SCFG rules with a bit vector corresponding to the parts of the parse using the original MR grammar

flat semantics?

I wonder if there would be any help to switch to a genuine flat semantics all the way (this stuff here is already pretty flat, I guess, with a little stuff done to assure the commutativity and associativity of conjunction). Would it buy us anything here? I keep thinking that maybe if you're doing phrase-based alignment on your texts and MRs, it would be a bit easier (or maybe not, maybe using MR productions is actually better). Also, another issue which I would have to struggle to wrap my head around is what you do about passing variables around.

trains

Slowly advancing on the train grammar. Realised, for one thing that the grammar we use needs to be a whole lot simpler than what the CRU paper provides. We don't need three logical forms for train -- caledonian_express will do just fine -- nor do we need two and orderings. But somehow there is a nagging thought that the MR grammar at the same time needs to be more general. For example, it ought to generate all permutations of literals, perhaps. Wonder if it's too flat, in the sense that it would make it non-obvious how to pass logical variables down.

Anyway. Found a little trick for alignment. I just repeat rules to add 'padding' so that if there is an english rule with 2 variants, and the equivalent MR one only has one, I copy and paste the MR one (make a twin). Wrote a little Haskell program to turn the two files into a corpus (using the Galois lightweight xml package).

This is where the hour of daydreaming kicked in.

underspecification

Trying to work out a good way to do underspecification for generation. First I'm trying to figure out what underspecification is for (in NLG)... what are we really after? This will be the 5th time I've read this tech report, trying to squeeze something out of it, refresh my crumbling memory. On my fifth reread, I notice these things:

• generation space != representation space (in AB language)
• underspecification seems to basically mean 1 representation = many possible realisations (and in that light, the input we gave to GenI for its multiple paraphrases can be thought of as an underspecified input wrt certain syntactic decisions)

Another thought is that transposing the CRU idea of just going up the syntax tree won't do the same thing, because the CRU grammar is meant to represent a generation space (or was it the other), where as the MRL grammar really is just representing the err... representational space. Or maybe I'm not using the right words. Or maybe I'm just babbling.

What are we really after?

Part of the problem is that I'm having trouble shifting my way of thinking. From the old Eric who basically focused on realisation, thought of it as a box providing an input and an output, to the CRU Eric who's trying to see this all as one space, single decision tree (despite my reservations), to the SCFG-based Eric. Yeah, it's all the same thing, but all this repackaging is confusing me. Mental habits that worked for one Eric have to be suppressed. Things don't mean the same thing.

What are we really after? Thinking about this just confuses me. Need more concrete.

downhill slope

Tomorrow, I should try to write my caledonian2 grammar as a wasp mrl-grammar, even if I have no idea if it makes sense as an input for such a small domain (it doesn't). Then I shall see what happens if we train on the corpus and try to parse some simple train strings.

still not working (but it is a sunday)

I thought I would come into the office this morning to make up for Friday's misdeeds. Have not been very good (although I have banned myself from contributing to darcs, with David's help by asking him to automatically reject any of patches for the next two weeks).

It's just hard for me to channel myself into one thing, hard to get started, muster the mental resources to get something done. Just sit around slack-jawed.

M+S

Well, I read a little bit of M+S, finishing the chapter on HMMs. Kinda flew over the Viterbi algorithm (the combination of better getting the forward/backward algorithm, and having seen this before made me less motivated to go through it). Also skimmed over the hill climbing Baum-Welch thing (because I saw a brief glimpse of EM in a Nesson and Shieber tech report). Fine. I think I get the picture, which isn't saying much because I'll just forget it again in a month.

Noted the practical aspects of implementing HMMs (because you're multiplying a bunch of < 1s, numbers will become very very small and then underflow [I guess, if you look at the wikipedia page on IEEE 754 floating point) you can take that to mean that 11 bits just aren't enough of an exponent... I wonder if it would be useful to have a super-low-precision floating point representation, an anti-IEEE with 11 bits for the fraction, and 52 bits for the exponent). They suggest to use logs. I didn't really understand the bit about the scaling co-efficients.

Also, I started the chapter on tagging (what the hell), but then I got bored and closed the book. I guess I should skip to the PCFG chapter, read that one and ch12 on parsing. Then look in Jurafsky and Martin. Not entirely sure what I'm trying to accomplish by doing this. So you want to learn about probabilistic parsing? Why? Does it particularly interest you? Are you just trying to "catch up" (where people have already long moved on)? Are you just trying to make it easier to understand the things you have to for work? Must bet the latter.

TIGs, TAGs and TSGs

Yesterday on the beach, in a cafe, on the couch at home, read a bit of Shieber and Nesson on synchronous TIGs and TAGs. I have lost all my intuitions about why we want TAGs. Yeah, I can recite the phrases "long distance dependencies" and "tight linguistic packages" (that comes from my thesis, just erician for "linguistically meaningful units"). I can even tell you about lexicalising context free grammars. But I've just completely lost the "ah-hah", the "getting" it. How can you un-get something? This is so embarrassing.

Word of advice to any PhD students who feel like I did: do not believe that you will stop feeling dumb afterwards. It only gets worse.

Sigh.

hierarchical phrase-based model for SMT

Reading David Chiang's ACL paper on SCFG based statistical MT. I really like the general idea here. SCFG for MT's been done before (Yamada and Knight, 2001). What's cute about the Chiang approach is that he doesn't try to go for the linguistically motivated grammar. Just formally, ok, fine, we need some notion of syntax. But lets learn our grammar from bi-text, not from annotations. Maybe I'm noticing the wrong things here, but I find it really really amusing that he just uses two categories, S and X, mostly the latter. We're just learning how to group things hierarchically [and order them], s'all. Oh, I guess that should be an obvious consequence of the we're-not-doing-linguistics bit, but it was just one of those minor 'ah-hah!' moments. Oh nevermind.

Conclusion says they want to look into adding in syntactic categories (more syntax). Still, it was useful for me to have de-blockage about the categories being necessary

playing with CRU and WASP

Just being so slow here. So now I've got an English grammar for the trains stuff, and a MR grammar. But (a) I haven't given any thought into what a reasonable MR grammar would be for trains -- still re-using the one that generates MRS-like expressions -- and (b) I haven't tried to solve the practical issue of how to align texts produced by both grammars if they have different ambiguities [uh, not the word I'm looking for... more than one way to say the same thing]. For example, the English grammar has two rules for "leaves" [leaves/departs], but the MR only has one [why would you need two?]. So rather than deal with that, I just commented out one of the rules. Likewise, the MR grammar has two rules for AND, for which the English grammar has only one. So I just commented out one of the rules. This way we have identical texts that we can just zip. (b) is just a practical problem, not a theoretical one. Like, aw man, do I have to write code myself to enumerate all the possible sentence pairs produced by an SCFG? [damnit, why have I become such a wuss? was I always like this? I think so, come to think of it]

Having a hard time trying to get myself to think about [1] what would be a sane MR for the trains grammar, simple domain as it is [2] what would be a reasonable notion of underspecification in the input? Why do we want underspecification again? Oh yeah, so that the guy giving you the input doesn't need to know too much stuff. On Friday, I wrote down something on the whiteboard which I thought was a useful insight, but now I don't really know what it means anymore. I said:

we only to generate from underspecified representations... which should be possible

statistical mumbo-jumbo

I say this not as an insult, but as a reference to the fact that I don't actually know what all this stuff means yet. Just note the phrase 'we depart from the traditional noisy-channel approach and use a more general log-linear model' (ok, I heard about the noisy-channel idea in 2001 and tell you about English having been garbled into French...). Right, so this sentence is a clue. It tells you that whatever the noisy-channel approach, and log-linear models are, they are sort of in opposition.

WASP λ-SCFG

I'm re-reading the λ-SCFG stuff (because it didn't stick the first two times around)... and I keep tripping over this bit:

If any of the arguments x_i appear in β as a free variable (i.e. not bound by any λ), then those free variables in β must be renamed before function application takes place

I guess that must be right, but it sure makes me feel uncomfortable... Err, maybe the trick is to realise that this is NOT λ-calculus, but just a variable-binding mechanism which is λ-calculus inspired. That's all there is to it, self. Nothing to be confused about. The rules are exactly as Wong makes them up.

So, ok, Eric, try not to get confused.

1. λ-SCFG rules look like this: <α,λx1..xN.β>
2. The non-terminals must be followed by some number of variables [arguments]
3. When we expand a non-terminal, it must be expanded into a λ-function of the same arity
4. [not in the paper, but should be clear] Once you've expanded a non-terminal, you can do function application.

That's all. There are no inner-λs, okay? The only λs are on the outside of each MR part of the rule. Non-terminals are typed (by arity). Type is declared in the SCFG rule (implicitly by the number of arguments you associate with the non-terminal).

That wasn't so hard. We'll leave it at that. Maybe it'll stick this time. Tomorrow, we figure out how this meshes with the statistical stuff.

Well, maybe I did so some work this weekend, after all. Guess I'll go to gardening now, then go home.

really really bad!

The story this week has been : wake up early and get to work before 9, sometimes before 8 (great!)... and then go hack on darcs (no! bad!)

I spent most of today preparing the ChangeLog for the upcoming darcs 2.0.1, which was really not what I set out to do for the day.

Otherwise, I did spend some time reading M+S

markov

Look, this is something I have seen a couple of times, (maybe 3 or 4), but maybe now I think I grasp it well enough that it will stick.

1. Markov Model: the past does not exist; only the present moment matters (probability of transitioning to state N+1 given state N, doesn't matter what states N-1 , etc are)
2. Hidden Markov Model: you don't have a way of telling for sure what state you are in

Actually pretty simple now that you "get" it. Sigh. Crazy soft drink machine: it would be a visible Markov Model if the machine had a little readout that said "I am now in Cola-Preferring mode", or if it as a rule, consistently spat out cola when it was in that mode. It's a Hidden MM because the drink you get doesn't actually tell you what state it's in. Just tendencies.

I think maybe the confusing thing is that people talk about "hidden states"... when what I think they really mean is that "you know what the states are and you can give them names; you just don't know which one you're in at the moment". I guess a priori one assumes that all states are connected to all states (although you may eventually learn that some of those connections have zero or little probability).

I'm a slow learner. I get side-tracked by things like that. But I hope that as the years roll by, I eventually figure out all the parameters behind Erician learning, and optimise things so that I pick things up quicker (and understand them more deeply, etc)

Also have the intuitions for the forward-algorithm, I think. Motivation being to find the probability of a sequence of observations. Crazy soda machine example again, worked through this with them. You could just add up all probabilities for all paths, but then you'd be running through each possible path one by one. Slow. Forward-algorithm is dynamic programming. Fine. Backward algorithm, my eyes sort of glazed over at first, but then you remember that "the past does not exist; only the present moment matters". If forward algorithm is about summing up all the previous states I could have come from, the backward algorithm is about summing up all the states I could go to. Talking this out in my own private language. The alphas and betas are the same as P(O)s... just incremental (up to time t, state i).

ehhhh

The good news is that I'm still getting into work earlier. Around 8h this time, although I would be happier if I could break that magical barrier. The bad news is that I'm still not getting very much done (although I did track down and fix a couple of darcs bugs).

Today got cut short because some friends-down-corridor wanted to got watch Germany-Croatia (around 16h30). I left at halftime. Then tried to get back in... but forgot my code... which I had thrown out whilst replacing my planner. Sigh. Oh well, it was 18h by that time.

Anyway, in the time that I wasn't doing darcs stuff, I managed to put together a CRU grammar for the trains input specification. This is kinda embarrassing since it was already published (brainwave was to copy-and-paste from pdf). But there's all sorts of little details (like you need tab-separation in the lexicon) which I had forgotten. Good. Now I understand the CRU stuff much better.

Also handled some admin stuff re: my staying in the UK. Got my forms downloaded, talked a bit with the Home Office.

relapse

Today was not super-disciplined.

It could have been worse, but I ended up spending 3 hours tracking down a darcs bug (as in triaging the bug tracker) instead of the 1 hour I thought I would allot myself.

wasp

It turns out that WASP was online after all! We found this out via a query to John Wong. Not only was it online, it was actually very easy to find (google: SCFG wasp) if I had only thought of the most obvious keywords. Doh. Things to note about it: it's in Java (bleah), it's GPL-3 or LGPL-3, not sure, and its got some pretty friendly documentation (i.e. with enough information to get you going). I ran into some trouble running GIZA++, so I'm waiting for some tech support there, if possible.

I think the following things are still shaky for me:

1. how do you go from SCFG to weighted SCFG and how do you use the weights? I guess the latter part is a relatively easy answer, Viterbi on the chart. As for the former part, it involves a log-linear model, and a set of features. But I have no idea what that means really
2. what's the deal with this lambda SCFG stuff. Does it affect parsing?
3. how does the chart generation work?

use DyALog?

Simple thought occurred to me on the way to work (from gardening) : if I'm so scared of implementing things now, then why don't I just use DyALog? Instant chart parser, right? And fast, too. In fact, I kind of wonder why I didn't just think of that during the phud.

Anyway, here's what's on my shopping list then:

1. getting DyALog running - sorta check... right now, I can run their test suite, and also run their example tag_parser directly. I was stuck for a while until I realised that the reason I wasn't getting any parses for "Jean mange une pomme" was that because the grammar only knows "Pierre" and "Paul". Doh. For what it's worth, the way I'm running their Test parser is tag_parser -forest - < foowhere foo is:

   'C'(0,'Pierre',1).
   'C'(1,'mange',2).
   'C'(2,'une',3).
   'C'(3,'pomme',4).
   'N'(4).

   
   
2. synchronous CFG with dyalog - at the very least, we can do this in a hacky hacky way... output derivations and then re-run them using the parallel grammar, or alternatively, just encode the thing as a DCG. Here is one attempt, modifying their DCG test case. Note that it's not very bi-directional anymore:

   ./fset - <anne-mange-une-pomme
   Answer:
    N = 5
    X = s(np(pn(Anne)), vp(v(eats), np(det(a), adj(green), n(apple))))
    I2 = [Anne,eats,a,green,apple|O2]

   
   The modified code just straightfowardly encodes the slave grammar as an extra parameter:

   np(np(det(Det),adj(Adj),n(N2)),In,Out) -->
    det(Det,In,Mid1),
    n(N2,Mid2,Out),
    adj(Adj,Mid1,Mid2).

   Note that I have simplified away their features here. Also, the difference lists here are for the target grammar (not the source one). Notice the weirdness in the lists for n and adj, though. I have to order the threading according to English syntax, which is why you get this permutation of lists. This could be a bit scary if you've not terminals you want to insert or delete (maybe)... but I guess the difference lists are just there for kicks anyway, and worst come to worst, I can ditch the lists and just parse the English tree.
3. chart generation with dyalog - instead of parsing from interval to interval [given X0..X1 + Y0..Y1, Z0..Z1 = X0..Y1] , we need to parse from non-overlapping bitvector to non-overlapping bitvector (given X and Y such that X && Y = 0, Z = X || Y) - this might be tricky
4. probabilistic disambiguation with dyalog - dealing with weighted grammars and picking the best parse using something like the Viterbi algorithm (no idea how easy this would be)
   
5. combining 2, 3 and 4 to get a probabilistic synchronous CFG generator

This won't solve the issue of obtaining the aligned grammar in the first place, or assigning weights to it, but it will get us 1/3 of the way there.

knowledge gap

I have a really uncomfortable knowledge gap here. I don't really know what a PCFG is. Oh yeah, I can guess (and remember) that it's just a context free grammar for which each alternative RHS for the same LHS is assigned a probability (all alternatives add up to 1), and that the best parse would be the most probable.

I remember seeing something about DyALog not supporting floats, which could, ah, be a problem. But then I suppose you could (sigh) hack up a generic tool that traverses the shared forests output by DyALog (cough... somebody's done this already).

knowledge gap 2

I'm still not super comfortable with the Lambda-SCFG stuff. But I'm guessing that if you want lambdas, just mucking around with Prolog (or a Prolog-like-language) can't be too bad a bet.

demain

I guess tomorrow I should look at that trains example again, see how reasonable the trains MRS-generating grammar is, and what simplifications we need.

being stupid today

Wow, I really didn't get much done today, despite trying to, like, work and stuff. Got here before 9h, which is ok, not great. Was trying to look at the SumTime grammar, figure out how it fit together (this was taking me a long time).

Had chat with AB about what I'm working on for the next fortnight:

1. building semantic grammar for trains domain
2. see what SCFG tools are out there (WF: her email)
3. think about underspecification re: #1
4. read more SCFG stuff, understand Wong stuff better... (maybe Melamed?)

Should be enough stuff to work on. I wonder where the time went, exactly. It just sort of... evaporated.

re-implementation

I'm scared of re-implementing stuff myself nowadays. I don't know if this means I've become a better (more realistic) programmer, or a worse one (more fearful). I suspect it's the latter. Why should this scare me? Parsing an SCFG is just parsing a CFG and then re-running the derivations on the flip-side. Ok, so we're dealing with a weighted SCFG so we have to use Viterbi in unpacking (?)... oh ok, we're doing chart generation and not chart parsing and the input is sorta like a flat semantics, but a little bit different, nothing you couldn't wrap your mind around, and oh there's the actual business of learning the grammar and learning the weights behind it. So why so nervous?

I could do it. I could just add an Earley-SCFG module to GenI. What am I so worried about? [maybe the fact that it took me an embarrassing 6 months, or was it longer? to do TAG CKY/Earley generation? but now all the refactoring work is done, so it's just a new module, right?] The probabilistic stuff scares me a bit. But come on! You know Viterbi is easy, if it's just te unpacking.

Oh wait, maybe I remember why I'm scared. It's all thedetails. Like, if I wanted to extend GenI, I would have to extend the parsing code so that it reads scfg-grammars (are these going to be in tree form?). Oh, and there's the whole Lambda-SCFG stuff. And and and all those command line switches to add (ok, that should be easy). The details.

more or less ok...

Yeah, I guess today was ok. (07h45, w00t!) . But what did I do exactly? I spent some time thinking about the input specification issue, and working some stuff out.

morning

Started putting together a 'chop an onion and fry it' semantic representation. Then I got confused about what exactly I'm trying to accomplish. I hate making design decisions like this. How many onions do we want here? How do we propagate the information? For now I'm going to hard-code it. You want 3 onions mentioned? (even if it's the same onion), you have to say onion(x1), onion(x1), onion(x1). But surely this does not make sense if the onion has properties you want to describe, like red(x1). You don't want to keep saying red onion, red onion, red onion in the input, do you? One thing I need to figure out is who would have thought about this before, and what did they decide?

afternoon

It goes back to the project goals. What do we want in terms of specificity, granularity and abstraction. ILG likes to think in row-column terms. Come to think of it, I don't think I fully grasped what she was saying about granularity (roughly, on the words vs documents level... but on the input?). We've got:

• low abstraction, coarse granularity : canned document as input
• low abstraction, coarse granularity, partial specification: NLU parsing task

Argh, lost her examples. Make a mental note that specification (in her terms) is not a scalar thing, just a question of coverage over the number of possible features. Abstraction is which 'level' of features you use. I guess our project is a high granularity (document level), and high abstraction (domain level input), but not sure about specification. To make things easy, we could go for high specification, but it may not be very reasonable. Dunno.

Sort of going around in thinking-circles, and confusing myself.

Got some constructive criticism today. Need to think more in terms of publishing and less in terms of project goals (secondary). "Better to start running in the wrong direction than sit still". Hard to explain.

Not exactly sitting still. Sort of tying my shoelaces, and then staring at them, and thinking, wait, what am I doing again? I have a lot of trouble turning things into tasks (I can deal with tasks, creating them is the hard part). This big mass of things we want to do, everything ill-defined. Can't just pick something and go, because to decide what to do about A, you need to decide what to do about B... which means you need something done about A... I just sit there stupefied.

evening

Sat myself down and tried to lay the project out. We've got corpus creation, input specification design [ugh!], and application of the parsing technology. Guess I'm going to grab the third part and try to do something with it. Maybe apply it to the weather forecast stuff (thanks, ILG, didn't think of that). Yeah, or something.

Passivity. How to react when you're spending all your energy trying to understand what people are saying? Just spent all my time here trying to wrap my head around stuff. Gah.

early but useless

Argh! I finally have time to sit down and do research (after all that scripting), but I'm just... not... doing it! I even got in early today, around 08:30, but proceeded to goof off (not really goofing off, dutifully triaging the darcs bug tracker, but what about dutifully doing my job?).

I spent the "productive" parts of my morning trying to pore over the GeoQuery MRL... am also slightly worried about these semantic indices. How exactly do we assure that the indices go in the right places (my basic scare scenario is how do you know you're not swapping and saying the exact opposite of what you mean, or something more subtly wrong?). Trying to turn this into recipe-talk. Oh, then the printer jammed and I spent a good half hour configuring my computer to use the other, colour printer [twas a tricky jam, was told to give up]. Took me a long time because I kept trying to convince my computer that the printer had a duplex unit. [it did]. Computer believed me, but the print dialog refused to offer me the option of double-sided printer.

Spent the early afternoon attending a talk in Sussex. I think I fell asleep during a crucial bit :-(

Late afternoon spent being a faceless darcs bureaucrat. Oh I tried to think about my meaning representation language (my original job) problem, but just kept wandering off. Tried to stay later so that I could maybe get something done, but no...

why darcs hacking == procrastination

Spent late afternoon doing more darcs stuff. It's really boring work too. For example, today, I sent out a mass-mailing trying to round up developers for a GPL exception. Basically, I check my mail, and note their "yes" in a text file. Boring stuff, but it's easier than sitting down and making myself concentrate on something, which is why I do it... the problem with this darcs stuff is that it makes you feel like you're accomplishing something.

Ah! I've submitted a patch, resolved a bug. Done. It feels like something gets done. Whereas sitting here trying to work something out, I tend to go around in stupid circles, probably because I haven't figured out what exactly I'm trying to work out in the first place -- just a vague idea -- and why I'm trying to do it.

Another thing about stuff like this is that it's easy, when you try to think for three seconds and then get fed up to just switch over, check your mail, transfer another "yes" to a list.

no M+S

Huh. Didn't get around to reading M+S today. Occurred to me last night that I shouldn't be afraid of not-understanding something (missing background information). So I'm going to skip to part III, because I realised also that I care a lot more about structures than words (for example, I'm more interested in learning how to work with a PCFG than I am in finding likely collocations)

extended

Oh, and some good news. I've been extended till end of December. Dunno why. It's not like I actually do anything useful here, except write shell scripts. Come on. 6 months left. Can I redeem myself in 6 months? Just maybe if I can make the going to bed and getting up early stick, I will have a chance. Just got to learn to focus.

GAH. I should put down a frowny for the darcs hacking, and a smiley for the extension. I guess they cancel out and go neutral.

chop the onion

Tomorrow, I'm going to figure out how to represent "chop the onion and fry it", and if I'm lucky, what a likely grammar would be for the input language.

could be better...

Had a good 7 hours last night. Think that made a difference. Finished reading the Wong thesis. Was easier to understand with not as much falling asleep. Reading just knocks me out in general. Brain dis-enagages.

So: synchronous CFG acquired from an unambiguous source CFG using word-alignment techniques with special tricks (use production rules, not tokens; do double-alignment, prune, re-add). Notion of lambda-SCFG (still a bit annoyed about this free variable thing). Lambda-SCFG for generation just uses standard chart generation instead of LR parsing of the LF. The overlap-bits come from the productions involved in the parse (using only the unambiguous grammar). I didn't actually read the mathy bits. Sorta ignored them. Going for the bits I can understand.

Working much more slowly through M+S. Also taking an attitude of not trying to understand the math. Just take it as given. Try to see the big picture, ideas. Why we want a null hypothesis, what we use the t and chi^2 test for (and not worry about what makes them tick).

Was slightly optimistic this afternoon, as though I was starting to figure out what I was doing, starting to learn how to learn, what to look for. I had the feeling all of a sudden that not-getting something no longer frightened me, that I could learn to manage and wield my ignorance, figure out how to figure out where to shine the flashlight.

But that burst of optimism vanished. Gah. So little time. So little brain. Got to use both wisely. Basic brain trick is that you've got to understand stuff in order to understand stuff. Compound-interest effect. So how do we use it? Quantity-not-quality is something to apply sometimes, just vacuum stuff up at a superficial level. Make connections, see how bits fit together without really deeply understanding the bits. But that doesn't always work. Sometimes you don't understand what the bits are until you've really worked through them. You can't always play the forest game. The trees are there for a reason.

Gah... so little time...

grouchy and tired

Well, had some trouble sleeping last night, so I'm pretty tired today. Spent too much time in the garden (going crazy with shears). Also worked a bit too much on darcs (trying to track down a bug). Intention was to stay late to make up for lost time, but just falling asleep and not understanding things.

Sigh.

Just so grumpy. I'm taking it out on the thesis I'm reading "ARRRRGH, I DON'T UNDERSTAND, AND IT'S ALL YOUR FAULT" (heh).

Finished chs 3 and 4 of M+S. Now on on the meaty stuff. Uh-oh.

earlier to bed

One of my recent thoughts is that my productivity problems are related to my poor sleeping habits, so I'm hoping to implement a 22h30 bedtime or maybe 23h if I'm feeling indulgent. I also like getting up early in the morning, puts me more in a mindset to get stuff done.

Today has been less than spectacular. Not too horrible. Plowed through most of M+S ch3 (much easier, no math). Did some admin (trying to get back into taking GTD seriously).

Spent a couple of hours talking with ILG about her work. Very interesting to see the big picture (the 13 year progression). Got notes down on our whiteboard, so maybe this time something will stick. I dislike it when I fail to understand what people are explaining to me about what they're doing. Lots of context like this helps.

Still reading Wong's PhD thesis. On ch4 now, on Lambda-WASP (already seen this in a paper, but I like re-reading and taking in the slower pace of a full thesis). Real aim is to understand the generation stuff.