evaluation sites cranked out

Trying out the new Blogger editor, which apparently doesn't mess up pre code so much, not that I'll be using any right here.


Spent a long day working from home (and catching up with some lost time from Monday).  The good news: all 5 evaluation sites cranked out, along with some slight enhancements to nltg-eval.  I feel a bit more justified in all that time I spent obsessively tidying, refactoring, changing things a little bit here and there, always ripping something out, renaming  something... and now finally it's paying off.  Finally, we're reaching a point where the process of creating a new site or of bootstrapping off an old one is relatively effortless and (the key point) Eric-proof.

The bad news is that things weren't Eric-proof enough.  I just realised that for one of the sites I was building in the previous batch, I stupidly used a whole new test set where the whole point was to keep the same one as experiments before then.  So I think that's pretty much a huge chunk of collected user data to throw out. SIGH. This is why I'm so obsessive about automating things.  It's all about the Eric-proofing.  Here, I'm not sure what I could have done, really just a matter of doing the Wrong thing.  I'm just prone to this kind of confusion, swapping true/false, left/right.  Don't cope with fiddly very well.

Also, in discussing how to allocate one of our trial sets, (10 systems = 45 pairs, on 30 scenarios), I realised that the documentation in nltg-eval for dimensioning latin square grids got things a bit wrong when it came to pairs.  Basically, I forgot to divide by 2.

Typical...

more evaluation sites

5 more evaluation sites to build. Will probably be updating them slightly to more closely match the nltg-eval tutorial (as I get slightly better ideas about what should constitute the state of the art, eg. putting the config settings into one big associative array near the top of the file, no helper variables).

I may have a user for nltg-eval lined up, but I'd like to spend some time generalising the identification system for it so that we've got more general usability outside our contexts

nltg-eval 1.0

Well, the NLTG evaluation toolkit is finally out the door. It may be useful to you if there is more than one way to do something, and you need to ask people questions to figure out which way is the best. Hopefully in the future, I'll have a chance to rework the authentication thing so that you can run your little evaluation apps as Facebook apps.

Hmm, 2 person weeks (at least) of documentation for just a silly little thing. Imagine what it must be like documenting a proper software project?

The writing may be horrible and clumsy, but I think at least overall I do make some kind of effort to be helpful. You get:

• a gallery/screenshots to give you an idea what sort of thing you use this for
• a sort of tutorial working you through a build-up of 6 increasingly complex examples
• live working demonstrators, showing all the examples from the tutorial
• API docs
• source code examples you can click your way through from the tutorial

Will it pay off? Will all this just have been a waste of time? We'll see... I hope my general must-invest-for-the-long-term thinking is the right one here. In other words, don't copy and paste that evaluation site, refactor it into a framework! Or don't just dump your framework on the web, write a little tutorial for it so that Future-You can remember how to use it later on, or even if you're lucky that somebody else will benefit from it.

But if a tree falls in a forest...

widgets done, now bunch of stuff

Right, so yesterday I wrote a pretty clumsy chunk of documentation for the two fancy widgets we ship for doing evaluations. It's not our code from the web perspective. That's the doing of WebFX and their JavaScript skills.

What we provide is a little bit of Perl that lets you express what you want on a little bit of a higher level than just printf'ing your HTML + JavaScript. In other words, if you want radio buttons, you write something like

my $w_competence = $widgets->widget("competence", "",

 [ "Incompetent"

 , "Unskilled"

 , "Neither skilled nor unskilled"

 , "Well executed"   

 , "Excellent"   ]);

whereas if you want a slider instead you write something like bit more like

my $w_competence = $widgets->widget("competence", "How competently was it executed?");

My hope is that this makes your experiment a little bit more abstract, a little bit closer to what you really want to say and also makes it easier to swap out one kind of experiment for another. Really it was all just written (organically grown rather) to keep me sane when managing a bunch of these evaluation sites.

In other news, I also worked out how to run pod2html and perltidy. Now you have links to API doc and also to example experiments used for the tutorial. The documentation is all a little bit crap, but it does try. Dunno if it shows. Dunno if it matters :-/

next up

Got myself a little list of bullet points, little things:

• clean up NLTG::EvaluationKit POD
• write up pair comparisons (Haiku-3B) documentation
• document miscellaneous other features
• demonstrator for all examples
• links to front page for examples
• make slider range customisable
• rewrite tutorial intro
• tarball

Maybe it's a bit optimistic to hope for a release by the end of the day. We're basically in the mop-up phase of releasing this thing, but we're going to need a pretty hefty mop.

done the latin square thing, now widgets

OK so I guess the part of the tutorial I've been most dreading (allocating latin squares) is now written. Now I need to flesh out the rest of the documentation and shoot for a release this week. Is it going to happen? Dunno, we've basically got widgets (radio, sliders), then pair-based evaluations and other features. That's a lot of stuff to write about. What's worrying is all the other tasks involved in making that first release that I may not have thought of....

...that and the possibility that nobody is going to use this toolkit and all this documentation effort would have been just a waste of time (although I guess it did get me thinking about the latin square stuff enough that I know how to do it myself now)

latin square documentation

Didn't make as much progress on the trial allocation documentation as I would have liked. Only managed to cover the sort of motivation for using a latin square. It was a bit tricky to write without running into a dead end.

My first mistake was laying out the square, initially by scenarios and systems (and then transposed). The problem with this layout is that while it's intuitive and useful for explaining, you later run into the practical reality that the toolkit lays them out by participant x scenario with systems in the cells. So that would mean that you need to do some kind of notational changeover in midstream. Using the more accurate notation from the get-go is a bit awkward (less intuitive) but it avoids a greater awkwardness later on.

My second mistake was in straw manning. I was presenting a latin square against some things-you-might-do and pointing out the flaws in the things-you-might-do. I think that's useful contrast, but the problem I ran into is that the kinds of flaws I ran into were kind of stupid/obvious. My choice of alternatives was inadequate because it leaves all sorts of "yeah, but I could have just done X" thoughts in the reader's head. This is partly because (until yesterday), I hadn't really pinned down why using a latin square is a good idea. I mean, I got the idea that it allows you to distribute evenly across both dimensions, but I never really worked out what the secret sauce was, what the minimal diff was wrt the next best alternative. Now I think I have a slightly better handle on it. Before I was going from Try 2 to 4, but now I realise you need a Try 1 to acknowledge the obviousness of not reusing trials [and Try 2 would be a regression against that], and a Try 3 to smooth the transition. So now we're going: random permutation (avoids re-use), grid with no constraints (pin down one dimension, but suffers re-use), grid with one constraint (best of both words), grid with two constraints (pin down both dimensions).

So that's it for yesterday. Today I need to knuckle down and figure out how to use my software again, hopefully get that latin square documentation written once and for all. The difficulty is all the dimensions swimming in my head, things I don't understand fully: number of systems, number of scenarios, number of participants, which must be divisible by what, how many rows, how many columns... head explodes.

more evaluation kit progress

I made a little bit of progress on Wednesday on the documentation, covering the core mechanism and also expanding on the use of an items directory. It's a bit tricky to motivate the latter with just a hello-world style example, sort of only becomes useful when you have more complicated evaluations to run. But I'd decided that it was a bit of a pre-req in building up to more complicated examples.

Today's work is what I consider the hard bit of the documentation: allocating trials and trialsets with the help of a latin square. Part of the trickiness is that I never really wrapped my head around the whole thing, although I have gotten enough bits and pieces of understanding to get it implemented. I'd be pretty happy if I can get just this one piece of documentation written. If it turns out that nobody outside of NLTG uses the kit, then maybe it'll be useful in the long run if we're going to be running more these runs.

nltg-eval homepage and darcs repo

No tarball yet, but on the NLTG eval homepage you can grab the source via darcs, and look at some screenshots and the hello world example walkthrough (should maybe put the example itself online).

Again, a bit embarrassing all this fuss for a tiny little bit of code. Just trying to keep up with our evaluation needs the only way I know how: cut duplication. I wish my refactoring wasn't so low-level, that I could really identify the sort of key abstractions you need in a problem, instead of always sweeping, sweeping piles of dirt together.

Today's work will consist in fleshing out the documentation as much as possible: trying to cover a lot of new ground, such as organising an evaluation experiment around a latin square. Goal is to have a tarball by mid-December.

evaluation sites back to 95% done? please?

Resolved my technical troubles yesterday. Also we're not going with the PHP recycle (due to time it would cost to set up a VM etc), but plan B which was to recreate that experiment using my framework. This I did rather sloppily/badly the first few tries (which is shameful since I'm a believer in the idea that whatever you do, you do as it must be done, no half-assery). Thing I've got it pretty much nailed down now, so we're back to the evaluation sites being 95% done.

I've also got the green light to put up the darcs repo and homepage of the NLTG eval kit. Now it's a bit embarrassing, for all the talk about it, it really is just a stupid thing. But it's a stupid thing that saves me from doing 10 stupider ones. Maybe others will benefit from it.

evaluation sites not 95% done

Wah.

Technical troubles.

In other news, I managed to import the MySQL database and get the PHP code running from an earlier version of one of the evaluation sites (written by somebody else). So now I feel obliged to use that code instead of mine to get identical results. This means mucking around with virtual machines a little bit.

evaluation sites 95% done

Almost ready to publish that 4th evaluation site. So today's agenda is to start on that NLTG evaluation kit release. The kit is basically a tool to help you build human evaluation websites like the ones I've been working on for the past 2 or 3 years.

An evaluation site is one where you want human beings to give you feedback on a collection of stuff. Suppose for instance you have a bunch of photographer friends who have each taken some pictures of kittens. You want to figure out which photographers tend to take kitten pictures that people like. One way to find out is to do an "intrinsic" evaluation, which basically consists in asking the people whether they like the kitten pictures or not (*). There are many different ways of doing this intrinsic kind of evaluation. What the evaluation toolkit helps you to do is to create websites using whatever evaluation method you want to cook up. You could, for example, use a simple 1-5 rating scale on different criteria like "cuteness" and "beauty". Or maybe you could present people with side-by-side comparisons: which kitteh is the cutest?

(*) an "extrinsic" evaluation on the other hand would be seeing how the kitten pictures affect them, for example, how many of them adopt kittens after looking at these pictures

Basically, the story behind the toolkit is that I hate building these websites. It's boring and stupid work. But they are a necessary evil. The toolkit is just something that arose out of a lot refactoring from a site, one little Perl script I inherited plus a chunk of Prolog. Now it's a set of Perl objects, shell scripts, makefiles, same chunk of Prolog and a bit of Haskell. The important part is the Perl though (the rest are just sort of accessories). I feel kind of guilty developing this toolkit because (A) I still know nothing about web programming and (B) there are lots of nice Perl web frameworks out there like Catalyst which I have not played with. Me, I'm just wrapping around your good old CGI module ,partly for technical reasons (I want it to work on our platform) and partly due to lack of time/interest to invest in becoming a good web programmer. Anyway, the toolkit is mainly about the science-y bits of evaluation (distributing the test sets to people, making sure the right bits and pieces and randomised), and not the web-by bits. I hope that once this thing goes online, somebody else can rip into it and tear out any web-by bits and make it possible to hook it into a proper web framework.

evaluation sites 80% done

I've built 3 out 4 of the evaluation sites. Fourth one is almost ready to go, but it's missing its latin square, so there'll be a bit of digging around in a MySQL database to get it up and running.

Today's agenda (if the getting the 4th site running is easy) is to think about the NLTG Evaluation Kit I've been putting together. What's our plan for getting this thing out the door by 17 December. I'm thinking it would involve these steps:

1. publish a darcs repository - this commits me to the code being public
2. create simple homepage for the toolkit, with link to repository, homepage should go live
3. create one working example using the toolkit
4. publish a preview-only tarball with the example and toolkit, no documentation
5. polish documentation for toolkit proper
6. think about examples some more and flesh out example set
7. write tutorial
8. publish proper tarball

evaluation site saga continues

Ok 4 sites, one of which external and written in Java. Been struggling with my lack of CSS fu lately. I want to have this effect:

LEFT	wide thingy	RIGHT
hello!

But despite several nice tutorials on CSS positioning, I can still only get an approximation. Oh well, at least now I know why I get zero-height divs when I use a float. Maybe I should give up and just use a table.

I keep going through this, trying to figure out how to achieve the layout I want with divs and CSS, and then finally saying "bah! I'm not a web programmer" and switching over to tables.

whoops, more evaluation sites to build

I've got three sites to get up and running today. They're actually almost exactly the same as last year (almost). It's tempting to just copy and paste and get it done, but as we all know Duplication is the Root of All Evil. So I'm going to go through dumbly and repeat my recent Perl and GNU make refactors, killing extra little bits of duplication that I missed from last year in the hope that it will pay off if I end up building more of these nearly identical (but not quite) evaluation sites in the future

avoiding duplication in evaluation sites

I've been working on evaluation campaigns this past few years, and juggling a bunch of variants/combinations of 3 evaluations. This year, I think I have something like 6 evaluation sites to manage, cut up into two projects which each use one of the 3 each. It's kind of crazy, especially since the sites are all basically the same but have some subtle difference in functionality.

It's been a constant battle against code duplication. Over the years, I've been shuffling things out of scripts and into standalone libraries, little tiny piece at a time because it's not always self-evident that code actually can be refactored, [particularly under deadline pressure] or if I need to tolerate a bit of duplication just a little longer to make it easier to slip in some subtle functionality difference. But slowly slowly, I'm finally getting to a point where at least as far as the CGI scripts are concerned, the duplication is down to a tolerable minimum.

Now we've got a problem with Makefiles. Today I finally got fed up with problems debugging my site on the live server, problems that I'd solved several times in the past, but lost the solution to due to code duplication. The solution was always buried away in one of the 6 sites, one of the makefiles. Oh you just need to copy library foo over. Oh, you just need to make sure you chmod that file. The reason why I had all this duplication was basically lack of mastery over GNU make. Didn't know how to make my rules parameterisable enough, and even if some Past Eric did, I didn't have the time/courage to go recover that knowledge to just get the job done. Eternal conflict between doing things now, or doing them right so they won't be so horrible in the future... a future you're even sure will arrive (because you might just dump this research prototype and never see it again anyway).

But now one of my server-related troubles finally annoyed me enough to roll up my sleeves and choose the side of Doing Things Right. No more makefile duplication!

Basic layout

1. project-root/makefile.lib: the rules refer to variables which are undefined, but that's OK - this is where our parameterisation comes in
2. project-root/site/Makefile : defines the variables that makefile.lib wants and has this block of code

   # die, duplication, die!
   include makefile.lib
   
   # complain if makefile.lib hasn't been symlinked in (see below)
   makefile.lib:
   $(error Looks like you need to run ./configure first)
   
   # use of :: instead of : means I can define all in makefile.lib
   # and just append a new pre-requisite to it (the | means order-only)
   all :: | makefile.lib

3. project-root/foo/configure is a stupid little bash script that just does ln -s ../makefile.lib .

So that's fairly tidy. The key trick here isn't so much that you can include bits of makefile in other makefiles, as that you should remember that variables in makefiles are only expanded on demand. This makes it possible to leave all sorts of holes in your common makefile that you instantiate in the outer file. Sort of thing you know, but fail to connect with your goals until you really sit down and set your mind to refactoring.

Aside about version control

This can partly be blamed on technical limitations with Darcs (incidentally a job which consists just in working on Darcs is potentially bad because it would deprive me of grist for the mill -- doing some sort of coding as work-work gives me a tiny bit more knowledge about my target audience, although then again, the temptation is to think that all programmers are like me, which is far from the truth!):

• [no repository nesting] I don't know how to set up some structure version control structure which reflects the fact that the 6 sites have similar content, while grouping them, say 3 at a time into distinct projects. I *could* treat each of them as individual repositories, and share patches between them. I've done this with other sites before and it works fairly well. In this particular context, though I'd really like my 3 sites to co-exist within a single container project.
• [no symlinks] My version control software does not manage symlinks

more evaluation website work

Today's challenge is to work out, given T texts and P peer systems, how to divvy the texts up among the systems so that they all get the same amount. Well, actually the answer came to me last night and I feel silly for this even being a problem in the first place:

1. scramble T
2. T1 to P1, T2 to P2, .. and so forth, looping around P as many times as you need

Otherwise, it's more cleaning up the NLTG web evaluation kit (which I'll probably rename again to NLTG evaluation kit).

I might look into Catalyst or some other Perl web framework, hoping that it will make my life a bit easier.

Also, one of the problems with the web evaluation kit is that it doesn't offer a good authentication mechanism, just a baked in password. So it would be nice to learn about some good ways to abstract over this. I'd like people to be able to, eg, use the toolkit to build Facebook or Mechanical Turk based evaluations.

evaluation websites

Building evaluation websites. Trying to work a little more on the NLTG web evaluation toolkit, mostly getting a set of semi-interchangeable widgets to reduce the cost of variation in experiments

back from holidays

I've been gone for basically a month:

• 13-19 Oct : darcs hacking sprint in Orleans, plus some talks
• 21-26 Oct: Google Summer of Code mentor summit in Mountain View (plus visiting San Francisco)
• 27 Oct to 10 Nov: visiting my family in Florida

Now back. Not sure I can really blog as much about work as before. Jetlagged and stupid, so just spending my time tidying office and upgrading Ubuntu. Not even triaging my inbox (except to delete the obvious spam) due to some technical stuff (servers being down, laptop not synched).

still working on corpus collection talk

Not much to report, mostly an issue of wrapping my head around what we did 6 months ago.

I'm experimenting with a tool called Prezi. One annoying thing about it is that it doesn't seem to support uploading through proxy servers on Linux. To work around this, I'm using a program called rdesktop to talk to a Windows machine.

3DC on hold, preparing talk

Have pretty much handed off my 3D-Coform work for the next month or so. Now preparing a talk on some work I did before. Will be visiting Orleans next week. Talking to undergrad computer science students about open source, and also to the NLP group about our corpus extraction work. Spent yesterday catching up with my email, and making a bit more progress on the open source talk and starting to think about the corpus extraction talk.

virtual machine working

Have finally gotten things a bit into gear with the virtual machine stuff. Two things that were tripping me up and also a new trick:

• I'd forgotten about the Debian udev persistent rules feature that made it tricky to set up networking when I cloned my virtual machines. The VirtualBox issue tracker describes a solution which basically tells it to ignore virtual network cards from VirtualBox.
• I'd forgotten to select a "bridging" networking adapter
• VBoxHeadless --startvm vmname (avoids some extra windows, once you have a working ssh server and can log into your box)

Once I'd gotten over those silly technical hurdles, the rest of it was relatively smooth sailing. The basic workflow is something like

1. Create stock VM, installing a fairly barebones Debian on it (I used a business card network install ISO and deselected all the tasks). This is probably going too far, but I wanted to minimise assumptions so that my INSTALL instructions are more likely to work on somebody else's machine.
2. Clone VM: test INSTALL instructions on VM. They'll probably fall flat on their face, so lots of README/INSTALL tweaking to make sure you get the details right
3. (If you can be bothered): throw away Clone and start #2 again -- should be much more straightforward

The idea is that by doing something like this, you'll given your INSTALL instructions a test drive, which gives you an idea about hidden assumptions and what not. Granted, you being the test driver is not ideal, but it's better than nothing...

Now this doesn't mean all is done and dusted. Unfortunately, I'm only up to halfway in the installation process for the Duct Tape Monster. Stuck on a random component that seems to be either slow or stuck. EDIT: not sure what went wrong, was stuck on reading the output. Some sort of deadlock, I guess. "Fixed" by using exactly the same version of the GATE framework as on my development machine.

virtual machine boredom

Yesterday, I was sluggishly trying to set up a virtual machine for our infrastructure. I found myself not really paying much attention to the task, taking each progress bar as an opportunity to wander off and do nothing useful. Should have been probably concentrating on the analysis, or maybe even doing Darcs stuff instead.

More of the same today, hopefully a bit more disciplined

louvre stuff sucked down

OK, so I managed to finish yesterday tagsoup adventure. Ran into some bits where in retrieving pages from the louvre site, I was getting funny characters. This was due to using String as the internal representation of the data but without actual proper Unicode code points in the Chars (I think). I struggled for a bit trying to get the HTTP library to use ByteString instead (it accepts both), but gave up on some type inference problem [:-( !]. Instead I just told the IO stuff to save the files as binary, ie. just treat the Chars as padded bytes rather than trying to encode them. Anyway, I now have a sort of parallel corpus of 275 texts in English, 303 in French (so some missing English translations). English texts have been processed with the automatic extractor.

Storing this thing is a bit awkward since I don't know the best practices. You have lots of tiny decisions to make like:

• how to store things? just files on the filesystem or through some mechanism like a database engine
• if the former (pro: Eric knows what he's doing), then how do you structure the directories/files? I have {fr,en}{,-orig,-processed} So note the choices there: relatively flat structure (no directory hierarchy), and files "repeated" across the directories. The same file at various stages of processing might be fr-orig/foo.html, fr/foo.txt, fr-processed/foo.txt.debug. Maybe this could have all gone into the same directory fr/.
• how do you name the files? I have my files named with a unique identifier from the Louvre, so I get things like fr/work-10134198673225779 and en/work-10134198673225779. The unique identifier is easy to derive (grab it from the URL parameters) and it allows you represent that a text has different translations (ie. it's the same name in fr/en). But it's also totally opaque to me as a human. Not good

So I guess there's three basic tracks I want to cover next:

1. Packaging everything up: updating the README and virtual machine to cover the latest infrastructure (I may be going away at the end of October, so I want this done early)
2. Analysing the output (this time focusing on the Louvre stuff and probably forgetting about Sussex Sculptures)... which I keep putting off because it's a Thinky task rather than a Do-ey one.
3. Improve the infrastructure (this sort of depends on #2). For example, the current RMRS based extractor is really dumb: it just assumes that any two grandchildren of a verb node are its arguments in a relation, without actually paying attention to which one should be the subject and which one should be the object. RMRS actually tells us enough to distinguish between the two, but I just wanted something quick and dirty which works. I could also (somewhat reluctantly) think about rewriting the extractor in Scala. This would minimise the number of languages being used in the infrastructure, allow for easier plugging into GATE (no spawning processes, just invoke a method), and also give me a chance to learn more Scala.

soupe aux balises

Managed to get some simple processing of the Louvre HTML with TagSoup. I love the TagSoup library because it gives the same kind of whipupitude you might expect from Perl, the ability to just throw together to do something that works by eyeball. You can use it as a sort of tag-aware grep, for example, taking a potentially malformed page and extracting only the interesting text:

The Chiusi funerary monuments

This plaque from an Etruscan funerary monument was discovered in 1843. It entered the Louvre's collections in 1863 with the purchase by Napoleon III of the Marquis Campana's antiquities collection. It comes from the region of Chiusi in Umbria, where it was created around 490-480 BC. At that time, the Chiusi workshops were particularly famous for their funerary monuments - stone columns that were used to indicate the placement of tombs or to hold urns. This plaque is characteristic of this production. The bas-relief is sculpted in a local stone that, when quarried, gives off a very disagreeable odor caused by the infiltration of sulfurous water. For this reason it is called "fetid limestone".

A funeral ceremony
...

The TagSoup model should remind you a bit of SAX. You get a list of objects like TagOpen "a" [("href","http://...)..] which you can trundle through with your basic Haskell stdlib higher order list functions. The kind of code you get is not just terse, but simple, very very simple. Just a pipeline of function compositions. Here's the pipeline I used (read it right-to-left, mathy style like f o g)

louvreText = intersperse (TagText "\n\n")
           . map trimText
           . filter isRealText
           . takeWhile (~/= "<div id=bibliographie>")
           . drop 1
           . dropWhile (\x -> x ~/= "Description" && x ~/= "Notice")

The two part (with Description) chops off any bits of the documentation before a bit of HTML text that either says "Description" or "Notice", the second part chops off that little bit of text, then we grab anything up to the bibliographie div, and finally we get rid of any boring text (boring here is defined as whitespace only -- isRealText is a small helper function I wrote), get rid of the whitespace surrounding the HTML text we do have, and then insert double newlines in between each paragraph.

There... so much for my Haskell propaganda. It's not so much about building short programs -- code golf is stupid -- but simple ones. Simple here meaning few moving parts, absence of bureaucracy, homogeneity (ie. just a pipeline). Understanding what a higher order function is may be not so simple at first, but once you're happy with that, you get to knock out little programs like the above without having to think so much.

Today, I intend to continue my TagSoup adventure, grabbing links to all 303 texts and fetching the corresponding documents. Then I'll have built up a little mini corpus which I can do some more interesting text processing on. I'm afraid I'm still doing this to procrastinate on the main job of analysing the output on the Sussex Sculptures texts... probably one of my personal failings, a tendency to prefer tasks that involve Doing rather than Thinking :-/

tag soup ahead

Mostly spent yesterday writing a Darcs email.

Otherwise, I took a bit of a look at the SketchEngine (mainly with an interest in using it as a corpus collection tool.

Also, I found the bit of the Louvre website where you can find the one text example I tend to lean on. So lots more samples ahead. Unfortunately, their content is not valid XHTML (says the W3C validator), so I'll have to use a TagSoup approach to processing it. Maybe I should mention this to the site admin if they have contact details.

I think I'll spend today processing some Louvre text with TagSoup and otherwise thinking about corpus collection and storage. I'm probably just doing this to put off doing the kind of analysis I've been promising since last week...

lundi

Not much to report as I wrote a Friday wrap-up. This week, I think I'll be tackling that analysis of automatic annotator outputs. I think I just have to grab some texts and look at them first, no point trying to work out what I'm looking for yet

debug output

Trying the approach of moving the Monday morning blog post to Friday evening (maybe with a shorter post on Monday) to maximise downward slopiness.

I didn't do any actual analysis because I got caught up with trying to tweak the output so that it was more useful for me.

The context is that I use XSLT to convert the GATE sausage machine output into something that AnnoMAD can read. I also have parallel scripts that convert the same sausage machine output to something friendlier for human debuggers. Today I wanted to tweak it, but then decided I was bothered by the duplication between the scripts (which makes maintainability hard). So I ended up spending my time refactoring and tuning stylesheets. The new approach is to translate everything into an intermediary pivot format and then convert from the pivot. This means that any knowledge-intensive stuff can be funnelled into the translate-to-pivot step.

Things I learned from the process:

• PXSL supports declaring element text on same line as element
• XSLT (?) has a normalize-space() function you can call in selectors
• XSLT include to merge stylesheets, import for merge + default override
• XPath: you can flatten by selecting on descendant

Here's what the debug output looks like on one of the Sussex Sculptures:

Long Man of [Wilmington]crm:E53.Place 


The figure either holds lances, staves or spears or the uprights possibly represent portals.
-- represent(upright,portal) --
-- hold(figure,lance) -- The uprights are almost symmetrical, giving the illusion of an unfinished rectangular frame.
-- give(illusion,frame) -- Both legs are now turned to the East but the feet were originally wider apart and both facing outwards as seen in the [1874]crm:E50.Date 
 photo. The figure is naked and featureless but there is a possibility that it was originally clothed – early drawings indicate this and also place facial features and make the head helmet shaped. Interestingly, the figure’s proportions only appear correct from the air. There is elongation to counteract foreshortening (unlike Cerne or the Uffington horse).
== Annotations ==
crm:E53.Place   Wilmington
crm:E50.Date    1874
== Relations ==

hold(figure,lance)
represent(upright,portal)
give(illusion,frame)

monster running

Yesterday felt productive, although perhaps in a semi-mindless sort of way. I was trying to figure out why the automatic annotation tool just spat back these argh go crazy type errors whereas my test script appeared to be working.

Turns out it was several issues including me being wrong about the test script working, and also a permissions problem. I'm sure there's another issue in there as well.

No more clicky clicky

I guess what was satisfying about yesterday was a feeling of overcoming mindlessness during the debugging process. I was being awfully slow-minded about it, but finally managed to get my shit together and start teasing the problem apart.

Here was this PHP script being called from the Flash UI, and the way I was feeding stuff to the PHP middle layer was by loading the flash script, loading files in the GUI and clicking on "annotate".

Silly. I ended up writing a small Haskell script to replace the Flash UI for testing:

import Network.URL
import Network.HTTP
import System.SimpleArgs

main = do
 html <- readFile =<< getArgs
 let req = postRequest . exportURL $ mkUrl 42 html
 print req
 rsp  <- simpleHTTP req
 print rsp
 rspTxt <- getResponseBody rsp
 putStrLn "=== Response body ==="
 putStrLn rspTxt

mkUrl i html =
 URL (Absolute (Host (HTTP False) "localhost" Nothing)) "HST/auto_annotate_text.php"
   [("id",show i), ("html_text",html)]
 where
  host = "localhost"

eNothing to sneeze at, just evidence of starting to learn a bit more about HTTP and CGI (ugh, do the parameters really all get encoded in the query string?), very slowly crawl out of my pathetic ignorance. It's like telneting to a web server and writing HTTP headers by hand, the kind of thing you ought to do to get a "feel" for what's underneath all the magic. Anyway, being able to just mash the up arrow on my terminal rather reloading the Flash UI and going through all these stupid mouse actions certainly improved my productivity.

Other little bits of getting my shit together: dumping intermediary outputs in a single temporary directory rather than a bazillion different temporary files (temp-9861/{foo,bar} rather than /foo-38x8, /tmp/bar-9zcg); redirecting output/stderr to a tmp file (because I wouldn't be able to see it from the web, duh); refactoring my scripts so there were fewer code paths to debug; replacing an intermediate shell script with just another spawn and pipe. Just tiny little things you do step back from the general hazy mindedness.

Internal analysis

Now comes the potentially tricky part. I've run the monster on all the sculptures in the Sussex Sculptures database. I have some stylesheets (in PXSL, of course), converting the output into something human readable and debuggable. Now I need to wrap my head around the problem of analysing the output to see how we're doing. Tricky... don't know where to start.

Basically, we're looking to answer two questions:

1. Sorta precision-y: How good are the annotations/relations that we find? (what to fix, etc)
2. Sorta recall-y: What annotations/relations are missing?

I guess I'll go with some sort of iterative approach. Think up a kind of prototype analysis, randomly select two texts. Try to analyse them. Re-think analysis, pick another two texts, and repeat until I've got a good sense of knowing what to look for. Then pick 10 texts and analyse them properly.

the monster as a diagram

After just a bit of procrastination (and struggling with Inkscape), I managed to create a diagram showing the current state of the duct tape monster. The purple bits are components that I wrote (AnnoMAD has a tiny bit of purple to indicate some modifications)

I derive some childish amusement from detailing the languages that went into this: Flex (Flash) for AnnoMAD (soon to be C++ and QT?), PHP for the annotate layer (sigh), Bash for the wrapper, Java for GATE, Scala for the RASP/extractor wrapper, Common Lisp for RASP and Haskell for the RMRS extractor. Why the languages? Well, I guess it goes to show that there's a certain amount of diversity in the domain -- different people know different languages -- but that doesn't necessarily stop us from trying to smoosh together our various bits of jigsaw puzzle (GATE in particular, as much as I resent the Java, being a fairly nice example of trying to work out a universal smooshing framework).

I keep hearing that actually Haskell on JVM has been done before. But I wonder what it would take for it to be practically useful. For example, I tend to depend on lots of packages on Hackage. Are the various Haskell on JVM approaches going to reach a state where I can continue to make use of them (perhaps they'd be installed as jar files?)

Today's work is to try and figure out why AnnoMAD is not happily talking to the monster. The shell script tests seem to work, but going the clicky-clicky route leads to a frowny face Eric. To do this, I'd like to refactor some of my shell scripts: I have a test version and a production version of the script, which really ought to be sharing more code so that I can be more confident that something that works in test will work in production.

If I manage to sort this out early enough, then I'd like to run the monster on 10 documents from the Sussex sculptures database... see what we get back

bright future, eh?

Yesterday I attended a university-local conference for Early Career Researchers, basically a sort of helping hand for folks like me who are in the relatively early stages of their academic careers, stuff like writing successful grant proposals and what not. One of the nice things about this university, a sort of attitude of nurturing and investing in their staff.

I think it was useful, although I must admit I tuned out a lot of the morning session because I was too busy scribbling notes on the computer game idea I got, called Early Career Researcher (what else?).

This week I've got two things on my plate:

1. Informal presentation, taking stock of where we are in 3D-Coform, basically presenting the current duct tape monster (presumably eschewing all the actual bits-of-string details), just showing over all where all the pieces fit together. This is basically a variant of the Write-To-Think exercise. Present-To-Think. Use the mini presentation to get a sense for how to go forward
2. Do an internal mini evaluation of the current annotation/relations extractor. Run it on a handful of texts and do an analysis of what it finds.

I've got these two tasks because I complained about not really knowing how to tackle the problems of linking relations to the ontology, nor to the annotations they're trying to relate. Very grateful to have something concrete to gain traction with!

RASP plus extractor now talking to GATE

At last! I should probably write the what-I-did-Friday blog post at the end of the day rather than Monday morning because now I have no idea what I worked on.

But I do know that with learning a bit of the GATE API and a bit of Scala, I was finally able to send the output from my RMRS graph-based relations extractor to GATE in the form of sentence level annotations. As input GATE gives us as input a set of annotations over a body of text (basically set of attribute-value pairs, plus left/right offsets). We grab all the input annotations marked "Sentence", sort them by offsets and moosh them down into RASP input. Off this goes to RASP and then to my relations extractor and comes back is a list of list of relations. So my plugin just does a very simple thing of pairing each input Sentence annotation with the corresponding list of relations, creating for each relation a new GATE "Relation" annotation with the same offsets as its sentence. Child's play if you know what you're doing, a week of hacking since I didn't.

Implications of having the output as GATE annotations? Well, now maybe it'll be easier to think about how on earth I'm going to align the NP annotations with the Relation ones. Also I get some nice little bonuses like being able to use whatever tokenisers or whatever GATE works with. We could even think about doing some kind of domain-specific-knowledge tricks like tagging all output from a certain Gazeteer as noun phrases for input to RASP...

RASP plugin now with actual parsing

Didn't manage to accomplish what I'd set out to yesterday: turns out I was wrong about the Digital Pebble plugin for RASP. I think it must have been emitting RASP tagged input for an earlier version of the parser. Took me a while to understand that the parser was simply not accepting the input (or rather, it was accepting it, but the input was causing it to crash). The fix turned out to be easy, just tweak the Java code so that it emits something slightly different.

Also, my brief foray into YAML was a bit of a failure. Turns out that for what I wanted to do, the JYaml parser had bug where it was confused by sequences of sequences of mappings like

- - a:b
  - c:d

Phooey. Oh well, at least in trying to work out what was going on, I got a taste of YAML and the kinds of features it supports.

I think today I'll actually have to try out what I said I would do yesterday. Only now I'm starting from actually being able to talk to the relations extractor. Guess I'll be switching to JSON

scala bits listening to Haskell bits but not understanding

Got the relations extractor to output YAML instead of the simple notation for triples pred(s,o). This is really a silly little job which you don't need something like YAML for, but like I said, it was mostly just a way of getting a concrete taste of the language and seeing what libraries are available. Right now I'm using data-object-yaml because it's convenient (high-level enough). I wonder why there isn't some sort of typeclass for it like the json library has; could maybe just make one myself (multiparameter?) but doesn't matter. I have all of two functions. One handy feature is data-object-yaml is that it shares an internal representation with data-object-json, so I can just treat them as two ways of rendering the same data.

The Scala wrapper now knows how to call the relations extractor and print its results back. I didn't really know what I was doing in terms of spawning a process, so I just imitated the original Java code using Threads and Runtime.getRuntime().exec(foo). I kind of wanted to play with Scala actors instead of threads, but I guess now is not the time.

Today's agenda may be a bit harder: I need to get the stuff from the relations extractor parsed and transformed into GATE annotations. I'll probably use a Java library called JYaml to do this. One bit of trickiness will be zipping the list of list of relations with the list of sentences. The main hard part will be learning enough of the GATE API and overall model for things.

more java/scala

Copied over the Java module and changed some of the private methods to public as I mentioned yesterday. Today's plan is to write my YAML-ised variant of the relations extractor -- shouldn't take more than an hour -- and set up a standalone pipeline that calls RASP, pipes output to said variant. If I get that done, then I can start forcing myself to learn enough Scala to make a system call and read/write output.

Resistance to learning. What am I, 40?

plugin plugged in

My little bit of progress yesterday was to successfully create a trivial GATE plugin in Scala. All it did was to subclass the RASP plugin from Digital Pebble, so when you ran it, it just output the same thing as the DP one. The one hiccup I encountered was some kind of Object Not Found error message, which basically meant that I needed to tell my plugin to include the scala-library.jar file, after which everything was OK.

So that's pretty cool, now with Scala I get to have a slightly less painful way to do Java stuff. Maybe it would be nice to have this working directly in Haskell, but it's better than nothing and learning a new language is probably not so bad for me anyway.

Today, my hope is to extend the plugin to do something a little different, maybe not exactly the relations extraction I have in mind. Even just appending "hello" to all annotations that the DP plugin would output would be a start.

One of the obstacles I've encountered is that functionality I need to make my plugin work is marked private by the DP plugin. So to grab at this functionality, I either need to:

• make a copy of the source code and just change the private to public
• re-implement the functionality in Scala

I think I'm going with the first option. This is the sort of thing that makes me feel like I have no idea how to do Object Oriented Programming. I vaguely understand why the practice of hiding functionality in modules is important (you want a nice clean API), but from a code re-use standpoint, it really gets in the way, and sometimes I feel like it's not always clear when it's right to hide and when it's right to expose functionality. I would tend to think that if you program in a functional style -- no mutation -- it'd be a lot safer to expose a lot more code, so you'd have a lot more liberty and can make the decision to hide stuff only where it really is beneficial from an API-design standpoint. Is people doing copy and paste programming (which I'm about to do) is a greater evil than the confusion from insufficiently black boxen?

the plan continues

Posting more thoughts now because I'm away for a 4-day weekend (AngloHaskell) and don't want to forget.

Right, so I've written that Scala hello world I wanted, and have also done a stupid transliteration of the GATE how-to-write-a-plugin Java template into Scala (I say transliteration in that I'm not so much translating as shuffling Java-sounding-things into Scala-sounding-things without really knowing what I'm doing). So, without any claim that this is right, here is writing a GATE plugin in Scala.

The next step is to actually try and do something useful with this. Basically, I think what I want to produce is something like this:

So you may notice the not just the completely gratuitous use of Scala but also of the YAML format as the output of the relations extractor and the input of the extractor plugin. (I suspect there aren't any mature YAML parsers for Scala and I'll just have to use a Java one instead). I think you can say two things about this: on the one hand, it's completely idiotic: the mission here isn't to go learn some new programming language or file metaformat..., it's to extract relations.

On the other hand, maybe it's not 100% foolish. OK, my tendency to neophilia and wanting to do things in the Right or Clean way can lead to a certain paralysis or time wasted just dicking around. But it provides me with skills that come in handy later, in completely different contexts. Programming in Scala and YAML, making presentations in Prezi may not be immediately useful, but they could pay off in the future. You don't always know what you're going to need to know when. It's good to have options. Anyway, that's the old investment argument.

A slight twist to that is that, it's actually kinda hard to make myself sit down and learn something just for the sake of learning it; I really need some kind of context, some kind of reason to do the learning. And so this is a semi-bad reason -- I don't really need Scala or YAML for the job -- but it is a reason.

Another argument is that the task at hand involves lots of boring stupid bits, unpacking things from one format and re-packing them in another, plugging one tool into another, etc, and you know I feel about boring stupid bits. Giving myself something to play with at least makes it easier for me to make progress because I can spend less time resisting the boring/stupid and more time exploiting the boring/stupid for play. There needs to be a lot of room for just playing with shit, know what I mean?

battle plan

Anyway, I've taken a look at the GATE plugins source, so what I think I'll try to do is something like this:

1. run pre-existing RASP plugin from GATE gui on a "john loves mary" example -- does this thing even work?
2. subclass the DigitalPebble RASP plugin (provided it works), or if it doesn't just do copy+paste programming
3. play with JYAML for YAML processing.

rasp now connected to annotator

Yesterday (after much procrastination), I did the very easy work of connecting my RMRS graph based relations extractor to the annotation tool. Easy means it was all mechanical nonsense:

• packaging up my library to be cabal installable (should stick it on patch-tag)
• copying a Haskell script to a different directory
• adapting it to output XML
• dealing with silly things like shell scripts and paths

Silly stuff that I could get done in like one hour if I could just sit down and focus on it. Instead, I had to stay late, sigh.

Anyway, it's done. So I now have relations extraction using an approach I'm a bit more comfortable with. I should really compare it with how something like MultiPAX works.

What I'd like to work on now is the second half of what I talked about yesterday: turning my relations extractor into a GATE component. I think the way I want to do it is:

1. Write hello world in Scala - I know it's silly for me to be learning a whole new language just to do some stupid Java scripting, but (A) I still find Java somewhat abhorrent even thought I'm not nearly a good enough programmer to be hating on Java as much as I do [I probably don't have good reasons, just dumb Eric ones like "gah, it's so verbose" and "I don't know how to deal with objects reasonable"and (B) learning new things tends to pay off in the long run
2. Find the links to the GATE API
3. Write a variant of the Haskell program that returns relations in a some sort of easy to parse line-based mini-language
4. Write a Scala program that makes a system call to the RASP shell script and which then feeds the output to my Haskell program. If I was more clever, I'd work out how to do this directly in Haskell :-)

relations from rmrs

I've got my toy RMRS-based relations extractor doing something potentially useful now. Recalling previous posts, the relations extractor snarfs up an RMRS output from the RASP parser, turns into into a graph representation and traverses the graph looking for things which may be relations (interesting ones, not just grammatical relations).

So far the definition of "interesting" is pretty stupid. Given an RMRS graph G, the set of interesting relations consists of the triples pred(subj, obj) where

• pred is some vertex in G with category "verb",
• subj/obj are two vertices either with category "noun" or type "constant"
• subj/obj do not have any paths leading out of them (they are "leaf" nodes)
• there is a path from the pred to subj, and from pred and obj
• there are exactly two paths from pred to a some noun leaf node

The definition is stupid in many ways. For example, we don't really care which one is the subject and which one is object, which is obviously the wrong thing to do. We also don't account for verb nodes which have paths to more than two noun nodes, because we (I) have no idea what to do with them. But doesn't matter for now. The priority is just to get the scaffolding in place. Later on, we can add refinements such as "one of the edges from pred to subj must be ARG1".

But at least we have something that works on the whole graph, that can think about relations beyond the immediate triple.

So revisiting our old friend "According to the inscription engraved on the belly, this oinochoe was part of a wine service belonging to an Etruscan named Karkana.", the relations we extract are

• engrave(inscription,belly)
• name(etruscan,karkana)

Here are my renderings of the RMRS. See if you can make something better of it.

l146
l1:_according_p(e2)
l3:_to_p(e4) l3:ARG2(x6)
l5:_the_q(x6) l5:RSTR(h40) l5:BODY(h41)
l7:_inscription_n(x6)
l9:_engrave_v(e10) l9:ARG2(x6) l9:ARGN(e12)
l11:_on_p(e12) l11:ARG2(x14)
l13:_the_q(x14) l13:RSTR(h20) l13:BODY(h21)
l15:_belly_n(x14)
l34:prpstn_m_rel(h35)
l54:_this_q(u55) l54:RSTR(h61) l54:BODY(h62)
l56:_oinochoe_n(x55)
l64:_be_v(e65)
l66:_part_n(x67) l66:ARG1(x71)
l68:_of_p(e69) l68:ARG2(x71)
l70:_a_q(x71) l70:RSTR(h119) l70:BODY(h120)
l72:_wine_n(x73)
l74:_service_n(x75)
l76:_belong_v(e77) l76:ARGN(e79)
l78:_to_p(e79) l78:ARG2(x81)
l80:_an_q(x81) l80:RSTR(h102) l80:BODY(h103)
l82:_etruscan_n(x81)
l84:_name_v(e85) l84:ARG2(x81) l84:ARG2(x87)
l86:proper_q_rel(x87) l86:RSTR(h89) l86:BODY(h90)
l88:named_rel(x87) l88:CARG(_karkana_)
l96:prpstn_m_rel(h97)
l135:prpstn_m_rel(h138)
l137:ARG1(x55)
h7 -ing- h34, h66 -ing- h68, h82 -ing- h96
h40 = l7, h35 = l9, h20 = l15
h138 = l134, h61 = l56, h119 = l115
h102 = l82, h97 = l84, h89 = l88

next up

The next tasks I'd like to take on are (A) hooking this up to the duct tape monster and (B) maybe making this extractor available as a GATE module. I want to do A in a really stupid way. Just take the raw text, push it to RASP, and then to my extractor and then dump the relations extractor back to the Monster. This is wrong because it gives us basically no hope of connecting the relations to the annotations they are supposed to relate. For that matter, connecting relations to the annotations is kind of a tough/boring problem for me, which makes me keep resisting working on this :-(

(B) is potentially a waste of time, but then that's what I worried about my playing with the RMRS stuff, and look maybe it bore fruit. Maybe the fantasy is true after all, that if academics just basically do what they feel like/what seems interesting to them, as long as it's work to them, they do useful things. For (B), I'm thinking I could go learn some Scala and see how easy it is to use an eminently Java-ish program with the supposedly compatible Scala. I wouldn't be doing anything really fancy, I hope, just glorified shell scripting. The hope would be to able to pair each sentence annotation in GATE with a relations annotation, a la MultiPAX. But the idea is that writing for loops and other stupid boilerplate, ie. Java, is boring. Maybe the ability to zip/map on a pair would make it more fun.

Still... in the back of my mind is the lingering worry. I have annotations. I have relations. But the two don't know anything about each other.... :-/

the small brown dog barks

I forgot to mention, here's a RMRS graph from a RASP parse of "the small brown dog barks":

creeping forward a little more

Yesterday I learned a little bit more about RMRS (mostly the stare-at-formulae kind of learning, not the read-a-paper kind), that:

• some "built-in" EPs have handles as arguments, eg. prpstn_m_rel, so I used QEQ constraints to connect them to other nodes and
• I had omitted constants from my model, eg, l53:named_rel(x52) l53:CARG(_napoleon_)

So took those into account and did a few tiny aesthetic tweaks to my graphviz converter. The result of which is that a sentence like This vase entered the Louvre in 1863 following the purchase by Napoleon III two years earlier of the collection of the Marquis Campana. now gets an RMRS graph like this:

Ignore the "WIBBLE"; I was trying to figure out how to include the original sentence as a caption (which is tricky because I generate this graph by using various graphviz tools to break up the original dot file into connected components, etc so I'd end up running into annoying issues like escaping strings for use within a shell expression -- sigh).

One note about this: it turns out that my graph builder sort of cheats, in that if a node A can be connected to multiple nodes, B1, B2.. BN, it will actually only connect to the first one it finds. This is why the graphs are semi-readable. I suppose what I could do to play it safer is to connect to all of the nodes -- I'd have to switch from the Maybe to the [] monad -- and then use a graphviz utility like tred to make the graph readable again.

I'm not entirely sure where to go from here. I think I should probably accept that I'm not going to be able to invest much more effort in learning about RMRS and should start trying to exploit what I have now. For example, I could take advantage of the fact that I have a fancy graph structure to extract relations out of indirectly connected nodes. Taking the example of the graph above, I wouldn't be interested in a relation like "enter(bare_div_q_rel, the)", but what may stand a chance of being a lot more interesting to me could simply walk down to edges and extract "enter(vase, louvre)" instead.

tiny bit of RMRS progress

Crept forward a millimeter on RMRS: After googling my way to Projecting RMRS from Tiger Dependencies, I know that the ING constraints stand for "In Group", and according to the footnote in that paper

Whenever two handles are related via an ing constraint, they can be understood to be conjoined. This is relevant, e.g., for intersective modification, since a quantifier that outscopes the modified
noun must also take scope over the modifier.

So if I want to see this in action, I suppose I should try RASP out on a fierce black cat example and see what the semantics is.

I looked at a few more outputs with this ING stuff in mind, but to my disappointment, didn't see anything that would help me connect the graphs more. So I wonder: what's with all these missing handles? Did my XML translation just go wrong? Is this what you get for parsing real world sentences? Is this some sort of underspecification thing? All I know is that I get a lot of disconnectedness in my graphs, clumps of connected components really, and some ing constraints on handles which never appear anywhere else

caaaaaatch up

No report for the past few days because I've been either away, ill, or plain catching up.

Got to wrap my head around those RMRS formulas a bit better, maybe take a look at the RASP code to see what these ing-{a,b} things are

rmrs via graphviz via fgl

I pretty much finished my RMRS visualiser yesterday, first transforming the XML files into a more readable text notation and then sucking the internal data structure into a graph representation using Martin Erwig's Functional Graph Library. A nice thing about using the fgl is that I can then just plug the stuff straight into Ivan Lazar Miljenovic's graphviz bindings. Nary lifting a finger (1.5 days of hacking), I've got both an ADT, a graph representation and a visualisation of an RMRS semantics coming out of a robust parser. Yay Haskell? :-)

Here's an example of such a semantic formula. The sentence being represented is According to the inscription engraved on the belly, this oinochoe was part of a wine service belonging to an Etruscan named Karkana.


I'm not sure how surprised I should be that the graph is not very connected. It could be that I'm just reverse-engineering (while scrolling through/squinting at the MRS paper) RMRS wrong. Or this could just be what you can expect out of a robust parser on real data.

the graphviz tip

By the way, digging through Google and the graphviz documentation, I found a useful trick for compacting your not-very-connected graphs: ccomps -x | dot | gvpack | neato -n2 (ccomps is a graphviz tool that breaks your graph into its connected components; dot just lays them out; I think gvpack somehow squishes everything together and neato I guess must place each of the subgraphs.

obligatory Haskell snippet

Also, to give an idea what the whole converter looks like, this is a sketch of mostly type signatures an a hint of code (this will be BSD3 or ISC licensed or some sort of permissive thing, but I'm not sure if it's worth putting it online yet):

rmrsXmlToGraphviz :: Xml.Element -> IO ()
rmrsXmlToGraphviz e =
 case getFormula e of
   Just e  -> do let g = toGraph e
                 writeFile "/tmp/foo.dot" (printIt $ graphToDot formatter g)
   Nothing -> fail "Not an XML doc"

data Formula = Formula Label [EP] [RArg] [Ing] [HConstraint]
-- plus another 40 lines fleshing out ADTs for EP, Arg, etc

getFormula :: Xml.Element -> Formula
-- about 60 lines of typeclass instances

toGraph :: Formula -> G.Gr EP RArg
-- about  20 lines converting my Formula ADT into an FGL graph

present :: Present a => String
-- a Show-like class, about another 50 lines of typeclass intstances

formatter :: GraphvizParams EP RArg () EP
formatter = nonClusteredParams { fmtNode = toLabel . snd
                               , fmtEdge = toLabel . thd3 }
 where
  toLabel = GV.Label . StrLabel . present

I've hidden away all the details about going from XML to my internal types to FGL. I mainly wanted to show how painless getting from FGL to graphviz was, basically a three-liner (although writing that 3-liner sure took a while as I tried to work my way through the API!)

So what next?

The whole point of Write First is to help you get your brain in gear, right? Now I'm not so sure what to do. I guess I ought to tweak this process a little bit so that I'm producing one-graph-per-sentence on a whole text file. Then I can sort of take a tour of the RMRS outputs for my whole example file and give myself a bit of an idea how I would go about doing relations extraction. Now that I've got a Haskell ADT and an explicit graph representation of the semantics, maybe this sort of thing won't be so hard.

PS:While I'm grateful for the free service/storage/etc, I must say that Blogger is a real pain in the butt with its eating your formatting on verbatim (well, almost verbatim) data. If I weren't so lazy, I guess I'd move to one of these Markdown-based blog engines where I'm not writing pseudo-HTML with interpreted line breaks but some proper text format.

bit of rmrs

Aside from a short meeting yesterday morning, I was mainly working on understanding the RMRS output from RASP.

I started work on a small Haskell script to convert the XML into something easier for me to look at. It doesn't get very far, but it does convert this angly blob:

<ep cfrom='-1' cto='-1'><realpred lemma='according' pos='p'/><label vid='1'/><var sort='e' vid='2'/></ep>
<ep cfrom='-1' cto='-1'><realpred lemma='to' pos='p'/><label vid='3'/><var sort='e' vid='4'/></ep>
<ep cfrom='-1' cto='-1'><realpred lemma='the' pos='q'/><label vid='5'/><var sort='x' vid='6'/></ep>
<ep cfrom='-1' cto='-1'><gpred>prpstn_m_rel</gpred><label vid='34'/><var sort='h' vid='35'/></ep>
<ep cfrom='-1' cto='-1'><realpred lemma='inscription' pos='n'/><label vid='7'/><var sort='x' vid='6'/></ep>
<ep cfrom='-1' cto='-1'><realpred lemma='engrave' pos='v'/><label vid='9'/><var sort='e' vid='10'/></ep>

into the much easier

l1:according(e2)
l3:to(e4)
l5:the(x6)
l34:_prpstn_m_rel_(h35)
l7:inscription(x6)
l9:engrave(e10)

Doing this kind of dumb translation has been somewhat useful because it helped to do (re)discover (R)MRS in a somewhat piecemeal fashion. It's about tiny details, right? For example, I was initially representing labels as data Label = Label String Int until I discovered on closer inspection than unlike variables, you only ever get "l" as a prefix for labels so I could just use data Label = Label Int (could be a newtype instead). It may sound like a triviality, but somehow getting to grips with these kinds of little details helps me to solidify the more important ideas.

review and millimeter forward

Didn't get all that much done yesterday. Now that I'm transitioning to a once-per-day email check, it makes sense for me to consolidate my email infrastructure a bit more. So I spent some time fiddling with getmail to check both my work and my gmail account (feeding everything a single procmail script). Also modified my mailer config to get rid of all the stuff that makes checking mail convenient. Finally, I did my weekly review of pending tasks and things I'm waiting for, something I'd neglected for over a month now. So lots of cobwebs to dust off.

With that out of the day, I proceeded to try and write that slightly-beyond-hello-world JAPE rule... but somehow couldn't really bring myself to it. I was just about to write a rule that would match X "entered" Y, but then I noticed that this is something that MultiPaX already picks up from MiniPar, so I thought that maybe it'd be worthwhile to write the little bit of code needed to transform MultiPaX output into what the UI wants. It's one of those cases where you get something that looks like it's doing what you want, but on just ever so slightly closer inspection, is pretty useless. But put more positively, this means that I've finally gotten a working interface between a relations extractor, the Duct Tape Monster and the clicky clicky GUI.

Another tiny bit of progress I made was to stare at RASP's RMRS output a little bit more and wonder to myself what I'd need to do to understand that XML. I need to refresh myself a bit on MRS, nothing too unfamiliar. I also need to understand what's behind the (R)obust. But maybe most of all, I just need to write some scripts. I think I tried installing the LKB (Delph-In) to see if there were any handy tools out there... but couldn't find anything. Maybe I'd better just write my own code.

Meanwhile, I was thinking that one nice thing about this JAPE language thing is that since I can pick up arbitrary annotations, I could actually maybe write some rules that go behind a proper parser. So you get a parser to pick out nice things like NPs, and you write rules like NP "entered" NP... Using some fairly low tech stuff to extract useful bits out of something higher tech and more powerful.

JAPE hello world

On Saturday, I managed to do two dinky little things:

1. Locate an hierarchical table of all the entities and relations in the CIDOC Conceptual Reference Model. It's just a 5 page table near the front of the PDF, the kind of thing I needed to get a birds eye view of the ontology and also a listing of entities I should be caring about.
2. Write Hello World in JAPE:

   Phase:firstpass // no idea what this does
   Input:Token
   Options:control = appelt
   
   Rule:JohnLoveMary
   Priority:20
   (
   { Token.string == "John" }
   { Token.string == "loves" }
   { Token.string == "Mary" }
   ): label
   -->
   :label.Relation = { subject="j", object="m", predicate="luuurve"
                    , rule = "JohnLoveMary" }

   This sort of thing should have been a cakewalk, but I getting tripped up over stupid stuff. For example, for the longest time, I couldn't figure out why my transducer was doing absolutely nothing. Then I figured out that in the clicky-clicky interface (as much as I'd hate to admit it, GUIs are realy nice for getting started when you have no clue what's going on) you have to re-initialise the transducer component if you make any changes to the file. Stupid stuff like that is why I have to stop worrying so much about what I'm trying to do -- relations extraction -- and start thinking more in terms of hello world.
   
   For what it's worth, this is very slightly more than Hello World, because it contains the ideas of (A) matching on a sequence of tokens instead of just one (B) dumping features into the output (like subject, object, predicate).

Next up

After this tiny little bit of progress, I want to combine the two strands; I just want to find one useful relation (as in using the CIDOC CRM) that I can extract using this sort of Hello World esque rule. Having done that, I'd like to plug it into the Duct Tape Monster, getting out of it a terribly useful automatic relations extractor that knows how to extract exactly one relation... and only just...

I might also like to go back to this RASP RMRS stuff, because, well semantic graphs are a lot more interesting than triplets.

PS: This is the second of my write-first exercises! I'm proud to say that this is the first time since thesis-writing days that the first thing I did in the morning was not to check my email

more MultiPaX and GATE stuff

On Friday, I made a bit more progress playing with MultiPaX for extracting relations out of text. I've managed to save my MultiPaX pipeline into a gapp file, and after going through a little bit of debugging (*) and API spelunking, run a MiniPar-to-MultiPAX without the graphical interface. Also learned that the XML output module seems to do something reasonably sane with the annotated relations, so at least I think I have a chance of turning the output XML into something I can use.

The only problem is the relations themselves. The output does not seem to have been very useful. It's all verb(subject, object) triplets, which is fine except that the subject and objects I get tend not to be very interesting. I've also managed to plug RASP into MultiPaX, but the output doesn't really seem to be any closer to what I'm looking for either. This is no fault of the authors, mind you; they probably never promised that they could do my job for me ;-)

So I guess I have three possible routes to go down:

1. I plan to learn how to write the hello world using the JAPE transducer package from GATE. JAPE is actually kind of powerful in that on the LHS, you can match tokens on features that other tools in the pipeline have marked up, and also because on the RHS you can use arbitrary Java, so well. Anyway, the hope is that I can at least write a stupid JAPE rule that extracts a triplet out of the sentence "John loves Mary". That stupid little bit would be progress for me.
2. I could think of turning the MultiPaX output triplets into the format I want, maybe mapping a subset verbs into relations from the ontology, but with the subjects and objects given, I'm not so inclined to do so.
3. I could go play more with RMRS output from RASP (which I think MultiPaX does not use, probably just the parse tree or something). I'd read somewhere that you can see an MRS as just being a sort of graph, which made perfect no-duh sense when I saw it, but which kind of makes me go "huh?" with a little distance. So I'd like to be able to reconstruct such a graph structure from an RMRS formula. Find or write a script that can turn an RMRS XML file into graphviz output (or at least the textual format). This sort of exploration would give me a better idea what I really can do with RASP.

(*) Sigh, Java and null pointer exceptions... I think the humble option type could make a huge difference in every day programming. Get all these Java hackers moving on to Scala already. That's the trouble with programming. It's so easy to make something that looks like it works, but then when your thing breaks because of a null pointer exception through some trivial unexpected use of your code, Somebody Else has to clean up the mess. At least the offending code was open source.

habitudes

This has been a busy week... unfortunately so much in terms of getting work-work done. Was coping with a hard disk failure on my laptop (saved by Time Capsule) and catching up with some Darcs stuff. It's been a strange cycle lately of not getting much done during the week and then feeling guilty and so coming in to work on the weekends... and then not getting much done during the week again.

Email diet

In the past couple of weeks, I've also gotten it into my head that I need to go on an email diet. I find myself spending just writing email on the time and this kind of needs to stop. What I'm trying to implement is a new policy of checking email only once a day (and being very very public about it in the hope that this will keep me honest).

This is my thinking on why checking email only once a day is a good idea:

1. It reduces overhead. Sometimes a lot of email triage just consists of deleting the message because it's some sort of automated noise or whatever. I reckon that such have clearly equivalent value whether you see them within the hour or within the day. We can say that the cost of checking your mail is C * N, where C is the task-switching cost (the huh-what? factor) and N is the number of times you do a check. Let's drive N down to 1.
2. It avoids me using email as a sort of self-distraction tool. OK bored now, email! now! now! now! I have a mental "ring fence" of websites that for some mysterious reason I have been able to enforce not looking at during work hours (facebook, reddit). Now I've added Quora to the list and am starting to shift email to it after my daily check. I suspect that part of the mysterious power of the ring fence is that it contains a very small number of obviously Bad (pathological-behaviour-inducing) sites. If I grew the list too much then I may find the discipline in not visiting these sites breaking down. Anyway, email goes in the ring fence.
3. It helps me get a better feel for how much time I spend doing email a day. If I'm always doing it in short little spurts here and there, I may not feel the actual cost. But if I sweep it all into one big pile, then it becomes crystal clear.
4. It avoids the slippery slope of compromise. Some of my friends think this is impractical (I hope they're wrong!) and then I should compromise by only limiting myself to once every few hours or something. But no, no, it doesn't work that way for slightly chaotic people like me. It has to be all or nothing because if I give myself an inch...
5. It lets problems take care of themselves. Sometimes you don't need to intervene, so you can just efficiently archive the whole thread and be done with it. In the case of public mailing lists (particularly, darcs-users which I feel a certain responsibility for), this gives a chance for other people to reply to messages first. I still have a bad habit of thinking I have to reply to everything, which is clearly insane.
   
   Interestingly, this bad habit is compounded by a Getting Things Done / Inbox Zero mentality where I kind of want to make sure everything that comes through my inbox has been tracked/processed/dealt with. The problem is that in the case of public mailing lists, it's not always clear who has the responsibility of answering what messages. If everybody leaves Somebody Else to do it, then does that mean messages go unanswered? Maybe there's a technical solution, some way of tracking messages like tickets. Or maybe the answer is more social, that we need to have clearer lines of responsibility, like X is responsible for answering user queries.
6. It makes me less of an information polluter. I tend to be very very wordy (how strange, I think a past version of myself was too terse for people). If Somebody Else responds to an email first, then hopefully they'll be more to the point. Also, if I only have one chance to get it right and if I know that there will be a 24 hour delay before my next chance to emit, perhaps this will make me more careful with my choice of words. Finally, doing the email in one big batch will hopefully also tamp down on my desire to weigh in on stuff, simply from the email fatigue.
7. Tricky one - It will hopefully train people I correspond with NOT to treat email as a form of instant communication. It's fine if you want to be a slave to your email; just don't drag me into it. You want an instant reply? Tough shit, you're not getting it. I'm not sure if this one will really work out and I'm slightly afraid that I may not be able to win this battle (which would ruin the diet). For now, I've put my phone number in my signature saying that this is what people should use if they want to contact me. I'd also put my jabber address, but I'm afraid of people confusing it with email. Sigh, I wish jabber addresses looked different.

We'll have to see how long this one lasts. I hear that out there in the Real World, some people only check their email like once or twice a week, which sounds pretty unbelievable to me. Note that I've been cheating a lot already, and in fact only yesterday was I really able to be very serious about the idea (although I did break down and peek around 18:00).

Write first

A new-old habit which occurred to me this morning is that I should bring back the write-first habit from the thesis-writing days, only this time making it a koweynlg posting. or something.

The idea is that (A) it's really sad that the very first thing I do in the morning is email... seems that something else should be on my mind instead (B) having email be a one-time dose-controlled thing means that I can just it as a reward for some other activity, so I thought maybe what I should implement is Write First...

I should start the day by writing what I've been working on (write-to-think) and what I want to do. This should help me get started a bit easier because I find that my work day tends to involve a lot of initial huh-what-am-I-trying-to-do-again? Hopefully there will also be some benefit to making this the first thing I think about when I get here and not some email thing. And then once I've got my little bit of work written down and sort of know what I need to be doing today, I can reward myself my with The Email Check.

Is any of this stuff going to last? We'll have to see. Getting Things Done has lasted, so maybe some of this other stuff will as well...

minipar, pax and GATE

Was trying to play with this JAPE transducer module that GATE supplies. While Googling a bit, I stumbled on a 3rd party GATE component called PAX that seems to do a bit of what I want, extracting something that looks a bit more like relations between entities.

A little bit of yak shaving involved (plugging MiniPar into GATE means realising that the plugin ships its own minipar binary, which is dynamically linked against an old libstdc++ but also supplies the source, which you have to recompile, blah de blah)... and well, it seems to work, but nothing particularly useful.

But dang, minipar is fast and it doesn't have this startup overhead that RASP does. Haven't quite worked out how to plug PAX into RASP yet (I can't tell if I need a custom script or something), but it's possible in principle.

Got to admit, this idea of providing a universal framework into which you can plug all sorts of 3rd party components that talk to each other is really attractive. Still not sure how well it really works in practice... getting the impression that it's mostly just components within the same family that share enough assumptions that combining them actually leads to something useful (say, POS tag sets). But maybe I'm wrong. Maybe any Haskell NLP components I make should come with little bits of Java (or Scala) that would let me plug into GATE...