duct-tape monster

Actually managed to get something moderately useful done before running off on holiday. Behold the duct-tape monster!

1. Flash UI sends HTMLish content to PHP script
2. [x] PHP script writes HTMLish content to temp-file-A and calls a shell script
3. [xx] Shell script runs an XSLT transform to convert HTMLish content in temp-file-A to temp-file-B
4. [xx] Shell script calls a trivial Java program (calls ANNIE via GATE API) on temp-file-B
5. [xxx] Java program outputs something that looks like XML (you have to wrap it in a tag or so)
6. [xx] Shell script runs an XSLT transform on the GATE XML to get HTML-subset (temp-file-C)
7. [xx] Shell script runs an XSLT transform on the GATE XML to get ad-hoc XML (temp-file-D) for unmarshalling into list of 5-tuples
8. [x] PHP script reads temp-file-C and temp-file-D (thank you SimpleXML for the latter)
9. [x] PHP saves HTMLish content and 5-tuples to MySQL database
10. [x] PHP marshals HTMLish + tuples on output
11. Flash UI displays new HTMLish and uses tuples for various clicky-clicky-bits

Like I how I give way too much detail to make something look harder than it really is? This thing I did was very very very easy and stupid.

It just involved me (as a grumpy Haskeller) picking up a lot of new minor skills, for example, teaching a button in Flash to only display if a static debugMode bool is set (hint: Bindable), or writing my very first SQL query (in PHP even):

function queryInsertAnnotation($doc_id,$annotation) {
return sprintf("INSERT INTO hst_annotations(doc_id, name, color, entity, comment) VALUES(%s,'%s','%s','%s','%s')",
  mysql_real_escape_string($doc_id),
  mysql_real_escape_string($annotation->name),
  mysql_real_escape_string($annotation->color),
  mysql_real_escape_string($annotation->entity),
  mysql_real_escape_string($annotation->comment));
}

As for the vexing question of how to preserve pre-existing annotations and paragraph boundaries when automatically annotating the text, we decided to punt. Just get something working first for traction. Made it real easy...

annotation sets and graphs

Hmm, still not much progress :-( Although then again, I had taken the day off to help some friends move only to find I was only needed the morning.

Anyway, reading the manual on embedded GATE, two new points:

• you can use the clicky-clicky interface to create a pipeline, which you then save into a sort of XML file that you can then run using the Gate API
• their representation of annotations is a DAG (looks familiar if you've used the connecting nodes notation for chart parsers):

I wonder why it's important that it's directed, or acyclic for that matter. Seems like all you really care about in an annotation is that nodes (a) and (b) are connected by annotation 42. I guess if you're going to say it's directed, acyclic makes sense.

not GATE

I should clarify that I'm not particularly attached to using GATE per se; I'm just going to assume they know what they're doing and borrow as much from them as possible, including model and any intermediary formats for storing things. If this makes it easier for me to plug into GATE (or plug GATE into what I'm trying to do, so much the better).

thinking in circles

So what's potentially tricky for me is still to imagine how all this is going to fit together. Say I have a translation layer between the annotation-backend and the linguistic layer. As input, the translation layer is getting HTML formatted text. This tells me where all the formatting information, pre-existing annotations, and paragraphs are; but if I want to stick to a node-id model and not a character-id model, I'm going to have to introduce some sort of tokenisation process. But OK, then how do I line up the original HTML with the output tokens?

Let's focus on paragraphs now. How I say where all the paragraphs are exactly? I don't care about your fancy bolds and italics, but if I squish your paragraphs together, you're going to be one unhappy manual annotator. I know which strings they wrap around, so one simple way is just to ditch all the formatting, and all the pre-existing annotations, and tokenise each paragraph separately.

Alternatively, we could perhaps try to take them into account by making the simplifying assumption that any pre-existing annotation necessarily introduces a token boundary. In other words, if you see hello, you've instantly got two tokens hel and lo. What you would do in effect is to start out by doing a crude tokenisation step that breaks up the text by whitespace (treating any markup node as introducing whitespace on both ends). This gives you an initial annotation graph with a list of tokens and paragraphs spanning some of those tokens. After this, you re-tokenise using your linguistic black magic... this means that some of your tokens may be merged or split, which has all sorts of potentially weird consequences for your paragraph. How would you know how to wibble your paragraph boundaries? Do you have to compare the token lists and somehow infer if they've been split or merged?

Now maybe the only thing to do is work on the character level at some early point, but still output token-based outputs. For that to work your tokeniser may have to tell you which characters it's spanning. That or you'd have to adopt some kind of whitespace folding scheme. Argh, thinking myself in circles.

How do you automatically annotate a text while preserving any pre-existing formatting and annotations on it? It sounds really simple: just use a stand-off annotation approach, but (having not a single clue what I'm doing), it seems that your choice of nodes (strings vs characters) could make things potentially tricky.

bit more GATE

Playing with GATE today, trying to plug heterogeneous components together (say the ANNIE sentence splitter, the RASP POS tagger and the OpenNLP chunker), but with little success. Largely recovering clicky-clicky discoveries from previous attempts at it.

It does seem to be really very well documented. This bit of their wiki offers, for example, copy+paste code to wrap up bits of ANNIE as a "sausage machine".

Also learned that the output format they use (when you click save as XML) seems to be based on token identifiers, as opposed to say, character-based (which makes me wonder: what happens when you have tools with different ideas about what constitutes a token?)

I'm not entirely sure why I'm choosing to do this. Sort of flipping back and forth between the dummy implementation of the auto-annotator and the off-the-shelf concrete instantiation of it. I think what I'm mainly after is to figure out how to go from that formatted HTML to 3rd-party output and back.

not much progress

Spent today reviewing a Darcs patch [I'm slow that way :-(], so not that much progress.

I've learned a little bit more about how the annotation tool behaves (when you select an annotation, it retrieves the document rich text from the DB and refreshes it). Last night, I saw that basically the pieces are that we'll need some sort of a fourth layer:

• the annotation tool (Flash) - clicky clicky
  
• the middle layer (PHP) - talks to the DB
  
• general purpose translator - HTML to raw text; standoff to HTML + annotations
  
• the NLP layer (Java, LISP, shell, etc) - raw text to standoff
  

So I'm thinking the general purpose translator receives HTML text from the middle layer, and then strips it down to raw text which it sends to the IE stuff. It receives data from the IE layer in some sort of standoff annotation format and wraps it all back up in HTML plus annotations. Tempting to write the general purpose translator in Haskell as a quick prototype...

code code

Slowly starting to wrap my head around the project.

First, the model for annotations is that certain text spans have identifiers (incidentally, colours) and the annotations point at those identifiers. Such spans are created in the GUI through mouse selection.

So we have the beginnings of an API coming together: you give me text, and I give you modified text (with spans) plus the annotations.:

annotate :: TreeWithSpans Char -> (TreeWithSpans Char, [Annotation])
data Annotation = Annotation { name :: String
                         , identifier :: String -- ^ colour
                         , entity :: String -- ^ the heart of the annotation
                         , comment :: String
                         }

Potentially interesting question: how do you augment a pre-formatted text with automatically derived annotations? I suppose you can model the formatted text as a sort of tree. My first naive guess would be to say that you have to walk the tree and basically mark each zone with text in it with an identifier. Then you have to associate character positions in the raw text with identifiers. Your annotation process results in a tree too (but a much lighter one, containing only new spans). You have to use the original zone identifiers to somehow merge the two trees together. Perhaps you can make some sort of simplifying assumptions when spans in one tree overlap spans in another (like you get to nuke something). Sounds really annoying!

(I guess this is sort of taking the standoff annotation view. You store 3 separate things, raw text, formatting which references character offsets [maybe with whitespace folding?], and other formatting... but then you have to somehow nicely pack that all back up into a tree again)

I'm going to completely disregard this question for now and just trash the original text formatting.

New tricks (or the refusal to learn them)

• Some basic PHP coding, functions, objects, array. Writing XML with XMLWriter
• Simple Makefile for compiling Flex stuff (I give up! Eclipse is too fancy for me. Luckily there are some decent vim compiler/syntax scripts that let me work with .mxml files)
• Some basic SQL (although maybe DESCRIBE table; is a MySQLism), encountering its use of ternary logic.
  
• ...and a tiny bit more Flex
  

3DC progress for 19 June

Not a productive week :-/

• Monday: getting software to work (learned a bit ActionScript needed to capture and printf (Alert.show) a fault event). Meetings
  
• Tuesday: playing with RASP? nosing around a bit for CIDOC CRM stuff (found an OWL formatted file I could download)
  
• Wednesday: playing with RASP notes, some planning
• Thursday [afternoon]: twitter talk, meeting
• Friday: fighting with FlexBuilder, setting up and tidying darcs repo, trying to learn more about interface between the Flex code and the PHP stuff. Added a button to the Flex GUI that just pops up a box saying "haha, you wish!" (well, ok, it's a bit more apologetic than that). Discovered that the demo directory contains (well, duh), a useful example to help me wrap my head around what I'm trying to do.
  
• Saturday [afternoon]: refactoring some PHP code for my sanity (little bits of code that establish a connection to a MySQL server; shovelling it all to one place so that it's easier to swap a server in and out, in the case of eg. two people collaborating and using their own test servers).

Still not sure I understand the relationship between the text and the annotations (working on an ontology-based editor akin to GATE's OAT). In the bit of investigations I did, I saw that there is a writeDocument function that just saves the rich text content of the document to the DB (as in with eg. bold tags, etc). Anxious about how annotations are anchored to text:

• word offsets? char offsets? id tags associated with annotations?
• what about overlapping?

As for Information Extraction, I found this little pipeline from the GATE user manual helpful showing what one particular IE system goes through to get useful bits and pieces out:


There's also this screenshot of their ontology based editor which seems intriguing:


Could be worth plugging in the CIDOC OWL file in just for kicks...

playing with RASP

Some more progress in my introduction to Information Extraction. It's useful to confront these capital letter terms and (hopefully) discover that there's actually not that much too them.

Apparently one way to think of IE is "well, OK natural language understanding is hard; could we make it easier by redefining the problem, ie. restricting ourselves to a pre-defined set of outputs?" I'm not sure how what proper Information Extraction people would make of that view, but it's certainly a very reassuring one for your standard, fresh out of school, I-took-some-master-level-courses-in-NLP postdoc, fancy certificate in hand and not a clue in your head.

Some sort of Information Extraction for Dummies could be nice, as if I'd ever get around to reading it.

RASPy notes

Right now, I'm having a look at RASP, a fairly local (as in kilometres down the road) "domain-independent, robust parsing system for English".

I pulled the classic "gee this thing sure takes a long time to run" ...18 hours later... "oh, it expects input on stdin".

Likes so far:

• Written in LISP, go functional programming!
• Seems quite fast (or fast enough?). Maybe I just have low expectations

  $ wc /opt/rasp/scripts/README
  198 1076 7400 /opt/rasp/scripts/README
  $ time scripts/rasp.sh < /opt/rasp/scripts/README > /tmp/rasp-output
  
  real 2m15.319s
  user 1m42.606s
  sys 0m1.960s

• Well documented: no blah blah, just the things you need to know, but all the things you need to know. When I grow up, I will write documentation like this.
• Self-demonstrating: one nice documentation trick they pulled off was to parse the README file. With apologies for tediously spelling it out: (a) you're including examples with your distribution, very convenient (b) the example is clearly real world and non-trivial. Shoot, what was (c)? Cute trick anyway.
  
• Very UNIXy: RASP appears to have independent modules that you can swap in and out at leisure, replacing them with your own home-grown tools. (You can also parameterise the wrapper shell script with environment variables). Modules are
  • sentence boundary detector
  • tokeniser
  • tagger
  • morphological analyser
  • parser
• parser knows about all sorts of output formats: by default you get grammatical relations, but you can ask for PARSEVAL style outputs or Penn Treebank-y outputs

Dislikes:

• custom non-libre software license. It's great that it's open-source and gratuit, and I pretty much don't care that it's non-libre (no commercial use), although libre would certainly make life easier for everybody. The main annoyance here is the Yet Another License factor. Mind you, this sort of thing usually is not the authors' fault, just a bunch of crap imposed from above.

  Attention Conservation Notice: bog standard rant about License Proliferation. Preaching to the choir for a moment, the problem is that licenses are inherently opaque (A: you have to wade through them, B: you have to understand the implications of the terms and not the terms themselves, like "can I plug this custom licensed product X into this custom licensed product Y and redistribute it under terms Z?"). The fewer licenses we use, the more effectively we resist license proliferation, the better things will be. Sticking to well-known licenses like the GPL or the 3-clause BSD would be great.

  Perhaps one step we could take for positive change is to develop a standard open-source non-free license, ie. restricting commercial use. At least we can then sweep everything up under the same banner that everybody can understand.

• Potential dependency on proprietary Allegro Common Lisp... but maybe not because I swore I saw an OpenMCL invocation for one of the architectures.

(For all my ranting, I'm not a principled FOSS geek, just a grumpy one. In other words, it's not so much about achieving Freedom with a capital F as it is moaning that "gah! things would just fricking work better if only..." No MacOS X for me if I were more ideologically pure.)

the boy saw a man with a railroad diagram

Sometimes you start reading a book like JavaScript: The Good Parts, and you can't help thinking "gee, I wonder what a toy natural language CFG would like as a railroad diagram". This is where somebody in the Haskell community steps into a time machine and gets a couple of his students to write Ebnf2ps 12 years ago.

A little yak-shaving later (details upon request)...








My capacity to be easily amused worries me sometime...

(I guess you'd want something less trivial to make it look more railroad-y. Or maybe you could make some sort of argument that NL grammars don't lend themselves well to this sort of visualisation because any practical grammar is going to be so highly factored that you'll just have a bunch of these little blobs).

uh-oh PHP

Looks like in addition to picking up some Flex (statically typed JS + an XML gui framework?), I may need to overcome my language chauvinism long enough to hack some PHP.

This is ridiculous. Why am I even a language chauvinist in the first place? I don't get it. PHP is the language that MediaWiki is written in, very real world people who get real things done. I haven't written a line of the code myself, but somehow, something (maybe the reputation?) just triggers a sort of instinctive revulsion that, say, Python does not. Well, better suck it up and deal.

I can say quite concretely and painfully that this sort of language chauvinism is a sort of handicap. (This is not some new revelation or anything, just something I need to articulate). The problem here is that I tend to think of Haskell as being a much better programming language than basically anything popular... which is fine, except that it's been almost 10 years since I stopped programming in $popular_language... which means that part of my belief system is just (A) reading what other people say and (B) not actually being very good at $popular_language. 10 years ago it was Java (before then C++ and a bit of Tcl). Now it's Perl. I've got crappy Perl skillz for sure, so it's not fair for me to talk about Haskell vs Perl or whatever simply because I don't actually have the concrete experience to speak from. Sure I can say "Haskell is a better language because Smart Guy X says so" and "Haskell is a better language because it's got a really great type system and higher order functions"... but then what? Part of the latter is simply that after 10 years my brain has rewired itself around recursion on algebraic data types and generally trying to avoid boilerplate. I have no idea what I'm saying. It's all belief.


There's also potentially some SQL involved.

Almost feels like I'm working in the real world...

a bit of this and a bit of that

Today I managed to do three things and try to answer one question.

Install Flex Stuff

Figure out how to install Eclipse, the Flex SDK and Flex Builder. By "figure out", I mean "Google and read blog entries by people who've been there and done that. The big stumbling block for me was FB wanting to know what the Eclipse directory, which I had to search to find out.

I also played around a bit with the EclipseDarcs plugin but didn't get very far (clicking darcs record seems to have on effect).

Write Hello World

I've now written hello world in Flex. By "write", I mean "Google and copy and paste, see that it doesn't work, Google some more and finally find out that the namespaces have changed you're supposed to use the 2006 version of the schema)."

I'm trying this out in FlexBuilder/Eclipse following advice from the Pragmatic Programmer that grumpy command-line+text-editor people like me should actually learn what it's like to use an IDE (I can see the appeal, actually :-/) and vice-versa.

I guess if I wasn't using an IDE, I would be (A) using PXSL to avoid writing all that XML and (B) putting together a Makefile

Do some IE on a text file with GATE

A lot of huh-WTF playing with GATE for the first time. At the moment, I'm still at the clicky-clicky stage of things. For data I fed in some files from some previous unrelated research. No rocket science.

The key trick is to watch this handy tutorial video

NLP for Dummies?

One of my technical but non-NLP friends wanted some sort of NLP for Dummies resource to get a very rough idea what was going on (in like 2 hours). I didn't know what to tell him. Suggested Jurafsky and Martin (with the warning that it wouldn't be two hours, but maybe some sort of flip-through would help). Also found out that the NLTK book is available online with a Creative Commons BY-NC-ND license.

Seems like maybe if we want this Haskell NLP community to take off we could actually start by going through that book systematically and making sure we can do everything that NLTK can do :-P

3D-Coform

I'm staying at Brighton for at least a little bit longer (a few months). This time, I'll be working on the 3D-Coform project, doing a little bit of Information Extraction. This should be interesting in the sense that I'll have no idea what I'm doing. Good for you.

Also good (grumble) to have to play with some of these non-Haskell technologies that people use in the real world like Flex.