corpus sifter - tool for building paired data/text corpora

Here's a small webapp I wrote yesterday. Since the app (as I envisoned it) was fairly simple, I figured I would invest some time to learn how to write web stuff in Haskell. I went with the Happstack framework just because it's the most well known.

Sales pitch

I call the app "corpus sifter". It could use a nicer name. The code is BSD licensed and I'm willing to stick the Darcs repository up somewhere, maybe even upload it to hackage if anybody is interested, but I doubt it's useful enough to justify such actions.

The scenario behind this application is as follows:

• You have some pieces of structured data
• Using a search engine, you have retrieved a text file for each piece of the data
• You don't know if the data you retrieved really corresponds to the data; there may be some automatic ways of finding out, but you need to spend a little time scrutinising the data to find out

Solution? Well, you're stuck with manually sifting through the data, so you might as well make this manual process as quick and painless as possible. To this end, we present the data and text file side-by-side, and three buttons "good", "bad" and "unsure". Clicking on one of these buttons moves the pair to the corresponding "good/bad/unsure" bin from the new bin; it also presents you with another item to classify.

Two features of this approach is that is (a) fairly general and (b) simple/robust (hopefully).

generality

There is nothing in corpus sifter that is specific to the corpus in question. The data must be represented as a JSON object (which corpus sifter will render as an HTML table), and the text will just be systematically dumped out as raw text (these are wikipedia entries that I'm using, but you may have other things in mind).

simplicity/robustness?

Part of corpus sifter's design is to account for incremental corpus construction. When you begin building the corpus, you may not exactly know all the best ways of (i) sifting through it or (ii) searching for texts. So over time, you may want to revise your data. Corpus sifter supports this by adopting a very simple and stupid model of processing: the bins are represented as directories, and moving items from one bin to another consists of literally moving them on the file system.

If you change your mind about a bin (say, you hit a back button), corpus sifter will notice and move the data pair from the corresponding bin.

If over time you add more data to the corpus that you want to sift through, just dump it in the "new" directory.

In a future version of corpus sifter (maybe tomorrow?), I plan to add a feature that lets it notice that you have put something in the "new" bin which already has been classified. This would account for cases where you have new, hopefully better search results. What the tool would hypothetically do is to create some sort of backup directory and move the old data from its bin to this directory. This will be handy for cases where you incrementally refine your search methodology, leading to different results for files that you have already seen. For example, working with Wikipedia entries, I've discovered better and better ways of coping with disambiguation pages, but I need to find some graceful solution for merging these new results in with my old sifted corpus. I certainly wouldn't want to have two copies of my data lying around!

Implementation notes

You can count on me to extol the virtues of Haskell programming at any slightest opportunity. Today, the features I'm going to gush about are common among many functional programming languages (Haskell, ML-ish languages and Scala). These features are algebraic data types and pattern matching.

How Haskell helped me here is to develop a simple and generic solution for rendering any JSON object as HTML. The code is as follows (renderer in 25 lines of Haskell, entire app in about 200 lines):

jsonToHtml JSNull = noHtml
jsonToHtml (JSBool b) = stringToHtml (show b)
jsonToHtml (JSRational float r) =
  stringToHtml $ if denominator r == 1 then show (numerator r)
                                       else show (fromRational r :: Float)
jsonToHtml (JSString js) =
 case fromJSString js of
   u@('h':'t':'t':'p':':':'/':'/':u') -> anchor (abbreviate u') ! [ href u ]
   s -> stringToHtml s
 where
   abbreviate = stringToHtml . (++ "...") . take 13
jsonToHtml (JSArray xs) = unordList (map jsonToHtml xs)
jsonToHtml (JSObject o) = jTable . concatHtml . altMap dark light . map mkRow $ fromJSObject o
 where
  mkRow  (k,v) = tr $ concatHtml [ td (stringToHtml k), td (jsonToHtml v) ]
  dark  = (! [ theclass "dark" ])
  light = (! [ theclass "light" ])
  jTable x = table x ! [ theclass "json" ]

-- | map on every other element
altMap _ _ []  = []
altMap f1 f2 [x] = [ f1 x ]
altMap f1 f2 (x1:x2:xs) = f1 x1 : f2 x2 : altMap f1 f2 xs

The basic idea is that JSON strings are escaped and turned into straight HTML, numbers are represented in decimal notation (where appropriate), lists are rendered as bullets (ul tag) and here's where it gets fun, objects (key-value pairs) as tables. The nice thing here for me is that I didn't have to go read a bunch of specs to know what I had to do. Since the author of the json library chose a nice simple sum type representation for JSON objects, I could simply write a function to pattern-match on each one of the cases individually...

These key is that algebraic types helped me to think about the problem because they (i) divided the problem into cases which (ii) they made very readily apparent and (iii) which the compiler could then check that I have exhaustively explored all the cases [eg, did you consider boolean values in JSON? what about the null case?]. Furthermore, the recursion on the the types (JSON objects contain JSON objects) was very simple to deal with as just a straightforward recursive function.

Pattern matching and algebraic data types... this kind of stuff makes me really reluctant to give up statically-typed functional programming languages. Of course, there must be other ways to solve this with other languages' tools; if this would have been easier in some other language, give me a shout in the comments.

incremental development + resource constraints = messiness

Say you've got a pipeline that searches for Wikipedia articles, retrieves them and maybe does some post-processing on them.

You later on decide that you need to do some extra work on the results, say parse the disambiguation pages, retrieve the disambiguation results and then do some post-processing on them.

Now, the pipeline takes a long time or a lot of resources to run, so you definitely do not want to run it again. You've already got your WP articles; no point fetching them again. If you had had the foresight to plan for the disambiguation stuff from the beginning, then it would all be part of one nice happy pipeline, but you didn't, tough luck.

So what do you do? You develop a separate post-processing script that operates over the results of your first processing step.

... and?

You've just created a mess. Now you've got yet another script in your pipeline that you need to tell somebody to run (editing your README file), or that you need to augment your pipeline with for future use (which you are not motivated to do because it does not solve any immediate problems). Not only that, you're faced with the dilemma of whether or not you should keep both the original data fetched from the pipeline, plus the additional disambiguation results [for future tracking/reproducibility]; or if you should just mash them all together [for simplicity].

This is the kind of organic incremental growth that leads to a general feeling of sprawl and out of controlness. I don't foresee any solutions.

• One could plan better and try to get things right the first time, but that's mostly like saying one should just be more intelligent, and capable of anticipating changes down the road... "Be more clever" is just not acceptable answer.
• One could use more pilot studies to flesh out as much of the toolchain as possible on small amounts of data (which eliminates the resource constraint problem). This is probably a good idea, but it only goes so far, in the sense that you sometimes need to work with large amounts of data before you can become aware of the problems that lead you to augment your pipeline.
  
• Or maybe one could design processes to easily accommodate these kinds of incremental changes in a tidy manner.

I haven't figured out the latter bit yet, and so I've got yet another Perl script in my burgeoning toolchain that I won't know what to do with three months from now.

subsumption

I mentioned that I tend to have a lot of difficulty with polarities. By this I mean, I tend to get left and right confused, along with true/false, plus/minus, subsumes/is subsumed by.

The good news is that I think I have finally figured out why I keep tripping over subsumption, as I have done ever since I encountered the concept in a beginner's Description Logic course.

In short, we say G ⊑ S (G subsumes S) more or less if G is a subset of S, right?

The reason I think this gives me difficulty is that there's a kind of subtle "contradiction" that my very sloppy and stupid brain simply cannot handle. It goes like this:

• G is a subset of S, so G is SMALLER than S, gotcha
• G is more general than S, so G is BIGGER than S, wait, WTF?

So the problem is that "G subsumes S" conjures up a mental image of a massive blob-like G swallowing up a helpless little S. I know the mental image isn't appropriate here, but I can't help it! You say the word, and I get the blob.

Of course, if I can keep my thinking cap on for long enough, I realise that the way out of this is that to remember that G is more general than S only by virtue of containing less stuff than it. Smaller, so more general.

But I think I'm the kind of guy that does NOT reason about things so much as trying to bash them into primitive mental constructs like "smaller" and "bigger". Hence the repeated run-ins with confusion about subsumption. How did I fool these people into giving me a fancy degree?

fractional B3

I think the PyCoLi B3 scorer would be better using the Python fractions module. No point messing around with floating point when we don't actually need it. If this works, I may send a patch.