off to France! (for a month)

It will be good to do some legitimate GenI hacking for a change.

I'd better make the most of it. I'm a bit anxious that I won't deliver the goods.

more real work, but very slowly

I'm half-heartedly working on a Pandoc parser for MediaWiki formatted files. The good news is that I can build off another intrepid soul's first attempt. He did all the hard work of figuring out how to deal with this insane idea of using 2 apostrophes to mean italics and 3 to mean bold.

So far I've added in:

• dots and dashes (without them necessarily being hyphens or ellipsis)
  
• paragraphs
• local links
• miscellaneous corrections
• simple test suite
  

I say half-heartedly because MediaWiki is one of those formats that's hard to formalise. Still it seems like using something which parses into a tree is much better than throwing a million regular expressions at the problem (even if the million-regexps solution is easiest to start with).

If I do succeed at making something that's halfway usable, then interesting possibilities open up. Pandoc is a universal translator for formats that uses an intermediate abstract representation of markup. Processing the intermediate abstract representation is just way easier to think about than dealing with markup itself. It also means that we can envisage converting MediaWiki data into other formats.

Hey maybe this can even double as GenI work then: I need to convert from MediaWiki to trac format to migrate my content to the new wiki.

real work

Making some progress on the Wikipedia retrieval stuff. Basically, I now have a thousand articles about British hills downloaded from Wikipedia (in the mediawiki format). It took me a while because I had to

• Extend the Haskell HTTP library with proxy settings detection (copy and pasted over from the cabal-install code). This extension has been accepted, so I guess we've got a little cross-package refactoring action going on here, bacterial DNA exchange.
• Tweak the Haskell mediawiki package to use said proxy detection (trivial)
• Learn how to use the Mediawiki API (and the corresponding Haskell package)
• Write a parser for hill names as they appear in the database (sigh)
  
• Decide what to search for (I just went with the simplest component of all names)
• Decide how to filter the results
• Parse mediawiki API output (I went with the XML output, lightweight parser)
  

The latest development was to just try and see what happens if I look for exact matches. Out of 5000 hills, this gives us 1000 articles... Not too bad, I guess. It really pays to start with the simplest stupidest thing you can possibly think of.

So we got a little bit of Haskell infrastructure out of this. There may be more to come. Now I want to look into the package for parsing Mediawiki syntax. I was initially tempted to work on merging this into Pandoc, but I guess on further reflection that would be very much Not My Problem (for now). All I want to do is parse this goop and extract the most interesting bits from it.

I'll also need to remember that I've also retrived timestamps and revision IDs along with the articles.

darcs for better NLG-human collaboration

It makes me think that there may be some sort of project out there on facilitating cooperation between human and machine editors of collaborative texts, through the use of fine-grained revision control (darcs with character patches).

The idea is that you want your machine to NLG-out some Wikipedia entries, and then have humans edit it, and have the machine make more modifications to the entries without clobbering the human changes unless absolutely necessary. Right now, this may be tricky because darcs and other revision control systems are line based (or maybe KISS, it doesn't actually matter in practice), but because darcs has the ability to deal with different patch types, you could imagine a future version of darcs which isn't line-based, but character based.... this gives you a much more fine-grained notion of merging.

There actually is a git/darcs-based wiki out there (it's called gitit), so you could imagine first an importer from mediawiki history into a darcs history and then a prototype based on this.