the Scala rewrite

This week I've switched over to one of the projects that's been funding my time at Brighton, 3D-Coform. Took a while to dust off my old work from last summer, but basically slowly starting to get into a groove.

One of my goals is to improve the packaging of some complex pipeline I'd built. It involves Flash (not my choice!), PHP (I'm innocent!), Java (for GATE), Scala and Haskell. The Scala and Haskell parts are my fault, mostly a refusal to go back to writing Java. What's funny about the pipeline though is that it's essentially Java, with a thin bit of Scala which is really just calling this one dinky component written in Haskell. Making people install a Haskell compiler just for that dinky component is stupid. In my defence, writing said dinky component in Haskell was probably the right thing to do, in other words, use the tools you know best to get the job done and out the door. But now that we have a little bit of time to put into packaging it up nicely, I'm trying to replace that little dinky with of Haskell with Scala. This way all the backend bits run on JVM and we have simpler dependencies.

It's also a good way for me to learn the language a little bit better.  I guess I've reached a stopping point for the week.  I've reimplemented the shallow part of my code which defines a bunch of data types, parses from XML and pretty prints the output.  If you want to take a look, the Haskell and Scala code are not identical (the XML parsing may be a bit better in Scala version, due to reimplementation?) but I think they are roughly comparable.

The deep part is the bit that does graph traversals, basically simplifying RMRS semantics into subject-object-predicate triples.  That bit I'll have to try and implement next week.

Things I've learned so far:

• always use case classes, never use new: I was wondering why for some classes, like Option, I could just call the constructor (eg. Some), but for others I had to write new Foo(...).  Turns out that it was just because I wasn't declaring my classes to be case classes.  Using case classes means I get to avoid all the confusing stuff that mutability gives you, for example, having to worry about the distinction between equality and identity.
• I may not be using traits right, but I think I like Haskell type classes better simply because with type classes, I can separate the instance declaration/definition from the class definition
• subtypes make for easy heterogeneous lists: in Haskell I'd wrap the things I want in some existential type E and then make a list of E.  In Scala, I make my types all extend the trait E.  For some reason, I find the latter a bit "easier", either more memorable/clear or just more convenient
• to get methods associated with classes rather than objects, just create a singleton object (that extends the desired trait)
• Scala lets you pattern match XML: it's kind of neat that you can interleave XML in your code and do things like pattern matching on it.

I get the impression that you basically have two kinds of Haskeller.  The first is the kind that uses Haskell because they can take on a lot of abstraction and use it to do amazing things, the smart Haskeller.  I think I fall more into the dumb category of Haskeller who tends to use it because I find the consequences of subtyping (Liskov substitution) and mutability to be too confusing.  And maybe there are also smart-dumb Haskellers... the kind who fall into both camps simultaneously.

makedo still paying off

Another day for confidence building.  I've been benefiting greatly from my build file being written in Haskell.  This the sort of situation I've been able to cope with way way better than my Bash script

problem A: recomputing LLR

1. fragment extraction takes as input: a sentence selection, data selection, and LLR
2. data selection takes as input: sentence selection and LLR
3. NEW: after data selection, we can recompute LLR (call it LLR2) - LLR2 takes data selection as input.  There are two different ways of recomputing LLR
4. What is effect of the LLR method on fragment extraction?

Solution: add a new LLRMethod argument to fragment extraction.  Fragment extraction now takes three arguments.  Furthermore, in computing the LLR, my fragment extraction function just passes in the data selection method it received in its parameters list.

data LlrMethod = LlrNoRecompute
               | LlrRecomputeDoc
               | LlrRecomputeSent

Problem B: decoupling token alignment

1. NEW: we want to change fragment extraction by separating an implicit token alignment step from it. Fragment extraction would no longer take a sentence/data dir, but a token dir
2. Token alignment depends on data selection, and LLR method...
3. There is more than one way to do token alignment

Solution B: add a TokenAlignmentMethod argument to fragment extraction.  Instead of passing in a sentence directory and data directory, I pass in a token aligned directory computed off the arguments passed to fragment extraction.

Here's what fragment extraction looks like now:

fragmentExtraction :: SentSelMethod -> DataSelMethod -> LlrMethod -> TokenAlignmentMethod -> ExactTarget
fragmentExtraction sm dm lm tm = ExactTarget path "" $ \ RedoArgs { rname } -> do
  redoIfChange [ tokenDir, llrDir ]
  createDirectoryIfMissing True rname
  runMyIO "detect-fragments" $ [ tokenDir, llrDir </> "lexicon", rname ] ++ parallelRTS
  recurseStampExt rname [ extEnChoice, extDataPerSentence ]
 where
  path       = d_FRAGMENTS </> showPC sm </> showPC dm </> showPC lm </> showPC tm
  tokenDir   = ePath (tokenAlignment sm dm lm tm)
  llrDir     = llr2Dir sm dm lm

Bonus to Solution B: I had forgotten that after fragment extraction comes a "peers directory" step.  Thanks to the Haskell type checker, I remember that I also need to extend the peers directory step with a token alignment method argument.

Before using my makedo library, I was trying to manage all this argument passing by deconstructing paths using basename and dirname.  Crazy!  Way way better to write your build files in a language that (A) has a notion of sum types or at least Enum types (B) lets you create your own types with minimal fuss (eg. I don't need a separate file per type, just declare them) and (C) statically type checks.  To be honest, the language for this probably does not need to be Haskell.  Even Java could do the job, although I think creating different enumerations like LLRMethod vs TokenAlignmentMethod might be more of a pain.  The advantage to using Haskell over Java (aside from the usual higher order functions, etc) is the HSH library, I think.  Maybe if there was a convenient way to write shell-script-like things in Java it could work.

But Bash or Perl script?  No way!

We missed the deadline for our paper by the way, probably because I was spending so much initial time working on makedo.  But I still think it was the right thing to do.  I would not have been able to understand what I'm working on as well as I could have if I was still wading knee deep in shell script.  Sometimes you just need to step aside and go abstract away some nonsense before you get to the meat of your job.  I hope that if we don't get a paper out of this, I can at least release my makedo library, and that somebody else benefits from it.

Take home message:

Statically typed shell scripts and Makefiles are good for you.

makedo pays off

Today was a good day, a confident day.

For a bit of context, I can be a bit insecure and defensive about the time I spend on "There Must Be a Better Way to Do This", time which I sometimes worry should be spent on "Doing This".  Unlike the smart Haskellers, my "There Must be A Better Way" does not tend to result in neat abstractions or boilerplate scrapping frameworks or what not.  A lot of time is just spent going around in circles, undoing past efforts, or just changing, changing from make to Redo + Bash, to Redo + Bash-in-Haskell to Redo + Haskell, change change change, churn churn churn.

I just spent a month scratching my head over my Complicated Sausage Machine, trying to figure out how not to make it hurt so much, and I think it finally paid off!  All that going around in circles, and I can now write this patch and this other patch.  I can now do the Thing I Was Actually Supposed to Do.  This is either something I would have had to do by hand (as in running commands by hand), or tried to stuff into my Complicated Sausage Machine, horrible fragile little piece of Bash.

So it's done... and I actually understand what the sausage machine is supposed to do.  The end result is much better.  Yesterday I remarked that Haskell version seems just as complex/long as the Bash one; turns out I just needed to bit of refactoring, refactoring which would likely have been harder in the Bash.  Again the beauty contest or Line of Code reduction isn't really the point.  It's about the clarity.  For the first time, I'm feeling confident that all my rewriting efforts were useful. The Haskell rewrite of the sausage machine now does more stuff than the original Bash one (not a superset, mind you) and is way easier to understand and by virtue of factorisation, has way fewer surprises in it, and is now much easier to extend.  OK the downside is that in order to extend it for future papers, you have to know a bit of Haskell.  But just in terms of mistakes avoided for this current paper, I think we have a net win.

Now we'll just have to see what this does for our paper.