fullstop - a simple sentence segmenter in Haskell

This is something I may release on hackage shortly. It splits texts into sentences using some very simple rules of thumb.

It's way too stupid to be considered a proper segmentation tool, but I'm hoping that what I upload will offend an NLP hacker enough for them to take over and submit something better.

To fetch the full source, including Cabal file, test suite runner, etc, just darcs get --lazy http://patch-tag.com/r/kowey/fullstop

-- ----------------------------------------------------------------------
module NLP.FullStop ( segment, testSuite ) where

import Data.List.Split -- lovely little library by Brent Yorgey
{- other imports elided -}

-- tests

testSuite :: Test.Framework.Test
testSuite =
 testGroup "NLP.FullStop"
  [ testGroup "basic sanity checking"
      [ testProperty "concat (segment s) == id s, modulo whitespace" prop_segment_concat
      ]
  , testGroup "segmentation"
     [ testCaseSegments "simple"  ["Foo.", "Bar."]   "Foo. Bar."
     , testCaseSegments "condense"  ["What?!", "Yeah"]   "What?! Yeah"
     , testCaseSegments "numbers" ["Hello.", "3.14"] "Hello. 3.14"
     , testCaseSegments "URLs"    ["Check out http://www.example.com.", "OK?"]
                                   "Check out http://www.example.com. OK?"
     , testCaseSegments "titles"  ["Mr. Doe, Mrs. Durand and Dr. Singh"]
                                   "Mr. Doe, Mrs. Durand and Dr. Singh"
     ]
  ]

testCaseSegments d xs x = testCase d $ assertEqual "" xs (segment x)

prop_segment_concat s =
   noWhite s == concatMap noWhite (segment s)
 where
   noWhite = filter (not . isSpace)

-- breaking things apart

segment :: String -> [String]
segment = map (dropWhile isSpace) . squish
        . ( split
          . condense       -- "huh?!"
          . dropFinalBlank -- strings that end with terminator
          . keepDelimsR    -- we want to preserve terminators
          $ oneOf stopPunctuation
          )

stopPunctuation = [ '.', '?', '!' ]
titles = [ "Mr.", "Mrs.", "Dr." ]

-- putting them back together

squish = squishBy (\_ y -> not (startsWithSpace y))
       . squishBy (\x _ -> any (`isSuffixOf` x) titles)
       . squishBy (\x y -> endsWithDigit x  && startsWithDigit y)
 where
  startsW f [] = False
  startsW f (x:_) = f x
  --
  startsWithDigit = startsW isDigit
  startsWithSpace = startsW isSpace
  --
  endsWithDigit xs =
    case reverse xs of
     ('.':x:_) -> isDigit x
     _ -> False

squishBy f = map concat . groupBy f

the "one-off" directory

Another useful convention when building pipelines or just doing any sort of computer-assisted (corpus driven?) researchy/academic task.

Problem

Sometimes you just need to do one-time transformations on your data. Just this once, you need to grep your data for all the pictures of people with big noses in them. Or just this once, you need to find the ones with funny hats.

Integrating it into your pipeline is kinda silly because you really only just need this thing fast, and like, just this once. It imposes a kind of cognitive overhead that leads to paralysis. Really, this is the sort of thing you just bang out on the command line and be done with it...

Except of course...

It's never really just the once is it?

Proposed solution

Within the project directory (eg. hills-corpus), create a directory called "one-off". The idea is to have a place where you can sweep up all your disposable code. Things you type out on the command line, you should really copy and paste and turn into "one-off/some-foo.sh".

Why I think this works

First: this means you preserve your operation for posterity. I've used some of my supposedly "one-off" scripts (be they in Bash or Haskell) dozens of times. Not so one-off after all.

Second: they reduce the cognitive overhead of trying to figure out what the "right" place to put them into your pipeline (eg. Makefile of Doom) is. You don't even really have to give them sensible names either, because they're really just swept up under the one-off rug, and you don't really expect to see them again anyway (oh but you will, my friend, you will...)

Third: it eliminates clutter. Previously, I was dumping these one-off scripts in my project root (for fear of losing them). The problem with clutter is that you come back to it 3 months later, and you have no idea what you can safely delete, on what you need to hang on to. Now they have some place to go. In the worst case, they go, but that's not so bad.