tidying and slogging (reg 6h)

Alright, now I think I'm done with REG stuff. I've cleaned up my scripts, consolidated my four Perl scripts plus 2 stylesheets, plus one Haskell program into two Perl scripts and some documentation. The Haskell program had bugs, as did a few of my scripts (bugs which would have been found by a type checker; the Haskell script ones were more logic bugs which I basically just got lucky on in the rewrite).

Am now slogging through Grosz and Sidner. Why am I reading this? I'm not entirely sure. I'm vaguely aware that it's influential because every one and their cousin twice removed Joe is citing it, so maybe I'm just being a tourist mobbing the Mona Lisa. Part of it is that I want to have a firmer understanding of 'intention' in discourse, so that I have a clearer picture of what I mean if I ever say "we're not doing that". Is that good enough a reason? I think making it through this paper will make the other stuff, like the Hovy stuff a little bit clearer, and maybe help me to understand those discourse lectures online. Lots of ideas in this paper, it seems. Head swimming. I wonder if reading this will make me better understand what people like AD and LB are working on.

See that one of the many things I hate about myself as a newbie researcher. I don't have a clear picture why I read things. I feel like some kid doing homework... not very goal-directed reading yet (which may be good in the like a box of chocolates sense). The other thing is that I never publish (or feel compelled to, other than having this vague anxiety about not doing it). Worry worry worry. When is it going to stop feeling that I'm doing this all wrong?

Right... linguistic structure, intentional structure and attentional structure. Vague memory that the Blackburn book in progress shows how to incorporate the notion of a focus space stack in DRT.

Oh yeah, and JV has been visiting us for a couple of weeks. It's kinda nice to have some company in the office.

Oh and saw a nice talk on AI in commercial games. Trying to look past the fancy graphics and see what kinds of things we can now do. JV asked some nice questions about how language technology would be relevant to the gaming industry.

slow, stupid but satisfying (reg 9h)

Alright, so maybe NOW the REG stuff is done (hehe, famous last words). Got my 'final' set of numbers out.

It turns out we have some extra time to do this, so this morning I went a lot slower on my 'last' task and tried to do it differently. Instead of using XLST or one-off HaXml filters, I decided to do everything in one script using the XML::Rules package from Perl. I also spent a bit more time reading the perldocs, getting better acquainted with the syntax (arrow) and the behaviour (autovivification) of references in Perl, and also the XML::Rules package. Looked up some more stuff on CPAN, like the File::Find module (which turns out to be a core library! stupid Eric for not knowing how to look in standard API documentation).

Going to do some rewriting and packaging tomorrow. I want geval to be all Perl, no XSLT, no Haskell much as I love it. Question of maintainability: need somebody to can hack Perl to look at it. And also prettiness, getting everything in a single tidy package instead of sprawled out in a bunch of tiny little programs (which is only good when you know how to break things up right, not when you've got this crazy mess of things calling other things). Oh yeah, there'll be some shell scripts around too. Also need to document the little details, like what filenames we're expecting, why we have the test data broken down in a certain file hierarchy.

Basic stitch in time saving nine deal. Trying to be nice to 2009 Eric or whatever poor schmuck is going to be looking after my stuff (sorry! sorry! I try, but I suck!)

I've been really really slow today. And going around in circles. But I think it's ok, because it's largely me sorting things out, getting a feel for where this codebase needs to be going. Such a small simple stupid thing and I've got it turned into a big production.

short day (reg 5h)

Got here at 10h, leaving at 18h; various tea and cake breaks in between. Almost done with the REG stuff. Got my tri-text framework setup (more and more scripts, some of them throwaway). I am very grateful for the automation, am thanking past-eric (of 1 week ago) for making the results file generator. Trying to do this by hand [what was I thinking] would have suicide.

Back to reading Grosz and Sidner.

More REG stuff tomorrow. Not done yet.

more scripting (reg 7h)

Been a full day of REG stuff. Filled out the submissions table (had missed a couple of systems). Finished the baseline generators, extended the submissions table generator to print fancy output, ran the baselines generators, generated a whole bunch of numbers. Started working on the extended version of the string accuracy script [boorrring ;-)]

I am a bit disappointed that I did not manage to kill this REG stuff dead by the end of week. Just this last script and my plate is (supposedly) empty for the project (until something new creeps up). It's really embarrassing that I've been spending all week on this. Easy stuff, just easy stuff, should I have blown through it?

Or maybe me going slow is a good thing. Being, uh, careful. Yeah. Careful. Or something.

Hey, that spamassassin thing works really well! No training really. Just had to launch spamd.

perl (reg 4h)

baseline generators

Finished writing the baseline generators. No testing, unit or otherwise, but lots of printfs to more or less assure myself that it's doing the right thing. I lack rigour. A Good Programmer would be [A] passing the code to somebody else for review [B] proving it correct [C] writing unit tests, or better yet, the equivalent to quickchecks [D] writing functional/regression tests [E] manually sanity checking the output (yes, and selecting the subset of activities which are most pertinent given the constraints. I have this idealised shifting version of the Good Programmer in my mind: Egoless, full of Rigour, thinks before acting, sees the big picture, puts the user first, never stops learning, etc... and s/he is not me.

Was a bit irritated at Perl, but will live. Lots of stuff that typechecking could have caught. Me scratching my head for nothing.

adam k

Saw a talk by AK on the Sketch Engine stuff. Neat. Too bad I fell asleep during the first third. Blame lunch. Blame Eric for not resisting the big lunch.

getmail, procmail, spamassissin

After much resistance, rolled my sleeves and set about installing a spam filtering solution:

1. I was using mutt's built-in IMAP stuff. I needed to switch to something third party. Chose getmail because I vaguely remember fetchmail sucking for one reason or another. Maybe I was wrong. My one annoyance is that I can't figure out how to get it to save my password temporarily, i.e. without me putting it into a file. But I guess typing my password every time I want to check my mail is more feature than bug. Resist the urge. Not that I have a big mail-checking urge at Uni (except my perso mail, which is different)
2. Had to configure mutt, binding the G key to getmail (using a macro).
3. Put in S's procmailrc simplified. Figured out that (A) it does not like having its procmailrc symlinked (custom directory), and (B) I have to stick a slash at the end of mailbox name to get it to understand maildir
4. Enabled spamd (/etc/defaults/spamassissin)

Still need to set up a macro to train the filter. Ugh. So much work for such a simple thing... but that's what I get for wanting to use mutt. You want the customisability (for example, my mutt/GTD stuff), you have to pay for it with some willingness to learn new tricks.

ennui (reg 8h)

Well, I guess I can't complain too much. Re-re-re-learnt how to use HaXml, one of those things I always forget. Task was to process some XML, ignoring case and whitespace in the text content. The resulting HaXml was real real short.

Hacked up my report-generator (that code is getting uglier by the minute) to parse evaluation output and print it out in the table. Complicated thing is that each track has its own kind of evaluation output, so the printing needs to be smart. Have the sneaking suspicion that I'm not doing a good job of finding abstractions and solving general problems. Feet on the ground, head in it. Sigh... will need to extend this again to get a different breakdown. Just one of those cases where a programming job isn't difficult in the intellectually stimulating sense, but in the c'est chiant sense.

But that's the difference between Good Programmers and Erics. Good Programmers recognise the challenge in every chiant task and find ways to make them less painful. If it's not intellectually stimulating, you aren't thinking hard enough. Surely there must be something deeper behind this? Why is this boring? See the patterns, get the insight, solve the general problem, learn a new skill. Don't just plow ahead. Well, I'm just plowing ahead.

My summary told me that there were some systems that I had not yet sanity checked, so I went ahead and did that. All systems accounted for.

Need to create some baseline generators. Wouldn't take very long if I was doing it in Haskell... but I figure that I should do it in Perl because it's the kind of thing we may want to distribute. Sigh. Resisted at first. Tried to do it with the old XLST+Perl trick, but then I realised we needed fancier handling of XML (and needed to output bits of it). This would be really really easy with HaXml. I shouldn't resist though.

Embrace all opportunities to learn new stuff.

So now I've a vague idea how to use the XML::Rules module, which is admittedly easy to use. There is a toXml function which looks promising.

social bad (reg 6h)

Did some nice gardening this morning, but got to work at 11h (really should have left home earlier). Spent a lot of time being social, i.e. long lunch and coffee, which I really shouldn't indulge in if I'm going to go gardening. Either one or the other, not both.

Otherwise, worked on the log summarisation stuff, which is slowly coming together. Got some systems evaluated, got reasonably readable output and have made the data entry basically painless. Just need to plug some stuff in tomorrow. But I'm tired.

Time to go home.

being bad (reg 3h)

Oooh, not so good today.

Had a reasonable start, slowly working my way through that Grosz and Sidner (figure it's a must-read, part of your standard NLP course?).

But then in the afternoon, for some unknown reason, I felt a darcs-hacking itch and indulged in a bit of light coding. Fixed a UI bug, and implemented issue792 so that you can pull/push from different default repositories (by setting pull remote-repo foo in _darcs/prefs/defaults). That took a couple of hours.

In a way, maybe that wasn't so horrible. I was starting to work on something slow and boring, and having done something completely different in between gave me a chance to realise that what I was doing was slow and boring precisely because I wasn't automating enough of it. So I spent the rest of the afternoon writing some code to process my CGI-generated log files and spit them out in a nice HTML format. Now, if only I had written those log files right in the first place (all-in-one), I wouldn't have to write this silly smelly merge code (I feel really bad about this code. It's something complicated for something simple. I must be doing something wrong!)

Meh.

Funny thing is that I know I was bad today, but I don't really feel bad, mostly because I've been doing stuff all day. Just not the right stuff.

meeting, census and planning

Let's see...

This morning I tried to summarise my Hovy reading on our big reading list. All this talk about text planning and communicative goals reminded me of koller+2007sgp, (Koller and Stone, Sentence generation as a planning problem) so I took some time to read that. A nice happy little paper. I wonder if one day when I'm all grown up, I too will write nice happy little papers.

Spent an hour in a meeting about the Prodigy stuff, and got the feeling that most of the stuff I thought about this week was more or less in the right direction (i) pretend we're using this in the context of a query system that is extracting arbitrary small slices of the data, (ii) restrict ourselves to one table (I chose primary means of traveling to work) and (iii) collect the table and format it in some usable way.

Not everything I said was right. ILG usefully pointed out that contrary to my initial inflexible/dogmatic thinking, different cities and different numbers really DO make a difference. It's not just X, Y, Z, and H, I, J, K. What if your city starts with a vowel? What if you've got two cities that share a bit of name, like "East Devon" and "Mid Devon", or what if you've got numbers which are next to zero. So we needn't worry too much about generating truly unique texts. Within the same kind of query, plugging in different data, there is lots of room for generating interesting/different stuff. I think we're all on a mostly compatible wavelength on this. Having these regular meetings is helping a lot, gives me a way to channel... gives me concrete objectives to work on.

Spent the early part of the afternoon generating 10 slices of the census data for sample inputs.

The rest of the afternoon was going back to reading koller+2007sgp (I get distracted very easily on Section 3, encoding generation as a planning problem). Now I need to figure out what is the best use of my time. Tempted to read the Grosz and Sidner stuff just for general education... it might not really help me understand what we need to do on the discourse level, but it would give me a better sense of context.

Oh and we had the fire alarm go off this afternoon. Two engines! And fluorescent yellow vests, which you see a lot of people (cyclists and fire wardens) using here. Nobody got hurt, I think, nor did anything burn down.

Huh. For some reason, I'm not being my usual morose self today. Maybe it's the sleep deprivation. Or maybe it's just the feeling that I'm actually doing what I've been asked to do, for a change. Let's slap a smiley face on the day for variety, then.

Edit 18:04 Reading G+S now. I wonder what "It is important to recognize that the linguistic structure is not strictly decompositional" means

hovy1993adg

Automated Discourse Generation Using Discourse Structure Relations

Why this was useful to me: gave me a better idea what text planning is (up to 1993; I'm guessing that text planners have gotten more sophisticated in the past 15 years).

My notes. It's just going to be a dumb outline. But I feel I need to do this thing. Doubt they'll be of any use to you.

Sigh. Did I learn anything from reading this? Or did it all just wash over me?

discourse structure

Motivates knowing about discourse structure, and having a notion of text coherence. You can't just string sentences together in any order, because they usually wouldn't make any sense, and what order you say the individual sentences has a great impact on what how the resulting text can be interpreted.

A note which could be useful for question of what kind of effect discourse relations can have on the output text:

Several discourse phenomena signal discourse structure, including clause juxtapositioning, pronoun and other reference use, quantifier scoping, focus shift, tense and aspect

Discourse structure matters because you don't want people to get the wrong idea. Making sure people get the right idea, I guess is where the notion of communicative intention comes in.

Hovy says there are two major approaches

formalist

(e.g. DRT) - explain pronominalisation and quantifier scoping effects... concentrate on formal properties of discourse structure itself. "The theories tend to be weak on the actual contents of the structure, such as the precise interrelationships between segments and the communicative purposes of the discourse

functionalist

Concentrate on the goals of the speaker and the way these goals are reflected in the discourse structure. "tend to be weakest on the precise form of the discourse structure". Goes back to Aristotle.

He also says that the Grosz and Sidner stuff is a combination of the two schools, which I'll just blindly accept because I don't really know it.

Points to McKeown 1985's TEXT system as one of the first text generators to take discourse structure into account. TEXT uses schema. Essential shortcoming of schema is "the lack of representation of the purpose of each part in the whole"... which makes it hard to replan or motivate bits of text... which makes it hard to do dynamic text generation. Not really sure I understand this, to be honest.

a first attempt: text structuring using RST

Describes an attempt at addressing the shortcomings of schema by using RST.

Tries to "operationalize" RST relations, which I guess means "turn them into something a computer could use". See page 10 for an example.

A system that uses these plans gets as input (1) a communicative goal and (2) a bunch of facts... it consults the set of possible plans, and builds an RST tree top-down. Each plan has a results field that matches against a bit of the communicative goal. A plan also has 'growth points', which get recursively planned. Each plan has an "order" field to control textual order, e.g. Order: (NUCLEUS SATELLITE)

general requirements for discourse structure

Here's what Hovy says is the rough consensus about plan-based English discourse:

discourse

structured collection of clauses. Clauses grouped into segments

purpose

each segment has a purpose (DSP)... each segment can be seen as step in the plan to achieve the overall goal

coherence

discourse only successful if mutually coherent; both speaker and hearer have to agree on how each segment relates to its neighbours

discourse segment

data DiscourseSegment =
DS { name :: String -- ^ unique identifier
   , purpose :: DSP
   , content :: [DiscourseSegment] }

discourse structure

the top level segment; i.e., not embedded in another

four central aspects of discourse structure relations

(1) text plans: content and format

Text planners not very sophisticated. Interesting subsection Text Plans vs. Intersegment Relations. Moore et al 90, 91 say that discourse structures as goals "erroneously conflates 'rhetorical' (i.e. structural) information with intentionality". I guess intentionality is just that which pertains to the speaker's communicative goals (and achieving them). Talks about work to generate intentional plans.

Still a debate going on (still? 2008?)... Basically, Hovy says you have to recognise that intentional plans and discourse relations do two different things and you need both simultaneously. Intentional plans let you figure out what things to communicate, and overall how to structure the discourse. Discourse relations [intersegment relations?] "ensure textual coherence, prevent unintended inferences, govern sentence formation, tense, pronominalization, and focus shift"... So he seems to say that you need intentional plans, big picture; but you also need the discourse relations on the ground to make sure Bad Things don't happen.

(2) a library of discourse relations

Heh. Was wondering about this one. Which ones? How many? Answers have ranged from 2 to tens of thousands.

profligate position

Here's the library of relations you're going to need, in the order of tens...

parsimonious position

Uh, you can't possibly draw up a library of relations; there's too many; here, just use these two: DOMINANCE and SATISFACTION-PRECEDENCE

Hovy says that the Parsimonious Position doesn't really work for NLG (lots of citations). Talks about building a taxonomy of relations (see appendix!). Three basic kinds of relation

semantic (ideational)

interpersonal

presentational (textual)

(I like Hovy's terminology better than the SFL because it sounds more like it it's in English... don't know why... the word 'ideational' just throws me off, and textual sounds so vague)

(3) schemas

Sometimes you want a little bit of canned text. Cuts down on variability (good in practice).

Schemas lack info about why each of the blanks are there, but you could just code that straight in so that the schema serves as a "fossilized discourse structure". One way of getting the schema effect from text plans is to make them more rigid, treating some suggestions as injunctions.

(4) focus shift

..."As illustrated by NOAH, ordering requirements cannot all be precompiled into plans, and some additional process has to exercise additional control". Example would be multinuclear RST relations. Some notion of order is still needed, but you have to figure it out on the fly.

Basically:

• not so good - it was red. it was large. it has a silver circle on top to reflect heat
  
• better - it was red. it has a silver circle on top to reflect heat. it was large

You can't bake that into the plan because this kind of thing comes bottom-up (from what you're trying to say). Here, putting the colour stuff together is smoother.

Talks about an attempt to generate simultaneous RST tree and Focus tree, using the focus tree to constrain/guide the growth of the RST tree.

three text planning tasks involving discourse structure relations

How having a discourse structure can help you in...

1. casting of syntactic roles
2. concept aggregation
3. text formatting

mount hovy

mountains

Spent the morning looking at our mountains text, and rewriting their representation to better reflect the actual text. Not sure if that was a useful activity. If we want to generate these texts, we might need recursive feature structures as a data representation, which is fine.

hovy

Reading the Hovy paper still. Ended up printing it out, all 45 pages of it, one page per side because the printer does something screwy with this paper if you try to nup and squeeze. So so so much easier than reading on screen. You can flip back and forth, for one thing.

One of useful things I get from this paper is a semi-unified view of the problem of representing text structure and intentional structure. For example, I had always sorta seen the Grosz and Sidner stuff (which I don't know very well) as an outlier in all my readings on discourse structure, them seeming to focus exclusively on intentional structure (did I get that right?). Hovy's section on building a library of discourse relations helps me to see them in context. He draws a contrast between a Profligate Position (here's the library of relations you're going to need, in the order of tens) and the Parsimonious Position (uh, you can't possibly draw up a library of relations; there's too many; here, just use these two: DOMINANCE and SATISFACTION-PRECEDENCE). I like that kind of comprehensive survey stuff.

Note his comment:

While the parsimonious relations may satisfactorily represent discourse structure for purposes of analysis, practical text generation experience, such as [...] has shown that they are insufficient and that planners need considerably more information of rhetorical and semantic nature to ensure successful communication

One thing I found amusing is this quote in the conclusion: "It is not unreasonable to expect the flexible planning and generation of coherent, high quality, multi-page texts in limited domains within the next five years". Paper was written in 1993. I wonder those systems ever materialised. I'm guessing... not...

Now I need to flip back to the beginning and figure out what the heck it was I just read.

Off to run some errands though. Guess I might try to write up a quick little report tomorrow, or if I come back to work, tonight.

censuseseses

why we like census data

Now less bored by this census data thing. I just realised that we're sitting on a good example of why you want NLG. Templates are fine if you're just writing some small number of well-defined queries. For example, the RamBot stuff is basically one query over a whole bunch of towns. But what if you want flexibility? What if you want to ask your census data all sorts of different questions and you want it in natural language form? You're never going to be able to write enough templates to keep up with these sort of on the fly demands.

So you've got census data, say, on how people get to work in England. Two dimensions: transportation means and city. What you want to do is to be able to extract any arbitrary slice of that data. For example, I want to compare bus ridership between Newcastle, Brighton and Plymouth. Or I want to see if people across the country prefer buses or trains. The key here is to be able to extract arbitrary slices, any dimension, and any subset of the dimension.

Well, ok, that answers my question of why we'd want NLG as opposed to canned text. It's an easy question. Any masters student with a Dale and Reiter book can tell you why.

But what I still don't really understand is why we'd even want to use language in the first place where you can just look at a stupid table. It's not my job to ask why, but you can't help wondering. My current guess is that if you've got a lot of data, you're either going to want to look at a table, or some kind of badass map visualisation thingy (with both a micro and macro view of the data, of course, ob Tufte)... [exception: blind, people in cars, etc].

Maybe, as I was saying yesterday, if the number of data points gets small enough, it becomes more efficient to use language because the terseness of tabular data imposes an overhead on the user to interpret the data. This overhead is very well compensated for with lots of data because you only pay it once (once you've got the table headers figured out, you're set)... but when you've only got a little bit, c'est pas rentable and you're better off having some computer put it into text rather than you translating it into language in your head. Hmm... nah, that sounds pretty silly.

I'm putting the cart before the horse. Here I am struggling to convince myself that the application will be useful, when really the application is just an excuse to work on the underlying technology. My job is the underlying technology, on s'en fout about the application.

On second thought, it's not so much a question of Why NLG is Useful for Applications. The thought I was trying to pursue is Why a Grammar is Useful Here, and the idea I was sort of trying to unfurl in my mind was that grammar is a factorised representation of all the things you're going to want to say with this data. And the question behind that is "what is our data representation? how feasible is it to use this? what kind of corpus size are we looking at", or to sum up "What do we want the corpus to look like?". So I wasn't deliberately trying to justify NLG. It really did start out on topic.

The answer which is slowly coalescing in my mind is that you want the texts to be very very small. You want the data given in the text to be light. (Hey, the project proposal says 1-3 sentences). That means that if you're going multi-dimensional, it really makes sense to restrict your data points [say, to buses and trains, and not the whole range of transport options, or teenagers]. You want it designed so that the factorisation that a grammar affords you actually becomes useful... why? Because we want to show that learning weights on the grammar can be useful.

[sigh, this made more sense when I was first thinking of it... but then I thought we were dealing with 3 dimensional data, where this whole notion of picking out arbitrary slices seems really important]

following up on this...

So it seems like your input representation to the NLG bits of the hypothetical census query tool are going to need three major relations. Say we got the following square we want to represent.

	bus	train
Brighton and Hove	8.38	12.46
Newcastle upon Tyne	0.94	18.87

We can represent the cells themselves in a pretty straightforward fashion: l11:cell(bn,bus,8.38),l12:cell(bn,train,12.46),l21:cell(nc,bus,0.94),l22:cell(nc,train,18.87).
Was thinking that the application is going to have to say what it's grouping and what it's comparing. Something like:

• compare(group(l11,l12),group(l21,l22)) gives you "In Brighton and Hove, 8.38% of people ages 17-64 usually travel to work by train, and 12.46 by bus, whereas in Newcastle upon Tyen, 0.94 % travel by train and3.81% travel by bus"
• compare(group(l11,l21),group(l12,l22)) gives you "8.38% of people in Brighton and Hove, and 12.46% in Newcastle upon Tyne usually travel to work by train. 12.46% in Brighton and 18.87 % in Newcastle upon Tyne travel by train."

Point is that how you group things is a non-linguistic decision, though it has ramifications.

csv

I never realised there was an RFC for CSV

Wrote up a tiny little script to extract a snippet of the KS15 table (the full thing) from the National Statistics people. I don't really understand their geographical hierarchies. I just arbitrarily picked everything that was on a 3rd level indent (there are smaller units below), basically,where I saw Brighton. The technical stuff doesn't really matter. We just want some data. Think I'll stick to the transport stuff for now. Maybe fold in other tables over time.

Numbers: We've got 378 autonomous political units to work with, 11 columns. So that's about 4000 single points. Arbitrary number of possible slices, really. How big do you want em? Doesn't really matter how many unique queries though. Point here is how many truly distinct corpus entries can we make out of this?

I guess I'm done looking at census data for this week. Maybe I won't have to stay quite so late tonight then, me and my not-working.

not very good (reg 5h)

Very sluggish today. Did a bit of gardening in the morning. Feeling sort of distracted. Got my scripts working for sanity checking the reg stuff. Good to have this automated, so that now I can run it whilst being brainless.

Looked a bit at the Canadian stuff, although I'm not sure if it's the right thing. Neat from a tech standpoint, producing descriptions of graphs that can be navigated (now I am looking at this part of the graph: you see... (like a text adventure))

Trying to go back to the stuff we're doing on census data, but can't really concentrate.

back at work (reg 2h)

Been on holiday for the past week, parental visit.

Looking at the census data, being bored [bad eric!].

I have this habit of jotting my notes on random scraps of paper, which is really good because I just lose them. Here's me transcribing them to something a little more permanent. How counter-intuitive that a bunch of bits hosted by some third party company are more permanent than an actual physical artifact.

• headings are largely already supplied by the people who made the tables (e.g., "People aged 16-74 who usually travel to work by: Underground, Metro, Light Rail or Tram"); might not be true for, say, religions
  
• snippets
• work mainly at home
  
• usually travel to work by train [almost texto]
  
• usually travel to work by driving a bus or a car [almost texto]
  
• usually travel to work on foot [get rid of 'by']
• usually travel to work by some other means [by: Other]
  
• presumably, once you know how to generate a kind of table, say transport tables, it's all template based from there on... there is no difference between the texts for Brighton and for Oxford, except for the city names and the actual numbers.... or is there? Maybe some subtle shifts because of the numbers themselves (for example, suppressing zero, singular/plural)
• NOMIS source a bit of a pain to work with, likely because our proxy server is caching the results (so two different requests just generates the same thing). All single point
• KS (key statistics), ST (standard tables), CAS (Census Area Statistics)
• postcode unit, OA (output area), SOA, ward
• some data has hierarchy (part time: <20,>
• prose does not really make sense when you have massive amounts of data (that's what tables were invented for). Need to restrict ourselves to small snippets of table before the intuitiveness of text is offset by its redundancy. Custom queries is where it's at, I guess.
• trying to think about what kind of entries we want in our corpus... do we stick to one topic (for example, transport?)... wouldn't the set of possible output texts be just really repetitive? what are we trying to get at here? For example, what's the point of having a transport data for a bunch of different cities if you're just going to have the same texts over and over and over again? I suppose you could have the texts generated by different people. Oh, the report suggests alternative verbal paraphrases.

alternative paraphrases

I guess you could have alternatives... single point

• 9.25% of people living in Brighton and Hove work mainly at home.
• Of people living in Brighton and Hove, 9.25% work mainly at home.
• People who work mainly at home account for 9.25% of people in Brighton and Hove
• 9.25% of Brighton and Hove residents mainly work at home
• 9.25% of people living in Brighton and Hove report working mainly from home

Now one about single dimensional?

• Of people aged 17-64 in employment in Brighton and Hove, 9.25% work mainly at home, 0.17 usually travel to work by underground, metro, light rail or tram, 8.35% by train, 12.46% by bus or coach, 0.81% by motorcycle or scooter, 43.15 by driving a car, 4.87 as a passenger in a car, 0.53% by taxi, 2.7% on foot and 0.52% by some other means

I'm not really sure what this tells me. We already have one of these texts for religions. Just seeing what it's making one for other kind of tables. Here, we see that it's actually quite useful to have NLG systems that know how to do some sort of formatting.

The thought of writing a comparison text just annoys me.

fstab

Added this to my /etc/fstab. Now I can mount USB keys by just typing mount /media/usbkey... no more launching nautilus.

/dev/sdb1       /media/usbkey   auto    rw,user,auto    0       0

lapata2003pts

Got bored looking at census data, and am now slogging through this paper Probabilistic Text Structuring: Experiments with Sentence Ordering (no offense to the author as usual, just eyes glazing over at the sight of formulae). I wonder why people working in this area like the word "operationalize" so much.

sanity check

Spent an hour before leaving working on more sanity checking scripts, on request. Hmm, kind of a day-dreamy day. Workwork!

meeting

Good. Concrete is good. Well-defined.

I'll be looking at census data. We want to know:

• source and modifications to text [n/a, since no text, but has to be human acquired?]
• data representation [should be straightforward?]
• corpus size [10^n ... what n]
• tentative data representation

Also need to contact Canadian census guy

research-lite (reg 4h)

Many bugs and scares later, we've published the submissions page.

hovy

Noticed this little snippet from the Hovy paper about how discourse structure can affect the output text (highlighting mine):

• Expression of the Satellites of the Elaboration relation as relative clauses: Knox, which is C4. . . instead of, say, Knox is C4. It is en route. . .. In English, this is the standard realization for the Elaboration Satellite.
• Use of the future tense in the final sentence. Since information provided by the data base was always based on the present time, anything that appeared in the Satellite of a temporal Sequence relation had to be in the future.
• Linkage of the last two clauses into a single sentence. Deciding to link clauses is easily done when a paragraph structure is available; the complexity of each subtree can readily be determined by counting the number of subnodes, and appropriate sentence-building decisions made.

Trying now to absorb this notion of an 'intentional' text plan...