Spring 2007 Team Project

Download the description of the project
Corpus
- Go the the site containing a collection of Europarl corpora extracted from proceedings of the European Parliament. This contains data for 11 European languages: Romance (French, Italian, Spanish, Portuguese), Germanic (English, Dutch, German, Danish, Swedish), Greek and Finnish.
  If I got the teams done correctly, someone can at least read French on both. The corpus we will work with for development is the French-English parallel corpus (103 MB, 04/1996-09/2003). When you unpack this collection you will find that it contains two directories, fr/ and en/. Each file fr/X.txt has a corresponding English translation en/X.txt (identical filename, different directory), and each X.txt appears to contain one or more CHAPTERs -- each "chapter" will be the unit we call a "document" for this project.
  If your team wants to do Spanish rather than French, that's ok. I am not expecting to see anything in this project that cares about the source language, as long as we use UTF-8 encoding. But let me know ASAP so I can get you the necessary resources. (I'm hoping to put together Chinese data while you're working on the project, and ideally we can just slot it in at the end (or right afterwards) to see what happens.)
- The Europarl files appear to use the ISO-8859-1 (Latin1) encoding. Convert all files you use to UTF-8. You can do this using the iconv utility on most Linux systems: iconv --f latin1 --t utf8 < in.txt > out.txt
- For purposes of this project, files from October to December 2000 should be set aside as unseen data for evaluation; the filename format is ep-YY-MM-DD.txt. (The other files can be used however you see fit for development.) Files fr/X.txt and en/X.txt are aligned line-by-line, so for testing you can treat a line of en/X.txt as the reference translation for the corresponding input line in fr/X.txt.
- Happily for us, the Europarl files already appear to have had sentences (or similar-sized units) broken up with newlines, and tokenization already applied in order to produce whitespace-delimited tokens.
Other resources
- French-English phrase-to-phrase dictionary (LARGE! >1Gbytes) Each line maps a French phrase to an English phrase, followed by a list of scores. The scores in order are:
  - p(f|e) - probability of French phrase given English phrase
  - lex(f|e) - score based on word-level conditional probabilities
  - p(e|f) - probability of French phrase given English phrase
  - lex(e|f) - score based on word-level conditional probabilities
  - math.e - ignore
  Note that this phrase table was automatically acquired by a statistical MT system and there is a lot of garbage in it! For efficiency you might want to filter out any entries where none of the words on the French side appear in the documents you're working with. Similarly, you might choose to ignore the large number of phrase to phrase mappings that involve punctuation.
- You can build a word-to-word dictionary containing tab-separated triples of French-word, English-word, score just by extracting one-word to one-word entries from the phrase-to-phrase dictionary. (I might provide a better dictionary in that format later on.)
- WordNet. I recommend Version 3.0. Note that there are various APIs written for WordNet in various programming languages.
- Named entities database. See the readme-entities file for description.
- In case they're useful here's a French stoplist in Latin1 (you'll need to convert to UTF8) and an English stoplist.
Advice (for what it's worth). For implementation purposes, think about starting by creating some artificial subsets of the data and resources, and implement/debug using that before you go to full scale. Although you do have to keep scalability issues in mind, this approach is likely to save you a lot of headaches and time.