BigFatLM V0.1
By Jonathan Clark
Carnegie-Mellon University -- School of Computer Science -- Language Technologies Institute
Released: March 31, 2011
Liscence: LGPL 3.0
"Embiggen your language models"
BACKGROUND
Provides Hadoop MapReduce training of state-of-the-art large-scale language models.
This allows building much larger models with commodity hardware. Other popular
packages either have memory usage on the order of the number of N-gram types
for the highest N (e.g. SRILM) or offer only approximate smoothing methods for
training that offer no theoretical guarantees as the number of nodes scales up
(e.g. IRSTLM). BigFatLM offers lower per-node memory usage on the order of the
vocabulary size and scales well as the data size increases by making use of
more nodes in a Hadoop cluster.
Modified Kneser-Ney with order-wise interpolation has become an accepted standard
of language model smoothing in the machine translation community (and likely has
comparably good performance on other tasks such as speech recognition). System builders
have also found that interpolating multiple models built on different corpora can often
provide benefits in terms of perplexity and end-task performance. BigFatLM supports
using both of these techniques with distributed training.**
(** MODEL INTERPOLATION IS STILL UNDER TESTING AND WILL NOT BE DOCUMENTED HERE UNTIL TESTED **)
For the sake of comparison, BigFatLM's results have been diffed with the output of
SRILM's ngram-count -kndiscount -gt3min 1 -gt4min 1 -gt5min 1 -order 5 -interpolate -kndiscount -unk
and found to be within floating point error.
REQUIREMENTS
* Java 6+
* A Hadoop cluster running Hadoop 20.1+ or, for much slower results, a local install of Hadoop.
* If you wish to build BigFatLM from source, you'll need the Apache Ant build system
* If you wish to filter the language model, you should also install KenLM's filter tool.
See https://kheafield.com/code/mt/filter.html.
* If you wish to run the regression tests, you'll need Python 2.6+
BUILDING
Just type "ant"
USAGE
Assume we have:
* A pre-processed, tokenized corpus on HDFS (the Hadoop distributed filesystem) 
  at /home/user/corpus
And we want to build:
* An order 3 LM
* Use order-wise interpolated modified Kneser-Ney smoothing (default)
And we want to put the resulting files:
* For the small local files, under ./corpus-3g-lm
* For the large HDFS files, under $HDFS_USER_HOME/corpus-3g-lm
* For the final local merged ARPA file, at /home/user/corpus-3g.arpa
# The most basic usage, to get an uncompressed ARPA file:
$ ./build-lm.sh 3 /home/user/corpus corpus-3g-lm /home/user/corpus-3g.arpa
# To gzip the resulting ARPA file using bash process substitution:
$ ./build-lm.sh 3 /home/user/corpus corpus-3g-lm >(gzip > /home/user/corpus-3g.arpa.gz)
REFERENCES
1. Chen, Stanley; Goodman J. An Empirical Study of Smoothing Techniques for Language Modeling.; 1998.
2. Brants T, Popat AC, Och FJ. Large Language Models in Machine Translation. Computational Linguistics. 2007;1(June):858-867.