MDL Reading

There is a large body of literature on the Minimum Description Length principle in the contexts of statistics, mathematics, machine learning, philosophy, etc. We give only a small selection of publications that we have found especially useful and important. More publications can be easily found using search engines such as Google, Google Scholar and CiteSeer.

Tutorials/Overviews

A.Barron, J.Rissanen, and B.Yu, The minimum description length principle in coding and modeling. IEEE Trans. Information Theory, vol. 44 (1998), no. 6, pp. 2743-2760.

P.Grünwald, A Tutorial introduction to the minimum description length principle. In: Advances in Minimum Description Length: Theory and Applications (edited by P. Grünwald, I.J. Myung, M. Pitt), MIT Press, 2005 (80 pages; [PS], [PDF]).

M.H.Hansen and B.Yu, Model selection and the principle of minimum description length. J. American Statistical Association, vol. 96 (2001), pp. 746-774. (available at Prof. Hansen's homepage)

Articles

A.Barron and T.M.Cover, Minimum complexity density estimation, IEEE Trans. Information Theory, vol. 37 (1991), no. 4, pp. 1034-1054. (available at Prof. Cover's homepage)

Q.Gao, M.Li, and P.M.B.Vitanyi, Applying MDL to learning best model granularity, Artificial Intelligence, vol. 121 (2000), no. 1-2, pp. 1-29. (available at Prof. Vitányi's homepage)

P.Grünwald, P.Kontkanen, P.Myllymäki, T.Silander, and H.Tirri, Minimum encoding approaches for predictive modeling. Proc. 14th Int. Conf. on Uncertainty in AI (UAI'98), G.Cooper and S.Moral (eds.), 1998, pp. 183-192. (available at CoSCo homepage)

A.D.Lanterman, Schwarz, Wallace, and Rissanen: Intertwining themes in theories of model selection. International Statistical Review, vol. 69 (2001), no. 2, pp. 185-212. (available at Prof. Lanterman's homepage)

I.J.Myung, V.Balasubramanian, and M.A.Pitt. Counting probability distributions: Differential geometry and model selection. Proc. National Academy of Sciences, USA, vol. 97 (2000), pp. 11170-11175. (available at Prof. Balasubramanian's homepage)

J.Rissanen, Modeling by shortest data description. Automatica, vol. 14 (1978), pp. 465-471.

J.Rissanen, A Universal prior for integers and estimation by minimum description length. Annals of Statistics, vol. 11(1983), no. 2, pp. 416-431.

J.Rissanen, Universal coding, information, prediction, and estimation, IEEE Trans. Information Theory, vol. 30 (1984), pp. 629-636.

J.Rissanen, Stochastic complexity. J. Royal Statistical Society, Series B, vol. 49 (1987), no. 3, pp. 223-239.

J.Rissanen, Stochastic complexity and modeling. Annals of Statistics, vol. 14 (1986), pp. 1080-1100.

J.Rissanen, Fisher information and stochastic complexity. IEEE Trans. Information Theory, vol. 42 (1996), pp. 40-47.

J.Rissanen, Hypothesis selection and testing by the MDL principle. The Computer Journal, vol. 42 (1999), no. 4, pp. 260-269. (available at Computer Journal)

J.Rissanen, MDL Denoising. IEEE Trans. Information Theory, vol. 46 (2000), no. 7, pp. 2537-2543.
Errata: 1. The last term in Eqs. (36) and (40) should be -ln k(n-k). 2. DJ signal in Fig. 1 incorrect.

J.Rissanen, Strong optimality of the normalized ML models as universal codes and information in data. IEEE Trans. Information Theory, vol. 47 (2001), no. 5, pp. 1712-1717.

J.Rissanen, Complexity of simple nonlogarithmic loss functions. IEEE Trans. Information Theory, vol. 49 (2003), no. 2, pp. 476-484.

N.K.Vereshchagin and P.M.B.Vitanyi, Kolmogorov's structure functions and model selection, IEEE Trans. Information Theory, vol. 50 (2004), no. 12, pp. 3265-3290. (available at Prof. Vitanyi's homepage)

P.M.B.Vitanyi and M.Li, Minimum description length induction, Bayesianism, and Kolmogorov complexity. IEEE Trans. Information Theory, vol. 47 (2000), pp. 446-464. (available at Prof. Vitányi's homepage)

K.Yamanishi, A Decision-theoretic extension of stochastic complexity and its applications to learning. IEEE Trans. Information Theory, vol. 44 (1998), pp. 1424-1439.

NEW: Jorma Rissanen, Information and Complexity in Statistical Modeling, Springer, 2007. Errata

NEW: Peter Grünwald, The Minimum Description Length Principle, MIT Press, 2007. Sample chapter: Preface

Peter Grünwald, In Jae Myung, and Mark Pitt (editors), Advances in Minimum Description Length: Theory and Applications, MIT Press, 2005.

Te Sun Han and Kingo Kobayashi, Mathematics of Information and Coding, Translations of Mathematical Monographs, vol. 203, American Mathematical Society, 2001.

Jorma Rissanen, Stochastic Complexity in Statistical Inquiry, World Scientific, 1989.

Lectures and Talks

Video lecture: Jorma Rissanen, MDL theory as a foundation for statistical modeling. MSRI Workshop on Information Theory, Mathematical Sciences Research Institute, Berkeley, February–March 2002.

Video lecture: Peter Grünwald, Universal modeling: Introduction to modern MDL. Machine Learning Summer School, Tubingen, 2003.

Slides: Peter Grünwald, Tutorial on modern MDL, NIPS 2001 Workshop on MDL: Developments in Theory and New Applications, Whistler, Canada, December 2001. (available at NIPS 2001)

Lecture notes: Jorma Rissanen, Lectures on statistical modeling theory, August 2005. (73 pages)

Slides: Jorma Rissanen, The Structure function and distinguishable models of data, 4th Annual Kolmogorov Lecture, Royal Holloway, London, February 2006.

Slides: Petri Myllymäki and Henry Tirri, On Minimum description length modeling, Three Concepts: Information, Dept. of Computer Science, University of Helsinki, January–May 2005.

Slides: Teemu Roos, "MDL Principle" (Part I, Part II), Three Concepts: Information, Dept. of Computer Science, University of Helsinki, September–December 2007.

Course web page: Peter Grünwald, The Minimum Description Length Principle for Learning and Prediction, University of Amsterdam, Spring 2005. (Featuring draft chapters from a book on MDL.)

Journals

IEEE Transactions on Information Theory
Annals of Statistics
Computer Journal (Special Issue on Kolmogorov Complexity)
Journal of the Royal Statistical Society: Series B

Conferences and Workshops

Conference on Learning Theory (COLT): 2003, 2004, 2005, 2006.
IEEE Information Theory Workshops: 2002, 2003, 2004, 2005, 2006.
Neural Information Processing Systems (NIPS).
    - NIPS 2001 Workshop on MDL: Developments in Theory and New Applications, December 2001.
    - NIPS 2002 Workshop on Universal Learning Algorithms and Optimal Search, December 2002.
Uncertainty in Artificial Intelligence (UAI).
MSRI Workshop on Information Theory, February-March 2002.
DIMACS Workshop on Complexity and Inference, June 2003.