MDL Reading

There is a large body of literature on the Minimum Description Length principle in the contexts of statistics, mathematics, machine learning, philosophy, etc. We give only a small selection of publications that we have found especially useful and important. More publications can be easily found using search engines such as Google, Google Scholar and CiteSeerx.

Tutorials/Overviews

A.Barron, J.Rissanen, and B.Yu, The minimum description length principle in coding and modeling. IEEE Trans. Information Theory, vol. 44 (1998), no. 6, pp. 2743-2760.

P.Grünwald, A tutorial introduction to the minimum description length principle. In: Advances in Minimum Description Length: Theory and Applications (edited by P. Grünwald, I.J. Myung, M. Pitt), MIT Press, 2005 (80 pages; [PS], [PDF]).

M.H.Hansen and B.Yu, Model selection and the principle of minimum description length. J. American Statistical Association, vol. 96 (2001), pp. 746-774. (available at Prof. Hansen's homepage)


Articles

P.Adriaans and P.Vitanyi, The power and perils of MDL, Proc. 2007 IEEE Intl. Symp. Information Theory (ISIT), pp.2216-2220. (available at Prof. Vitányi's homepage)

A.Barron and T.M.Cover, Minimum complexity density estimation, IEEE Trans. Information Theory, vol. 37 (1991), no. 4, pp. 1034-1054. (available at Prof. Cover's homepage)

Q.Gao, M.Li, and P.M.B.Vitanyi, Applying MDL to learning best model granularity, Artificial Intelligence, vol. 121 (2000), no. 1-2, pp. 1-29. (available at Prof. Vitányi's homepage)

P.Grünwald, P.Kontkanen, P.Myllymäki, T.Silander, and H.Tirri, Minimum encoding approaches for predictive modeling. Proc. 14th Int. Conf. on Uncertainty in AI (UAI'98), G.Cooper and S.Moral (eds.), 1998, pp. 183-192. (available at CoSCo homepage)

A.D.Lanterman, Schwarz, Wallace, and Rissanen: Intertwining themes in theories of model selection. International Statistical Review, vol. 69 (2001), no. 2, pp. 185-212. (available at Prof. Lanterman's homepage)

I.J.Myung, V.Balasubramanian, and M.A.Pitt. Counting probability distributions: Differential geometry and model selection. Proc. National Academy of Sciences, USA, vol. 97 (2000), pp. 11170-11175. (available at Prof. Balasubramanian's homepage)

J.Rissanen, Modeling by shortest data description. Automatica, vol. 14 (1978), pp. 465-471.

J.Rissanen, A Universal prior for integers and estimation by minimum description length. Annals of Statistics, vol. 11(1983), no. 2, pp. 416-431.

J.Rissanen, Universal coding, information, prediction, and estimation, IEEE Trans. Information Theory, vol. 30 (1984), pp. 629-636.

J.Rissanen, Stochastic complexity. J. Royal Statistical Society, Series B, vol. 49 (1987), no. 3, pp. 223-239.

J.Rissanen, Stochastic complexity and modeling. Annals of Statistics, vol. 14 (1986), pp. 1080-1100.

J.Rissanen, Fisher information and stochastic complexity. IEEE Trans. Information Theory, vol. 42 (1996), pp. 40-47.

J.Rissanen, Hypothesis selection and testing by the MDL principle. The Computer Journal, vol. 42 (1999), no. 4, pp. 260-269. (available at Computer Journal)

J.Rissanen, MDL Denoising. IEEE Trans. Information Theory, vol. 46 (2000), no. 7, pp. 2537-2543.
Errata: 1. The last term in Eqs. (36) and (40) should be -ln k(n-k). 2. DJ signal in Fig. 1 incorrect.

J.Rissanen, Strong optimality of the normalized ML models as universal codes and information in data. IEEE Trans. Information Theory, vol. 47 (2001), no. 5, pp. 1712-1717.

J.Rissanen, Complexity of simple nonlogarithmic loss functions. IEEE Trans. Information Theory, vol. 49 (2003), no. 2, pp. 476-484.

N.K.Vereshchagin and P.M.B.Vitanyi, Kolmogorov's structure functions and model selection, IEEE Trans. Information Theory, vol. 50 (2004), no. 12, pp. 3265-3290. (available at Prof. Vitanyi's homepage)

P.M.B.Vitanyi and M.Li, Minimum description length induction, Bayesianism, and Kolmogorov complexity. IEEE Trans. Information Theory, vol. 47 (2000), pp. 446-464. (available at Prof. Vitányi's homepage)

K.Yamanishi, A Decision-theoretic extension of stochastic complexity and its applications to learning. IEEE Trans. Information Theory, vol. 44 (1998), pp. 1424-1439.


Books

NEW: Jorma Rissanen, Optimal Estimation of Parameters, Cambridge University Press, 2012.

Peter Grünwald, Petri Myllymäki, Ioan Tabus, Marcelo Weinberger, and Bin Yu (editors), Festschrift in Honor of Jorma Rissanen on the Occasion of his 75th Birthday, Tampere International Center for Signal Processing, TICSP Series #38, 2008. (PDF, 29.0 MB)

Jorma Rissanen, Information and Complexity in Statistical Modeling, Springer, 2007. Errata

Peter Grünwald, The Minimum Description Length Principle, MIT Press, 2007. Sample chapter: Preface

Peter Grünwald, In Jae Myung, and Mark Pitt (editors), Advances in Minimum Description Length: Theory and Applications, MIT Press, 2005.

Te Sun Han and Kingo Kobayashi, Mathematics of Information and Coding, Translations of Mathematical Monographs, vol. 203, American Mathematical Society, 2001.

Jorma Rissanen, Stochastic Complexity in Statistical Inquiry, World Scientific, 1989.


Dissertations

Steven de Rooij, Minimum Description Length Model Selection: Problems and Extensions, University of Amsterdam, the Netherlands, 2008.

Teemu Roos, Statistical and Information-Theoretic Methods for Data Analysis, University of Helsinki, Finland, 2007.

Tim van Erven, When Data Compression and Statistics Disagree: Two Frequentist Challenges for the Minimum Description Length Principle, Leiden University, the Netherlands, 2010.


Lectures and Talks

Video lecture: Jorma Rissanen, MDL theory as a foundation for statistical modeling. MSRI Workshop on Information Theory, Mathematical Sciences Research Institute, Berkeley, February–March 2002.

Video lecture: Peter Grünwald, Universal modeling: Introduction to modern MDL. Machine Learning Summer School, Tubingen, 2003.

Slides: Peter Grünwald, Tutorial on modern MDL, NIPS 2001 Workshop on MDL: Developments in Theory and New Applications, Whistler, Canada, December 2001. (available at NIPS 2001)

Lecture notes: Jorma Rissanen, Lectures on statistical modeling theory, August 2005. (73 pages)

Slides: Jorma Rissanen, The Structure function and distinguishable models of data, 4th Annual Kolmogorov Lecture, Royal Holloway, London, February 2006.

Video lectures: Teemu Roos, "MDL Principle", Lectures 9 & 10 of the Information-Theoretic Modeling course, Dept. of Computer Science, University of Helsinki, September–October 2009.

Lecture notes: Teemu Roos, Introduction to Information-Theoretic Modeling, April 2011.


Journals

IEEE Transactions on Information Theory
Annals of Statistics
Computer Journal (Special Issue on Kolmogorov Complexity)
Journal of the Royal Statistical Society: Series B