The Combinatorial Basis of Entropy (“MaxProb”) 22nd Canberra International Physics Summer School ANU, Canberra 11 December 2008 by Robert K. Niven Marie Curie Incoming International Fellow, 2007-2008 Niels Bohr Institute, University of Copenhagen, Denmark School of Aerospace, Civil and Mechanical Engineering The University of New South Wales at ADFA Canberra, ACT, Australia
60
Embed
The Combinatorial Basis of Entropy (“MaxProb”)people.physics.anu.edu.au/~ccs106/SUMMERSCHOOLS/SS22/RNiven/… · The Combinatorial Basis of Entropy ... • Combinatorial basis
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
The Combinatorial Basis of Entropy (“MaxProb”)
22nd Canberra International Physics Summer School ANU, Canberra
11 December 2008
by Robert K. Niven
Marie Curie Incoming International Fellow, 2007-2008 Niels Bohr Institute, University of Copenhagen, Denmark
School of Aerospace, Civil and Mechanical Engineering
The University of New South Wales at ADFA Canberra, ACT, Australia
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 2
Lectures
1. The Combinatorial Basis of Entropy (“MaxProb”)
2. Jaynes’ MaxEnt, Riemannian Metrics and the Principle of Least Action
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 3
Contents • Historical overview
- combinatorics - probability theory
• Combinatorial basis of entropy / MaxProb principle
generalised combinatorial definitions of entropy and cross-entropy
explanation of MaxEnt / MinXEnt
• Applications 1. Multinomial systems (asymptotic vs non-asymptotic) 2. (In)distinguishable particles or categories 3. “Neither independent nor identically distributed” sampling
• Future applications …
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 4
(Advertisement)
Courses at UNSW@ADFA, Canberra:
• Short course in “Maximum Entropy Analysis”, 14-15 May 2009
(fee paying $1270).
• Masters course: ZACM8327 Maximum Entropy Analysis, semester 2,
2009 (fee paying or UNSW@ADFA enrolled student)
- 3 hours of lectures + tutorials per week
- based on similar course at Niels Bohr Institute
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 5
Historical Overview
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 6
Combinatorics Knowledge is very old! (Edwards, 2002)
(a) Number-patterns
Pythagoras (500BC)
Egypt (300BC)
Theon of Smyrna, Nicomachus (100AD)
Higher dimensions: Tartaglia (1523, publ. 1556)
figurate numbers fk
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 7
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 56
Acknowledgments:
Thanks to:
• The University of New South Wales, Australia
• The European Commission, for Marie Curie Incoming
International Fellowship at University of Copenhagen
• Dr Bjarne Andresen + Dr Flemming Topsøe
• COSNET, ANU and (Prof. R. Dewar)2 for opportunity to present
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 57
References Acharya, R., Narayana Swamy, P. (1994) J. Phys. A: Math. Gen., 27: 7247-7263. Boltzmann, L. (1872) Sitzungsberichte Akad. Wiss., Vienna, II, 66: 275-370; English transl.: Brush,
S.G. (1966) Kinetic Theory: Vol. 2 Irreversible Processes, Permagon Press, Oxford, 88-175. Boltzmann, L. (1877), Wien. Ber., 76: 373-435, English transl., Le Roux, J. (2002), 1-63,
http://www.essi.fr/~leroux/. Clausius, R. (1865) Poggendorfs Annalen 125: 335; English transl.: R.B. Lindsay, in J. Kestin (ed.)
(1976) The Second Law of Thermodynamics, Dowden, Hutchinson & Ross, PA, (1976) 162. Clausius, R. (1876) Die Mechanische Wärmetheorie (The Mechanical Theory of Heat), F. Vieweg,
Braunschwieg; English transl.: W.R. Browne (1879), Macmillan & Co., London. Edwards, A.W.F. (2002) Pascal’s Arithmetical Triangle: The Story of a Mathematical Idea, 2nd ed.,
John Hopkins U.P., Baltimore. Grendar, M., Grendar, M. (2001) What is the question that MaxEnt answers? A probabilistic
interpretation, in A. Mohammad-Djafari (ed.) Bayesian Inference and Maximum Entropy Methods in Science and Engineering, AIP (Melville), 83.
Grendar, M., Niven, R.K. (in submission), http://arxiv.org/abs/cond-mat/0612697. Jaynes, E.T. (1957), Physical Review, 106: 620-630. Jaynes, E.T. (Bretthorst, G.L., ed.) (2003) Probability Theory: The Logic of Science, Cambridge
U.P., Cambridge. Körner, J., Longo, G. (1973) IEEE Trans. Information Theory IT-19(6): 778. Körner, J., Orlitsky, A., (1998) IEEE Trans. Information Theory 44(6) 2207. Kullback, S., Leibler, R.A. (1951), Annals Math. Stat., 22: 79-86.
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 58
Lilly, S. (2002), A Practical Guide to Runes, Caxton Editions, London. Niven, R.K. (2005), Physics Letters A, 342(4): 286-293. Niven, R.K. (2006), Physica A, 365(1): 142-149. Niven, R.K. (in submission) CTNEXT07, 1-5 July 2007, Catania, Sicily, Italy, http://arxiv.org/
abs/0709.3124. Niven, R.K. (2005-07) Combinatorial information theory: I. Philosophical basis of cross-entropy
and entropy, cond-mat/0512017. Niven, R.K., Suyari, H. (in submission) Combinatorial basis and finite forms of the Tsallis entropy
function. Pascal, B. (1654), Traité du Triangle Arithmétique, Paris. Paxson, D.L. (2005) Taking Up the Runes, Red Wheel/Weiser, York Beach, ME, USA. Pennick, N. (2003) The Complete Illustrated Guide to Runes, HarperCollins, London. Planck, M. (1901) Annalen der Physik 4: 553. Sanov, I.N. (1957) Mat. Sb. 42, 11-44; English transl. Selected Transl. Math. Stat. Prob. 1 (1961),
213-224. Shannon, C.E. (1948), Bell System Technical Journal, 27: 379-423; 623-659. Suyari, H. (2006), Physica A 368(1): 63. Vincze, I, (1974) Progress in Statistics, 2: 869-895. Historical references prior to 1800AD are given in Edwards (2002).
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 59
- used across Germanic + central Europe, Britain + Scandinavia, 5th-10th cent.; in Sweden to 17th cent.
- derived from Etruscan alphabet (not Greek or Roman) - each rune has symbolic meaning Anglo-Saxon h (“Haegl”) = old German h (“Hagalaz”) = hail, hailstones - symbolic of destructive force of Nature, but melts and gives
new life - evokes need to accept what is inevitable; to “go with the flow”;
i.e. rune of transformation
R.K. Niven, UNSW 22nd Canberra International Physics Summer School 60