Probability Theory : The Logic of Science

Rating:
Summary: A brilliant logical foundation for physical science but...
Review: Jaynes has done a brilliant job of constructing a logical framework for incorporating and explicating the crucial differences between causal(physical)independence and logical(epistemological)independence that will enter into the assumptions underlying applications of the probability calculus(statistical inference)to scientific analysis in physical science(physics,engineering,chemistry,biology,etc.).Jaynes shows repeatedly how one goes about applying maximum entropy and/or Shannon's information approach to a host of problems in the physical sciences.Jaynes also incorporates valuable historical commentary on a host of individuals,from Laplace and Boole to Keynes,Ramsey,Fisher,Neyman and Pearson,Jeffreys,Savage,etc.For instance,on pp.564-65,he corrects the mythology and misdirected criticism directed at Laplace's calculation of the probability that the sun will rise tomorrow,given that it has already risen a certain number of times before,using the rule of succession.Laplace made it clear that he was not using any of the extensive ,relevant background knowledge ,available a priori, in his calculations.Thus,relative to the evidence specified only,there is nothing incorrect about the answer arrived at by Laplace.Jaynes,however, needed to add some chapters that would deal explicitly with social science and liberal arts,as well as disciplines like education,educational psychology,economics and business.I will spend the rest of the review commenting on the broader aspects of the work of John Maynard Keynes and Benoit Mandelbrot that Jaynes has appeared to have overlooked.First,Keynes intended that his interval estimate approach to probability,based on partial orders that do not satisfy the assumption of a sigma algebra,was applicable to all areas of life.Thus,Keynes obtains a general theory that is applicable everywhere.Nothing written by Keynes in the A Treatise on Probability(1921) contradicts the work of Jeffreys or Jaynes since the specific areas of science that Jeffreys and Jaynes seek to apply a logical approach to probability to satisfy the conditions need to specify a continuous mapping of the real numbers into each other.Single number probabilities under such a mapping require that a sigma algebra be specified in order to calculate the appropriate sums of unions and intersections.The basic data of physical and life science(molecules,cells,genes,chromosomes,atomic and subatomic partcles,electrons,etc.)is generally independent ,homogeneous, and invariant through time.When one turns to the other fields mentioned above,this is not the case.Keynes complemented his interval,approximation approach to decision making by systematically constucting a conventional coefficient of risk and weight that is able to deal with the nonlinear types of effects generated by the type of data available in the social sciences.Keynes's c coefficient equals p/(1+q)[2w/(1+w)].Define A to be an outcome.The decision maker maximizes cA,as opposed to the expected value rule,maximize pA or the expected utility rule,maximize pU(A),where U is a utility function and p is a probability(p+q=1).w represents the weight of the evidence.It measures the completeness of the actual and potential available ,relevant evidence upon which an estimate of probability will be based.It is an index that is normalized on the unit interval,0<=w<=1.Keynes's approach explains and gives solutions for all of the socalled paradoxes of decision theory.I now turn to the theoretical and empirical work of Benoit Mandelbrot.Based on a massive amount of data analysis from a number of different countries drawn from a number of different financial markets(cotton,commodity,stock,money,currency,bond),Mandelbrot has shown empirically that price movements in these markets demonstrate both long and short run dependence and discontinuity over time.Mandelbrot has incorporated variables representing these effects along with variables representing skewness and kurtosis into a generalized model that simplifies under special conditions to the normal distribution .Unfortunately,it is the normal probability distribution(and its relatives,the t,F,and chi square)that is used practically everywhere in the social sciences.Jaynes discussion of turbulence effects in his two page discourse on economics(7.21,pp.233-234)is suggestive that he also is somewhat aware that a different approach to analyzing social data that is nonhomogeneous and subject to abrupt and discontinuous change over time is needed in the social sciences.

Rating:
Summary: Ontological and Epistomological Probability
Review: I read this book before it was published; I downloaded it from a WU website. It has been of immense use to me in my career, it is a very practical book. Other reviews that say Dr. Jaynes' ideas are at odds with traditional measure theoretic probability are mistaken. Dr. Jaynes is a true Baysian. A Baysian is one who believes that probabilities do not model serendipity in nature, but do model subjective certainty. The Bayesian concept of probability is epistomological, i.e. the uncertainty is in our minds, not in objective reality. Traditional probability takes the reverse view: probabilities model unpredictable events, they are a model of objective reality like any science, i.e. probabilities are ontological. The trick is to realize the two are not mutually exclusive! There can be true ontological randomness in nature, and our minds can have uncertainty from incomplete knowledge as well. Probability theory as a branch of mathematics makes no claim what it models. The beauty is that probabiltity distributions integrate the two seamlessly. Thus, it is perfectly valid to put a distribution on an unknown parameter, epistomologically unknown, and derive that distribution from an experiment with, presumably, ontological randomness. Dr. Jaynes' book is well worth reading for the many case studies he presents. His background as a physicist is key to understanding some of the esoteric philisophical points.

Rating:
Summary: Truthful
Review: it offers a mathematical discussion of probability
from the point of view of information theory. It argues
against the frequentist approach. The author is absolutely
right: probability has meaning only as incomplete
knowledge (but still objective). Any frequentist approach
is problematic. A real diamond.

Rating:
Summary: The Basic Reference on Probability.
Review: Jaynes' work on probability has inspired many student and academics over the years. Jaynes advocates probability as a degree of belief. In the first two chapters he refers to the two axioms of Robert Cox relating probability to plausible reasoning and comments on "subjective" vs. "objective" reasoning. He moves on to discuss many aspects of classical statistics such as hyptohesis testing and parameter estimation from a Bayesian view. He also presents his famous work on prior probabilities and builds on Shannon's entropy definition to present the maximum entropy principle. A lot of people have been waiting for this book to be published. It should be on the bookshelf of every person who is interested in probability.

Rating:
Summary: The Basic Reference on Probability.
Review: Jaynes' work on probability has inspired many students and academics over the years. Jaynes advocates probability as a degree of belief. In the first two chapters he refers to the two axioms of Robert Cox relating probability to plausible reasoning and comments on "subjective" vs. "objective" reasoning. He moves on to discuss many aspects of classical statistics such as hyptohesis testing and parameter estimation from a Bayesian view. He also presents his famous work on prior probabilities and builds on Shannon's entropy definition to present the maximum entropy principle. A lot of people have been waiting for this book to be published.It should be on the bookshelf of every person who deals with probability.

Rating:
Summary: This book is the REAL new kind of science
Review: Outline

This book develops probability theory from first principles as an extension of deductive logic. In deductive logic, propositions can have only three possible truth values: true, false, and irremediable uncertainty. Therefore, the goal of the book is to describe a consistent extended logic that assigns real numbers to the plausibility of propositions. The requirements for such a system are derived from five simple desiderata, which serve as the postulates of this theory - and it turns out that *any* such system is equivalent to probability theory, to within a monotonic transformation.

Probability theory is then developed through applications to problems which grow more and more complex. The author demonstrates its use in direct sampling problems and so-called inverse problems, aka Bayesian probability. He derives procedures for multiple hypothesis testing, parameter estimation, and significance testing, and shows that although there are close connections between probability and frequency of occurrence in a large number of trials, no probability is *simply* a frequency.

Following this, the author presents solutions to the problem of assigning prior probabilities, and develops decision theory as an adjunct to probability theory. The author then compares and contrasts mainstream or "orthodox" statistical theory with probability theory as extended logic, and (perhaps unsurprisingly) finds severe deficiencies in the orthodox methods. The final chapters concern even more advanced applications.

Math Requirements

Readers should be well versed in simple calculus and multivariate calculus; some familiarity with convolution integrals and finite combinatorics is also an asset, but not essential. In isolated places, the author uses or refers to the calculus of variations and the theory of function spaces (in this case Hilbert spaces); but lack of familiarity with these branches of mathematics will not seriously hamper the reader.

Critical Review

This book represents a major step forward in the understanding of what probability theory is and how to use it. In particular, a lack of solutions to the problem of prior probabilities is the main reason that for the past 100 years, mainstream probability theory was taught as a theory of frequencies instead of as an extenstion of logic; therefore, having solutions to the problem of assigning priors in a textbook is a great step forward in the development of probability theory.

The book is a pleasure to read, with a text-to-equation ratio that is uncharacteristically high for a textbook of probability theory. That is not to say that the equations are simplistic; on the contrary, solutions to quite challenging problems are presented. In addition, the author's polemics against orthodox theory are quite entertaining (and convincing); he wields an acerbic pen when describing the efforts of those who actively reject probability theory as extended logic.

One negative feature of the book is its incompleteness: the author passed away before finishing the book, so occasionally large chunks of planned text are missing. The editor has cleverly mitigated this flaw by inserting "Editor's Problem Boxes", which challenge the reader to fill in the missing text. Still, as one reads the book, one gets the vaguely disquieting feeling that the author wanted to include much, much more information, but didn't have the chance.

Rating:
Summary: The most important book on probability theory in decades
Review: Reading this book is an exhilarating intellectual adventure. I found that it shed light on many mysteries and answered questions that had long troubled me. It contains the clearest exposition of the fundamentals of probability theory that I have ever encountered, and its chatty style is a pleasure to read. Jaynes the teacher collaborates fully with Jaynes the scientist in this book, and at times you feel as if the author is standing before you at the blackboard, chalk in hand, giving you a private lesson. Jaynes's advice on avoiding errors in the application of probability theory -- reinforced in many examples throughout the book -- is by itself well worth the price of the book.

If you deal at all with probability theory, statistics, data analysis, pattern recognition, automated diagnosis -- in short, any form of reasoning from inconclusive or uncertain information -- you need to read this book. It will give you new perspectives on these problems.

The downside to the book is that Jaynes died before he had a chance to finish it, and the editor, although capable and qualified to fill in the missing pieces, was understandably unwilling to inject himself into Jaynes's book. One result is that the quality of exposition suffers in some of the later chapters; furthermore, the author is not in a position to issue errata to correct various minor errors. Volunteer efforts are underway to remedy these problems -- those who buy the book may want to visit the "Unofficial Errata and Commentary" website for it, or check out the etjaynesstudy mailing list at Yahoo groups.

Rating:
Summary: The most important book on probability theory in decades
Review: Reading this book is an exhilarating intellectual adventure. I found that it shed light on many mysteries and answered questions that had long troubled me. It contains the clearest exposition of the fundamentals of probability theory that I have ever encountered, and its chatty style is a pleasure to read. Jaynes the teacher collaborates fully with Jaynes the scientist in this book, and at times you feel as if the author is standing before you at the blackboard, chalk in hand, giving you a private lesson. Jaynes's advice on avoiding errors in the application of probability theory -- reinforced in many examples throughout the book -- is by itself well worth the price of the book.

If you deal at all with probability theory, statistics, data analysis, pattern recognition, automated diagnosis -- in short, any form of reasoning from inconclusive or uncertain information -- you need to read this book. It will give you new perspectives on these problems.

The downside to the book is that Jaynes died before he had a chance to finish it, and the editor, although capable and qualified to fill in the missing pieces, was understandably unwilling to inject himself into Jaynes's book. One result is that the quality of exposition suffers in some of the later chapters; furthermore, the author is not in a position to issue errata to correct various minor errors. Volunteer efforts are underway to remedy these problems -- those who buy the book may want to visit the "Unofficial Errata and Commentary" website for it, or check out the etjaynesstudy mailing list at Yahoo groups.

Rating:
Summary: Invaluable
Review: This book has been on the web in unfinished form for a number of years and has shaped my scientific thinking more than any other book. I believe it constitutes one of the most important scientific texts of the last hundred years. It convincingly shows that "statistics", "statistical inference", "Bayesian inference", "probability theory", "maximum entropy methods" , and "statistical mechanics" are all parts of a large coherent theory that is the unique consistent extension of logic to propositions that have degrees of plausibility attached to them. This is already a theoretical accomplishment of epic proportions. But in addition, the book shows how one actually solves real world problems within this frame work, and in doing so shows what a vastly wider array of problems is addressable within this frame work than in any of the forementioned particular fields.
If you work in any field where on needs to "reason with incomplete information" this book is invaluable.

As others have already mentioned, Jaynes never finished this book. The editor decided to "fill in" the missing parts by putting excercises that, when finished by the reader, provide what (so the editor guesses) Jaynes left out. I find this solution a bit disappointing. The excercises don't take away the impression that holes are left in the text. It would have been better if the editor had written the missing parts and then printed those in different font so as to indicate that these parts were not written by Jaynes. Better still would have been if the editor had invited researchers that are intimately familiar with Jaynes' work and the topic of each of the missing pieces to submit text for the missing pieces. The editor could then have chosen from these to provide a "best guess" for what Jaynes might have written.

Finally, there is the issue of Jaynes' writing style. This is of course largely a matter of taste. I personally like his writing style very much because it is clear, and not as stifly formal as most science texts. However, some readers may find his style too belligerent and polemic.

Rating:
Summary: Brilliant but attended by many misunderstandings
Review: To "pure" mathematicians, probability theory is measure theory in spaces of measure 1. To the extent to which you remain a "pure" mathematician, this book will be incomprehensible to you.

To frequentist statisticians, probability theory is the study of relative frequencies or of proportions of a population; those are "probabilities".

To Bayesian statisticians, probability theory is the study of degrees of belief. Bayesians may assign probability 1/2 to the proposition that there was life on Mars a billion years ago; frequentists will not do that because they cannot say that there was life on Mars a billion years ago in precisely half of all cases -- there are no such "cases".

To _subjective_ Bayesians, probability theory is about subjective degrees of belief. A subjective degree of belief is merely how sure you happen to be.

"Noninformative" _objective_ Bayesians assign "noninformative" probability distributions when they deal with uncertain propositions or uncertain quantities, and replace them with "informative" distributions only when they update them because of "data". "Data", in this sense, consists of the outcomes of random experiments.

"Informative" _objective_ Bayesians -- a rare species -- ask what degree of belief in an uncertain proposition is logically necessitated by whatever information one has, and they don't necessarily require that information to consist of outcomes of random experiments.

Jaynes is an "informative" objective Bayesian. This book is his defense of that position and his account of how it is to be used.

"Pure" mathematicians will not find that this book resembles that branch of "pure" mathematics that they call probability theory.

Jaynes rails against those he disagrees with at great length. Often he is right. But often he simply misunderstands them. For example, writing in the 1990s, he said that pure mathematicians reject the use of Dirac's delta function and its derivatives, and related topics. That is nonsense; the delta function has long been considered highly respectable, and required material in the graduate curriculum. Unfortunately Jaynes's misunderstandings may cause some others to misunderstand him when he is right. Statisticians are more informed than "pure" mathematicians and will disagree with Jaynes for better reasons. _Some_ statisticians will agree with him.

Jaynes has many flaws, made all the more annoying by the fact that we need to overlook them in order to understand him. His message is important.