beyond objective and subjective statistics a discussion Christian P. Robert Universit´ e Paris-Dauphine (CEREMADE) & University of Warwick (Dept. of Statistics) http://xianblog.wordpress.com,seriesblog.net Royal Statistical Society, April 2017
beyond objective and subjective statisticsa discussion
Christian P. Robert
Universite Paris-Dauphine (CEREMADE)& University of Warwick (Dept. of Statistics)
http://xianblog.wordpress.com,seriesblog.netRoyal Statistical Society, April 2017
the elephant in the room...
Statistical analysis invariably starts
with the unquestioned premise of therandom nature of the data
which differs from the assumption of aprobabilistic model generating thedata [not much discussed therein]
& connects with repeatability ofobservations, which is almost alwayswrong
How do we address this blatantly wrong start?!
[perspective that led Keynes to abandon statistics and move toeconomics!]
the elephant in the room...
Statistical analysis invariably starts
with the unquestioned premise of therandom nature of the data
which differs from the assumption of aprobabilistic model generating thedata [not much discussed therein]
& connects with repeatability ofobservations, which is almost alwayswrong
How do we address this blatantly wrong start?!
[perspective that led Keynes to abandon statistics and move toeconomics!]
...and the tortoise in the next room
Is focus on wrong issue
arguing between ourselves about the best way to solve thewrong problem,
while users seek approximate solutions with some modicum ofefficiency
i.e., satisfied with imprecise inference
E.g., how many statistical problems “solved” Amazon in aday, compared with uncovering new fundamental particles???
...and the tortoise in the next room
Is focus on wrong issue
arguing between ourselves about the best way to solve thewrong problem,
while users seek approximate solutions with some modicum ofefficiency
i.e., satisfied with imprecise inference
E.g., how many statistical problems “solved” Amazon in aday, compared with uncovering new fundamental particles???
a forensic illustration
Vote on 10 April by US National Commission on Forensic Scienceon how forensic analysts should testify about evidence:
Analysts must
explain
how they examined evidence
what statistical analyses theychose
inherent uncertainties in theirmeasurements
[R. Mejia, Nature, 4 April 2017]
a forensic illustration
Vote on 10 April by US National Commission on Forensic Scienceon how forensic analysts should testify about evidence:
Analysts must
never claim with certainty thatanything on a crime scene islinked to a suspect
quantify the probability thatobserved similarities occurred bychance
[R. Mejia, Nature, 4 April 2017]
the ultimate issue with statistics
“...the ultimateinaccessibility of a realitythat is truly independent ofobservers is a basic humancondition.” A. Gelman& C. Hennig
Except for the most [basic] scientificsettings, there is not reality behindstatistical models [Box], hence aninaccessible consensus is the rule
”Keynesians who focus on more subjective factors...”
Read in John Maynard Keynes’s A Treatise on Probability (1921):
“...where general statistics areavailable, the numericalprobability which might bederived from them is inapplicablebecause of the presence ofadditional knowledge withregards to the particular case...”
”Keynesians who focus on more subjective factors...”
Read in John Maynard Keynes’s A Treatise on Probability (1921):
“...until a prima facie case hasbeen established for the existenceof a stable probable frequency,we have but a flimsy basis forany statistical induction...”
models, models, models
missing: trial-and-error way ofbuilding a statistical modeland/or analysis,while subjective inputs fromoperator(s) and regulators foundat all stages of constructionand should be spelled out ratherthan ignored (or rejected)
99% of the Universe
discussion on foundations (§5) thorough [objective judgement!] but
does not expand on issue of “default” or all-inclusivestatistical solutions
used through “point-and-shoot” software by innumeratepractitioners
while under impression of conducting a statistical analysis
false feeling also occurs in treatment of statistical expertise bymedia, courts, and scientific journals
[We are the 1%!!!]
99% of the Universe
discussion on foundations (§5) thorough [objective judgement!] but
does not expand on issue of “default” or all-inclusivestatistical solutions
used through “point-and-shoot” software by innumeratepractitioners
while under impression of conducting a statistical analysis
false feeling also occurs in treatment of statistical expertise bymedia, courts, and scientific journals
[We are the 1%!!!]
99% of the Universe
discussion on foundations (§5) thorough [objective judgement!] but
does not expand on issue of “default” or all-inclusivestatistical solutions
used through “point-and-shoot” software by innumeratepractitioners
while under impression of conducting a statistical analysis
false feeling also occurs in treatment of statistical expertise bymedia, courts, and scientific journals
[We are the 1%!!!]
Relativity
Pertains to awareness of multiple perspectives but also to stability
Fundamental dependence of inferential output on statisticalframework(s)
difference in outcomes perfectly acceptable
inner assessment of model feasible by producing pseudo data
comparison of frameworks only to the extent of predictivecharacteristics
Relativity
Pertains to awareness of multiple perspectives but also to stability
Bayesian analysis [both objective and subjective] well-suited tothis purpose / rescued by relativity
move to quantum theory where objectivity is impossible dueto the presence of the observer/experimenter [??]
reproducibility fraught with danger: production of similarresults depends on rigid framework, relates to fact thatstatistics not experimental science
rise of the machines
relevance of machinelearning for [model-free]learning
dismissal ofmachine-learning perspectivedisappointing
given robustness [againstmodels] of machine learningprediction
...if missing in uncertaintyassessment
the five pillars of statistical wisdom
frequentism, subjective about the Universe and its probabilitydistribution, and about the choice of an inference procedure,with falsifiability only achievable by replicating data viasimulation, but this only rejects poor models
[agreeing with Davies, 2014]
the five pillars of statistical wisdom
frequentism, subjective about the Universe and its probabilitydistribution, and about the choice of an inference procedure,with falsifiability only achievable by replicating data viasimulation, but this only rejects poor models
[agreeing with Davies, 2014]
subjective Bayesianism is quite objective and open about itssubjectivity or relativity, working in parallel universes that donot need to intersect, provided they produce such universes(discussion seems to swerve towards a common prior principlethat I do not understand!)
the five pillars of statistical wisdom
subjective Bayesianism is quite objective and open about itssubjectivity or relativity, working in parallel universes that donot need to intersect, provided they produce such universes(discussion seems to swerve towards a common prior principlethat I do not understand!)
very subjective concept of objective Bayes when it reduces toJaynes’s! and missing the point of “objective” Bayesprinciples which is not to be unique but rather define a genericprinciple of derivation of a prior from the likelihood functionwithout a particular justification
the five pillars of statistical wisdom
very subjective concept of objective Bayes when it reduces toJaynes’s! and missing the point of “objective” Bayesprinciples which is not to be unique but rather define a genericprinciple of derivation of a prior from the likelihood functionwithout a particular justification
I do not feel the gap between the above and the falsificationistBayesian perspective expressed in Section 5.5, hence soundsvery subjective to me. Unless the school reduces to Gelmanand Shalizi (2013)?!
embracing uncertainty with a subjective hug
basic realism and uncertain nature of data call for an absence ofhard decisions like tests and model choices, but rather fordescriptive performances of the suggested procedures, acceptingimperfection and variability in the answer produced locally
“Its not so hard to move awayfrom hypothesis testing andtoward a Bayesian approach ofembracing variation andaccepting uncertainty.” A.Gelman
conclusion
exposes the need to spell out the various inputs leading to astatistical analysis
reinforces the call for model awareness:
critical stance on all modelling inputs, including priors!,disbelief that any model is true
potential if realistic outcome would be to impose not onlyproduction of all conscious choices but also through theposting of (true or pseudo-) data and of relevant code for allpublications involving a statistical analysis
conclusion
proposal too idealistic in that most users (and most makers)of statistics cannot or would not spell out their assumptionsand choices, being unaware of or unapologetic about those
central difficulty with statistics as a service discipline, namelythat almost anyone anywhere can produce an estimate or ap-value without ever being proven wrong
how epistemological argument here going to profit statisticalmethodology
add layers of warning that the probability behind the model isnot connected with the phenomenon
a debate to be continued, hopefully
See you at O’Bayes17:International Workshop on Objective Bayes Methodology
held in Austin, Texas, Su, December 10 through We, December13, 2017
[https://sites.google.com/site/obayes2017/]