Statistical aspects of Higgs analyses
Post on 25-Feb-2016
56 Views
Preview:
DESCRIPTION
Transcript
Statistical aspects of Higgs analyses
W Verkerke(NIKHEF)
Introductionbull Enormous effort to search for Higgs signature in many
decay channelsbull Results many plots with signal
background expectations each with (systematic) uncertainties and data
bull Q How do you conclude from this that yoursquove seen the Higgs (or not)ndash Want answer of type
lsquoWe can exclude that the Higgs exist at 95 CLrdquo or ldquoProbabilitythat background only caused observedexcess is 310-7
bull Here a short guide through how this is (typically) done
Quantifying discovery and exclusion ndash Frequentist approach
bull Consider the simplest case ndash a counting experimentndash Observable N (the event count)ndash Model F(N|s) Poisson(N|s+b) with b=5 known exactly
bull Predicted distributions of N for various values of s
s=0
s=5
s=10s=15
bull Now make a measurement N=Nobs (example Nobs=7)bull Can now define p-value(s) eg for bkg hypothesis
ndash Fraction of future measurements with N=Nobs (or larger) if s=0
Frequentist p-values ndash excess over expected bkg
)230()0(
obsNb dNbNPoissonp
s=0
s=5s=10
s=15
bull p-values of background hypothesis is used to quantify lsquodiscoveryrsquo = excess of events over background expectation
bull Another example Nobs=15 for same model what is pb
ndash Result customarily re-expressed as odds of a Gaussian fluctuation with equal p-value (35 sigma for above case)
ndash NB Nobs=22 gives pb lt 2810-7 (lsquo5 sigmarsquo)
Frequentist p-values - excess over expected bkg
)000220()0(
obsNb dNbNPoissonp
s=0
s=5s=10
s=15
Upper limits (one-sided confidence intervals)bull Can also defined p-values for hypothesis with signal ps+b
ndash Note convention integration range in ps+b is flipped
bull Convention express result as value of s for which p(s+b)=5 ldquosgt68 is excluded at 95 CLrdquo
Wouter Verkerke NIKHEF
obsN
bs dNsbNPoissonp )(
p(s=15) = 000025p(s=10) = 0007p(s=5) = 013
p(s=68) = 005
Modified frequentist upper limitsbull Need to be careful about interpretation p(s+b) in terms
of inference on signal onlyndash Since p(s+b) quantifies consistency of signal plus backgroundndash Problem most apparent when observed data
has downward stat fluctations wrt background expectationbull Example Nobs =2
bull Modified approach to protectagainst such inference on sndash Instead of requiring p(s+b)=5
require
ps+b(s=0) = 004
sge0 excluded at gt95 CL s=0
s=5
s=10s=15
51
b
bsS p
pCL
for N=2 exclude sgt34 at 95 CLs for large N effect on limit is small as pb0
p-values and limits on non-trivial analysis
bull Typical Higgs search result is not a simple number counting experiment but looks like this
bull Any type of result can be converted into a single number by constructing a lsquotest statisticrsquo ndash A test statistic compresses all signal-to-background
discrimination power in a single numberndash Most powerful discriminators
are Likelihood Ratios(Neyman Pearson)
)ˆ|()|(ln2
dataLdataLq
- Result is a distribution not a single number
- Models for signal and background have intrinsic uncertainties
The likelihood ratio test statistic
bull Definition μ = signal strength signal strength(SM)ndash Choose eg likelihood with nominal signal strength in numerator (μ=1)
bull Illustration on model with no shape uncertainties
)ˆ|()1|(ln21
dataLdataLq
μ is best fit value of μ^
lsquolikelihood of best fitrsquo
lsquolikelihood assuming nominal signal strengthrsquo
770~1 q 3452~
1 q
On signal-like data q1 is small On background-like data q1 is large
Distributions of test statisticsbull Value of q1 on data is now the lsquomeasurementrsquobull Distribution of q1 not calculable
But can obtain distribution from pseudo-experimentsbull Generate a large number of pseudo-experiments with a given value of mu
calculate q for each plot distribution
bull From qobs and these distributions can then set limitssimilar to what was shown for Poisson counting example
)ˆ|()1|(ln2~
1
dataLdataLq
q1 for experiments with signal
q1 for experiments with background only
Note analogyto Poisson
counting example
Incorporating systematic uncertaintiesbull What happens if models have uncertainties
ndash Introduction of additional model parameters θ that describe effect of uncertainties
)|()|(
dataLdataL
)~())()(|()|( pbsNPoissondataL iii
xx θ2θ1
Jet Energy ScaleQCD scaleluminosity
Incorporating systematic uncertainties
)~())()(|()|( pbsNPoissondataL iii
xx θ2θ1
Likelihood includes auxiliary measurement terms that constrains the nuisance parameters θ(shape is flat log-normal gamma or Gaussian)
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are
incorporated in test statistic using a profile likelihood ratio
bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments
ˆ0(with a constraint )
lsquolikelihood of best fitrsquo
lsquolikelihood of best fit for a given fixed value of μrsquo
)ˆ|()|(ln2
dataLdataLq
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are
incorporated using a profile likelihood ratio test statistic
bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments
ˆ0(with a constraint )lsquolikelihood of best fitrsquo
lsquolikelihood of best fit for a given fixed value of μrsquo
μ=035 μ=10 μ=18 μ=26
Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models
the data for a given true Higgs mass hypothesis
bull Construct test statistic
bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value
and signal exclusion limitbull Repeat for each assumed mH
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ|(
ln2)(~
H
HH mdataL
mdataLmq
Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV
Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV
Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]
Expected exclusion limitfor background-only hypothesis
Combining Higgs channels (and experiments)bull Procedure define joint likelihood
bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration
bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
)()()( CMSCMSATLASATLASLHC LLL
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)
have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and
delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
lsquollll110rootrsquo lsquogg110rootrsquo
Example ndash joint ATLASCMS Higgs exclusion limit
Wouter Verkerke NIKHEF
Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ0|(ln2~ 0
0
dataLdataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming background onlyrsquo
)ˆˆ|()ˆ|(
ln2~
dataL
dataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming μ signal strengthrsquo
770~1 q
lsquodiscoveryrsquolsquoexclusionrsquo
7734~0 q
0~0 q3452~
1 q
simulateddata with signal+bkg
simulateddata with bkg only
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
Note that lsquopeakrsquo around 160 GeV reflects increased
experimental sensitivity not SM prediction of Higgs mass
Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)
But need to be careful with local p-values
Search is executed for a wide mH mass range
Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1
Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction
Conclusionsbull We are early awaiting more data
Wouter Verkerke NIKHEF
- Statistical aspects of Higgs analyses
- Introduction
- Quantifying discovery and exclusion ndash Frequentist approach
- Frequentist p-values ndash excess over expected bkg
- Frequentist p-values - excess over expected bkg
- Upper limits (one-sided confidence intervals)
- Modified frequentist upper limits
- p-values and limits on non-trivial analysis
- The likelihood ratio test statistic
- Distributions of test statistics
- Incorporating systematic uncertainties
- Incorporating systematic uncertainties (2)
- Dealing with nuisance parameters in the test statistic
- Dealing with nuisance parameters in the test statistic (2)
- Putting it all together for one Higgs channel
- Example ndash 95 Exclusion limit vs mH for HWW
- Combining Higgs channels (and experiments)
- A word on the machinery
- Example ndash joint ATLASCMS Higgs exclusion limit
- Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
- Conclusions
Introductionbull Enormous effort to search for Higgs signature in many
decay channelsbull Results many plots with signal
background expectations each with (systematic) uncertainties and data
bull Q How do you conclude from this that yoursquove seen the Higgs (or not)ndash Want answer of type
lsquoWe can exclude that the Higgs exist at 95 CLrdquo or ldquoProbabilitythat background only caused observedexcess is 310-7
bull Here a short guide through how this is (typically) done
Quantifying discovery and exclusion ndash Frequentist approach
bull Consider the simplest case ndash a counting experimentndash Observable N (the event count)ndash Model F(N|s) Poisson(N|s+b) with b=5 known exactly
bull Predicted distributions of N for various values of s
s=0
s=5
s=10s=15
bull Now make a measurement N=Nobs (example Nobs=7)bull Can now define p-value(s) eg for bkg hypothesis
ndash Fraction of future measurements with N=Nobs (or larger) if s=0
Frequentist p-values ndash excess over expected bkg
)230()0(
obsNb dNbNPoissonp
s=0
s=5s=10
s=15
bull p-values of background hypothesis is used to quantify lsquodiscoveryrsquo = excess of events over background expectation
bull Another example Nobs=15 for same model what is pb
ndash Result customarily re-expressed as odds of a Gaussian fluctuation with equal p-value (35 sigma for above case)
ndash NB Nobs=22 gives pb lt 2810-7 (lsquo5 sigmarsquo)
Frequentist p-values - excess over expected bkg
)000220()0(
obsNb dNbNPoissonp
s=0
s=5s=10
s=15
Upper limits (one-sided confidence intervals)bull Can also defined p-values for hypothesis with signal ps+b
ndash Note convention integration range in ps+b is flipped
bull Convention express result as value of s for which p(s+b)=5 ldquosgt68 is excluded at 95 CLrdquo
Wouter Verkerke NIKHEF
obsN
bs dNsbNPoissonp )(
p(s=15) = 000025p(s=10) = 0007p(s=5) = 013
p(s=68) = 005
Modified frequentist upper limitsbull Need to be careful about interpretation p(s+b) in terms
of inference on signal onlyndash Since p(s+b) quantifies consistency of signal plus backgroundndash Problem most apparent when observed data
has downward stat fluctations wrt background expectationbull Example Nobs =2
bull Modified approach to protectagainst such inference on sndash Instead of requiring p(s+b)=5
require
ps+b(s=0) = 004
sge0 excluded at gt95 CL s=0
s=5
s=10s=15
51
b
bsS p
pCL
for N=2 exclude sgt34 at 95 CLs for large N effect on limit is small as pb0
p-values and limits on non-trivial analysis
bull Typical Higgs search result is not a simple number counting experiment but looks like this
bull Any type of result can be converted into a single number by constructing a lsquotest statisticrsquo ndash A test statistic compresses all signal-to-background
discrimination power in a single numberndash Most powerful discriminators
are Likelihood Ratios(Neyman Pearson)
)ˆ|()|(ln2
dataLdataLq
- Result is a distribution not a single number
- Models for signal and background have intrinsic uncertainties
The likelihood ratio test statistic
bull Definition μ = signal strength signal strength(SM)ndash Choose eg likelihood with nominal signal strength in numerator (μ=1)
bull Illustration on model with no shape uncertainties
)ˆ|()1|(ln21
dataLdataLq
μ is best fit value of μ^
lsquolikelihood of best fitrsquo
lsquolikelihood assuming nominal signal strengthrsquo
770~1 q 3452~
1 q
On signal-like data q1 is small On background-like data q1 is large
Distributions of test statisticsbull Value of q1 on data is now the lsquomeasurementrsquobull Distribution of q1 not calculable
But can obtain distribution from pseudo-experimentsbull Generate a large number of pseudo-experiments with a given value of mu
calculate q for each plot distribution
bull From qobs and these distributions can then set limitssimilar to what was shown for Poisson counting example
)ˆ|()1|(ln2~
1
dataLdataLq
q1 for experiments with signal
q1 for experiments with background only
Note analogyto Poisson
counting example
Incorporating systematic uncertaintiesbull What happens if models have uncertainties
ndash Introduction of additional model parameters θ that describe effect of uncertainties
)|()|(
dataLdataL
)~())()(|()|( pbsNPoissondataL iii
xx θ2θ1
Jet Energy ScaleQCD scaleluminosity
Incorporating systematic uncertainties
)~())()(|()|( pbsNPoissondataL iii
xx θ2θ1
Likelihood includes auxiliary measurement terms that constrains the nuisance parameters θ(shape is flat log-normal gamma or Gaussian)
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are
incorporated in test statistic using a profile likelihood ratio
bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments
ˆ0(with a constraint )
lsquolikelihood of best fitrsquo
lsquolikelihood of best fit for a given fixed value of μrsquo
)ˆ|()|(ln2
dataLdataLq
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are
incorporated using a profile likelihood ratio test statistic
bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments
ˆ0(with a constraint )lsquolikelihood of best fitrsquo
lsquolikelihood of best fit for a given fixed value of μrsquo
μ=035 μ=10 μ=18 μ=26
Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models
the data for a given true Higgs mass hypothesis
bull Construct test statistic
bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value
and signal exclusion limitbull Repeat for each assumed mH
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ|(
ln2)(~
H
HH mdataL
mdataLmq
Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV
Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV
Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]
Expected exclusion limitfor background-only hypothesis
Combining Higgs channels (and experiments)bull Procedure define joint likelihood
bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration
bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
)()()( CMSCMSATLASATLASLHC LLL
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)
have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and
delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
lsquollll110rootrsquo lsquogg110rootrsquo
Example ndash joint ATLASCMS Higgs exclusion limit
Wouter Verkerke NIKHEF
Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ0|(ln2~ 0
0
dataLdataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming background onlyrsquo
)ˆˆ|()ˆ|(
ln2~
dataL
dataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming μ signal strengthrsquo
770~1 q
lsquodiscoveryrsquolsquoexclusionrsquo
7734~0 q
0~0 q3452~
1 q
simulateddata with signal+bkg
simulateddata with bkg only
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
Note that lsquopeakrsquo around 160 GeV reflects increased
experimental sensitivity not SM prediction of Higgs mass
Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)
But need to be careful with local p-values
Search is executed for a wide mH mass range
Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1
Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction
Conclusionsbull We are early awaiting more data
Wouter Verkerke NIKHEF
- Statistical aspects of Higgs analyses
- Introduction
- Quantifying discovery and exclusion ndash Frequentist approach
- Frequentist p-values ndash excess over expected bkg
- Frequentist p-values - excess over expected bkg
- Upper limits (one-sided confidence intervals)
- Modified frequentist upper limits
- p-values and limits on non-trivial analysis
- The likelihood ratio test statistic
- Distributions of test statistics
- Incorporating systematic uncertainties
- Incorporating systematic uncertainties (2)
- Dealing with nuisance parameters in the test statistic
- Dealing with nuisance parameters in the test statistic (2)
- Putting it all together for one Higgs channel
- Example ndash 95 Exclusion limit vs mH for HWW
- Combining Higgs channels (and experiments)
- A word on the machinery
- Example ndash joint ATLASCMS Higgs exclusion limit
- Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
- Conclusions
Quantifying discovery and exclusion ndash Frequentist approach
bull Consider the simplest case ndash a counting experimentndash Observable N (the event count)ndash Model F(N|s) Poisson(N|s+b) with b=5 known exactly
bull Predicted distributions of N for various values of s
s=0
s=5
s=10s=15
bull Now make a measurement N=Nobs (example Nobs=7)bull Can now define p-value(s) eg for bkg hypothesis
ndash Fraction of future measurements with N=Nobs (or larger) if s=0
Frequentist p-values ndash excess over expected bkg
)230()0(
obsNb dNbNPoissonp
s=0
s=5s=10
s=15
bull p-values of background hypothesis is used to quantify lsquodiscoveryrsquo = excess of events over background expectation
bull Another example Nobs=15 for same model what is pb
ndash Result customarily re-expressed as odds of a Gaussian fluctuation with equal p-value (35 sigma for above case)
ndash NB Nobs=22 gives pb lt 2810-7 (lsquo5 sigmarsquo)
Frequentist p-values - excess over expected bkg
)000220()0(
obsNb dNbNPoissonp
s=0
s=5s=10
s=15
Upper limits (one-sided confidence intervals)bull Can also defined p-values for hypothesis with signal ps+b
ndash Note convention integration range in ps+b is flipped
bull Convention express result as value of s for which p(s+b)=5 ldquosgt68 is excluded at 95 CLrdquo
Wouter Verkerke NIKHEF
obsN
bs dNsbNPoissonp )(
p(s=15) = 000025p(s=10) = 0007p(s=5) = 013
p(s=68) = 005
Modified frequentist upper limitsbull Need to be careful about interpretation p(s+b) in terms
of inference on signal onlyndash Since p(s+b) quantifies consistency of signal plus backgroundndash Problem most apparent when observed data
has downward stat fluctations wrt background expectationbull Example Nobs =2
bull Modified approach to protectagainst such inference on sndash Instead of requiring p(s+b)=5
require
ps+b(s=0) = 004
sge0 excluded at gt95 CL s=0
s=5
s=10s=15
51
b
bsS p
pCL
for N=2 exclude sgt34 at 95 CLs for large N effect on limit is small as pb0
p-values and limits on non-trivial analysis
bull Typical Higgs search result is not a simple number counting experiment but looks like this
bull Any type of result can be converted into a single number by constructing a lsquotest statisticrsquo ndash A test statistic compresses all signal-to-background
discrimination power in a single numberndash Most powerful discriminators
are Likelihood Ratios(Neyman Pearson)
)ˆ|()|(ln2
dataLdataLq
- Result is a distribution not a single number
- Models for signal and background have intrinsic uncertainties
The likelihood ratio test statistic
bull Definition μ = signal strength signal strength(SM)ndash Choose eg likelihood with nominal signal strength in numerator (μ=1)
bull Illustration on model with no shape uncertainties
)ˆ|()1|(ln21
dataLdataLq
μ is best fit value of μ^
lsquolikelihood of best fitrsquo
lsquolikelihood assuming nominal signal strengthrsquo
770~1 q 3452~
1 q
On signal-like data q1 is small On background-like data q1 is large
Distributions of test statisticsbull Value of q1 on data is now the lsquomeasurementrsquobull Distribution of q1 not calculable
But can obtain distribution from pseudo-experimentsbull Generate a large number of pseudo-experiments with a given value of mu
calculate q for each plot distribution
bull From qobs and these distributions can then set limitssimilar to what was shown for Poisson counting example
)ˆ|()1|(ln2~
1
dataLdataLq
q1 for experiments with signal
q1 for experiments with background only
Note analogyto Poisson
counting example
Incorporating systematic uncertaintiesbull What happens if models have uncertainties
ndash Introduction of additional model parameters θ that describe effect of uncertainties
)|()|(
dataLdataL
)~())()(|()|( pbsNPoissondataL iii
xx θ2θ1
Jet Energy ScaleQCD scaleluminosity
Incorporating systematic uncertainties
)~())()(|()|( pbsNPoissondataL iii
xx θ2θ1
Likelihood includes auxiliary measurement terms that constrains the nuisance parameters θ(shape is flat log-normal gamma or Gaussian)
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are
incorporated in test statistic using a profile likelihood ratio
bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments
ˆ0(with a constraint )
lsquolikelihood of best fitrsquo
lsquolikelihood of best fit for a given fixed value of μrsquo
)ˆ|()|(ln2
dataLdataLq
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are
incorporated using a profile likelihood ratio test statistic
bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments
ˆ0(with a constraint )lsquolikelihood of best fitrsquo
lsquolikelihood of best fit for a given fixed value of μrsquo
μ=035 μ=10 μ=18 μ=26
Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models
the data for a given true Higgs mass hypothesis
bull Construct test statistic
bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value
and signal exclusion limitbull Repeat for each assumed mH
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ|(
ln2)(~
H
HH mdataL
mdataLmq
Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV
Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV
Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]
Expected exclusion limitfor background-only hypothesis
Combining Higgs channels (and experiments)bull Procedure define joint likelihood
bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration
bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
)()()( CMSCMSATLASATLASLHC LLL
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)
have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and
delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
lsquollll110rootrsquo lsquogg110rootrsquo
Example ndash joint ATLASCMS Higgs exclusion limit
Wouter Verkerke NIKHEF
Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ0|(ln2~ 0
0
dataLdataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming background onlyrsquo
)ˆˆ|()ˆ|(
ln2~
dataL
dataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming μ signal strengthrsquo
770~1 q
lsquodiscoveryrsquolsquoexclusionrsquo
7734~0 q
0~0 q3452~
1 q
simulateddata with signal+bkg
simulateddata with bkg only
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
Note that lsquopeakrsquo around 160 GeV reflects increased
experimental sensitivity not SM prediction of Higgs mass
Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)
But need to be careful with local p-values
Search is executed for a wide mH mass range
Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1
Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction
Conclusionsbull We are early awaiting more data
Wouter Verkerke NIKHEF
- Statistical aspects of Higgs analyses
- Introduction
- Quantifying discovery and exclusion ndash Frequentist approach
- Frequentist p-values ndash excess over expected bkg
- Frequentist p-values - excess over expected bkg
- Upper limits (one-sided confidence intervals)
- Modified frequentist upper limits
- p-values and limits on non-trivial analysis
- The likelihood ratio test statistic
- Distributions of test statistics
- Incorporating systematic uncertainties
- Incorporating systematic uncertainties (2)
- Dealing with nuisance parameters in the test statistic
- Dealing with nuisance parameters in the test statistic (2)
- Putting it all together for one Higgs channel
- Example ndash 95 Exclusion limit vs mH for HWW
- Combining Higgs channels (and experiments)
- A word on the machinery
- Example ndash joint ATLASCMS Higgs exclusion limit
- Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
- Conclusions
bull Now make a measurement N=Nobs (example Nobs=7)bull Can now define p-value(s) eg for bkg hypothesis
ndash Fraction of future measurements with N=Nobs (or larger) if s=0
Frequentist p-values ndash excess over expected bkg
)230()0(
obsNb dNbNPoissonp
s=0
s=5s=10
s=15
bull p-values of background hypothesis is used to quantify lsquodiscoveryrsquo = excess of events over background expectation
bull Another example Nobs=15 for same model what is pb
ndash Result customarily re-expressed as odds of a Gaussian fluctuation with equal p-value (35 sigma for above case)
ndash NB Nobs=22 gives pb lt 2810-7 (lsquo5 sigmarsquo)
Frequentist p-values - excess over expected bkg
)000220()0(
obsNb dNbNPoissonp
s=0
s=5s=10
s=15
Upper limits (one-sided confidence intervals)bull Can also defined p-values for hypothesis with signal ps+b
ndash Note convention integration range in ps+b is flipped
bull Convention express result as value of s for which p(s+b)=5 ldquosgt68 is excluded at 95 CLrdquo
Wouter Verkerke NIKHEF
obsN
bs dNsbNPoissonp )(
p(s=15) = 000025p(s=10) = 0007p(s=5) = 013
p(s=68) = 005
Modified frequentist upper limitsbull Need to be careful about interpretation p(s+b) in terms
of inference on signal onlyndash Since p(s+b) quantifies consistency of signal plus backgroundndash Problem most apparent when observed data
has downward stat fluctations wrt background expectationbull Example Nobs =2
bull Modified approach to protectagainst such inference on sndash Instead of requiring p(s+b)=5
require
ps+b(s=0) = 004
sge0 excluded at gt95 CL s=0
s=5
s=10s=15
51
b
bsS p
pCL
for N=2 exclude sgt34 at 95 CLs for large N effect on limit is small as pb0
p-values and limits on non-trivial analysis
bull Typical Higgs search result is not a simple number counting experiment but looks like this
bull Any type of result can be converted into a single number by constructing a lsquotest statisticrsquo ndash A test statistic compresses all signal-to-background
discrimination power in a single numberndash Most powerful discriminators
are Likelihood Ratios(Neyman Pearson)
)ˆ|()|(ln2
dataLdataLq
- Result is a distribution not a single number
- Models for signal and background have intrinsic uncertainties
The likelihood ratio test statistic
bull Definition μ = signal strength signal strength(SM)ndash Choose eg likelihood with nominal signal strength in numerator (μ=1)
bull Illustration on model with no shape uncertainties
)ˆ|()1|(ln21
dataLdataLq
μ is best fit value of μ^
lsquolikelihood of best fitrsquo
lsquolikelihood assuming nominal signal strengthrsquo
770~1 q 3452~
1 q
On signal-like data q1 is small On background-like data q1 is large
Distributions of test statisticsbull Value of q1 on data is now the lsquomeasurementrsquobull Distribution of q1 not calculable
But can obtain distribution from pseudo-experimentsbull Generate a large number of pseudo-experiments with a given value of mu
calculate q for each plot distribution
bull From qobs and these distributions can then set limitssimilar to what was shown for Poisson counting example
)ˆ|()1|(ln2~
1
dataLdataLq
q1 for experiments with signal
q1 for experiments with background only
Note analogyto Poisson
counting example
Incorporating systematic uncertaintiesbull What happens if models have uncertainties
ndash Introduction of additional model parameters θ that describe effect of uncertainties
)|()|(
dataLdataL
)~())()(|()|( pbsNPoissondataL iii
xx θ2θ1
Jet Energy ScaleQCD scaleluminosity
Incorporating systematic uncertainties
)~())()(|()|( pbsNPoissondataL iii
xx θ2θ1
Likelihood includes auxiliary measurement terms that constrains the nuisance parameters θ(shape is flat log-normal gamma or Gaussian)
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are
incorporated in test statistic using a profile likelihood ratio
bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments
ˆ0(with a constraint )
lsquolikelihood of best fitrsquo
lsquolikelihood of best fit for a given fixed value of μrsquo
)ˆ|()|(ln2
dataLdataLq
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are
incorporated using a profile likelihood ratio test statistic
bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments
ˆ0(with a constraint )lsquolikelihood of best fitrsquo
lsquolikelihood of best fit for a given fixed value of μrsquo
μ=035 μ=10 μ=18 μ=26
Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models
the data for a given true Higgs mass hypothesis
bull Construct test statistic
bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value
and signal exclusion limitbull Repeat for each assumed mH
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ|(
ln2)(~
H
HH mdataL
mdataLmq
Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV
Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV
Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]
Expected exclusion limitfor background-only hypothesis
Combining Higgs channels (and experiments)bull Procedure define joint likelihood
bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration
bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
)()()( CMSCMSATLASATLASLHC LLL
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)
have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and
delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
lsquollll110rootrsquo lsquogg110rootrsquo
Example ndash joint ATLASCMS Higgs exclusion limit
Wouter Verkerke NIKHEF
Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ0|(ln2~ 0
0
dataLdataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming background onlyrsquo
)ˆˆ|()ˆ|(
ln2~
dataL
dataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming μ signal strengthrsquo
770~1 q
lsquodiscoveryrsquolsquoexclusionrsquo
7734~0 q
0~0 q3452~
1 q
simulateddata with signal+bkg
simulateddata with bkg only
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
Note that lsquopeakrsquo around 160 GeV reflects increased
experimental sensitivity not SM prediction of Higgs mass
Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)
But need to be careful with local p-values
Search is executed for a wide mH mass range
Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1
Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction
Conclusionsbull We are early awaiting more data
Wouter Verkerke NIKHEF
- Statistical aspects of Higgs analyses
- Introduction
- Quantifying discovery and exclusion ndash Frequentist approach
- Frequentist p-values ndash excess over expected bkg
- Frequentist p-values - excess over expected bkg
- Upper limits (one-sided confidence intervals)
- Modified frequentist upper limits
- p-values and limits on non-trivial analysis
- The likelihood ratio test statistic
- Distributions of test statistics
- Incorporating systematic uncertainties
- Incorporating systematic uncertainties (2)
- Dealing with nuisance parameters in the test statistic
- Dealing with nuisance parameters in the test statistic (2)
- Putting it all together for one Higgs channel
- Example ndash 95 Exclusion limit vs mH for HWW
- Combining Higgs channels (and experiments)
- A word on the machinery
- Example ndash joint ATLASCMS Higgs exclusion limit
- Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
- Conclusions
bull p-values of background hypothesis is used to quantify lsquodiscoveryrsquo = excess of events over background expectation
bull Another example Nobs=15 for same model what is pb
ndash Result customarily re-expressed as odds of a Gaussian fluctuation with equal p-value (35 sigma for above case)
ndash NB Nobs=22 gives pb lt 2810-7 (lsquo5 sigmarsquo)
Frequentist p-values - excess over expected bkg
)000220()0(
obsNb dNbNPoissonp
s=0
s=5s=10
s=15
Upper limits (one-sided confidence intervals)bull Can also defined p-values for hypothesis with signal ps+b
ndash Note convention integration range in ps+b is flipped
bull Convention express result as value of s for which p(s+b)=5 ldquosgt68 is excluded at 95 CLrdquo
Wouter Verkerke NIKHEF
obsN
bs dNsbNPoissonp )(
p(s=15) = 000025p(s=10) = 0007p(s=5) = 013
p(s=68) = 005
Modified frequentist upper limitsbull Need to be careful about interpretation p(s+b) in terms
of inference on signal onlyndash Since p(s+b) quantifies consistency of signal plus backgroundndash Problem most apparent when observed data
has downward stat fluctations wrt background expectationbull Example Nobs =2
bull Modified approach to protectagainst such inference on sndash Instead of requiring p(s+b)=5
require
ps+b(s=0) = 004
sge0 excluded at gt95 CL s=0
s=5
s=10s=15
51
b
bsS p
pCL
for N=2 exclude sgt34 at 95 CLs for large N effect on limit is small as pb0
p-values and limits on non-trivial analysis
bull Typical Higgs search result is not a simple number counting experiment but looks like this
bull Any type of result can be converted into a single number by constructing a lsquotest statisticrsquo ndash A test statistic compresses all signal-to-background
discrimination power in a single numberndash Most powerful discriminators
are Likelihood Ratios(Neyman Pearson)
)ˆ|()|(ln2
dataLdataLq
- Result is a distribution not a single number
- Models for signal and background have intrinsic uncertainties
The likelihood ratio test statistic
bull Definition μ = signal strength signal strength(SM)ndash Choose eg likelihood with nominal signal strength in numerator (μ=1)
bull Illustration on model with no shape uncertainties
)ˆ|()1|(ln21
dataLdataLq
μ is best fit value of μ^
lsquolikelihood of best fitrsquo
lsquolikelihood assuming nominal signal strengthrsquo
770~1 q 3452~
1 q
On signal-like data q1 is small On background-like data q1 is large
Distributions of test statisticsbull Value of q1 on data is now the lsquomeasurementrsquobull Distribution of q1 not calculable
But can obtain distribution from pseudo-experimentsbull Generate a large number of pseudo-experiments with a given value of mu
calculate q for each plot distribution
bull From qobs and these distributions can then set limitssimilar to what was shown for Poisson counting example
)ˆ|()1|(ln2~
1
dataLdataLq
q1 for experiments with signal
q1 for experiments with background only
Note analogyto Poisson
counting example
Incorporating systematic uncertaintiesbull What happens if models have uncertainties
ndash Introduction of additional model parameters θ that describe effect of uncertainties
)|()|(
dataLdataL
)~())()(|()|( pbsNPoissondataL iii
xx θ2θ1
Jet Energy ScaleQCD scaleluminosity
Incorporating systematic uncertainties
)~())()(|()|( pbsNPoissondataL iii
xx θ2θ1
Likelihood includes auxiliary measurement terms that constrains the nuisance parameters θ(shape is flat log-normal gamma or Gaussian)
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are
incorporated in test statistic using a profile likelihood ratio
bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments
ˆ0(with a constraint )
lsquolikelihood of best fitrsquo
lsquolikelihood of best fit for a given fixed value of μrsquo
)ˆ|()|(ln2
dataLdataLq
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are
incorporated using a profile likelihood ratio test statistic
bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments
ˆ0(with a constraint )lsquolikelihood of best fitrsquo
lsquolikelihood of best fit for a given fixed value of μrsquo
μ=035 μ=10 μ=18 μ=26
Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models
the data for a given true Higgs mass hypothesis
bull Construct test statistic
bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value
and signal exclusion limitbull Repeat for each assumed mH
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ|(
ln2)(~
H
HH mdataL
mdataLmq
Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV
Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV
Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]
Expected exclusion limitfor background-only hypothesis
Combining Higgs channels (and experiments)bull Procedure define joint likelihood
bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration
bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
)()()( CMSCMSATLASATLASLHC LLL
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)
have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and
delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
lsquollll110rootrsquo lsquogg110rootrsquo
Example ndash joint ATLASCMS Higgs exclusion limit
Wouter Verkerke NIKHEF
Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ0|(ln2~ 0
0
dataLdataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming background onlyrsquo
)ˆˆ|()ˆ|(
ln2~
dataL
dataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming μ signal strengthrsquo
770~1 q
lsquodiscoveryrsquolsquoexclusionrsquo
7734~0 q
0~0 q3452~
1 q
simulateddata with signal+bkg
simulateddata with bkg only
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
Note that lsquopeakrsquo around 160 GeV reflects increased
experimental sensitivity not SM prediction of Higgs mass
Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)
But need to be careful with local p-values
Search is executed for a wide mH mass range
Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1
Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction
Conclusionsbull We are early awaiting more data
Wouter Verkerke NIKHEF
- Statistical aspects of Higgs analyses
- Introduction
- Quantifying discovery and exclusion ndash Frequentist approach
- Frequentist p-values ndash excess over expected bkg
- Frequentist p-values - excess over expected bkg
- Upper limits (one-sided confidence intervals)
- Modified frequentist upper limits
- p-values and limits on non-trivial analysis
- The likelihood ratio test statistic
- Distributions of test statistics
- Incorporating systematic uncertainties
- Incorporating systematic uncertainties (2)
- Dealing with nuisance parameters in the test statistic
- Dealing with nuisance parameters in the test statistic (2)
- Putting it all together for one Higgs channel
- Example ndash 95 Exclusion limit vs mH for HWW
- Combining Higgs channels (and experiments)
- A word on the machinery
- Example ndash joint ATLASCMS Higgs exclusion limit
- Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
- Conclusions
Upper limits (one-sided confidence intervals)bull Can also defined p-values for hypothesis with signal ps+b
ndash Note convention integration range in ps+b is flipped
bull Convention express result as value of s for which p(s+b)=5 ldquosgt68 is excluded at 95 CLrdquo
Wouter Verkerke NIKHEF
obsN
bs dNsbNPoissonp )(
p(s=15) = 000025p(s=10) = 0007p(s=5) = 013
p(s=68) = 005
Modified frequentist upper limitsbull Need to be careful about interpretation p(s+b) in terms
of inference on signal onlyndash Since p(s+b) quantifies consistency of signal plus backgroundndash Problem most apparent when observed data
has downward stat fluctations wrt background expectationbull Example Nobs =2
bull Modified approach to protectagainst such inference on sndash Instead of requiring p(s+b)=5
require
ps+b(s=0) = 004
sge0 excluded at gt95 CL s=0
s=5
s=10s=15
51
b
bsS p
pCL
for N=2 exclude sgt34 at 95 CLs for large N effect on limit is small as pb0
p-values and limits on non-trivial analysis
bull Typical Higgs search result is not a simple number counting experiment but looks like this
bull Any type of result can be converted into a single number by constructing a lsquotest statisticrsquo ndash A test statistic compresses all signal-to-background
discrimination power in a single numberndash Most powerful discriminators
are Likelihood Ratios(Neyman Pearson)
)ˆ|()|(ln2
dataLdataLq
- Result is a distribution not a single number
- Models for signal and background have intrinsic uncertainties
The likelihood ratio test statistic
bull Definition μ = signal strength signal strength(SM)ndash Choose eg likelihood with nominal signal strength in numerator (μ=1)
bull Illustration on model with no shape uncertainties
)ˆ|()1|(ln21
dataLdataLq
μ is best fit value of μ^
lsquolikelihood of best fitrsquo
lsquolikelihood assuming nominal signal strengthrsquo
770~1 q 3452~
1 q
On signal-like data q1 is small On background-like data q1 is large
Distributions of test statisticsbull Value of q1 on data is now the lsquomeasurementrsquobull Distribution of q1 not calculable
But can obtain distribution from pseudo-experimentsbull Generate a large number of pseudo-experiments with a given value of mu
calculate q for each plot distribution
bull From qobs and these distributions can then set limitssimilar to what was shown for Poisson counting example
)ˆ|()1|(ln2~
1
dataLdataLq
q1 for experiments with signal
q1 for experiments with background only
Note analogyto Poisson
counting example
Incorporating systematic uncertaintiesbull What happens if models have uncertainties
ndash Introduction of additional model parameters θ that describe effect of uncertainties
)|()|(
dataLdataL
)~())()(|()|( pbsNPoissondataL iii
xx θ2θ1
Jet Energy ScaleQCD scaleluminosity
Incorporating systematic uncertainties
)~())()(|()|( pbsNPoissondataL iii
xx θ2θ1
Likelihood includes auxiliary measurement terms that constrains the nuisance parameters θ(shape is flat log-normal gamma or Gaussian)
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are
incorporated in test statistic using a profile likelihood ratio
bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments
ˆ0(with a constraint )
lsquolikelihood of best fitrsquo
lsquolikelihood of best fit for a given fixed value of μrsquo
)ˆ|()|(ln2
dataLdataLq
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are
incorporated using a profile likelihood ratio test statistic
bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments
ˆ0(with a constraint )lsquolikelihood of best fitrsquo
lsquolikelihood of best fit for a given fixed value of μrsquo
μ=035 μ=10 μ=18 μ=26
Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models
the data for a given true Higgs mass hypothesis
bull Construct test statistic
bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value
and signal exclusion limitbull Repeat for each assumed mH
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ|(
ln2)(~
H
HH mdataL
mdataLmq
Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV
Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV
Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]
Expected exclusion limitfor background-only hypothesis
Combining Higgs channels (and experiments)bull Procedure define joint likelihood
bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration
bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
)()()( CMSCMSATLASATLASLHC LLL
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)
have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and
delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
lsquollll110rootrsquo lsquogg110rootrsquo
Example ndash joint ATLASCMS Higgs exclusion limit
Wouter Verkerke NIKHEF
Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ0|(ln2~ 0
0
dataLdataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming background onlyrsquo
)ˆˆ|()ˆ|(
ln2~
dataL
dataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming μ signal strengthrsquo
770~1 q
lsquodiscoveryrsquolsquoexclusionrsquo
7734~0 q
0~0 q3452~
1 q
simulateddata with signal+bkg
simulateddata with bkg only
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
Note that lsquopeakrsquo around 160 GeV reflects increased
experimental sensitivity not SM prediction of Higgs mass
Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)
But need to be careful with local p-values
Search is executed for a wide mH mass range
Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1
Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction
Conclusionsbull We are early awaiting more data
Wouter Verkerke NIKHEF
- Statistical aspects of Higgs analyses
- Introduction
- Quantifying discovery and exclusion ndash Frequentist approach
- Frequentist p-values ndash excess over expected bkg
- Frequentist p-values - excess over expected bkg
- Upper limits (one-sided confidence intervals)
- Modified frequentist upper limits
- p-values and limits on non-trivial analysis
- The likelihood ratio test statistic
- Distributions of test statistics
- Incorporating systematic uncertainties
- Incorporating systematic uncertainties (2)
- Dealing with nuisance parameters in the test statistic
- Dealing with nuisance parameters in the test statistic (2)
- Putting it all together for one Higgs channel
- Example ndash 95 Exclusion limit vs mH for HWW
- Combining Higgs channels (and experiments)
- A word on the machinery
- Example ndash joint ATLASCMS Higgs exclusion limit
- Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
- Conclusions
Modified frequentist upper limitsbull Need to be careful about interpretation p(s+b) in terms
of inference on signal onlyndash Since p(s+b) quantifies consistency of signal plus backgroundndash Problem most apparent when observed data
has downward stat fluctations wrt background expectationbull Example Nobs =2
bull Modified approach to protectagainst such inference on sndash Instead of requiring p(s+b)=5
require
ps+b(s=0) = 004
sge0 excluded at gt95 CL s=0
s=5
s=10s=15
51
b
bsS p
pCL
for N=2 exclude sgt34 at 95 CLs for large N effect on limit is small as pb0
p-values and limits on non-trivial analysis
bull Typical Higgs search result is not a simple number counting experiment but looks like this
bull Any type of result can be converted into a single number by constructing a lsquotest statisticrsquo ndash A test statistic compresses all signal-to-background
discrimination power in a single numberndash Most powerful discriminators
are Likelihood Ratios(Neyman Pearson)
)ˆ|()|(ln2
dataLdataLq
- Result is a distribution not a single number
- Models for signal and background have intrinsic uncertainties
The likelihood ratio test statistic
bull Definition μ = signal strength signal strength(SM)ndash Choose eg likelihood with nominal signal strength in numerator (μ=1)
bull Illustration on model with no shape uncertainties
)ˆ|()1|(ln21
dataLdataLq
μ is best fit value of μ^
lsquolikelihood of best fitrsquo
lsquolikelihood assuming nominal signal strengthrsquo
770~1 q 3452~
1 q
On signal-like data q1 is small On background-like data q1 is large
Distributions of test statisticsbull Value of q1 on data is now the lsquomeasurementrsquobull Distribution of q1 not calculable
But can obtain distribution from pseudo-experimentsbull Generate a large number of pseudo-experiments with a given value of mu
calculate q for each plot distribution
bull From qobs and these distributions can then set limitssimilar to what was shown for Poisson counting example
)ˆ|()1|(ln2~
1
dataLdataLq
q1 for experiments with signal
q1 for experiments with background only
Note analogyto Poisson
counting example
Incorporating systematic uncertaintiesbull What happens if models have uncertainties
ndash Introduction of additional model parameters θ that describe effect of uncertainties
)|()|(
dataLdataL
)~())()(|()|( pbsNPoissondataL iii
xx θ2θ1
Jet Energy ScaleQCD scaleluminosity
Incorporating systematic uncertainties
)~())()(|()|( pbsNPoissondataL iii
xx θ2θ1
Likelihood includes auxiliary measurement terms that constrains the nuisance parameters θ(shape is flat log-normal gamma or Gaussian)
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are
incorporated in test statistic using a profile likelihood ratio
bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments
ˆ0(with a constraint )
lsquolikelihood of best fitrsquo
lsquolikelihood of best fit for a given fixed value of μrsquo
)ˆ|()|(ln2
dataLdataLq
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are
incorporated using a profile likelihood ratio test statistic
bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments
ˆ0(with a constraint )lsquolikelihood of best fitrsquo
lsquolikelihood of best fit for a given fixed value of μrsquo
μ=035 μ=10 μ=18 μ=26
Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models
the data for a given true Higgs mass hypothesis
bull Construct test statistic
bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value
and signal exclusion limitbull Repeat for each assumed mH
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ|(
ln2)(~
H
HH mdataL
mdataLmq
Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV
Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV
Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]
Expected exclusion limitfor background-only hypothesis
Combining Higgs channels (and experiments)bull Procedure define joint likelihood
bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration
bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
)()()( CMSCMSATLASATLASLHC LLL
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)
have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and
delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
lsquollll110rootrsquo lsquogg110rootrsquo
Example ndash joint ATLASCMS Higgs exclusion limit
Wouter Verkerke NIKHEF
Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ0|(ln2~ 0
0
dataLdataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming background onlyrsquo
)ˆˆ|()ˆ|(
ln2~
dataL
dataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming μ signal strengthrsquo
770~1 q
lsquodiscoveryrsquolsquoexclusionrsquo
7734~0 q
0~0 q3452~
1 q
simulateddata with signal+bkg
simulateddata with bkg only
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
Note that lsquopeakrsquo around 160 GeV reflects increased
experimental sensitivity not SM prediction of Higgs mass
Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)
But need to be careful with local p-values
Search is executed for a wide mH mass range
Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1
Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction
Conclusionsbull We are early awaiting more data
Wouter Verkerke NIKHEF
- Statistical aspects of Higgs analyses
- Introduction
- Quantifying discovery and exclusion ndash Frequentist approach
- Frequentist p-values ndash excess over expected bkg
- Frequentist p-values - excess over expected bkg
- Upper limits (one-sided confidence intervals)
- Modified frequentist upper limits
- p-values and limits on non-trivial analysis
- The likelihood ratio test statistic
- Distributions of test statistics
- Incorporating systematic uncertainties
- Incorporating systematic uncertainties (2)
- Dealing with nuisance parameters in the test statistic
- Dealing with nuisance parameters in the test statistic (2)
- Putting it all together for one Higgs channel
- Example ndash 95 Exclusion limit vs mH for HWW
- Combining Higgs channels (and experiments)
- A word on the machinery
- Example ndash joint ATLASCMS Higgs exclusion limit
- Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
- Conclusions
p-values and limits on non-trivial analysis
bull Typical Higgs search result is not a simple number counting experiment but looks like this
bull Any type of result can be converted into a single number by constructing a lsquotest statisticrsquo ndash A test statistic compresses all signal-to-background
discrimination power in a single numberndash Most powerful discriminators
are Likelihood Ratios(Neyman Pearson)
)ˆ|()|(ln2
dataLdataLq
- Result is a distribution not a single number
- Models for signal and background have intrinsic uncertainties
The likelihood ratio test statistic
bull Definition μ = signal strength signal strength(SM)ndash Choose eg likelihood with nominal signal strength in numerator (μ=1)
bull Illustration on model with no shape uncertainties
)ˆ|()1|(ln21
dataLdataLq
μ is best fit value of μ^
lsquolikelihood of best fitrsquo
lsquolikelihood assuming nominal signal strengthrsquo
770~1 q 3452~
1 q
On signal-like data q1 is small On background-like data q1 is large
Distributions of test statisticsbull Value of q1 on data is now the lsquomeasurementrsquobull Distribution of q1 not calculable
But can obtain distribution from pseudo-experimentsbull Generate a large number of pseudo-experiments with a given value of mu
calculate q for each plot distribution
bull From qobs and these distributions can then set limitssimilar to what was shown for Poisson counting example
)ˆ|()1|(ln2~
1
dataLdataLq
q1 for experiments with signal
q1 for experiments with background only
Note analogyto Poisson
counting example
Incorporating systematic uncertaintiesbull What happens if models have uncertainties
ndash Introduction of additional model parameters θ that describe effect of uncertainties
)|()|(
dataLdataL
)~())()(|()|( pbsNPoissondataL iii
xx θ2θ1
Jet Energy ScaleQCD scaleluminosity
Incorporating systematic uncertainties
)~())()(|()|( pbsNPoissondataL iii
xx θ2θ1
Likelihood includes auxiliary measurement terms that constrains the nuisance parameters θ(shape is flat log-normal gamma or Gaussian)
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are
incorporated in test statistic using a profile likelihood ratio
bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments
ˆ0(with a constraint )
lsquolikelihood of best fitrsquo
lsquolikelihood of best fit for a given fixed value of μrsquo
)ˆ|()|(ln2
dataLdataLq
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are
incorporated using a profile likelihood ratio test statistic
bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments
ˆ0(with a constraint )lsquolikelihood of best fitrsquo
lsquolikelihood of best fit for a given fixed value of μrsquo
μ=035 μ=10 μ=18 μ=26
Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models
the data for a given true Higgs mass hypothesis
bull Construct test statistic
bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value
and signal exclusion limitbull Repeat for each assumed mH
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ|(
ln2)(~
H
HH mdataL
mdataLmq
Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV
Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV
Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]
Expected exclusion limitfor background-only hypothesis
Combining Higgs channels (and experiments)bull Procedure define joint likelihood
bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration
bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
)()()( CMSCMSATLASATLASLHC LLL
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)
have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and
delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
lsquollll110rootrsquo lsquogg110rootrsquo
Example ndash joint ATLASCMS Higgs exclusion limit
Wouter Verkerke NIKHEF
Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ0|(ln2~ 0
0
dataLdataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming background onlyrsquo
)ˆˆ|()ˆ|(
ln2~
dataL
dataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming μ signal strengthrsquo
770~1 q
lsquodiscoveryrsquolsquoexclusionrsquo
7734~0 q
0~0 q3452~
1 q
simulateddata with signal+bkg
simulateddata with bkg only
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
Note that lsquopeakrsquo around 160 GeV reflects increased
experimental sensitivity not SM prediction of Higgs mass
Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)
But need to be careful with local p-values
Search is executed for a wide mH mass range
Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1
Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction
Conclusionsbull We are early awaiting more data
Wouter Verkerke NIKHEF
- Statistical aspects of Higgs analyses
- Introduction
- Quantifying discovery and exclusion ndash Frequentist approach
- Frequentist p-values ndash excess over expected bkg
- Frequentist p-values - excess over expected bkg
- Upper limits (one-sided confidence intervals)
- Modified frequentist upper limits
- p-values and limits on non-trivial analysis
- The likelihood ratio test statistic
- Distributions of test statistics
- Incorporating systematic uncertainties
- Incorporating systematic uncertainties (2)
- Dealing with nuisance parameters in the test statistic
- Dealing with nuisance parameters in the test statistic (2)
- Putting it all together for one Higgs channel
- Example ndash 95 Exclusion limit vs mH for HWW
- Combining Higgs channels (and experiments)
- A word on the machinery
- Example ndash joint ATLASCMS Higgs exclusion limit
- Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
- Conclusions
The likelihood ratio test statistic
bull Definition μ = signal strength signal strength(SM)ndash Choose eg likelihood with nominal signal strength in numerator (μ=1)
bull Illustration on model with no shape uncertainties
)ˆ|()1|(ln21
dataLdataLq
μ is best fit value of μ^
lsquolikelihood of best fitrsquo
lsquolikelihood assuming nominal signal strengthrsquo
770~1 q 3452~
1 q
On signal-like data q1 is small On background-like data q1 is large
Distributions of test statisticsbull Value of q1 on data is now the lsquomeasurementrsquobull Distribution of q1 not calculable
But can obtain distribution from pseudo-experimentsbull Generate a large number of pseudo-experiments with a given value of mu
calculate q for each plot distribution
bull From qobs and these distributions can then set limitssimilar to what was shown for Poisson counting example
)ˆ|()1|(ln2~
1
dataLdataLq
q1 for experiments with signal
q1 for experiments with background only
Note analogyto Poisson
counting example
Incorporating systematic uncertaintiesbull What happens if models have uncertainties
ndash Introduction of additional model parameters θ that describe effect of uncertainties
)|()|(
dataLdataL
)~())()(|()|( pbsNPoissondataL iii
xx θ2θ1
Jet Energy ScaleQCD scaleluminosity
Incorporating systematic uncertainties
)~())()(|()|( pbsNPoissondataL iii
xx θ2θ1
Likelihood includes auxiliary measurement terms that constrains the nuisance parameters θ(shape is flat log-normal gamma or Gaussian)
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are
incorporated in test statistic using a profile likelihood ratio
bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments
ˆ0(with a constraint )
lsquolikelihood of best fitrsquo
lsquolikelihood of best fit for a given fixed value of μrsquo
)ˆ|()|(ln2
dataLdataLq
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are
incorporated using a profile likelihood ratio test statistic
bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments
ˆ0(with a constraint )lsquolikelihood of best fitrsquo
lsquolikelihood of best fit for a given fixed value of μrsquo
μ=035 μ=10 μ=18 μ=26
Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models
the data for a given true Higgs mass hypothesis
bull Construct test statistic
bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value
and signal exclusion limitbull Repeat for each assumed mH
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ|(
ln2)(~
H
HH mdataL
mdataLmq
Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV
Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV
Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]
Expected exclusion limitfor background-only hypothesis
Combining Higgs channels (and experiments)bull Procedure define joint likelihood
bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration
bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
)()()( CMSCMSATLASATLASLHC LLL
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)
have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and
delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
lsquollll110rootrsquo lsquogg110rootrsquo
Example ndash joint ATLASCMS Higgs exclusion limit
Wouter Verkerke NIKHEF
Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ0|(ln2~ 0
0
dataLdataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming background onlyrsquo
)ˆˆ|()ˆ|(
ln2~
dataL
dataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming μ signal strengthrsquo
770~1 q
lsquodiscoveryrsquolsquoexclusionrsquo
7734~0 q
0~0 q3452~
1 q
simulateddata with signal+bkg
simulateddata with bkg only
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
Note that lsquopeakrsquo around 160 GeV reflects increased
experimental sensitivity not SM prediction of Higgs mass
Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)
But need to be careful with local p-values
Search is executed for a wide mH mass range
Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1
Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction
Conclusionsbull We are early awaiting more data
Wouter Verkerke NIKHEF
- Statistical aspects of Higgs analyses
- Introduction
- Quantifying discovery and exclusion ndash Frequentist approach
- Frequentist p-values ndash excess over expected bkg
- Frequentist p-values - excess over expected bkg
- Upper limits (one-sided confidence intervals)
- Modified frequentist upper limits
- p-values and limits on non-trivial analysis
- The likelihood ratio test statistic
- Distributions of test statistics
- Incorporating systematic uncertainties
- Incorporating systematic uncertainties (2)
- Dealing with nuisance parameters in the test statistic
- Dealing with nuisance parameters in the test statistic (2)
- Putting it all together for one Higgs channel
- Example ndash 95 Exclusion limit vs mH for HWW
- Combining Higgs channels (and experiments)
- A word on the machinery
- Example ndash joint ATLASCMS Higgs exclusion limit
- Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
- Conclusions
Distributions of test statisticsbull Value of q1 on data is now the lsquomeasurementrsquobull Distribution of q1 not calculable
But can obtain distribution from pseudo-experimentsbull Generate a large number of pseudo-experiments with a given value of mu
calculate q for each plot distribution
bull From qobs and these distributions can then set limitssimilar to what was shown for Poisson counting example
)ˆ|()1|(ln2~
1
dataLdataLq
q1 for experiments with signal
q1 for experiments with background only
Note analogyto Poisson
counting example
Incorporating systematic uncertaintiesbull What happens if models have uncertainties
ndash Introduction of additional model parameters θ that describe effect of uncertainties
)|()|(
dataLdataL
)~())()(|()|( pbsNPoissondataL iii
xx θ2θ1
Jet Energy ScaleQCD scaleluminosity
Incorporating systematic uncertainties
)~())()(|()|( pbsNPoissondataL iii
xx θ2θ1
Likelihood includes auxiliary measurement terms that constrains the nuisance parameters θ(shape is flat log-normal gamma or Gaussian)
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are
incorporated in test statistic using a profile likelihood ratio
bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments
ˆ0(with a constraint )
lsquolikelihood of best fitrsquo
lsquolikelihood of best fit for a given fixed value of μrsquo
)ˆ|()|(ln2
dataLdataLq
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are
incorporated using a profile likelihood ratio test statistic
bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments
ˆ0(with a constraint )lsquolikelihood of best fitrsquo
lsquolikelihood of best fit for a given fixed value of μrsquo
μ=035 μ=10 μ=18 μ=26
Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models
the data for a given true Higgs mass hypothesis
bull Construct test statistic
bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value
and signal exclusion limitbull Repeat for each assumed mH
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ|(
ln2)(~
H
HH mdataL
mdataLmq
Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV
Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV
Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]
Expected exclusion limitfor background-only hypothesis
Combining Higgs channels (and experiments)bull Procedure define joint likelihood
bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration
bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
)()()( CMSCMSATLASATLASLHC LLL
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)
have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and
delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
lsquollll110rootrsquo lsquogg110rootrsquo
Example ndash joint ATLASCMS Higgs exclusion limit
Wouter Verkerke NIKHEF
Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ0|(ln2~ 0
0
dataLdataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming background onlyrsquo
)ˆˆ|()ˆ|(
ln2~
dataL
dataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming μ signal strengthrsquo
770~1 q
lsquodiscoveryrsquolsquoexclusionrsquo
7734~0 q
0~0 q3452~
1 q
simulateddata with signal+bkg
simulateddata with bkg only
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
Note that lsquopeakrsquo around 160 GeV reflects increased
experimental sensitivity not SM prediction of Higgs mass
Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)
But need to be careful with local p-values
Search is executed for a wide mH mass range
Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1
Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction
Conclusionsbull We are early awaiting more data
Wouter Verkerke NIKHEF
- Statistical aspects of Higgs analyses
- Introduction
- Quantifying discovery and exclusion ndash Frequentist approach
- Frequentist p-values ndash excess over expected bkg
- Frequentist p-values - excess over expected bkg
- Upper limits (one-sided confidence intervals)
- Modified frequentist upper limits
- p-values and limits on non-trivial analysis
- The likelihood ratio test statistic
- Distributions of test statistics
- Incorporating systematic uncertainties
- Incorporating systematic uncertainties (2)
- Dealing with nuisance parameters in the test statistic
- Dealing with nuisance parameters in the test statistic (2)
- Putting it all together for one Higgs channel
- Example ndash 95 Exclusion limit vs mH for HWW
- Combining Higgs channels (and experiments)
- A word on the machinery
- Example ndash joint ATLASCMS Higgs exclusion limit
- Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
- Conclusions
Incorporating systematic uncertaintiesbull What happens if models have uncertainties
ndash Introduction of additional model parameters θ that describe effect of uncertainties
)|()|(
dataLdataL
)~())()(|()|( pbsNPoissondataL iii
xx θ2θ1
Jet Energy ScaleQCD scaleluminosity
Incorporating systematic uncertainties
)~())()(|()|( pbsNPoissondataL iii
xx θ2θ1
Likelihood includes auxiliary measurement terms that constrains the nuisance parameters θ(shape is flat log-normal gamma or Gaussian)
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are
incorporated in test statistic using a profile likelihood ratio
bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments
ˆ0(with a constraint )
lsquolikelihood of best fitrsquo
lsquolikelihood of best fit for a given fixed value of μrsquo
)ˆ|()|(ln2
dataLdataLq
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are
incorporated using a profile likelihood ratio test statistic
bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments
ˆ0(with a constraint )lsquolikelihood of best fitrsquo
lsquolikelihood of best fit for a given fixed value of μrsquo
μ=035 μ=10 μ=18 μ=26
Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models
the data for a given true Higgs mass hypothesis
bull Construct test statistic
bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value
and signal exclusion limitbull Repeat for each assumed mH
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ|(
ln2)(~
H
HH mdataL
mdataLmq
Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV
Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV
Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]
Expected exclusion limitfor background-only hypothesis
Combining Higgs channels (and experiments)bull Procedure define joint likelihood
bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration
bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
)()()( CMSCMSATLASATLASLHC LLL
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)
have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and
delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
lsquollll110rootrsquo lsquogg110rootrsquo
Example ndash joint ATLASCMS Higgs exclusion limit
Wouter Verkerke NIKHEF
Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ0|(ln2~ 0
0
dataLdataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming background onlyrsquo
)ˆˆ|()ˆ|(
ln2~
dataL
dataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming μ signal strengthrsquo
770~1 q
lsquodiscoveryrsquolsquoexclusionrsquo
7734~0 q
0~0 q3452~
1 q
simulateddata with signal+bkg
simulateddata with bkg only
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
Note that lsquopeakrsquo around 160 GeV reflects increased
experimental sensitivity not SM prediction of Higgs mass
Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)
But need to be careful with local p-values
Search is executed for a wide mH mass range
Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1
Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction
Conclusionsbull We are early awaiting more data
Wouter Verkerke NIKHEF
- Statistical aspects of Higgs analyses
- Introduction
- Quantifying discovery and exclusion ndash Frequentist approach
- Frequentist p-values ndash excess over expected bkg
- Frequentist p-values - excess over expected bkg
- Upper limits (one-sided confidence intervals)
- Modified frequentist upper limits
- p-values and limits on non-trivial analysis
- The likelihood ratio test statistic
- Distributions of test statistics
- Incorporating systematic uncertainties
- Incorporating systematic uncertainties (2)
- Dealing with nuisance parameters in the test statistic
- Dealing with nuisance parameters in the test statistic (2)
- Putting it all together for one Higgs channel
- Example ndash 95 Exclusion limit vs mH for HWW
- Combining Higgs channels (and experiments)
- A word on the machinery
- Example ndash joint ATLASCMS Higgs exclusion limit
- Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
- Conclusions
Incorporating systematic uncertainties
)~())()(|()|( pbsNPoissondataL iii
xx θ2θ1
Likelihood includes auxiliary measurement terms that constrains the nuisance parameters θ(shape is flat log-normal gamma or Gaussian)
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are
incorporated in test statistic using a profile likelihood ratio
bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments
ˆ0(with a constraint )
lsquolikelihood of best fitrsquo
lsquolikelihood of best fit for a given fixed value of μrsquo
)ˆ|()|(ln2
dataLdataLq
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are
incorporated using a profile likelihood ratio test statistic
bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments
ˆ0(with a constraint )lsquolikelihood of best fitrsquo
lsquolikelihood of best fit for a given fixed value of μrsquo
μ=035 μ=10 μ=18 μ=26
Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models
the data for a given true Higgs mass hypothesis
bull Construct test statistic
bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value
and signal exclusion limitbull Repeat for each assumed mH
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ|(
ln2)(~
H
HH mdataL
mdataLmq
Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV
Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV
Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]
Expected exclusion limitfor background-only hypothesis
Combining Higgs channels (and experiments)bull Procedure define joint likelihood
bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration
bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
)()()( CMSCMSATLASATLASLHC LLL
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)
have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and
delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
lsquollll110rootrsquo lsquogg110rootrsquo
Example ndash joint ATLASCMS Higgs exclusion limit
Wouter Verkerke NIKHEF
Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ0|(ln2~ 0
0
dataLdataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming background onlyrsquo
)ˆˆ|()ˆ|(
ln2~
dataL
dataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming μ signal strengthrsquo
770~1 q
lsquodiscoveryrsquolsquoexclusionrsquo
7734~0 q
0~0 q3452~
1 q
simulateddata with signal+bkg
simulateddata with bkg only
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
Note that lsquopeakrsquo around 160 GeV reflects increased
experimental sensitivity not SM prediction of Higgs mass
Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)
But need to be careful with local p-values
Search is executed for a wide mH mass range
Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1
Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction
Conclusionsbull We are early awaiting more data
Wouter Verkerke NIKHEF
- Statistical aspects of Higgs analyses
- Introduction
- Quantifying discovery and exclusion ndash Frequentist approach
- Frequentist p-values ndash excess over expected bkg
- Frequentist p-values - excess over expected bkg
- Upper limits (one-sided confidence intervals)
- Modified frequentist upper limits
- p-values and limits on non-trivial analysis
- The likelihood ratio test statistic
- Distributions of test statistics
- Incorporating systematic uncertainties
- Incorporating systematic uncertainties (2)
- Dealing with nuisance parameters in the test statistic
- Dealing with nuisance parameters in the test statistic (2)
- Putting it all together for one Higgs channel
- Example ndash 95 Exclusion limit vs mH for HWW
- Combining Higgs channels (and experiments)
- A word on the machinery
- Example ndash joint ATLASCMS Higgs exclusion limit
- Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
- Conclusions
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are
incorporated in test statistic using a profile likelihood ratio
bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments
ˆ0(with a constraint )
lsquolikelihood of best fitrsquo
lsquolikelihood of best fit for a given fixed value of μrsquo
)ˆ|()|(ln2
dataLdataLq
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are
incorporated using a profile likelihood ratio test statistic
bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments
ˆ0(with a constraint )lsquolikelihood of best fitrsquo
lsquolikelihood of best fit for a given fixed value of μrsquo
μ=035 μ=10 μ=18 μ=26
Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models
the data for a given true Higgs mass hypothesis
bull Construct test statistic
bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value
and signal exclusion limitbull Repeat for each assumed mH
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ|(
ln2)(~
H
HH mdataL
mdataLmq
Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV
Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV
Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]
Expected exclusion limitfor background-only hypothesis
Combining Higgs channels (and experiments)bull Procedure define joint likelihood
bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration
bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
)()()( CMSCMSATLASATLASLHC LLL
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)
have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and
delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
lsquollll110rootrsquo lsquogg110rootrsquo
Example ndash joint ATLASCMS Higgs exclusion limit
Wouter Verkerke NIKHEF
Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ0|(ln2~ 0
0
dataLdataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming background onlyrsquo
)ˆˆ|()ˆ|(
ln2~
dataL
dataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming μ signal strengthrsquo
770~1 q
lsquodiscoveryrsquolsquoexclusionrsquo
7734~0 q
0~0 q3452~
1 q
simulateddata with signal+bkg
simulateddata with bkg only
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
Note that lsquopeakrsquo around 160 GeV reflects increased
experimental sensitivity not SM prediction of Higgs mass
Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)
But need to be careful with local p-values
Search is executed for a wide mH mass range
Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1
Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction
Conclusionsbull We are early awaiting more data
Wouter Verkerke NIKHEF
- Statistical aspects of Higgs analyses
- Introduction
- Quantifying discovery and exclusion ndash Frequentist approach
- Frequentist p-values ndash excess over expected bkg
- Frequentist p-values - excess over expected bkg
- Upper limits (one-sided confidence intervals)
- Modified frequentist upper limits
- p-values and limits on non-trivial analysis
- The likelihood ratio test statistic
- Distributions of test statistics
- Incorporating systematic uncertainties
- Incorporating systematic uncertainties (2)
- Dealing with nuisance parameters in the test statistic
- Dealing with nuisance parameters in the test statistic (2)
- Putting it all together for one Higgs channel
- Example ndash 95 Exclusion limit vs mH for HWW
- Combining Higgs channels (and experiments)
- A word on the machinery
- Example ndash joint ATLASCMS Higgs exclusion limit
- Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
- Conclusions
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
Dealing with nuisance parameters in the test statisticbull Uncertainty quantified by nuisance parameters are
incorporated using a profile likelihood ratio test statistic
bull Interpretation of observed value of qμ in terms of p-value again based on expected distribution obtainedfrom pseudo-experiments
ˆ0(with a constraint )lsquolikelihood of best fitrsquo
lsquolikelihood of best fit for a given fixed value of μrsquo
μ=035 μ=10 μ=18 μ=26
Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models
the data for a given true Higgs mass hypothesis
bull Construct test statistic
bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value
and signal exclusion limitbull Repeat for each assumed mH
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ|(
ln2)(~
H
HH mdataL
mdataLmq
Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV
Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV
Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]
Expected exclusion limitfor background-only hypothesis
Combining Higgs channels (and experiments)bull Procedure define joint likelihood
bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration
bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
)()()( CMSCMSATLASATLASLHC LLL
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)
have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and
delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
lsquollll110rootrsquo lsquogg110rootrsquo
Example ndash joint ATLASCMS Higgs exclusion limit
Wouter Verkerke NIKHEF
Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ0|(ln2~ 0
0
dataLdataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming background onlyrsquo
)ˆˆ|()ˆ|(
ln2~
dataL
dataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming μ signal strengthrsquo
770~1 q
lsquodiscoveryrsquolsquoexclusionrsquo
7734~0 q
0~0 q3452~
1 q
simulateddata with signal+bkg
simulateddata with bkg only
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
Note that lsquopeakrsquo around 160 GeV reflects increased
experimental sensitivity not SM prediction of Higgs mass
Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)
But need to be careful with local p-values
Search is executed for a wide mH mass range
Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1
Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction
Conclusionsbull We are early awaiting more data
Wouter Verkerke NIKHEF
- Statistical aspects of Higgs analyses
- Introduction
- Quantifying discovery and exclusion ndash Frequentist approach
- Frequentist p-values ndash excess over expected bkg
- Frequentist p-values - excess over expected bkg
- Upper limits (one-sided confidence intervals)
- Modified frequentist upper limits
- p-values and limits on non-trivial analysis
- The likelihood ratio test statistic
- Distributions of test statistics
- Incorporating systematic uncertainties
- Incorporating systematic uncertainties (2)
- Dealing with nuisance parameters in the test statistic
- Dealing with nuisance parameters in the test statistic (2)
- Putting it all together for one Higgs channel
- Example ndash 95 Exclusion limit vs mH for HWW
- Combining Higgs channels (and experiments)
- A word on the machinery
- Example ndash joint ATLASCMS Higgs exclusion limit
- Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
- Conclusions
Putting it all together for one Higgs channelbull Result from data D(x)bull PDF F(x|mHμθ) that models
the data for a given true Higgs mass hypothesis
bull Construct test statistic
bull Obtain expected distributions of qμ for various μndash Determine lsquodiscoveryrsquo p-value
and signal exclusion limitbull Repeat for each assumed mH
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ|(
ln2)(~
H
HH mdataL
mdataLmq
Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV
Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV
Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]
Expected exclusion limitfor background-only hypothesis
Combining Higgs channels (and experiments)bull Procedure define joint likelihood
bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration
bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
)()()( CMSCMSATLASATLASLHC LLL
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)
have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and
delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
lsquollll110rootrsquo lsquogg110rootrsquo
Example ndash joint ATLASCMS Higgs exclusion limit
Wouter Verkerke NIKHEF
Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ0|(ln2~ 0
0
dataLdataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming background onlyrsquo
)ˆˆ|()ˆ|(
ln2~
dataL
dataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming μ signal strengthrsquo
770~1 q
lsquodiscoveryrsquolsquoexclusionrsquo
7734~0 q
0~0 q3452~
1 q
simulateddata with signal+bkg
simulateddata with bkg only
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
Note that lsquopeakrsquo around 160 GeV reflects increased
experimental sensitivity not SM prediction of Higgs mass
Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)
But need to be careful with local p-values
Search is executed for a wide mH mass range
Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1
Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction
Conclusionsbull We are early awaiting more data
Wouter Verkerke NIKHEF
- Statistical aspects of Higgs analyses
- Introduction
- Quantifying discovery and exclusion ndash Frequentist approach
- Frequentist p-values ndash excess over expected bkg
- Frequentist p-values - excess over expected bkg
- Upper limits (one-sided confidence intervals)
- Modified frequentist upper limits
- p-values and limits on non-trivial analysis
- The likelihood ratio test statistic
- Distributions of test statistics
- Incorporating systematic uncertainties
- Incorporating systematic uncertainties (2)
- Dealing with nuisance parameters in the test statistic
- Dealing with nuisance parameters in the test statistic (2)
- Putting it all together for one Higgs channel
- Example ndash 95 Exclusion limit vs mH for HWW
- Combining Higgs channels (and experiments)
- A word on the machinery
- Example ndash joint ATLASCMS Higgs exclusion limit
- Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
- Conclusions
Example ndash 95 Exclusion limit vs mH for HWWExample point asymp3 x SM HWW cross-section excluded at mH=125 GeV
Example point asymp05 x SM HWW cross-section excluded at mH=165 GeV
Higgs with 10x SM cross-section excluded at 95 CL for mH in range [150~187]
Expected exclusion limitfor background-only hypothesis
Combining Higgs channels (and experiments)bull Procedure define joint likelihood
bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration
bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
)()()( CMSCMSATLASATLASLHC LLL
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)
have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and
delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
lsquollll110rootrsquo lsquogg110rootrsquo
Example ndash joint ATLASCMS Higgs exclusion limit
Wouter Verkerke NIKHEF
Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ0|(ln2~ 0
0
dataLdataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming background onlyrsquo
)ˆˆ|()ˆ|(
ln2~
dataL
dataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming μ signal strengthrsquo
770~1 q
lsquodiscoveryrsquolsquoexclusionrsquo
7734~0 q
0~0 q3452~
1 q
simulateddata with signal+bkg
simulateddata with bkg only
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
Note that lsquopeakrsquo around 160 GeV reflects increased
experimental sensitivity not SM prediction of Higgs mass
Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)
But need to be careful with local p-values
Search is executed for a wide mH mass range
Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1
Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction
Conclusionsbull We are early awaiting more data
Wouter Verkerke NIKHEF
- Statistical aspects of Higgs analyses
- Introduction
- Quantifying discovery and exclusion ndash Frequentist approach
- Frequentist p-values ndash excess over expected bkg
- Frequentist p-values - excess over expected bkg
- Upper limits (one-sided confidence intervals)
- Modified frequentist upper limits
- p-values and limits on non-trivial analysis
- The likelihood ratio test statistic
- Distributions of test statistics
- Incorporating systematic uncertainties
- Incorporating systematic uncertainties (2)
- Dealing with nuisance parameters in the test statistic
- Dealing with nuisance parameters in the test statistic (2)
- Putting it all together for one Higgs channel
- Example ndash 95 Exclusion limit vs mH for HWW
- Combining Higgs channels (and experiments)
- A word on the machinery
- Example ndash joint ATLASCMS Higgs exclusion limit
- Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
- Conclusions
Combining Higgs channels (and experiments)bull Procedure define joint likelihood
bull Correlations between θWWθγγ etc and between θATLASθCMS requires careful consideration
bull The construction profile likelihood ratio test statistic from joint likelihood and proceed as usual
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
)()()( CMSCMSATLASATLASLHC LLL
)ˆˆ|()ˆ|(
ln2~
dataLdataL
q
A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)
have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and
delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
lsquollll110rootrsquo lsquogg110rootrsquo
Example ndash joint ATLASCMS Higgs exclusion limit
Wouter Verkerke NIKHEF
Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ0|(ln2~ 0
0
dataLdataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming background onlyrsquo
)ˆˆ|()ˆ|(
ln2~
dataL
dataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming μ signal strengthrsquo
770~1 q
lsquodiscoveryrsquolsquoexclusionrsquo
7734~0 q
0~0 q3452~
1 q
simulateddata with signal+bkg
simulateddata with bkg only
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
Note that lsquopeakrsquo around 160 GeV reflects increased
experimental sensitivity not SM prediction of Higgs mass
Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)
But need to be careful with local p-values
Search is executed for a wide mH mass range
Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1
Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction
Conclusionsbull We are early awaiting more data
Wouter Verkerke NIKHEF
- Statistical aspects of Higgs analyses
- Introduction
- Quantifying discovery and exclusion ndash Frequentist approach
- Frequentist p-values ndash excess over expected bkg
- Frequentist p-values - excess over expected bkg
- Upper limits (one-sided confidence intervals)
- Modified frequentist upper limits
- p-values and limits on non-trivial analysis
- The likelihood ratio test statistic
- Distributions of test statistics
- Incorporating systematic uncertainties
- Incorporating systematic uncertainties (2)
- Dealing with nuisance parameters in the test statistic
- Dealing with nuisance parameters in the test statistic (2)
- Putting it all together for one Higgs channel
- Example ndash 95 Exclusion limit vs mH for HWW
- Combining Higgs channels (and experiments)
- A word on the machinery
- Example ndash joint ATLASCMS Higgs exclusion limit
- Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
- Conclusions
A word on the machinerybull Common tools (RooFitRooStats - all available in ROOT)
have been developed in past 23 years to facilitate these combinationsndash Analytical description of likelihood of each component stored and
delivered in a uniform language (lsquoRooFit workspacesrsquo)ndash Construction of joint likelihood technically straightforward
Wouter Verkerke NIKHEF
)()()()( HZZZZHWWWWHcomb LLLL
lsquollll110rootrsquo lsquogg110rootrsquo
Example ndash joint ATLASCMS Higgs exclusion limit
Wouter Verkerke NIKHEF
Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ0|(ln2~ 0
0
dataLdataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming background onlyrsquo
)ˆˆ|()ˆ|(
ln2~
dataL
dataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming μ signal strengthrsquo
770~1 q
lsquodiscoveryrsquolsquoexclusionrsquo
7734~0 q
0~0 q3452~
1 q
simulateddata with signal+bkg
simulateddata with bkg only
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
Note that lsquopeakrsquo around 160 GeV reflects increased
experimental sensitivity not SM prediction of Higgs mass
Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)
But need to be careful with local p-values
Search is executed for a wide mH mass range
Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1
Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction
Conclusionsbull We are early awaiting more data
Wouter Verkerke NIKHEF
- Statistical aspects of Higgs analyses
- Introduction
- Quantifying discovery and exclusion ndash Frequentist approach
- Frequentist p-values ndash excess over expected bkg
- Frequentist p-values - excess over expected bkg
- Upper limits (one-sided confidence intervals)
- Modified frequentist upper limits
- p-values and limits on non-trivial analysis
- The likelihood ratio test statistic
- Distributions of test statistics
- Incorporating systematic uncertainties
- Incorporating systematic uncertainties (2)
- Dealing with nuisance parameters in the test statistic
- Dealing with nuisance parameters in the test statistic (2)
- Putting it all together for one Higgs channel
- Example ndash 95 Exclusion limit vs mH for HWW
- Combining Higgs channels (and experiments)
- A word on the machinery
- Example ndash joint ATLASCMS Higgs exclusion limit
- Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
- Conclusions
Example ndash joint ATLASCMS Higgs exclusion limit
Wouter Verkerke NIKHEF
Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ0|(ln2~ 0
0
dataLdataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming background onlyrsquo
)ˆˆ|()ˆ|(
ln2~
dataL
dataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming μ signal strengthrsquo
770~1 q
lsquodiscoveryrsquolsquoexclusionrsquo
7734~0 q
0~0 q3452~
1 q
simulateddata with signal+bkg
simulateddata with bkg only
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
Note that lsquopeakrsquo around 160 GeV reflects increased
experimental sensitivity not SM prediction of Higgs mass
Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)
But need to be careful with local p-values
Search is executed for a wide mH mass range
Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1
Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction
Conclusionsbull We are early awaiting more data
Wouter Verkerke NIKHEF
- Statistical aspects of Higgs analyses
- Introduction
- Quantifying discovery and exclusion ndash Frequentist approach
- Frequentist p-values ndash excess over expected bkg
- Frequentist p-values - excess over expected bkg
- Upper limits (one-sided confidence intervals)
- Modified frequentist upper limits
- p-values and limits on non-trivial analysis
- The likelihood ratio test statistic
- Distributions of test statistics
- Incorporating systematic uncertainties
- Incorporating systematic uncertainties (2)
- Dealing with nuisance parameters in the test statistic
- Dealing with nuisance parameters in the test statistic (2)
- Putting it all together for one Higgs channel
- Example ndash 95 Exclusion limit vs mH for HWW
- Combining Higgs channels (and experiments)
- A word on the machinery
- Example ndash joint ATLASCMS Higgs exclusion limit
- Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
- Conclusions
Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
Wouter Verkerke NIKHEF
)ˆˆ|()ˆ0|(ln2~ 0
0
dataLdataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming background onlyrsquo
)ˆˆ|()ˆ|(
ln2~
dataL
dataLq
lsquolikelihood of best fitrsquo
lsquolikelihood assuming μ signal strengthrsquo
770~1 q
lsquodiscoveryrsquolsquoexclusionrsquo
7734~0 q
0~0 q3452~
1 q
simulateddata with signal+bkg
simulateddata with bkg only
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
Note that lsquopeakrsquo around 160 GeV reflects increased
experimental sensitivity not SM prediction of Higgs mass
Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)
But need to be careful with local p-values
Search is executed for a wide mH mass range
Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1
Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction
Conclusionsbull We are early awaiting more data
Wouter Verkerke NIKHEF
- Statistical aspects of Higgs analyses
- Introduction
- Quantifying discovery and exclusion ndash Frequentist approach
- Frequentist p-values ndash excess over expected bkg
- Frequentist p-values - excess over expected bkg
- Upper limits (one-sided confidence intervals)
- Modified frequentist upper limits
- p-values and limits on non-trivial analysis
- The likelihood ratio test statistic
- Distributions of test statistics
- Incorporating systematic uncertainties
- Incorporating systematic uncertainties (2)
- Dealing with nuisance parameters in the test statistic
- Dealing with nuisance parameters in the test statistic (2)
- Putting it all together for one Higgs channel
- Example ndash 95 Exclusion limit vs mH for HWW
- Combining Higgs channels (and experiments)
- A word on the machinery
- Example ndash joint ATLASCMS Higgs exclusion limit
- Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
- Conclusions
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
Note that lsquopeakrsquo around 160 GeV reflects increased
experimental sensitivity not SM prediction of Higgs mass
Expected p-value for backgroundhypothesis of a data samplecontaining SM Higgs boson
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)
But need to be careful with local p-values
Search is executed for a wide mH mass range
Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1
Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction
Conclusionsbull We are early awaiting more data
Wouter Verkerke NIKHEF
- Statistical aspects of Higgs analyses
- Introduction
- Quantifying discovery and exclusion ndash Frequentist approach
- Frequentist p-values ndash excess over expected bkg
- Frequentist p-values - excess over expected bkg
- Upper limits (one-sided confidence intervals)
- Modified frequentist upper limits
- p-values and limits on non-trivial analysis
- The likelihood ratio test statistic
- Distributions of test statistics
- Incorporating systematic uncertainties
- Incorporating systematic uncertainties (2)
- Dealing with nuisance parameters in the test statistic
- Dealing with nuisance parameters in the test statistic (2)
- Putting it all together for one Higgs channel
- Example ndash 95 Exclusion limit vs mH for HWW
- Combining Higgs channels (and experiments)
- A word on the machinery
- Example ndash joint ATLASCMS Higgs exclusion limit
- Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
- Conclusions
Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)Example point at mHasymp150 GeV probability to obtain observed event count (or larger) here from background only is ~00015 (~3σ)
But need to be careful with local p-values
Search is executed for a wide mH mass range
Odds to find a local p-value of eg 1 anywhere in mass range (the lsquoglobal p-valuersquo) is larger than 1
Can a estimate trial factor (global plocal p)and obtain estimate of global significanceby using trial factor as correction
Conclusionsbull We are early awaiting more data
Wouter Verkerke NIKHEF
- Statistical aspects of Higgs analyses
- Introduction
- Quantifying discovery and exclusion ndash Frequentist approach
- Frequentist p-values ndash excess over expected bkg
- Frequentist p-values - excess over expected bkg
- Upper limits (one-sided confidence intervals)
- Modified frequentist upper limits
- p-values and limits on non-trivial analysis
- The likelihood ratio test statistic
- Distributions of test statistics
- Incorporating systematic uncertainties
- Incorporating systematic uncertainties (2)
- Dealing with nuisance parameters in the test statistic
- Dealing with nuisance parameters in the test statistic (2)
- Putting it all together for one Higgs channel
- Example ndash 95 Exclusion limit vs mH for HWW
- Combining Higgs channels (and experiments)
- A word on the machinery
- Example ndash joint ATLASCMS Higgs exclusion limit
- Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
- Conclusions
Conclusionsbull We are early awaiting more data
Wouter Verkerke NIKHEF
- Statistical aspects of Higgs analyses
- Introduction
- Quantifying discovery and exclusion ndash Frequentist approach
- Frequentist p-values ndash excess over expected bkg
- Frequentist p-values - excess over expected bkg
- Upper limits (one-sided confidence intervals)
- Modified frequentist upper limits
- p-values and limits on non-trivial analysis
- The likelihood ratio test statistic
- Distributions of test statistics
- Incorporating systematic uncertainties
- Incorporating systematic uncertainties (2)
- Dealing with nuisance parameters in the test statistic
- Dealing with nuisance parameters in the test statistic (2)
- Putting it all together for one Higgs channel
- Example ndash 95 Exclusion limit vs mH for HWW
- Combining Higgs channels (and experiments)
- A word on the machinery
- Example ndash joint ATLASCMS Higgs exclusion limit
- Switching from lsquoexclusionrsquo to lsquodiscoveryrsquo formulation
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo)
- Comb p-value of background-only hypothesis (lsquodiscoveryrsquo) (2)
- Conclusions
top related