Course Schedule (cont) two Principles of Testing Principles of Testing Principles of Assessment Principles of Assessment Conducting a PPC Conducting a.

Course Schedule Course Schedule (con’t)(con’t)

twotwo

Principles of TestingPrinciples of Testing Principles of AssessmentPrinciples of Assessment Conducting a PPCConducting a PPC

PPC TolerancesPPC Tolerances

Course ScheduleCourse Scheduletwotwo

Outline Outline General PrinciplesCommon Rating ErrorsSelf-study handout

Principles of TestingPrinciples of Testing

Outline General PrinciplesCommon Rating ErrorsSelf-study handout

General PrinciplesGeneral Principles A good test should be:A good test should be:

1) Valid2) Reliable3) Comprehensive4) Discriminating5) Objective

Validity

Does test measure what it is Does test measure what it is supposedsupposed

to measure?to measure?1) make the test realistic2) make the test

complete3) apply standards

carefully4) no test of items

outside std

ValidityValidity

Which is the Which is the mostmost valid? valid?1) describe action in

briefing2) simulate action in

aircraft3) do action in aircraft4) do action in simulator

Validity ExampleValidity Example

Reliability

How accurately does it How accurately does it measuremeasure

what it measures?what it measures?Must be consistent:1) Avoid extremes

- distractions- aircraft- weather

2) Plan compensations (eg. V1)- beforehand- carefully

ReliabilityReliability

Which test is Which test is mostmost reliable? reliable?

1) PPC in marginal VFR weather

2) PPC in good weather, light winds

3) PPC in good weather, strong winds

4) PPC in IFR weather near minima

Reliability ExampleReliability Example

Comprehensive

The test must include all theThe test must include all therelevant sequencesrelevant sequences

- important "need to know" info

- do not focus on details- vary the program from test to

test- change approaches,

sequences, emergencies (where

discretion)

ComprehensivenessComprehensiveness

Which test is Which test is mostmost comprehensive?comprehensive?

1) departure, steep turns, hold, NDB, ILS

2) dep, hold, ILS, ILS, VOR3) dep, hold, ILS, VOR4) dep, steep turns, LOC (BC), PAR

Comprehensiveness ExampleComprehensiveness Example

Discriminating

Detects different levels ofDetects different levels ofachievementachievement

- separate qualified from unqualified

- this is main goal of test- rank those who are qualified- use overall performance for

borderline cases

DiscriminationDiscrimination

Which test is Which test is mostmost discriminating?discriminating?

1) no oral test before flight2) oral test before flight with

general briefing3) oral test and specific test

sequences 4) oral test before and during

flight

Discrimination ExampleDiscrimination Example

Objective

Test outcome must not be Test outcome must not be affected byaffected by

personal examinerpersonal examiner

1) do not impose your own standards2) adhere to standards in FM, SOP, AOM and DCP Manual3) do not let rating errors affect your

assessment4) carefully record events as they occur

ObjectivityObjectivity

Which written assessment is Which written assessment is most objective?most objective?

1) poor judgment during emergency2) excessive swing during engine fail3) altitude 100ft low inbd to beacon4) airspeed unstable on final

Objectivity ExampleObjectivity Example

Outline General PrinciplesCommon Rating

ErrorsSelf-study handout

1) Error of Standards2) Error of Central Tendency3) Error of Halo4) Error of Logic5) Error of Familiarity6) Error of Narrow Criterion

Common Rating ErrorsCommon Rating Errors

Error of centraltendancy

Rate of performance at centre of scale

Hesitant to give excellent or poor ratings

Affects all assessments by this observer

Most common with inexperiencedobservers

Error of Central TendencyError of Central Tendency

Error ofStandards

Use personal standards to rate others

If good at it - rate harderIf poor at it - rate easierAffects all assessments by this

observer

Error of StandardsError of Standards

Error of Halo

Allow general impression to affect rating

Can be favourable or unfavourableAffects only certain persons rated Can include stereotypes

Error of haloError of halo

Error of Logic

When several similar traitsTraits are related in mind of

observerCould relate engine fail at V1 to SE

landingObserver generally not aware of

Error of LogicError of Logic

Error of Familiarity

Error of FamiliarityError of Familiarity

Grading objectivity weakenedGrading objectivity weakened Accustomed to weaknesses and Accustomed to weaknesses and

eccentricitieseccentricities Occurs over long period of timeOccurs over long period of time Stop and regain perspectiveStop and regain perspective

Error of Narrow Criterion

Error of Narrow CriterionError of Narrow Criterion

Related to assigning ratingRelated to assigning rating Often occurs in assessing two-crew or Often occurs in assessing two-crew or

multi-crew assessmentsmulti-crew assessments One who does the best of perhaps a One who does the best of perhaps a

marginal performance obtains a high ratingmarginal performance obtains a high rating

Course ScheduleCourse Scheduleday 2day 2

Outline Outline PPC general guidelinesAssessment formatOverall assessment

Principles of AssessmentPrinciples of Assessment

Outline PPC general

guidelinesAssessment formatOverall assessment

1) Take note soon after event2) Terminate test if failure3) Allowance for turbulence4) No advice, criticisms, training

or demonstrations

5) Create realistic scenarios6) Do not simulate multiple,

unrelated failures

PPC general guidelinesPPC general guidelines

7) PPC - Use Designated Check Pilot Manual

8) Simulated weather at or below minimums

9) Need a "U" to fail10) Take weather info into

account in scenarios11) PNF responsible if gross

deviations, violations

PPC general guidelines (con’t)PPC general guidelines (con’t)

According to the following According to the following publications:publications:

• DCP ManualPPC - AFM

- AOM, Pilot Operating Handbook

- Operations Manual- SOP Manual

IFT - Instrument Procedures Manual- Approach Procedures (Jepp)- Company Procedures

Performance AssessmentPerformance Assessment

Outline PPC general guidelinesAssessment formatOverall assessment

S exercise flown within acceptable tolerances

and procedures correct and effective

SB a brief excursion from the criteria ora deviation from standard practices.Corrected by acknowledging the

deviationduring de-briefing; or regaining the

criteriaduring flight

U gross excursions from acceptable tolerances or a violation of airspace or clearance or a fault in procedure that could create an unsafe or ineffective situation

PPC Assessment FormatPPC Assessment Format

S - Minor errors only

- Airspeed and altitude control acceptable

- Handling and knowledge are acceptable and safe considering the experience of the candidate

U - Endanger aircraft, passenger, crew;

- It results in a crash;- Multiple errors made in one exercise;- Violates ATC clearance or an altitude;- Major deviation from standard

procedures or practices or safety was jeopardized;

- Candidate required continued prompting or help from other candidate;

- It exceeds aircraft limitations; or- Candidates demonstrates

unsatisfactory knowledge of systems, equipment, or procedures

SB - Aircraft handling and knowledge are safe but of a lower standard that can be corrected by briefing;

- Brief excursion from tolerances but corrected promptly;

- Deviated from SOP but did not create a more hazardous situation and is clarified by candidate

- Candidate experienced some difficulty or required slight prompting to accomplish the task

Outline PPC general guidelinesAssessment formatOverall assessment

• is it safe and effective?• are criteria broken?• are any fail points done?• is crew concept used?• are crew in control of situation?• are procedues known and executed?• are limitations adhered to?• does an error create an unsafe situation?• are errors recognized and corrected?• would you send your family with this

pilot?

Overall PerformanceOverall Performance

PPC/IFT TolerancesPPC/IFT Tolerances

Outline Outline Flight test requirements

- PPCPre-flight preparation/briefingFlight test scenariosSafe checking practicesPost-flight briefingWritten comments

Conducting a PPCConducting a PPC

Outline Outline Flight test requirementsPPCPre-flight preparation/briefingFlight test scenariosSafe checking practicesPost-flight briefingWritten comments

PRE-FLIGHTPRE-FLIGHT- Flt planning - weather, fuel, NOTAMs,

W&B- IFR x-country exercise for initials- Cockpit checks and ident of NAVAIDS- Read back and interpretation of an

ATC clearance- Take-off and passenger briefings

DEPARTURE AND ENROUTEDEPARTURE AND ENROUTE- Transition to instruments- Climb and level flight- Use of NAVAIDS and adherence to

ATC clearance

PPC RequirementsPPC Requirements

AIRWORK - Applicant Should AIRWORK - Applicant Should Perform:Perform:

• Medium and steep turns• Hold• Tracking on VOR and NDB• Failure of AI and DI

- timed turns, S&L- climbs and descents- optional for renewals

PPC Requirements (con’t)PPC Requirements (con’t)

TERMINAL PROCEDURES• Transition to approach should be

flown• At least 2 approaches to minima

- ILS and NPA required for initials

- different approaches if possible

- NDB recommended• Missed approach• Circling (optional)• Landing from an instrument

approachEMERGENCIES

• One asymmetric approach should be done

• No multiple unrelated failures

PPC Requirements (con’t)PPC Requirements (con’t)

Typical Large Type Aircraft• 2 T/O's• 2 landings (one

asymmetric)• asymmetric instrument

approach• engine fail after T/O• RTO (mandatory in SIM)• engine fail on rejected

ldg/MAP• circling if company

limits<1000/3refer to DCP Manual Chap 7

Schedule I

PPC Additional RequirementsPPC Additional Requirements

Typical Small Type Aircraft• 2 T/O's• 2 landings (one assymetric)• steep turns• approach to stall - ldg and

T/O• RTO (mandatory in SIM)

refer to DCP Manual Chap7Schedule II

PPC Additional RequirementsPPC Additional Requirements

- PPCPre-flight

preparation/briefingFlight test scenariosSafe checking practicesPost-flight briefingWritten comments

• Before meeting candidate- check weather- applicant's info- plan IFR x-country exercise- plan oral questions

• After meeting candidate- ensure admission req'ts met- ensure aircraft is ready for IFR

flight- check accuracy of log, flt plan,

W&B- test knowledge of weather and

NOTAMs- test IFR knowledge and aircraft limitations

Pre-Flight PreparationPre-Flight Preparation

• Valuable part of any flight test• Can be used to identify changes• Can provide feedback on weak

training areas• Ask "why" questions• Cover only "need-to-know" material• Vary questions from test to test

Oral QuestionsOral Questions

"Need to know" a/c speeds if not "Need to know" a/c speeds if not familiar with type:familiar with type:

Vref VrHold VmcaEngine out approach

• Use these briefed speeds to assess speed control

Aircraft LimitationsAircraft Limitations

- Radio, 2 nav radios, aircraft instruments

- C of A, C of R,- Journey log book- Compass swing (12 months)- Static pressure check (24 months)- Altimeter certified (24 months)- Weight and Balance report- Radio station license

Aircraft IFR RequirementsAircraft IFR Requirements( general aviation requirements)

• private location, calm atmosphere• entire flight is IMC at minimum• clear who is PIC• description of major test

components• responsibility for radios and clncs• cockpit checks to be done per SOP• crew responsibilities and

coordination• emergencies - use of

"SIMULATED"• how engine failure to be simulated• "FIELD IN SIGHT" or overshoot• candidates QUESTIONS

Pre-Flight BriefingPre-Flight Briefing

- PPCPre-flight preparation/briefingFlight test scenarios -

ExerciseSafe checking practicesPost-flight briefingWritten comments

• Make efficient use of time• Create realistic scenarios• Liaise with ATC early• No overloading• Minimize re-positioning in Sim• Use proper ATC terminology

Flight Test ScenariosFlight Test Scenarios

• Check Pilot assuming crew duties• RTO• Engine fail after V1

• Rejected landing with engine fail• Engine out landing• Approach to stall

Safe Checking PracticesSafe Checking Practices

• Advise points in order• Include location, atmosphere, tone• Quiet location no distraction

De-Brief After Flight TestDe-Brief After Flight Test

Post-Flight BriefingPost-Flight BriefingPassPass• advise applicant of outcome• establish a positive atmosphere• highlight above-average performance• de-brief as per written notes• identity major (possibly minor) errors

FailFail• advise applicant of outcome• establish a positive atmosphere• highlight above average performance• explain reasons for

failure• de-brief other major errors• advise regarding training and re-test

• Should be factual• Clear and concise• Must support assessment• Required for "U" and "SB"• No sermons or teaching• Relate assessments to PPC

tolerances

Written CommentsWritten Comments

Outline Outline StatisticsDCP ManualPPC fail pointsDCP Manual tolerances: 6.8.8

Outline Outline StatisticsACP ManualIFT fail pointsPPC fail pointsACP Manual tolerances

PPC/IFT TolerancesPPC/IFT Tolerances

IFT Failure ItemsIFT Failure Items

RENEWALRENEWAL

NDB 28%HOLD 23%ILS 13%TRANSITION 8%LOC 5%VOR 5%DEPARTURE 5%

INITIALINITIAL

NDB 29%HOLD 26%ILS 26%DEPARTURE 7%TRANSITION 6%FLT. PLANNING 6%HANDLING 4%

Outline Outline StatisticsDCP ManualPPC fail pointsDCP Manual tolerances

from DCP Manual 6.8.8

-assigned headings within 10 degrees-assigned tracks and bearings within 10 degrees-altitude within 100 ft except at MDA when accurate

altitude control is required-airspeed within 10 knots for holding, approach, and

missed approach;and-not more than half scale deflection, as appropriate to

aircraft type, of course deviation indicators during instrument approaches

-assumes no unusual circumstances; wx,turbulence, malfunction(s), and type of approach may modify tolerances

DCP CriteriaDCP Criteria

• Outside DCP Manual, AFM, IFR Procedures• No check of flt instruments before flight• No check or incorrect use of icing systems• No check of electrical charging system• Wrong altimeter setting• Vital actions checklist item missed• Non-compliance with ATC clearance• No communication with ATC when required• No ident or monitor of NAVAIDs• Comm fail procedures• Inability to establish cleared holding pattern

fail Pointsfail Points

• Loss of directional control• Wrong corrective action for swing• Exceeding airframe or engine

limitations• Rotation before Vr• Failure to comply with departure

clearance• Vital action check missed

Tolerance Example 1Tolerance Example 1

Take-Take-offoff

• Loss of separation due to failure to follow clearance

• No prompt corrective action when DCP criteria exceeded

• Unable to complete a successful landing

• Exceeding airframe or engine limitations

Intstrument Intstrument ApproachesApproaches

Systems MalfunctionsSystems Malfunctions • Vital action check missed• Missing a check which could create an unsafe situation• Exceeding airframe or engine limitations

Missed ApproachMissed Approach• Not following published procedure or clearance

Questions?

Course Schedule (cont) two Principles of Testing Principles of Testing Principles of Assessment Principles of Assessment Conducting a PPC Conducting a.

objectivity slide

perspective slide

std validity slide

good test

oral test

general briefing

test of items

test outcome

Documents

Testing Principles and Concepts - Overview

Testing of Current Carrying Capacity of Conducting Tracks in...

Software Testing and Analysis: Process, Principles,...

Principles of Shear Testing -

Dariusz Kozon - Performance testing principles

Conducting Usability Testing with a Team of One

Unit testing, principles

05 Principles Tox Testing Factors

Principles for Conducting Critical Realist Case Study ...

Model-based Testing Principles

Software Testing- Principles of testing- Mazenet Solution

Software Testing Principles

1 2a Principles Interchangeability Testing

Planning and conducting summative usability testing

Georgia Professional Standards Commission Educator Ethics...

Principles Semiconductor Testing