TSB Recommendation A90-86

Evaluation of pilot practical decision-making skills

The Department of Transport devise and implement a means of regularly evaluating the practical decision-making skills of commercially-employed pilots engaged in small air carrier operations.

Air transportation safety investigation report
Date the recommendation was issued
Date of the latest response
October 2017
Date of the latest assessment
February 2018
Rating of the latest response
Fully Satisfactory
File status
Closed

All responses are those of the stakeholders to the TSB in written communications and are reproduced in full. The TSB corrects typographical errors in the material it reproduces without indication but uses brackets [ ] to show other changes or to show that part of the response was omitted because it was not pertinent.

Recommendation A90-86 in PDF [150.41 KB]

Summary of the occurrence

Accidents in which the aircraft was operated under Visual Flight Rules (VFR) into adverse weather conditions occur regularly, claiming a disproportionately high number of fatalities each year. They involve professional pilots, private pilots and business pilots who fly general aviation aircraft and chartered commercial aircraft, including fixed-wing aircraft and helicopters.

The regularity with which these accidents have occurred, and the seriousness of the continuing loss of life, prompted the Canadian Aviation Safety Board (CASB) to initiate a comprehensive and systematic examination of the issue. In March 1990, when this examination was nearing completion, the CASB was replaced by the Transportation Safety Board of Canada (TSB), under whose auspices the report on this examination (Aviation Safety Study 90-SP00) was published on 13 November 1990.

During the two decades that preceded the release of this safety study, a number of foreign government agencies have undertaken measures to more fully understand these types of accidents. Studies have emphasized both the complex decisional nature of continued VFR flight into adverse weather and the often fatal consequences. The Aviation Safety Study 90-SP002 was the first comprehensive review of the topic in Canada in recent years, and built upon earlier works.

The Board authorized the release of Recommendation A90-86 as part of its Aviation Safety Study entitled Report of a safety study on VFR flight into adverse weather (90-SP002) on 13 November 1990.

Rationale for the recommendation

Board Recommendation A90-86 (December 1990)

At least one-third of the studied accidents involved operations which were subject to Transport Canada (TC) audits. These audits seek to ensure that companies and pilots are adhering to minimum safety standards. Section 5.0 of the Aviation Safety Study 90-SP002 dealt with the Board's concerns about industry practices, many of which are not influenced by regulatory criteria; this section addresses the effectiveness of regulatory procedures to evaluate an important aspect of a company's safety, the pilot's inflight skills. At present, the skill of commercially-employed pilots operating small, multi-engine fixed-wing aircraft in VFR operations is evaluated by an annual pilot proficiency check (PPC). The PPC focuses on aircraft handling skills and essential technical knowledge required for the safe operation of the aircraft.

Technical piloting skills were seldom found wanting in the accidents examined in this study, suggesting that the present method of evaluating pilots' skills do not address the root causes of most commercial VFR-into-IMC accidents. The study indicates that without some means of evaluating pilots' decision-making skills, professional inadequacies will go undetected until after an accident has occurred. This principle has led to a number of recent initiatives in the aviation industry. Line Oriented Flight Training (LOFT) and Cockpit Resource Management (CRM) have improved pilot decision-making skills throughout the world in larger commercial operations. Transport Canada is presently undertaking measures to incorporate an evaluation of pilot decision-making skills into the Private Pilot Licence flight test. The Board supports this, and it believes that similar initiatives to train and evaluate pilots employed in smaller commercial operations in decision-making skills would reduce the incidence of VFR-into-IMC accidents. Accordingly, the Board recommends that:

The Department of Transport devise and implement a means of regularly evaluating the practical decision-making skills of commercially-employed pilots engaged in small air carrier operations.
Transportation Safety Recommendation A90-86

Previous responses and assessments

March 1991: Transport Canada's response to Recommendation A90-86

Transport Canada has reviewed the report and concluded that the implementation of the recommendations regarding visual flight rules would result in a major change of the concepts in the conduct of visual flight operations. The regulatory actions required to institute these changes will require extensive consultation with the aviation community as a normal part of the rule making process.

Transport Canada therefore intends to establish a VFR Working Group to address and develop, in conjunction with suitable representation from the aviation community, the changes required to incorporate the TSB recommendations regarding VFR operations into the Air Regulations.

This working group will be considering the TSB recommendations A90-65, A90-66, A90-67, A90-68, A90-69, A90-70, and A90-71 relating to VFR Rules as well as recommendations A90-78 and A90-81 on licence privileges and will also include recommendations A90-83 and A90-84 concerning the mandatory equipment for rotary winged aircraft. Transport Canada will provide the responses to these recommendations when the Working Group has concluded its activities.

With the exception of the responses that depend on the Working Group Transport Canada is pleased to provide the responses as required under subsection 24(6) of the CTAISB Act to the remainder of the 14 TSB recommendations contained in this study.

Considerable emphasis has been placed on the decision-making skills of pilots in recent years. Air carriers have been encouraged to include Pilot Decision-Making (PDM) courses in their training programs. It has been Transport Canada's position that the benefits of this training were intrinsic in the enhanced performance of the pilot and that a properly planned and executed Pilot Proficiency Check would provide a practical and realistic assessment of a pilot's ability to make reasoned and timely decisions when faced with a simulated emergency situation. We will continue to keep abreast of developments in the field of decision-making training and assessment and will not hesitate to introduce improvements in our present system should they become available.

June 1991: Board assessment of the response to Recommendation A90-86

This recommendation aimed at addressing the large proportion of accidents occurring to small commercial operations and attributable, in part, to faulty decision-making. Technical pilot skills were seldom found wanting in the accidents examined in the study, suggesting that the present method of evaluating pilots' skills do not address the root causes of most commercial VFR‑into‑IMC accidents. Whereas flight test procedures for commercial pilots operating large aircraft and for private pilots flying smaller aircraft have been, or are in the process of being, modified to evaluate decision-making skills, no such modifications have been made for commercial pilots flying small aircraft, a group who figured so predominantly in VFR-into-IMC accidents. The TSB recommendation sought remedial action.

Transport Canada officials have responded to this recommendation by asserting that the proficiency pilot check (PPC) flight test, conducted annually on some commercial pilots operating small aircraft, successfully evaluates pilot decision-making skills. The information from the safety study has not led them to reconsider the existing methods of evaluating the skills required for safe flight, and they will not be exploring means by which the existing PPC could be modified to evaluate judgment and decision-making skills.

In that Transport Canada has not acknowledged the existence of the safety deficiency, nor proposed methods of addressing the conditions which led the Board to issue the recommendation, TSB staff propose that the Transport Canada response to Recommendation A90-86 be considered Unsatisfactory.

November 1996: Board reassessment of the response to Recommendation A90-86

Transport Canada still feels that the PPC flight test, conducted annually on some commercial pilots operating small aircraft, successfully evaluates pilot decision-making skills. No further action is planned by Transport Canada.

Therefore, the assessment remains Unsatisfactory.

As such, “Further Action is Unwarranted” with respect to Recommendation A90-86 and the status is set to Inactive.

April 2014: Board review of Recommendation A90-86 deficiency file status

The Board requested that Recommendation A90-86 be reviewed to determine if the Deficiency File Status was appropriate. After an initial evaluation, it was determined that the safety deficiency addressed by Recommendation A90-86 is addressed by the more recent recommendations A95-11, A95-12, A00-06, and A09-02.

It is therefore appropriate to follow the progress on CRM and PDM safety issues through recommendations A00-06 and A09-02.

Therefore, the assessment remains Unsatisfactory.

Recommendations A95-11, A95-12, A00-06, and A09-02 remain Active and include the safety deficiency identified in Recommendation A90-86. The TSB will continue to monitor the progress of TC to mitigate the risks associated with the safety deficiency identified in recommendations A00-06 and A09-02. Consequently, the status of Recommendation A90-86 will remain Active until such time as the Board assigns a Deficiency File Status of Closed to recommendations A95-11, A95-12, A00-06, and A09-02.

July 2015: Transport Canada's response to Recommendation A90-86

TCCA agrees with the intent of this recommendation. A training module (TP 13897) was published on our website regarding pilot decision making (PDM).

Work continues on the development of updating PDM, which will be incorporated in the new Standards for crew resource management (CRM) in all Commercial Sub Parts of the Canadian Aviation Regulations (CARs). A Preliminary Issue & Consultation Assessment (PICA) was recently consulted on the CARAC website regarding CRM. Notices of Proposed Amendment (NPAs) will be drafted by December 2015. Transport Canada suggests updating progress through updates to the TSB's CRM recommendations and therefore suggests closing this item.

November 2015: Transport Canada's response to related Recommendation A09-02

Transport Canada agrees with the intent of the recommendation.

Work continues on the development of standards for crew resource management (CRM). A Notice of Proposed Amendment on CRM Standards was developed and will be published in 2016.

March 2016: Board reassessment of the response to Recommendation A90-86

Until all regulatory changes proposed by TC are enacted and the standards are amended and fully implemented, the deficiency identified in Recommendation A90-86 will continue to exist. However, the proposed changes, if fully implemented, will substantially reduce or eliminate the risks associated with the safety deficiency identified in Recommendation A90-86.

Therefore, the response to Recommendation A90-86 is assessed as Satisfactory Intent.

January 2017: Transport Canada's response to Recommendation A90-86

TCCA agrees with the intent of this recommendation. TCCA proposes to go beyond the scope of the recommendation and require crew resource management (CRM) training for CAR 702 (Aerial work) operations as well.  A revised Notice of Proposed Amendment for CRM was posted to the CARAC Activity Reporting System under Activity #2014-021 and emailed to all CARAC stakeholders on May 9, 2016.

The new CRM Standard will be published in May/June 2017, together with guidance material for industry in the form of an Advisory Circular being published at the same time. Industry stakeholders will be briefed before publication.

The new standard will come into effect 30 days after publication.

March 2017: TSB reassessment of Transport Canada's response to Recommendation A90-86

TC's latest response suggests that its revised Notice of Proposed Amendment (NPA 2014-021) addresses the deficiency identified in Recommendation A90-86. The NPA states that TC's current framework for CRM training does not reflect many contemporary CRM training concepts. The NPA's objective is to integrate such concepts into commercial aviation crew training programs. Additionally, the proposed amendments will see a broader application of these updated CRM training requirements to include Commercial Air Service Standards (CASS) Subparts 722, 723, 724 and 725.

Progress toward mitigation of the risks associated with this recommendation has been slow. Such extended delays have prompted the TSB to add an item to its key safety issues Watchlist that calls for both TC and the Government of Canada to move towards an improved and accelerated process for taking action on TSB recommendations.

The Board is encouraged that amendments to the CASS, and guidance material for industry, are planned to be published in May/June 2017 and come into effect 30 days following their publication.

The Board is pleased that the intent of NPA 2014-021 is to update CRM training standards across all CASS, including Aerial Work operations, which were not included in the original recommendation. Consequently, if fully implemented, the proposed changes should serve to mitigate the risks identified in Recommendation A90-86.

However, until the new CRM Standards are fully implemented, the risks associated with the safety deficiency identified in Recommendation A90-86 will continue to exist.

Therefore, the response to Recommendation A90-86 is assessed as Satisfactory Intent.

Latest response and assessment

October 2017: Transport Canada's response to Recommendation A90-86

TC agrees in principle with the recommendation.

The latest amendment of the approved check pilot (ACP) manual (TP6533) provides the following guidance to check pilots in assessing candidate decision-making skills.

5.13 Non-Technical Skill Element - Decision Making

  1. Decision making is a non-technical skill element.
  2. Decision making is defined as the process of making a judgment call or choosing an option.
  3. Various decision points differ enormously in what they demand of the crew, what options and supports exist in SOPs and policies for making decisions, and what features may make the situation difficult or error-prone.
  4. Decision-making is comprised of four sub-elements:
    1. Problem Definition / Diagnosis
      1. Accurately defining a problem is dependent upon one's situational awareness and attending to critical information. It also relies upon avoiding perception errors.
      2. Gathering information to identify a problem is observed.
      3. Reviewing causal factors with other crew members.
    2. Option Generation
      1. Generating options through an unbiased collective effort where possible.
      2. Stating alternative options.
      3. Seeking opinions from crew members.
    3. Risk Assessment
      1. Assessing risks by way of an unbiased and collective effort where possible, subject to time available.
      2. Considering and sharing risks of alternative options.
      3. Talking about potential risks in terms of crew limitations.
    4. Option Selection
      1. Confirming and stating selected option and/or agreed upon action.
    5. Outcome Review
      1. Incorporating a measure of evaluation when a decision is implemented.
      2. hecking outcomes against a plan.
  5. Assessments of decision making are qualitative.

Additionally, the new standards for crew resource management (CRM) have been published in the Canada Gazette on 31 July 2017. Industry stakeholders were briefed prior the publication and the new standards will come into effect on 31 January 2019.

TC believes that the amendments to the ACP Manual and the requiring initial and recurrent CRM training satisfy this recommendation.

February 2018: TSB reassessment of Transport Canada's response to Recommendation A90-86

TC has taken a number of actions to address the safety deficiency identified in Recommendation A90-86, regarding the evaluation of practical decision-making skills. These include the following:

  • In June 2017, TC published an amendment to the approved check pilot (ACP) manual (TP6533, Tenth Edition). With this amendment, specific guidance is provided to ACPs for the evaluation of pilot decision-making skills; and
  • New crew resource management (CRM) standards are scheduled to come into effect on 31 January 2019, and industry stakeholders have been briefed. The new standards will require annual CRM training for all commercially-employed pilots and include a decision-making component.

The Board believes that the actions taken by TC will substantially reduce the risk associated with the safety deficiency identified in Recommendation A90-86, once the new CRM standards come into effect.

Therefore, the Board considers the response to the recommendation to be Fully Satisfactory.

File status

The TSB will monitor TC’s progress on its planned actions.

This deficiency file is Closed.