AAP element 11

Objectives and Targets, Monitoring and Measurement

The eight most expensive words in aviation; ‘But we have always done it this way‘.

An organisation needs to give itself a health check. Looking inwards is a means to achieve this.  When an organisation becomes aware of gaps in safety, or negative safety trends, positive steps can be taken before an incident or accident occurs.

There are numerous ways to measure and monitor safety, but a mandatory and voluntary reporting system is critical.  Combined with a fair and just culture, hazard and incident reporting provides steady input, and a reliable database for the risks an organisation is exposed to.

Reporting and self-auditing

A voluntary and mandatory reporting system provides a systematic means for an organisation to record occurrences, incidents and accidents in operations. Mandatory reporting criteria may include:

  • Any equipment or personnel failure airborne that compromises the safety of flight
  • Any equipment or personnel failure on the ground that compromises the safe operation of the aircraft
  • Any breach (unintentional or otherwise) of SOPs
  • Any Human Factors transgressions, airborne or on the ground that compromise safety.
Red Ring Binder with Inscription Database.

Voluntary reporting can be any other occurrence that personnel feel could offer a learning experience to others.

Feedback to management from the SMS is required to measure and monitor safety performance. Safety performance evaluation defines the required changes.

The SAG and SRC will include performance of safety on a regular basis.

SMS audits, both internal and external, provide an assessment of performance and should be conducted as detailed in SPT/SPIs.

Compliance  link provides practical examples and templates for internal audits.

Effective reporting culture:

  • A good safety culture encourages reporting of aviation hazards
  • Feedback! When people report – provide feedback (‘thank you for your report…as a consequence we have done/or will do…‘).
  • Lack of reporting feedback triggers the ‘why bother’ attitude
  • A just culture ensures fairness and draws a line in the sand between errors, violations, recklessness and wilful disobedience
  • When carrying out internal safety investigations as a result of a report – concentrate on the why, how and what, rather than the who.

Reasons for reporting – and reasons for non-reporting

Yes - I should report itNo - I will hide it
I don’t want an accident to occurI might get fired
I know I’m supposed toNothing will change
It’s easy to fill out a reportI’ll look foolish
It’s the lawI might lose my licence
The system will support meThey will look for someone to blame
It might happen againIt’s too hard
We can learn from thisIt’s unimportant
Somebody saw me…Nobody saw me

Reporting of Near Misses is an indication of an improving or a good safety culture.

If the question is asked, ‘should I report this?‘ – the answer is always, ‘Yes!

Safety Objectives, SPIs and SPTs

Performance should not remain static.  Expectations should increase over time and as the SMS matures.  The phases to consider are:

  • Implementation phase – performance required for compliance or entry control
  • Improvement phase – performance after the SMS has completed a full cycle
  • Learning phase – performance with a mature SMS after multiple cycles.

Outcome based indicators are measurable. (eg, the number of runway incursions or unstable approaches).

Process based indicators contribute to the achievement of an outcome. (eg, training provided to airfield drivers or pilots to conducting approaches).

Leading and lagging performance indicators

Aviation safety measures have historically been biased towards SPIs that reflect ‘low frequency – high consequence‘ outcomes. This is understandable in that accidents and serious incidents are high profile events and are easy to count. However, from a safety performance management perspective, here are drawbacks in an overreliance on accidents and serious incidents as a reliable indicator of safety performance. For instance, accidents and serious incidents are infrequent (there may be only one accident in a year, or none) making it difficult to perform statistical analysis to identify trends. This does not necessarily indicate that the system is safe. A consequence of a reliance on this sort of data is a potential false sense of confidence that an organization’s or system’s safety performance is effective, when it may in fact be perilously close to an accident.

Leading indicators are measures that focus on processes and inputs that are being implemented to improve or maintain safety. These are also known as ‘activity or process SPIs‘ as they monitor and measure conditions that have the potential to become or to contribute to a specific outcome.

From ICAO SMM 4th edition

Measuring Safety Performance

  • Safety objectives should address your most significant risks
  • Safety performance indicators are linked to the safety objectives
  • However, targets can be misleading:
    • They can drive the wrong behaviour…’we reached our goal, no need to do any more/ SPI’s and bonuses linked to targets‘…
    • They can cap innovation and continuous improvement
    • They can divert focus from quality to quantity
    • Measurement can become what is easy rather than what is important.
  • Good presentation of data is as important as the content. It:
    • Keeps the audience engaged and helps them to remember key points
    • Enables management to make necessary decisions
    • Considers the end user and what information is important to them and how much detail is appropriate.
  • Poor presentation of data can:
    • Lead to confusion (If you can’t understand it, can anyone else?)
    • Result in the analysis being ignored
    • Result in misinterpretation of key points.