Auditing Techniques for aviation

Gathering Audit Evidence … cont.

• Statistical sampling

This is the process of choosing a representative sample
from a target population and collecting data from that
sample in order to understand something about the
population as a whole.

Uses a sample selection process based on probability
theory.

• Attribute-based sampling

This means that an item being sampled either will or won’t
possess certain qualities or attributes.

is used when there are only two possible sample outcomes
for each sample (e.g. correct/incorrect or pass/fail).

Audit Execution M3/S21

Gathering Audit Evidence … cont.

An auditor selects a certain number of records to estimate
how many times a certain feature will show up in a
population;

When using attribute sampling, the sampling unit is a single
record or document;

Auditors typically use attribute sampling to test internal
controls.

Audit Execution M2/S22

Four Concepts of Audit Evidence

The Four Concepts of Audit Evidence

• Audit evidence consists of the documents you use during an
audit to substantiate your audit opinion.

• Nature: The form of the evidence — for example, physical,
oral, visual or written.

• Appropriateness: The quality, relevancy and reliability of the
evidence.

• Sufficiency: The quantity of audit evidence — enough
evidence to allow you to form an opinion.

• Evaluation: A decision on whether the evidence is
compelling enough to allow you to form an opinion.

Audit Execution M2/S23

Conducting Interviews – Using Checklists

• Use a checklist to control the pace of the audit;

• Create a checklist that is adapted to the process of the
organization and not too generic;

• Checklists should be updated on a regular basis to ensure
they remain relevant;

• Ask “open” ended questions based on the checklist;

• The use of a checklist will ensure the audit scope is being
followed;

• Checklists provide objective evidence that the audit was
performed.

Audit Execution M2/S24

Conducting interviews

Type of Question Sample Questions Relative value of
response to auditor
Leading
You do check the tyre ―
pressure every day?

Yes / No Do you have an approved √
Close-ended SMS Manual? √
Open-ended √√√
What is you current
fleet size?

How are maintenance
personnel selected?

How do you know- How do you ensure that
staff comply with duty √√√√
Demonstration rest period?

Audit Execution M2/S25

Conducting interviews

Behaviours Do not rush the interview
To Do not get into debates
Do not amplify criticism given
Avoid
Do not use sarcasm
Audit Execution Avoid humor

M2/S26

Conducting interviews

Behaviours Do not rush into conclusions
To
Do not overreact
Avoid Do not communicate
unsubstantiated findings
Do not use an audio recorder

Audit Execution M2/S27

When Conducting the Audit

• Put the person being audited at ease;
• Interview personnel at their normal place of work;
• Always explain reason for the audit;
• Explain the method of recording information gained during

the audit;
• Ask personnel to “show me” the processes being audited;
• Avoid leading questions;
• Verify information gained with personnel before reaching

final conclusions;
• Thank the personnel for their cooperation.

Overview of Auditing M1S28

Audit Protocol

Creates a road map that can be used to:

Planning • Assist in formal planning
process of the audit

Record • Assists in recording the audit procedures
Closing (Planned and completed)

• Changes/variations to the audit scope,
procedures, etc

• Review and comment on
an audit

Audit Execution M2/S29

Audit Protocol

Audit
procedures

Essential Audit
agenda
tools

Audit
checklists

Audit Execution M2/S30

Audit protocol

Allows adequate Defines acceptable
communication conduct of auditors
between auditors and
auditee

Benefits

Facilitates an Enhances factual
objective audit reporting of findings
process

Audit Execution M2/S31

Audit Approach

The 3 phased approach

Pre-audit On-Site audit Post Audit
activities Activities Activities

Audit Execution M2/S32

Audit Approach

Road to Use auditing skills and techniques
effective and Prioritizing the areas to review
efficient audits Reviewing management systems

Audit Execution Developing verification strategies
Communicating effectively

M2/S33

Application of Auditing Technique

The location factor

Auditor and Auditor Location Remote
Auditee On-site
involvement
Human  Interviews  Interactive communication:
Interaction  Checklists and questionaires  Conduct interviews
 Complete checklists and
No Human with auditee
interaction  Sampling questionnaires
 Conduct document review
Audit Execution  Document review (records,
data analysis) with auditee

 Observation of work  Document review (records,
performed
data analysis)
 Checklists  Observation of work
 Sampling
performed via surveillance

(need to consider social and

legal requirements)
 Analyze data

M2/S34

Audit Strategies To assess Horizontal
effectiveness of
Vertical Assess one
a common activity/process
Assess all approach across multiple
activities in
departments
a given
department

Upstream Downstream

Audit Execution M2/S35

Audit Strategies

Audit Execution M2/S36

Good auditing practices

Auditing Mutually agree on audit dates
Do’s
Clearly identify scope of audit

Communicate effectively before,
during and after the audit

Record findings based on a factual
approach

Make reports within a
reasonable time

Hold entry and exit meetings

Audit Execution M2/S37

Bad auditing practices

Auditing Inadequate notification
Dont’s Unannounced audits (though

sometimes necessary)

Demanding/arrogant attitude

Misuse of internal audit results

Use of recording devices during
interview

Unclear findings/statements

Audit Execution M2/S38

Process of collecting and analyzing information

Collect and Audit Review the Record Evaluate the Make audit
verify Evidence evidence audit findings conclusion
information findings against
audit
• Identify criteria
source

• Use of an
appropriate
sample

Audit Execution M2/S39

Gathering Audit Evidence

Audit Execution M2/S40

Gathering Audit Evidence

Audit Execution M2/S41

Gathering Audit Evidence

Audit Execution M2/S42

Audit Stages… cont.

Generating Audit Findings

What is a finding?

Findings are where;

The process does not conform to the requirements of the applicable
procedure;

Nonconformities and their supporting audit evidence should be
recorded. They may be graded as per the Safety Risk Matrix – Low,
Medium, High.

Observations/recommendations – are raised when the auditor wishes
to propose improvements to the system or process. They should also
appear in the final audit report.

Overview of Auditing M1S43

Evaluating an Audit Finding

A comparison of evidence
Vs

requirement within the
criteria

Non-conformity Conformity

Record supporting Grade where Record supporting
evidence required evidence of good

Audit Execution practice

M2/S44

Recording a Non-Conformance

Audit Execution M2/S45

Recording a Non-Conformance

Audit Execution M2/S46

Recording a Non-Conformance

Audit Execution M2/S47

Evaluating an Audit Finding

There is an Everything is
issue in order

Regulatory Management Good Local
non-conformity system non- management attention item
conformity
practice
observation

Audit Execution M2/S48

Audit Findings

Audit findings

• audit evidence should be evaluated against the audit criteria
in order to determine audit findings

• When specified by the audit plan, individual audit findings
should include conformity and good practices along with
their supporting evidence, opportunities for improvement,
and any recommendations to the auditee.

Audit Execution M2/S49

Classes of audit findings

• Praises or noteworthy efforts

• Areas seen as excellent examples of implementation of a requirement
according to audit criteria

• Noteworthy efforts are also given when the practices are seen as best in
class. These are done to show to the organization areas where they can
feel proud of.

• Positives

• findings that conform to the audit criteria

• Observations

• Areas noted by the auditor as being in compliance but very close to
becoming a nonconformance or that given additional evidence could
transform into non conformance. Can be looked at as “accident waiting
to happen”.

• Non-conformances

• NCs are areas where the organization’s management system does not
comply with one of the requirements of the criteria.

Audit Execution M2/S50

Recording Conformities

• The following should be considered:

• identification of the audit criteria against which conformity is
shown;

• audit evidence to support conformity;
• declaration of conformity, if applicable.

Audit Execution M2/S51

Recording Non-conformities

For records of nonconformity, the following should be
considered:

• description of or reference to audit criteria;
• nonconformity declaration;
• audit evidence;
• related audit findings, if applicable.

Audit Execution M2/S52

Audit Stages… cont.

Preparing Audit Conclusions

Audit conclusions can lead to recommendations regarding
improvements, business relationships, certification/registration or
future auditing activities.

Audit conclusions can address issues such as the extent of conformity of
the management system with the audit criteria as well as any effective
implementation, maintenance and improvement of the management
system.

Overview of Auditing M1S53

Preparing audit conclusions

• Auditors review

The audit team should confer prior to the closing meeting in
order to:

• review the audit findings, and any other appropriate information
collected during the audit, against the audit objectives;

• agree on the audit conclusions, taking into account the uncertainty
inherent in the audit process;

• prepare recommendations, if specified by the audit plan;
• discuss audit follow-up, as applicable.

Audit Execution M2/S54

Audit conclusions

Is used to address the following;
• the extent of conformity with the audit criteria and robustness of

the management system, including the
• effectiveness of the management system in meeting the stated

objectives;

• the effective implementation, maintenance and improvement of the
management system;

• the capability of the management review process to ensure the
continuing suitability, adequacy,

• effectiveness and improvement of the management system;

• achievement of audit objectives, coverage of audit scope, and fulfilment
of audit criteria;

• root causes of findings, if included in the audit plan;
• similar findings made in different areas that were audited for the

purpose of identifying trends.

• If specified by the audit plan, audit conclusions can lead to
recommendations for improvement, or future auditing activities.

Audit Execution M2/S55

Closing meeting Audit team
leader
Facilitated by the
Lead auditor

Audit team Auditee
members personnel for
areas audited
Required
participants

Other Management
Interested of Auditee

parties

Audit Execution M2/S56

Audit Stages… cont.

Conducting the Closing Meeting

• Facilitated by the audit team leader, should be held to
present the audit findings & conclusions in such a manner
that they are understood & acknowledged by the auditee.

• Participants in the closing meeting should include;

• the management of the auditee
• those responsible for the functions or processes which have been

audited,
• may also include the audit client and other parties.

• If defined in the management system or by agreement with
the audit client, the participants should agree on the time
frame for an action plan to address audit findings.

Audit Execution M2/S57

Agenda of the Closing Meeting

The following should be explained to the auditee in the
closing meeting:

• advising that the audit evidence collected was based on a sample
of the information available;

• the method of reporting;

• the process of handling of audit findings and possible consequences;

• presentation of the audit findings and conclusions in such a manner that
they are understood and acknowledged by the auditee’s management;

• any related post-audit activities (e.g. implementation of corrective actions,
audit complaint handling, appeal process).

• Any diverging opinions regarding the audit findings or conclusions
between the audit team and the auditee should be discussed and,
if possible, resolved. If not resolved, this should be recorded.

• If specified by the audit objectives, recommendations for
improvements may be presented. It should be emphasized that
recommendations are not binding.

Audit Execution M2/S58

Role Play Exercise

• Auditor: Group 1
• Auditee: Instructor
• Tools: Interview, review of docs, observations
• Scenario: Audit of ABC Airlines Airport Baggage Handling
• Checklist Standard:

All bags shall be securely attached to the baggage trolley using
approved baggage straps prior to leaving the baggage sorting area.

All staff shall display their passes when at work

Audit Execution M2/S59

Audit Stages… cont.

5. Preparing, Approving & Distributing the Audit Report (Off Site)

The audit team leader should report the audit results in
accordance with the audit program procedures.

• The audit report should provide a complete, accurate, concise
and clear record of the audit, and should include or refer to the
following:

• the audit objectives;
• the audit scope, particularly identification of the organizational and

functional units or processes audited;
• identification of the audit client;
• identification of audit team and auditee’s participants in the audit;
• the dates and locations where the audit activities were conducted;
• the audit criteria;
• the audit findings and related evidence;
• the audit conclusions;
• a statement on the degree to which the audit criteria have been fulfilled.

Audit Execution M2/S60

Content of an Audit report

• The audit report can also include or refer to the following, as
appropriate:

• the audit plan including time schedule;
• a summary of the audit process, including any obstacles encountered

that may decrease the reliability of the audit conclusions;
• confirmation that the audit objectives have been achieved within the

audit scope in accordance with the audit plan;
• any areas within the audit scope not covered;
• a summary covering the audit conclusions and the main audit findings

that support them;
• any unresolved diverging opinions between the audit team and the

auditee;
• opportunities for improvement, if specified in the audit plan;
• good practices identified;
• agreed follow-up action plans, if any;
• a statement of the confidential nature of the contents;
• any implications for the audit program or subsequent audits;
• the distribution list for the audit report.

Audit Execution M2/S61

Distributing the Audit Report

• The audit report should be issued within an agreed period
of time. If it is delayed, the reasons should be
communicated to the auditee and the person managing the
audit program.

• The audit report should be dated, reviewed and approved,
as appropriate, in accordance with audit program
procedures.

• The audit report should then be distributed to the recipients
as defined in the audit procedures or audit plan.

Audit Execution M2/S62

Audit Stages… cont.

6. Completing the Audit

The audit is completed when all activities described in the audit plan
have been carried out and the approved audit report has been
distributed.

7. Corrective Action & Audit Follow-up

The conclusions of the audit may indicate the need for corrective,
preventive or improvement actions, as applicable.

Such actions are usually decided and undertaken by the auditee
within an agreed timeframe. The Corrective Action has to be
approved by the auditor prior to implementation.

The auditee should keep the audit client informed of the status of
these actions.

Overview of Auditing M1S63

Audit Follow-up

Follow-Up Audits

• A product, process, or system audit may have findings that
require correction and corrective action. Since most
corrective actions cannot be performed at the time of the
audit, the audit program manager may require a follow-up
audit to verify that corrections were made and corrective
actions were taken.

• Due to the high cost of a single-purpose follow-up audit, it is
normally combined with the next scheduled audit of the
area. However, this decision should be based on the
importance and risk of the finding.

Overview of Auditing M1S64

Audits & Inspections

• Distinguish between audits and inspection

Audit Execution M2/S65

Answer

• An inspection is a critical examination usually following a
predefined checklist, and with a defined frequency (daily,
weekly, monthly, bimonthly, etc.) in order to verify
compliance of a particular unit, work environment, process
or a product.

• An audit is a tool to assess compliance against an existing
written Management System, Standard, and/or Process; in
order to verify documentation, implementation and
effectiveness. It is an end to end in depth review of a
process or system and is performed less frequently. E.g.
annually or every 2 yrs.

Audit Execution M2/S66

Summary

An audit report is the formal summary of the audit and any
findings.

The audit report is the end result of an audit and can be
used by the recipient person or organization as a tool for
altering operations, enforcing accountability, or making
decisions.

An effective audit report is essential to making sure the
results of your audit are presented in a way that is useful to
the party receiving the audit.

Audit Execution M2/S67

• MASTERY M2/S68

Audit Execution

Review…

• Re-cap of Module 3 Mastery

1. Criteria is …

2. The objectives should be a sentence stating why you are
auditing that scope. In order to verify what?

3. Standards or regulations are in the form of a statement or
question and may contain shall or should in the narrative..
SUPPLEMENT 33 - Air Navigation Services.pdf (reg 41)

4. CNS Audit Checklist-Eldoret Airport-2018 (1).docx

5. IOSA Standards Manual (ISM) Edition 12.pdf

6. ..\..\..\..\OneDrive - Kenya Airways PLC\desktop MAY
2018\IOSA Edn 11\ism-org-en.docm org 1.1.3

Corrective Action & Audit Follow up M4/S1

MODULE 4

CORRECTIVE ACTION & AUDIT FOLLOW UP