QUALITY ASSURANCE TASKS
Table of Contents
OVERVIEW OF INVESTIGATION QUALITY ASSURANCE
DATA AND DATA SOURCES REQUIRED
PART I. IQA FOR MES-BASED INVESTIGATIONS. PART II. IQA FOR NON MES-BASED INVESTIGATIONS.
GENERAL IQA FRAMEWORK. IQA TECHNICAL PROCEDURE
2. Checking your EBs.
3. Organize the highlighted EBs.
4. Apply sequencing tests
5. Add partial EBs.
6 Determine causal relationships among EBs
7. Enter causal links.
Tutorial (with example and school solutions): See Investigation Catalyst software Help Menu
ASSURING INVESTIGATION QUALITY
Assuring the quality of investigations and investigation work products is another continuing challenge to incident investigators. This task requires a consistent, disciplined and objective procedures. The MES Matrix development procedures and rules provide for the quality of the investigation. This Guide describes how to assess the investigation reports.
Procedures for this task are contained in this Guide. See also the Help Menu in the Investigation Catalyst software.
Traditional investigations lack objective quality assurance procedures for either the investigation work products or the investigation process itself. Quality of work products is controlled primarily by one of several variations of the peer review procedures, and validity is decided by power of persuasion in what is essentially an adversarial process. A second general approach is the "fly-fix-fly" approach, where conclusions are tested by repeating the experiment or occurrence or simulating it, and determining from the outcome if the investigation findings "reproduced" the occurrence.
The multilinear events sequence-based (MES-based) investigation process provides new opportunities for assuring the quality of the investigation and its work products, including the description and explanation of what happened, recommendations flowing from the investigation, investigation reports, and effectiveness of the assurance actions implemented. The primary quality assurance vehicles are the MES-based Matrix and work products developed from that Matrix. These Investigation Quality Control (IQA) procedures constitute an essential element of the MES-based investigation process, and provide a way to check reports resulting from other methods.
The objective of this Guide is to present a quality assurance procedure that can be used to
determine the quality of non-MES-based investigation work products.
This investigation quality assurance procedure is applicable to all kinds of investigation work products purporting to describe and explain an occurrence, and to the analyses of proposed needs and actions flowing from that description. It can be applied to any investigation and work product to ensure confidence in the outputs.
The investigation quality assurance process utilizes the data from the investigation and resultant work products. For MES-based investigations, data on Matrixes, interim work products and source references are required. For non-MES-based investigations, only the investigation work products and, if available, source references are used. Data in the check lists presented in this Guide may also be used.
Procedures for MES-based and non-MES-based investigations are similar in that EBs and Matrixes form the basis for both. They differ in that Matrixes must be created for non-MES-based in investigations, whereas Matrixes are developed as part of the MES-based investigation process. The effect is that the IQA person will perform EB and Matrix development tasks (Guides 1 and 2) for non-MES-based investigations, whereas investigators perform these tasks during MES-based investigations. This is an important difference, because it means the IQA person must be capable of performing the EB and Matrix tasks as well as the IQA tasks. For this Guide, the EB and Matrix development tasks are not repeated in their entirety, but only where they differ due to the IQA requirements.
IQA procedures are implemented during the investigation process. They begin with the creation of the first EB, and do not end until the final investigation based work product is completed. The IQA tasks include the following, if the guidance in Guides 1-9 has been implemented and a Matrix is available for IQA review. If the Matrix is not available, treat the IQA task the same as a non-MES-based investigation work product, and go to Part II.
Examine each EB an the Matrix to verify that the format and content are consistent with Guide 1 specifications. If they are not, note the discrepancies on the Matrix.
Examine every event pair on the Matrix for the sequential logic of the EB placement. If EBs are not properly ordered, note the discrepancies on the Matrix.
Examine each event set on the Matrix for the necessary and sufficient logic of the EB links displayed, and try to visualize what is being described from the data presented. Try not to let your prior knowledge of the system get into your way. Consult with someone having systems operations knowledge if you can't visualize the process being described. If you find logic fallacies or gaps, note the discrepancies on the Matrix.
Examine each countermeasure tab, needs statement and recommended action to ensure they are tied to the correct EBs describing the occurrence, and do not conflict with each other or introduce new, unknown relationships.
Review the Matrix for countermeasure tabs to locate the EBs or relationships that were concluded to be "needs" and determine that this decision was made by the proper person or persons. This should not become an exercise in second guessing, but rather review to ascertain that the needs identified are indeed validly defined and reported.
Review each countermeasure diamond on the Matrix to verify that the need identified MEETS THE SPECIFICATIONS described in Guide 8, and that the rationale is described in terms of the EBs involved, rather than some abstractions or judgment call based on past experience, past occurrences or other unsupported rationale. Note any problems with the rationale from those specifications on the Matrix.
Over time, the IQA procedures conclude with a quality check of the final documented work products resulting from the recommended action(s) implemented. Depending on the scope of the investigation program, the last document may be a report on the success of the action implemented.
The IQA procedures begin by recasting information from the work products into the MES-based Event Building Block (EB) format. Then an MES-based Matrix is created with those EBs. Then the IQA process proceeds as described above. Before attempting to perform an IQA on a set of documents, the reviewer should become skilled in the procedures for creating and testing Matrixes in Guides 1 and 2.
Kipling's faithful servants: WHO-WHAT-WHEN-WHERE-WHY form the general framework for IQA procedures.
As a first step, a quality investigative output will give every person or object one name and use only that name throughout the rest of the occurrence description or discussion.
Secondly, the output will present the occurrence data and occurrence description in the "active voice" ( ball struck floor) where the name of the actor who or which does something comes before any words describing what the actor did (not floor was struck by ball.) An abbreviated way to state this is to ensure that data are transformed into an Actor + Action (ball + struck) format. This is the basic "EB" used to describe what happened.
The description will describe when each EB occurred, relative to some other EB or reference point. Review each EB to verify that it is in the proper temporal sequence based on the logic of the EB flow.
In addition to confirming that the EBs are in their proper order, the EBs must be arranged in their spatial sequence to make sense. Don't add actions that require someone to fall up two long flights of stairs, for example.
This is probably the most subtle and the most abused quality assurance criterion. For one EB to affect another EB, a cause-effect relationship - or linkage - between the EBs must exist. This cause-effect linkage must be established for all EBs in the description, because the cause-effect links define why the occurrence continued to its conclusion - the harmful outcome after it started. If there is no cause-effect linkage, the EB may be nice to know, but it is not needed to understand and explain the occurrence! Cause connotes an if/then relationship: if A occurs then B follows. If B does not always follow A, then A is not the cause of B. Rather A + An are the cause of B, or A causes B + Bn.
In an occurrence description this kind of critical logic testing can be used to select EBs that must be reported to describe what happened and why it happened. Tests for cause-effect linkages are well know to critical thinkers. The tests require application of the necessary and sufficient reasoning from logic. They provide a basis for establishing investigation quality assurance to determine and ensure that precede/ follow logic governs the occurrence description.
These ideas provide the essential criteria to permit a check of the quality of occurrence descriptions and resultant work products.
The following technical procedure provides a way to do a quality assurance check of an occurrence description, using MES-based criteria to ensure that the description is valid and complete, explain why the occurrence occurred, eliminate unnecessary data from the description, and identify uncertainties. Although relevant, this quality assurance procedure does not address subsequent uses of the occurrence description, judgments or opinions about the occurrence, recommendations, or other aspects of the investigation quality assurance issue. It is based on the premise that the quality of other aspects of the investigation depend on the quality of the occurrence description, because if the quality of the description is unsatisfactory, any subsequent uses of the occurrence description are tainted. Quality criteria and procedures for those aspects of an investigation require separate consideration
This procedure is designed to ensure that the who, what, when, where and why are properly reported, using the concepts just described. To apply the procedure, start with any description of an occurrence and take the following steps.
Highlight or underline each actor and concrete action set reported in documents. Each actor/action set is an EB. These EBs become the basis for assessment of the data quality. If the data are not properly transformed into the actor/action format, you will experience difficulties when you go to use it. You may have to dig hard to find all these actor + action (EB) sets, because the actor and the action may be separated by many words, - or in bad cases, sentences - or be described ambiguously or in the passive voice. For whatever the investigator's reasons, sometimes the name of the actor is not stated when an action is reported. "Passive voice" sentences ("was" or "were") or pronouns or other ambiguous actor names, or abstract action words like failed, erred, etc., obstruct formulation of your mental movie of what happened. DO NOT assume or infer complete EBs from these entries. LET THE REPORT PREPARER DO THAT. Rather, use the names of people or objects in the report, or the action words, by recording the given part of the EB only, leaving the other part of the EB either blank or represented by a question mark.
The quality assurance process first addresses the adequacy of the building blocks created during the investigation. If they are flawed, the investigation is likely to be flawed. Each EB used on a Matrix should be checked for content and form, as it is placed on the Matrix, and again when the Matrix has been competed prior to its use for problem definition and assurance purposes.
ACTOR + ACTION
ACTOR + ACTION
When an EB has passed these checks, it is ready for use in subsequent IQA tasks.
After you have underlined or circled (highlighting also is OK too) all the "EBs" in the document, go to the next step.
For a tutorial with examples, go to the on-line tutorial file at Starline Software Ltd.
Record all the EB building blocks (actor + action + any descriptive words needed to describe the action) on a medium like "POST-IT"&tm; sticky note pads. As you record them, put each new EB on a Matrix laid out to provide time and actor coordinates. Use one row for each actor, and a time line across the top with tick marks to indicate approximate times under which the EBs are aligned. Arrange the EBs so their order from left to right represents the relative time when they occurred. As this display progresses, look at each EB relative to each other EB to be sure each is in the right sequence. If two actors were doing something at the same time, the EBs should be placed one above or below the other, to indicate that the actors represented by the rows did something at the same time - in parallel, rather than in series.
When all the highlighted EB building blocks have been placed in their proper rows and in their proper time "columns" along the rows, review the Matrix to see if they are aligned properly according to their spatial location during the occurrence.
Now add EBs for which you have only the actor or the action - not both - and place those EBs on your Matrix. At this time, it is also convenient to review again the source of your EBs, and add EBs that can be inferred reasonably. When all the EBs are entered, review each pair to ascertain whether the order in which they are displayed is logical with respect to time and space. If not rearrange the EBs to array them into their proper order. When both time and spatial relationships have been tested and are satisfactory, proceed to the next step.
This next step is a critical quality assurance test step. (See Guide 2 for additional information about this procedure.) Add linking arrows to describe the sequences or flow of the changes that produced the outcome you are investigating, from the first EB in the occurrence process through the last EB (outcome) in the process. An inability to add cause-effect links shows where you have a quality assurance problem - in the form of an investigation data, data processing or reporting failure - and shows where you have problems with the occurrence description or the reported data.
The procedure is based on working with EB pairs or sets on your Matrix. You begin with the right-most (earliest in time) EB pair on the Matrix and examine the EB pair for a cause-effect relationship. If the left EB had to occur to make the right EB occur, you have established an initial "necessary" cause-effect relationship between the EBs. In other words, the right EB could not have occurred until the left EB occurred IN THIS OCCURRENCE. (Do NOT introduce expected relationships yet; forget about what usually happens and focus on what the report says did happen in this occurrence.)
The next step is to examine the same EBs to establish a "sufficient" logical relationship between the EBs. If the left EB was the only EB that was required for the right EB to occur, you have established an initial "sufficient" cause-effect relationship between the EBs. In other words, the left EB was both necessary and sufficient to produce the right EB of the pair, and in this occurrence did so.
At that point, you draw a linking arrow from the left EB to the right EB, signifying a cause-effect link between the two EBs. Each linking arrow on your work sheet shows you that you have a cause-effect relationship between two EBs.
Often you will find that more than one EB is must occur before another EB will occur. This is similar to an "AND" logic gate in a fault tree (See Figure 1.) "OR" logic gates are not permitted in a final occurrence description because they indicate uncertainty about what happened. It is better to use a question mark if the cause-effect links can not be established, to indicate the uncertainty in the description. If you find that the left EB was not sufficient by itself to produce the right EB, you begin to look for the other EB(s) that HAD TO OCCUR to produce the right EB. For each linked EB pair, draw a linking arrow between them.
Necessary and sufficient tests may produce cause-effect linked pairs (1), converging EB sets (2) or diverging EBs sets (3). Uncertainties (4) are indicated by a question mark between the EBs, indicating an unmet data need. Uncertainties are not objectionable if they are faithfully represented in the text of the report.
After you establish all the necessary and sufficient causal relationships among the initial EBs pair, repeat the same logical reasoning for each pair of EBs on the Matrix. If any EBs are not linked to the other EBs, those EBs identify problems with the occurrence description that you should resolve. At the conclusion of this process, you know what the occurrence description contains, and how well it enables users to visualize and understand the occurrence process. If the Matrix display has gaps or flawed logic, they are apparent from your IQA Matrix. The output can be returned to the preparer with concrete shortcomings very visible for all to see ( and correct.)
Upon completion of the Matrix IQA tasks, the needs statements and recommended actions tasks described in the preceding section are implemented to complete the IQA task.
IQA reviewers quickly discover several kinds of problems after doing the causal links for the occurrence. If you are an independent IQA reviewer, try to get the investigator to fix any problems you note.
By now the vital role of the MES Matrix - the flow chart of the incident or operation - should be evident.
 See Hendrick & Benner "Investigating Accidents with STEP" Marcel Dekker, NY 1987 for discussions of these aspects of investigations and assurance of their quality. See also Benner, "Introduction to Investigations", available from www.starlinesw.com for additional background presentation about investigation tasks.