Matrix Quality Check.
Needs Definition Quality Check.
Action Option Quality Check.

IQA PROCESS OVERVIEW.
GENERAL IQA FRAMEWORK.
Who did what When Where Why
IQA TECHNICAL PROCEDURE
a. The IQA procedure
1. Find the reported EBs.
2. Checking your EBs.
3. Organize the highlighted EBs.
4. Apply sequencing tests
5. Add partial EBs.
6 Determine causal relationships among EBs
7. Enter causal links.

b. Continue IQA process for Needs and Recommended Actions.
HANDLING PROBLEMS DISCLOSED BY THE IQA TASK

Tutorial (with example and school solutions): See Investigation Catalyst software Help Menu

Assuring the quality of investigations and investigation work products is another continuing challenge to incident investigators. This task requires a consistent, disciplined and objective procedures. The MES Matrix development procedures and rules provide for the quality of the investigation. This Guide describes how to assess the investigation reports.

The general output quality assurance steps are to:

1. note explicit actions described in the document
2. note implicit actions described in the document
3. transform the actions into EBs
4. array the EBs on the MES Matrix
5. link the interactions
6. gaps and ? identify quality problems.

Procedures for this task are contained in this Guide. See also the Help Menu in the Investigation Catalyst software.

OVERVIEW OF INVESTIGATION QUALITY ASSURANCE

Traditional investigations lack objective quality assurance procedures for either the investigation work products or the investigation process itself. Quality of work products is controlled primarily by one of several variations of the peer review procedures, and validity is decided by power of persuasion in what is essentially an adversarial process. A second general approach is the "fly-fix-fly" approach, where conclusions are tested by repeating the experiment or occurrence or simulating it, and determining from the outcome if the investigation findings "reproduced" the occurrence.

The multilinear events sequence-based (MES-based) investigation process provides new opportunities for assuring the quality of the investigation and its work products, including the description and explanation of what happened, recommendations flowing from the investigation, investigation reports, and effectiveness of the assurance actions implemented. The primary quality assurance vehicles are the MES-based Matrix and work products developed from that Matrix. These Investigation Quality Control (IQA) procedures constitute an essential element of the MES-based investigation process, and provide a way to check reports resulting from other methods.

OBJECTIVE

The objective of this Guide is to present a quality assurance procedure that can be used to

ensure valid, effective investigations and work products as an investigation proceeds.

determine the quality of non-MES-based investigation work products.

APPLICABILITY

This investigation quality assurance procedure is applicable to all kinds of investigation work products purporting to describe and explain an occurrence, and to the analyses of proposed needs and actions flowing from that description. It can be applied to any investigation and work product to ensure confidence in the outputs.

Note: The procedure is also applicable to hazard and risk investigations where the system operation is described with MES Matrixes. If not so described, the analyses can be recast into that format, and the reformatted process flow and analysis conducted as described.

DATA AND DATA SOURCES REQUIRED

The investigation quality assurance process utilizes the data from the investigation and resultant work products. For MES-based investigations, data on Matrixes, interim work products and source references are required. For non-MES-based investigations, only the investigation work products and, if available, source references are used. Data in the check lists presented in this Guide may also be used.

IQA PROCEDURES

Procedures for MES-based and non-MES-based investigations are similar in that EBs and Matrixes form the basis for both. They differ in that Matrixes must be created for non-MES-based in investigations, whereas Matrixes are developed as part of the MES-based investigation process. The effect is that the IQA person will perform EB and Matrix development tasks (Guides 1 and 2) for non-MES-based investigations, whereas investigators perform these tasks during MES-based investigations. This is an important difference, because it means the IQA person must be capable of performing the EB and Matrix tasks as well as the IQA tasks. For this Guide, the EB and Matrix development tasks are not repeated in their entirety, but only where they differ due to the IQA requirements.

IQA procedures are implemented during the investigation process. They begin with the creation of the first EB, and do not end until the final investigation based work product is completed. The IQA tasks include the following, if the guidance in Guides 1-9 has been implemented and a Matrix is available for IQA review. If the Matrix is not available, treat the IQA task the same as a non-MES-based investigation work product, and go to Part II.

MATRIX QUALITY CHECK.

Examine each EB an the Matrix to verify that the format and content are consistent with Guide 1 specifications. If they are not, note the discrepancies on the Matrix.

Examine every event pair on the Matrix for the sequential logic of the EB placement. If EBs are not properly ordered, note the discrepancies on the Matrix.

Examine each event set on the Matrix for the necessary and sufficient logic of the EB links displayed, and try to visualize what is being described from the data presented. Try not to let your prior knowledge of the system get into your way. Consult with someone having systems operations knowledge if you can't visualize the process being described. If you find logic fallacies or gaps, note the discrepancies on the Matrix.

Examine each countermeasure tab, needs statement and recommended action to ensure they are tied to the correct EBs describing the occurrence, and do not conflict with each other or introduce new, unknown relationships.

NEEDS DEFINITION QUALITY CHECK.

Review the Matrix for countermeasure tabs to locate the EBs or relationships that were concluded to be "needs" and determine that this decision was made by the proper person or persons. This should not become an exercise in second guessing, but rather review to ascertain that the needs identified are indeed validly defined and reported.

Review each countermeasure diamond on the Matrix to verify that the need identified MEETS THE SPECIFICATIONS described in Guide 8, and that the rationale is described in terms of the EBs involved, rather than some abstractions or judgment call based on past experience, past occurrences or other unsupported rationale. Note any problems with the rationale from those specifications on the Matrix.

1. Review the needs/problem statement to ensure that the EXPECTED PERFORMANCE and the source of those expectations is identified in terms of specific actions by specific actors! Again note any deficiencies in the identification of the expectations on the Matrix. Challenge any ..ly words or allegations of error or failure if they do not include the preexisting expected performance sources in a concrete way.

2. Review the process by which the author determined that the need/problem merits attention, which is implied by stating it, to ensure that the rationale is presented, and is based on an agreed measure of significance or some agreed-upon criteria, as described in Guide 8.

3. Then make sure the selection decision(s) for the needs that will be addressed by recommended action was made by the proper "authority" in the organization.

4. If the IQA check does not support its selection as a need or problem to address, note your rationale on the Matrix, relating it to the diamond number.

ACTION OPTION CHECK.

1. Review the needs selected for action, and the candidate action options suggested. Ideally, the options should be related to some principle to provide the supporting rationale for its candidacy. That principle should be readily apparent or stated.

2. For each candidate action, review the tradeoffs identified in favor of its selection, and opposed to its selection. Tradeoffs should address the predicted relative effectiveness, resources required, technical feasibility, organizational impact or other criteria of significance to the decision maker. At a minimum, the predicted effects of the action on similar future occurrences should be described. Review the weights assigned to each trade-off to assess their validity. Since this is a judgment call, variances in weights are to be expected. Negotiations to reach a consensus on the trade-offs and weights is a useful exercise during the IQA process, because it helps develop weighting criteria over the long term. Notations on the trade-off Matrix are recommended.

3. Review the predicted effects to ascertain that they provide a way to track the predicted results. Remember, if the predicted results are not defined so this can be done, the proposed action is suspect, and should be challenged. Again, make notes of your findings on the Matrix.

4. Review the recommendation selection process to verify that the options recommended are the higher rated options among the candidates suggested, using the criteria described in Guide 8.

5. Finally, review the proposed actions against the IQA check list (Figure 10-1)as a final test of the recommendations.

Over time, the IQA procedures conclude with a quality check of the final documented work products resulting from the recommended action(s) implemented. Depending on the scope of the investigation program, the last document may be a report on the success of the action implemented.

IQA PROCESS OVERVIEW.

The IQA procedures begin by recasting information from the work products into the MES-based Event Building Block (EB) format. Then an MES-based Matrix is created with those EBs. Then the IQA process proceeds as described above. Before attempting to perform an IQA on a set of documents, the reviewer should become skilled in the procedures for creating and testing Matrixes in Guides 1 and 2.

GENERAL IQA FRAMEWORK.

Kipling's faithful servants: WHO-WHAT-WHEN-WHERE-WHY form the general framework for IQA procedures.

Who did what

As a first step, a quality investigative output will give every person or object one name and use only that name throughout the rest of the occurrence description or discussion.

Secondly, the output will present the occurrence data and occurrence description in the "active voice" ( ball struck floor) where the name of the actor who or which does something comes before any words describing what the actor did (not floor was struck by ball.) An abbreviated way to state this is to ensure that data are transformed into an Actor + Action (ball + struck) format. This is the basic "EB" used to describe what happened.

When

The description will describe when each EB occurred, relative to some other EB or reference point. Review each EB to verify that it is in the proper temporal sequence based on the logic of the EB flow.

Where

In addition to confirming that the EBs are in their proper order, the EBs must be arranged in their spatial sequence to make sense. Don't add actions that require someone to fall up two long flights of stairs, for example.

Why

This is probably the most subtle and the most abused quality assurance criterion. For one EB to affect another EB, a cause-effect relationship - or linkage - between the EBs must exist. This cause-effect linkage must be established for all EBs in the description, because the cause-effect links define why the occurrence continued to its conclusion - the harmful outcome after it started. If there is no cause-effect linkage, the EB may be nice to know, but it is not needed to understand and explain the occurrence! Cause connotes an if/then relationship: if A occurs then B follows. If B does not always follow A, then A is not the cause of B. Rather A + An are the cause of B, or A causes B + Bn.

In an occurrence description this kind of critical logic testing can be used to select EBs that must be reported to describe what happened and why it happened. Tests for cause-effect linkages are well know to critical thinkers. The tests require application of the necessary and sufficient reasoning from logic. They provide a basis for establishing investigation quality assurance to determine and ensure that precede/ follow logic governs the occurrence description.

These ideas provide the essential criteria to permit a check of the quality of occurrence descriptions and resultant work products.

IQA TECHNICAL PROCEDURE

The following technical procedure provides a way to do a quality assurance check of an occurrence description, using MES-based criteria to ensure that the description is valid and complete, explain why the occurrence occurred, eliminate unnecessary data from the description, and identify uncertainties. Although relevant, this quality assurance procedure does not address subsequent uses of the occurrence description, judgments or opinions about the occurrence, recommendations, or other aspects of the investigation quality assurance issue. It is based on the premise that the quality of other aspects of the investigation depend on the quality of the occurrence description, because if the quality of the description is unsatisfactory, any subsequent uses of the occurrence description are tainted. Quality criteria and procedures for those aspects of an investigation require separate consideration[1]

a. The IQA procedure

This procedure is designed to ensure that the who, what, when, where and why are properly reported, using the concepts just described. To apply the procedure, start with any description of an occurrence and take the following steps.

1. Find the reported EBs.

Highlight or underline each actor and concrete action set reported in documents. Each actor/action set is an EB. These EBs become the basis for assessment of the data quality. If the data are not properly transformed into the actor/action format, you will experience difficulties when you go to use it. You may have to dig hard to find all these actor + action (EB) sets, because the actor and the action may be separated by many words, - or in bad cases, sentences - or be described ambiguously or in the passive voice.[2] For whatever the investigator's reasons, sometimes the name of the actor is not stated when an action is reported. "Passive voice" sentences ("was" or "were") or pronouns or other ambiguous actor names, or abstract action words like failed, erred, etc., obstruct formulation of your mental movie of what happened. DO NOT assume or infer complete EBs from these entries. LET THE REPORT PREPARER DO THAT. Rather, use the names of people or objects in the report, or the action words, by recording the given part of the EB only, leaving the other part of the EB either blank or represented by a question mark.

2. Checking your EBs.

The quality assurance process first addresses the adequacy of the building blocks created during the investigation. If they are flawed, the investigation is likely to be flawed. Each EB used on a Matrix should be checked for content and form, as it is placed on the Matrix, and again when the Matrix has been competed prior to its use for problem definition and assurance purposes.

1. It contains the required content - the actor, action and descriptors, or ? if appropriate in lieu of these three required elements.

2. The name for each actor is unique and used consistently in all EBs for that actor.

3. Each EB is free of the following poison words (From Guide 1):
   
   
   

4. Each EB has sufficient information to identify about when it happened, at least relative to other EBs, so the EB can be placed in the right column on a Matrix.

5. Each EB with a ? in the box has been reviewed to be sure the data are not available, or that provision has been made to acquire the missing data, or that MES-Trees or other form of logic tree structure will be used to try to postulate what probably occurred, before closing the investigation.

6. Any codes used are identified and recorded in a master list of codes.

7. The sources have all been identified and retained where necessary or when so instructed.

When an EB has passed these checks, it is ready for use in subsequent IQA tasks.

After you have underlined or circled (highlighting also is OK too) all the "EBs" in the document, go to the next step.

For a tutorial with examples, go to the on-line tutorial file at Starline Software Ltd.

3. Organize the highlighted EBs.

Record all the EB building blocks (actor + action + any descriptive words needed to describe the action) on a medium like "POST-IT"&tm; sticky note pads. As you record them, put each new EB on a Matrix laid out to provide time and actor coordinates. Use one row for each actor, and a time line across the top with tick marks to indicate approximate times under which the EBs are aligned. Arrange the EBs so their order from left to right represents the relative time when they occurred. As this display progresses, look at each EB relative to each other EB to be sure each is in the right sequence. If two actors were doing something at the same time, the EBs should be placed one above or below the other, to indicate that the actors represented by the rows did something at the same time - in parallel, rather than in series.

4. Apply sequencing tests

When all the highlighted EB building blocks have been placed in their proper rows and in their proper time "columns" along the rows, review the Matrix to see if they are aligned properly according to their spatial location during the occurrence.

5. Add partial EBs.

Now add EBs for which you have only the actor or the action - not both - and place those EBs on your Matrix. At this time, it is also convenient to review again the source of your EBs, and add EBs that can be inferred reasonably. When all the EBs are entered, review each pair to ascertain whether the order in which they are displayed is logical with respect to time and space. If not rearrange the EBs to array them into their proper order. When both time and spatial relationships have been tested and are satisfactory, proceed to the next step.

6 Determine cause-effect relationships among EBs

This next step is a critical quality assurance test step. (See Guide 2 for additional information about this procedure.) Add linking arrows to describe the sequences or flow of the changes that produced the outcome you are investigating, from the first EB in the occurrence process through the last EB (outcome) in the process. An inability to add cause-effect links shows where you have a quality assurance problem - in the form of an investigation data, data processing or reporting failure - and shows where you have problems with the occurrence description or the reported data.

The procedure is based on working with EB pairs or sets on your Matrix. You begin with the right-most (earliest in time) EB pair on the Matrix and examine the EB pair for a cause-effect relationship. If the left EB had to occur to make the right EB occur, you have established an initial "necessary" cause-effect relationship between the EBs. In other words, the right EB could not have occurred until the left EB occurred IN THIS OCCURRENCE. (Do NOT introduce expected relationships yet; forget about what usually happens and focus on what the report says did happen in this occurrence.)

7. Enter cause-effect links.

The next step is to examine the same EBs to establish a "sufficient" logical relationship between the EBs. If the left EB was the only EB that was required for the right EB to occur, you have established an initial "sufficient" cause-effect relationship between the EBs. In other words, the left EB was both necessary and sufficient to produce the right EB of the pair, and in this occurrence did so.

At that point, you draw a linking arrow from the left EB to the right EB, signifying a cause-effect link between the two EBs. Each linking arrow on your work sheet shows you that you have a cause-effect relationship between two EBs.

Often you will find that more than one EB is must occur before another EB will occur. This is similar to an "AND" logic gate in a fault tree (See Figure 1.) "OR" logic gates are not permitted in a final occurrence description because they indicate uncertainty about what happened. It is better to use a question mark if the cause-effect links can not be established, to indicate the uncertainty in the description. If you find that the left EB was not sufficient by itself to produce the right EB, you begin to look for the other EB(s) that HAD TO OCCUR to produce the right EB. For each linked EB pair, draw a linking arrow between them.

Necessary and sufficient tests may produce cause-effect linked pairs (1), converging EB sets (2) or diverging EBs sets (3). Uncertainties (4) are indicated by a question mark between the EBs, indicating an unmet data need. Uncertainties are not objectionable if they are faithfully represented in the text of the report.

After you establish all the necessary and sufficient causal relationships among the initial EBs pair, repeat the same logical reasoning for each pair of EBs on the Matrix. If any EBs are not linked to the other EBs, those EBs identify problems with the occurrence description that you should resolve. At the conclusion of this process, you know what the occurrence description contains, and how well it enables users to visualize and understand the occurrence process. If the Matrix display has gaps or flawed logic, they are apparent from your IQA Matrix. The output can be returned to the preparer with concrete shortcomings very visible for all to see ( and correct.)

b. Continue IQA process for Needs and Recommended Actions.

Upon completion of the Matrix IQA tasks, the needs statements and recommended actions tasks described in the preceding section are implemented to complete the IQA task.

HANDLING PROBLEMS DISCLOSED BY THE IQA TASK

IQA reviewers quickly discover several kinds of problems after doing the causal links for the occurrence. If you are an independent IQA reviewer, try to get the investigator to fix any problems you note.

1. A typical problem is that Matrixes have gaps in the links between the first and last EBs. Work with the IQA sponsor (bill payer) to determine if you should

   ask for more data through further investigation,

   report the gaps in the documentation, or

   request hypotheses to provide potential descriptions to fill the gaps using systematic methods like MES-based logic trees, etc., or

   reduce the scope of the review or report.

   If you accept the gaps, recognize you are depriving users of potential corrective action choices and the investigation is more likely to promote less effective recommendations, or misdirect action efforts.

2. A second kind of problem is extraneous EBs that are left over after completing the causal links. What do you do with them? The best course of action is to remove them from the description, because they mislead rather than illuminate users of the reports, and provide "hooks" for others to grasp and raise irrelevant, unnecessary and invalid questions about the occurrence. At their worst, they divert efforts to assurance future performance from bona fide needs demonstrated by the occurrence.

3. A third kind of problem arises when EBS are flawed. The report does not transform the occurrence data into useful building blocks, so you can't establish or validate causal links or EBs displays. This is a clear warning that the report is of unacceptable quality.

4. A fourth kind of problem is that some managers demand a "simple" description of a complex occurrence, like a "root cause" or "causal factors." The Matrix provides a way to present the complexity of the occurrence that most managers will accept when it is demonstrated. Remember, a properly completed MES Matrix describes all the necessary and sufficient events - and only those events - that had to occur to produce the outcome. Presenting the most effective, efficient changes helps refocus managers attentions in desirable channels.

5. A fifth kind of problem is that individuals keeping occurrence statistics insist that investigators completing their forms even though the occurrence data do not fit the blanks on forms. This can be addressed by providing the IQA Matrixes with the forms, and letting whoever wants the forms completed fill in the forms on a best efforts basis. Unfortunately, the statistician's solution is to raise the level of abstraction of the data and blur detail until the investigator's data can be fit into the blank. The consequences of this on the statistical outputs are obvious - GIGO.

   Most forms will not fare well with this quality assurance method, but be assured the problem is with the forms, rather than the IQA method. For example, how does one state the time and date of an accident when the accident stretches out over several days, as occurs in some hazardous materials transportation incidents. "Cause of the accident" blanks create similar problems.
   It is uncommon for the narrative section of forms to fare much better when IQA checked this way, but that problem can be ameliorated by providing the Matrix to the investigator, forms preparer and user.

6. Another problem is that individuals who want to make unwarranted judgments or draw unjustified conclusions, or propose recommendations to solve last year's problems instead of the problems demonstrated by this occurrence, or insist on a single cause or root cause of the occurrence will get quite upset with your IQA results. Your IQA outputs with their multiple causal links and "diamond" checks expose sloppy logic, loose interpretations of data, or unsubstantiated hypotheses, and are very frustrating to hip shooters used to presenting flawed outputs and abstractions. You will just have to endure their frustrations and hope they will begin to understand your better way over time.

7. The problem of hidden assumptions is one of the more subtle and difficult problems to detect during the quality assurance checks. For example, a report will propose more training, which assumes the additional training will be valid. Or it assumes prior training was valid, but just not frequent enough. Investigators must guard against these logic failures by challenging each statement in a report that is not clearly supported by an event building block or event set or links showing relevant interactions.

[1]  See Hendrick & Benner "Investigating Accidents with STEP" Marcel Dekker, NY 1987 for discussions of these aspects of investigations and assurance of their quality. See also Benner, "Introduction to Investigations", available from www.starlinesw.com for additional background presentation about investigation tasks.

[2]  For a further explanation of this procedure, see Quality Assurance Tutorial