Opinion-based data is the foundation of qualitative assessments. Qualitative assessments are used in various applications, including asset management, risk management, human reliability analysis, and customer surveys. The usefulness of any qualitative assessment is a function of design, analysis, and administration. This article summarizes a review of ten risk matrices performed by facility owners or their consultants.
Qualitative Assessment Scales
Likert scales are five-point ordinal scales where participants select a label (or numerical value) that equates to an opinion or attitude. Labels are paired (i.e., agree-disagree, strongly agree-strongly disagree) symmetrically around a neutral center. Today, we see 5-, 7-, and 9-point versions of Likert scales in everything from customer surveys to risk assessments.
Ten Risk Matrices Evaluated
Ten risk assessments were randomly evaluated that used consequences of failure (CoF) and likelihood of failure (LoF) in a two-axis matrix. In this case, random means from my files of numerous projects I have reviewed over the past two decades. All of the examples are related to some form of facilities or infrastructure.
Seven of the examples were developed under the leadership of consultants and three were developed by facility staff. One consulting company appeared twice; however, the lead consultant and the geographic regions were different.
Focus of Design and Analysis
The review focused on the design and analysis of the qualitative assessments. In most cases, the approaches and techniques of administering the assessments were not known.
Poor Application of Risk Matrices
Eight of the ten risk matrices had flawed analysis or design. In theory, these flaws produce significant impacts on the results.
Scales (and bad math)
In seven of the ten risk matrix evaluations examined, applying scalar properties to ordinal data was found to be a major source of error. The ordinal data from 5- and 9-point Likert risk surveys were treated parametrically. In five of the seven cases, the numbers (ratings) were multiplied as if the data were continuous.
The correct analysis in these cases is for the scales to be handled non-parametrically and, if combined, treated additively. A comparative analysis of the results indicated much different risk rankings and prioritization depending on how the data was treated.
From a practical perspective, the ordinal data can be treated as interval data if the scales. This was done correctly in only three of the ten cases. The incorrect treatment of ordinal data is consistent with similar reviews I have performed over the past twenty years. The improper analysis of ordinal scales underscores a significant practitioner problem.
Flawed Internal Consistency
Eight of the risk matrices examined involved the concept of vertical consistency. By way of example, in one risk matrix involving the consequences of failure, the value associated with a human life was 10. However, a 10 was also correlated to a budget impact of $150,000 or a customer experiencing five days without utility service. Clearly, these are not equivalent.
The results in eight of the ten cases were blindly weighted and aggregated without regard to vertical equivalency. Technically, this is a violation of the measurement principle of representativeness. From a practical perspective, it skews many lesser important systems to be treated as equivalent to those where human health and safety should be dominant.
Similar to the improper treatment of ordinal scales, flawed internal consistency is common and a significant practitioner problem.
Limited Practical Ramifications
Although eight of the ten cases had some form of poor assessment design and analysis, there were few practical ramifications. In all ten cases, I reviewed the final prioritization and discussed the results with an owner’s representative one to three years after the work was implemented.
Common Sense
In four of the eight cases, the risk matrix was technically performed wrong from a design or analysis perspective. Still, the final selections of risk treatment yielded acceptable results for the organization. It appears that common sense or maybe even luck overcame poor analytical techniques.
Insignificant Magnitude
In four of the eight cases, the owners acknowledged that the assessment produced a risk evaluation that included or excluded some wrong items. The costs of the associated implemented items (projects) were in the range of hundreds of thousands to over ten million dollars.
However, the facility and infrastructure system values ranged from a half billion to more than one billion dollars. While acknowledging some degree of undesirability, the impacts were not insignificant or outside the “normal” process for the organization.
Extremes Remained the Same
In four of seven cases, it was possible to re-analyze the ordinal data were treated as continuous data and parametrically. Meaningful distinctions were made in the average scores (calculated to two to three decimal places) that were applied in the risk matrices. When the data was re-evaluated as ordinal and analyzed non-parametrically, the order of the prioritization resulted in significant differences in the risk matrices and associated project prioritizations.
However, items that ranked at the extremes, either high or low, were considered properly regardless of the strict correctness of the assessment design and analysis. In all cases, the issues were most significant in the middle, where an action (funding) cut-off level would have impacted projects on the “bubble.”
Overall, most organizations stated they could not significantly fund projects that were not high priorities. In those cases, the fine-tuning in the middle made no practical difference.
Acceptable, Not Optimal, Results
One conclusion that can be taken from the examination of ten cases is that risk matrices yield acceptable but not optimal results.
Some more specific conclusions include the following:
- A minority of risk matrices are designed, analyzed, and administered properly.
- When done correctly, the results of risk matrices are acceptable for initial prioritization purposes. (more analysis and decision making may be needed, depending on the context)
- There is a significant practitioner problem in analyzing ordinal scales and associated data.
- There is a significant practitioner problem in designing and analyzing multi-attribute variables, particularly related to the consequences of failure (CoF).
- Common sense or luck often prevails in the finalized prioritization.
- Organizations have some slop in the normal selection of priorities, and using risk matrices produces no worse than normal results.
- Many organizations can only afford to invest in high priorities, so the fine differences in the middle often do not matter.
Applying It with FINESSE
Do the fine points of risk matrices really matter? Yes, because someone is paying a trained professional to do the technical assessment correctly. The risk matrix is often not constructed as it should be.
Qualitative assessments are cost-effective and highly flexible tools. Using Likert-type questionnaires as a principal evaluation test must be accompanied by the appropriate design, analysis, and administration. Do it correctly, and communicate it with FINESSE, or do not do it at all.
Communicating with FINESSE is a not-for-profit community of technical professionals dedicated to being highly effective communicators and facilitators. Learn more about our publications, webinars, and workshops. Join the community for free.
Leave a Reply