Opinion-based data is the foundation of qualitative assessments. Qualitative assessments are used in various applications, including asset management, risk management, human reliability analysis, and customer surveys. The usefulness of any qualitative assessment is a function of design, analysis, and administration. The article provides best practices for improving administering and facilitating qualitative assessments.
A Long History with Many Forms
The modern basis of opinion-based data’s scientific use can be traced from the western hemisphere to the late 1800s. Educators and psychologists were seeking to quantify their clinical observations of human behavior. A similar movement was underway in the fields of natural science and statistics.
Rensis Likert is credited with creating one of the first data instruments, which employs the 5-point ordinal scales currently used in most opinion-based surveys. There are five major qualitative measurement scales: Likert, ranking, Thurstone, Guttman, and semantic differential.
Proper Administration and Facilitation
According to the Institute of Defense Analysis (IDA), “It is true that surveys measure subjective experiences (i.e., thoughts) and they are not accurate measures of anything other. However, when designed, administered, and analyzed correctly, surveys provide objective, reliable, and valid measurement of these subjective experiences. There is a substantial body of research on how to collect accurate survey data (e.g., Likert, 1932; Babbitt & Nystrom, 1989).”
IDA adds, “The motivations of the respondents is affected by what the respondents were asked to do before the survey, their beliefs as to what will happen with the survey data, how interested they are in the topic of the survey, and how frequently they are surveyed. Motivation can be increased or decreased by the design of the test (e.g., completing the same survey multiple times), the design of the survey (e.g., length, question wording), and the administration of the survey (e.g., when administered).”
Administration and facilitation constraints, which come from things such as the test design, the survey delivery method (paper, electronic, verbal), and the time of day, can impact results.
Likert’s Survey Administration Recommendations
Likert provides few recommendations in terms of survey administration. However, he clearly believes that the survey should be evaluated for reliability (repeatability) and used the Spearman-Brown method as a primary tool. He used data from resurveys of a given group of respondents up to 30 days following the initial survey.
Modern References
Several modern-era sources of good practices are available. In addition to the author’s own experiences, the basis for the ones provided relates closely to the field of reliability engineering. It includes The Handbook of Human Factors Testing and Evaluation, the Institute for Defense Analysis, and the International Handbook of Survey Methodology.
- avoid giving surveys at the beginning or end of a day or work shift; avoid administering detailed surveys at the beginning or end of a work week
- facilitate the questions orally
- use facilitators that are available throughout the time(s) the survey is being completed to answer questions
- provide written instructions or frequently asked questions so that each respondent has a common basis of knowledge
- any single session/scenario should be able to be completed by a respondent working independently in 15 minutes.
Additional Observations
Qualitative assessments are cost-effective and highly flexible tools. They measure the opinions, attitudes, knowledge, perceived behaviors, observations, beliefs, and experiences of individuals who use a system the most. Human factors should be included in any system evaluation, as well as aspects of system performance that we cannot or do not understand how to measure.
Solomon and Benson use some best practices related to risk evaluations of infrastructure systems, some of which include:
- beginning and ending the session with “easier” subsystems to account for mental warm-up and to avoid the potential effects of fatigue
- using a combination of paper and electronic forms
- going through each question as a group to avoid the potential for respondents to feel as if they were in a mental race
- providing a comment section for each question to allow respondents to give more detail and to avoid potential frustration if a question is not well understood
Facilitating with FINESSE
There are greater opportunities than at any time in our history to leverage different types and combinations of data and utilize this information to improve decision quality. All data potentially leads to knowledge, and knowledge can lead to greater understanding.
Qualitative assessments are used in various applications, including asset management, risk management, human reliability analysis, and customer surveys. The usefulness of any qualitative assessment is a function of design, analysis, and administration. This article explores aspects and good practices of survey administration.
Are you Facilitating with FINESSE?
References
Rensis Likert, R.S. Woodruff, editor, Archives of Psychology, Columbia University, New York, Volume XXII, Nos. 146-146, 1932-1933, pp.4-43.
Handbook of Human Factors Testing and Evaluation, 2nd edition, edited by S.G. Charlton and T.G. O’Brien, publishers Lawrence Erlbaum Associates, Inc., 2002.
Institute of Defense Analysis, “ICH Q9 Briefing Pack II”, July 2006.
European Association of Methodology (EAM), International Handbook of Survey Methodology, edited by E.D. de Leeuw, J.J. Hox, and D.A. Dillman, 2008.
J.D. Solomon, Daniel Vallero, and Kathryn Benson, “Evaluating Risk: A Revisit of the Scales, Measurement Theory, and Statistical Analysis Controversy,” Proceedings of the 2017 international Reliability and Maintainability Symposium.
JD Solomon Inc provides solutions for facilitation, asset management, and program development at the nexus of facilities, infrastructure, and the environment. Founded by JD Solomon, Communicating with FINESSE is the community of technical professionals dedicated to being highly effective communicators and facilitators. Learn more about our publications, webinars, and workshops. Join the community for free.
Leave a Reply