Opinion-based data is the foundation of qualitative assessments. Qualitative assessments are used in various applications, including asset management, risk management, human reliability analysis, and customer surveys. The usefulness of any qualitative assessment is a function of design, analysis, and administration.
The article provides tips for improving qualitative assessment design. Facilitators develop and use qualitative assessments in the execution of their work. Facilitators should be aware of qualitative assessment design as they seek to bring a group of participants to solutions that are created, understood, and accepted by all.
A Long History with Many Forms
The modern basis of the scientific use and evaluation of such opinion-based data can be traced from the western hemisphere to the late 1800s. Educators and psychologists were seeking to quantify their clinical observations of human behavior. A similar movement was underway in the fields of natural science and statistics.
Many good practices for qualitative assessments that were developed during this time period are still applicable today. From hotel stays to dining experiences to equipment condition, opinion-based qualitative assessments are used successfully every day in a variety of fields.
Rensis Likert is credited with creating one of the first ordinal data instruments, which employs the 5-point scales currently used in most opinion-based surveys. There are five major qualitative measurement scales: Likert, ranking, Thurstone, Guttman, and semantic differential.
Survey versus Questionnaire
Remember that survey is a broad term to describe any content collection. A questionnaire is any written set of questions. A questionnaire is a subset of a survey.
Validity and Reliability
The concepts of validity and reliability are introduced as key concepts that apply to surveys, including those associated with risk management. A survey is considered valid if it can be shown to measure the variable that it is intended to measure and not others. Survey reliability refers to the extent the same results are obtained with the same question when repeated to the same group of respondents.
Likert’s Qualitative Assessment Design Recommendations
Likert provides the following recommendations concerning survey design:
- All statements must be expressions of value or desired behavior and not statements of fact
- It is necessary to state each question in a clear, concise, and straightforward manner
- The scale should be created where the modal reaction is in the approximate middle of the responses
- Approximately one-half of the questions should be in the affirmative (strongly approve) range and approximately half in the negative (strongly disapprove) range
- The ONE is assigned consistently to the negative end of the scale and the FIVE to the positive end
- In terms of measurement, it is immaterial what the extremes are called as long as the respondents’ understanding of the description is the same
- Internal consistency within the survey is important
- Select of the most differentiating statements for use in the final form of the survey
Modern References
Several modern-era sources of good practices are available. In addition to the author’s own experiences, the basis for the ones provided relates closely to the field of reliability engineering. It includes The Handbook of Human Factors Testing and Evaluation, the Institute for Defense Analysis, and the International Handbook of Survey Methodology.
Survey (general design)
- Demographics: Do not ask for extraneous demographic information that may serve to alienate the respondent. In most cases, provide a statement that demographics will be used only to group responses (i.e., by department or job function).
- Language: Separate versions may be needed for non-English (native language) speakers.
- Simplicity. Keep it simple.
- Length: Keep it short! The rule-of-thumb is 8 to 10 questions that can be completed in 15 minutes or less. Most computer-based survey tools can estimate total response time.
- Open-ended items: Include a few open-ended items where the responses are not structured. The plus of doing this is obtaining information in the respondent’s words.
- Pilot test: Pilot test your questionnaire with a small group to ensure your items are understood as intended.
Questionnaire (written form)
- White space: Do not clutter pages with codes or images that do not help respondents.
- Physical form: Construct in accordance with accessibility standards. Perform an accessibility check of the final draft questionnaire.
- Questionnaire look: Avoid the look of marketing material or junk mail. If transmitted electronically, send a test email to verify that user browsers are allowing access.
- Reminders: If mailed or electronically transferred, send reminders from the original sender and executive sponsor weekly.
- Workplace distribution: Provide executive commitment (a cover letter from the executive sponsor), including the importance of responding and assurances of confidentiality.
- Live distribution: If the questionnaire is provided in a work session, use a polling system (audience response system) to ensure participation and confidentiality.
Additional Observations
These observations are provided from much severe experience.
- Good descriptor: The response scale and its descriptor will determine the form of the answer. This enables both representatives and uniqueness to be fulfilled from the scale-measurement perspective.
- Balanced Scales: Balanced scales are particularly valued by many analysts because they tend to produce normal distribution and, in turn, reduce the importance of selecting the most appropriate parametric or non-parametric statistical techniques for the analysis.
- Discriminability: Clear discriminability can be obtained with no more than seven values (5- or 7-point Likert scales are sufficient in most cases).
- Question construction: Most modern references cite approaches similar to Likert.
- speak to the level of the individuals who will be answering the questionnaire
- avoid using jargon, acronyms, or overly technical terms that the respondents may misunderstand
- use positive phrases wherever possible
- avoid posing two questions simultaneously
- avoid using leading or emotional questions.
- Question validity: Use multiple question types to keep participants interested. Multiple question types, such as a mix of Likert, multiple choice, true-false, and ranking, provide the opportunity to ask the question in slightly different ways to validate what is being measured.
Facilitating with FINESSE
In the new age of meta-data, there are greater opportunities than at any time in our history to leverage data and utilize this information to improve decision quality. All data potentially leads to knowledge, and knowledge can lead to greater understanding.
Qualitative assessments are used in various applications, including asset management, risk management, human reliability analysis, and customer surveys. The usefulness of any qualitative assessment is a function of design, analysis, and administration. This article explores aspects and good practices of survey design. System-based approaches like FINESSE are grounded in applying qualitative assessments properly to bring people to solutions that are created, understood, and accepted by all.
Are you Facilitating with FINESSE?
References
Rensis Likert, R.S. Woodruff, editor, Archives of Psychology, Columbia University, New York, Volume XXII, Nos. 146-146, 1932-1933, pp.4-43.
Handbook of Human Factors Testing and Evaluation, 2nd edition, edited by S.G. Charlton and T.G. O’Brien, publishers Lawrence Erlbaum Associates, Inc., 2002.
Institute of Defense Analysis, “ICH Q9 Briefing Pack II”, July 2006.
European Association of Methodology (EAM), International Handbook of Survey Methodology, edited by E.D. de Leeuw, J.J. Hox, and D.A. Dillman, 2008.
J.D. Solomon, Daniel Vallero, and Kathryn Benson, “Evaluating Risk: A Revisit of the Scales, Measurement Theory, and Statistical Analysis Controversy,” Proceedings of the 2017 international Reliability and Maintainability Symposium.
JD Solomon Inc provides solutions for facilitation, asset management, and program development at the nexus of facilities, infrastructure, and the environment. Founded by JD Solomon, Communicating with FINESSE is the community of technical professionals dedicated to being highly effective communicators and facilitators. Learn more about our publications, webinars, and workshops. Join the community for free.
Leave a Reply