Accendo Reliability

Your Reliability Engineering Professional Development Site

  • Home
  • About
    • Contributors
  • Reliability.fm
    • Speaking Of Reliability
    • Rooted in Reliability: The Plant Performance Podcast
    • Quality during Design
    • Critical Talks
    • Dare to Know
    • Maintenance Disrupted
    • Metal Conversations
    • The Leadership Connection
    • Practical Reliability Podcast
    • Reliability Matters
    • Reliability it Matters
    • Maintenance Mavericks Podcast
    • Women in Maintenance
    • Accendo Reliability Webinar Series
    • Asset Reliability @ Work
  • Articles
    • CRE Preparation Notes
    • on Leadership & Career
      • Advanced Engineering Culture
      • Engineering Leadership
      • Managing in the 2000s
      • Product Development and Process Improvement
    • on Maintenance Reliability
      • Aasan Asset Management
      • CMMS and Reliability
      • Conscious Asset
      • EAM & CMMS
      • Everyday RCM
      • History of Maintenance Management
      • Life Cycle Asset Management
      • Maintenance and Reliability
      • Maintenance Management
      • Plant Maintenance
      • Process Plant Reliability Engineering
      • ReliabilityXperience
      • RCM Blitz®
      • Rob’s Reliability Project
      • The Intelligent Transformer Blog
    • on Product Reliability
      • Accelerated Reliability
      • Achieving the Benefits of Reliability
      • Apex Ridge
      • Metals Engineering and Product Reliability
      • Musings on Reliability and Maintenance Topics
      • Product Validation
      • Reliability Engineering Insights
      • Reliability in Emerging Technology
    • on Risk & Safety
      • CERM® Risk Insights
      • Equipment Risk and Reliability in Downhole Applications
      • Operational Risk Process Safety
    • on Systems Thinking
      • Communicating with FINESSE
      • The RCA
    • on Tools & Techniques
      • Big Data & Analytics
      • Experimental Design for NPD
      • Innovative Thinking in Reliability and Durability
      • Inside and Beyond HALT
      • Inside FMEA
      • Integral Concepts
      • Learning from Failures
      • Progress in Field Reliability?
      • Reliability Engineering Using Python
      • Reliability Reflections
      • Testing 1 2 3
      • The Manufacturing Academy
  • eBooks
  • Resources
    • Accendo Authors
    • FMEA Resources
    • Feed Forward Publications
    • Openings
    • Books
    • Webinars
    • Journals
    • Higher Education
    • Podcasts
  • Courses
    • 14 Ways to Acquire Reliability Engineering Knowledge
    • Reliability Analysis Methods online course
    • Measurement System Assessment
    • SPC-Process Capability Course
    • Design of Experiments
    • Foundations of RCM online course
    • Quality during Design Journey
    • Reliability Engineering Statistics
    • Quality Engineering Statistics
    • An Introduction to Reliability Engineering
    • An Introduction to Quality Engineering
    • Process Capability Analysis course
    • Root Cause Analysis and the 8D Corrective Action Process course
    • Return on Investment online course
    • CRE Preparation Online Course
    • Quondam Courses
  • Webinars
    • Upcoming Live Events
  • Calendar
    • Call for Papers Listing
    • Upcoming Webinars
    • Webinar Calendar
  • Login
    • Member Home

by JD Solomon Leave a Comment

FINESSE Facilitation: Why Qualitative Assessment Design Is Important

FINESSE Facilitation: Why Qualitative Assessment Design Is Important

Opinion-based data is the foundation of qualitative assessments. Qualitative assessments are used in various applications, including asset management, risk management, human reliability analysis, and customer surveys. The usefulness of any qualitative assessment is a function of design, analysis, and administration.

The article provides tips for improving qualitative assessment design. Facilitators develop and use qualitative assessments in the execution of their work. Facilitators should be aware of qualitative assessment design as they seek to bring a group of participants to solutions that are created, understood, and accepted by all.

A Long History with Many Forms

The modern basis of the scientific use and evaluation of such opinion-based data can be traced from the western hemisphere to the late 1800s. Educators and psychologists were seeking to quantify their clinical observations of human behavior. A similar movement was underway in the fields of natural science and statistics.

Many good practices for qualitative assessments that were developed during this time period are still applicable today. From hotel stays to dining experiences to equipment condition, opinion-based qualitative assessments are used successfully every day in a variety of fields.

Rensis Likert is credited with creating one of the first ordinal data instruments, which employs the 5-point scales currently used in most opinion-based surveys. There are five major qualitative measurement scales: Likert, ranking, Thurstone, Guttman, and semantic differential.

Survey versus Questionnaire

Remember that survey is a broad term to describe any content collection. A questionnaire is any written set of questions. A questionnaire is a subset of a survey.

Validity and Reliability

The concepts of validity and reliability are introduced as key concepts that apply to surveys, including those associated with risk management. A survey is considered valid if it can be shown to measure the variable that it is intended to measure and not others. Survey reliability refers to the extent the same results are obtained with the same question when repeated to the same group of respondents.

Likert’s Qualitative Assessment Design Recommendations

Likert provides the following recommendations concerning survey design:

  • All statements must be expressions of value or desired behavior and not statements of fact
  • It is necessary to state each question in a clear, concise, and straightforward manner
  • The scale should be created where the modal reaction is in the approximate middle of the responses
  • Approximately one-half of the questions should be in the affirmative (strongly approve) range and approximately half in the negative (strongly disapprove) range
  • The ONE is assigned consistently to the negative end of the scale and the FIVE to the positive end
  • In terms of measurement, it is immaterial what the extremes are called as long as the respondents’ understanding of the description is the same
  • Internal consistency within the survey is important
  • Select of the most differentiating statements for use in the final form of the survey

Modern References

Several modern-era sources of good practices are available. In addition to the author’s own experiences, the basis for the ones provided relates closely to the field of reliability engineering. It includes The Handbook of Human Factors Testing and Evaluation, the Institute for Defense Analysis, and the International Handbook of Survey Methodology.

Survey (general design)

  • Demographics: Do not ask for extraneous demographic information that may serve to alienate the respondent. In most cases, provide a statement that demographics will be used only to group responses (i.e., by department or job function).
  • Language: Separate versions may be needed for non-English (native language) speakers.
  • Simplicity. Keep it simple.
  • Length: Keep it short! The rule-of-thumb is 8 to 10 questions that can be completed in 15 minutes or less. Most computer-based survey tools can estimate total response time.
  • Open-ended items: Include a few open-ended items where the responses are not structured. The plus of doing this is obtaining information in the respondent’s words.
  • Pilot test: Pilot test your questionnaire with a small group to ensure your items are understood as intended.

Questionnaire (written form)

  • White space: Do not clutter pages with codes or images that do not help respondents.
  • Physical form: Construct in accordance with accessibility standards. Perform an accessibility check of the final draft questionnaire.
  • Questionnaire look: Avoid the look of marketing material or junk mail. If transmitted electronically, send a test email to verify that user browsers are allowing access.
  • Reminders: If mailed or electronically transferred, send reminders from the original sender and executive sponsor weekly.
  • Workplace distribution: Provide executive commitment (a cover letter from the executive sponsor), including the importance of responding and assurances of confidentiality.
  • Live distribution: If the questionnaire is provided in a work session, use a polling system (audience response system) to ensure participation and confidentiality.

Additional Observations

These observations are provided from much severe experience.

  • Good descriptor: The response scale and its descriptor will determine the form of the answer. This enables both representatives and uniqueness to be fulfilled from the scale-measurement perspective.
  • Balanced Scales: Balanced scales are particularly valued by many analysts because they tend to produce normal distribution and, in turn, reduce the importance of selecting the most appropriate parametric or non-parametric statistical techniques for the analysis.
  • Discriminability: Clear discriminability can be obtained with no more than seven values (5- or 7-point Likert scales are sufficient in most cases).
  • Question construction: Most modern references cite approaches similar to Likert.
    • speak to the level of the individuals who will be answering the questionnaire
    • avoid using jargon, acronyms, or overly technical terms that the respondents may misunderstand
    • use positive phrases wherever possible
    • avoid posing two questions simultaneously
    • avoid using leading or emotional questions.
  • Question validity: Use multiple question types to keep participants interested. Multiple question types, such as a mix of Likert, multiple choice, true-false, and ranking, provide the opportunity to ask the question in slightly different ways to validate what is being measured.

Facilitating with FINESSE

In the new age of meta-data, there are greater opportunities than at any time in our history to leverage data and utilize this information to improve decision quality. All data potentially leads to knowledge, and knowledge can lead to greater understanding.  

Qualitative assessments are used in various applications, including asset management, risk management, human reliability analysis, and customer surveys. The usefulness of any qualitative assessment is a function of design, analysis, and administration. This article explores aspects and good practices of survey design. System-based approaches like FINESSE are grounded in applying qualitative assessments properly to bring people to solutions that are created, understood, and accepted by all.

Are you Facilitating with FINESSE?

 

 

References

Rensis Likert, R.S. Woodruff, editor, Archives of Psychology, Columbia University, New York, Volume XXII, Nos. 146-146, 1932-1933, pp.4-43.

Handbook of Human Factors Testing and Evaluation, 2nd edition, edited by S.G. Charlton and T.G. O’Brien, publishers Lawrence Erlbaum Associates, Inc., 2002.

Institute of Defense Analysis, “ICH Q9 Briefing Pack II”, July 2006.

European Association of Methodology (EAM), International Handbook of Survey Methodology, edited by E.D. de Leeuw, J.J. Hox, and D.A. Dillman, 2008.

J.D. Solomon, Daniel Vallero, and Kathryn Benson, “Evaluating Risk: A Revisit of the Scales, Measurement Theory, and Statistical Analysis Controversy,” Proceedings of the 2017 international Reliability and Maintainability Symposium.


JD Solomon Inc provides solutions for facilitation, asset management, and program development at the nexus of facilities, infrastructure, and the environment.  Founded by JD Solomon, Communicating with FINESSE is the community of technical professionals dedicated to being highly effective communicators and facilitators. Learn more about our publications, webinars, and workshops. Join the community for free.

Filed Under: Articles, Communicating with FINESSE, on Systems Thinking

« FMEA Occurrence Risk- Insights and Advices
Biodiesel Incidents Trend »

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Headshot of JD SolomonArticles by JD Solomon
in the Communicating with FINESSE article series

Join Accendo

Receive information and updates about articles and many other resources offered by Accendo Reliability by becoming a member.

It’s free and only takes a minute.

Join Today

Recent Posts

  • Risk Prioritization in FMEA – a Summary
  • What Are Best Practices for Facilitating Qualitative Assessments?
  • So, What’s Still Wrong with Maintenance
  • Foundation of Great Project Outcomes – Structures
  • What is the Difference Between Quality Assurance and Quality Control?

© 2023 FMS Reliability · Privacy Policy · Terms of Service · Cookies Policy

This site uses cookies to give you a better experience, analyze site traffic, and gain insight to products or offers that may interest you. By continuing, you consent to the use of cookies. Learn how we use cookies, how they work, and how to set your browser preferences by reading our Cookies Policy.