9: Validation

Article by Mike Freier

In the last blog post, we discussed how incorporating Agile principles can improve your hardware product development process. Not only will you be able to accelerate your schedule, but you can also develop higher quality products with lower costs and reduced risks. Furthermore, Agile frameworks and techniques can fit into your existing new product introduction (NPI) and stage gate process, so your high-level standard operating procedures do not have to change.  

Once you have engineering and beta units in hand, validation and verification can begin. In this stage, engineering teams should test your product against the engineering design specification to verify that your product meets the performance, reliability, regulatory, manufacturing test, and safety goals.

Performance

Performance is a broad category that specifies how your product should work at your customer site(s) or in the field. Using industrial robots as an example, here are a few things your NPI team might test for.

• Accuracy
• Precision
• Payload
• Droop
• Linearity 
• Hysteresis
• Stability
• Reach (min-max)
• Cycle time

Performance testing is often done by a team of design and test engineers, with some support from quality management to confirm the product is compliant with the specifications outlined in the engineering design specifications. Although it’s common to do performance testing on-site, some tests may be performed at outside third-party labs, if specialized equipment is required.

Reliability Strategy and Plan 

Another important part of validation and verification testing is deciding how you are going to validate or confirm your product meets the reliability targets, often specified as mean time (or cycles) before failure (MTBF).  To achieve your reliability targets, dedicate a couple of early production units to your long term reliability testing program.  

Good reliability strategies identify and test for the challenges your product might face in the real world.  

Elements of a reliability plan include: 

• Clear cycle definition (or agreement)
• Method to measure and report total number of cycles
• Controlled environment, if it relates to performance (e.g., temperature) 
• Agreement on how to handle interrupts or hard failures
• Plan for root-cause analysis and product improvements 

Regulatory

Agency and environmental standards compliance is another critical part of product development.

Ideally, your NPI team discusses regulations and the testing requirements early in the development process and documents an agreed-upon test plan. Planning ahead is advantageous and can minimize issues downstream.  

Agencies testing requirements can be well documented, but confusing, because there are multiple standards and multi-tiered sections that may apply to your product. Furthermore, if your NPI team is not intimately familiar with each agency’s standards and test procedures, it’s helpful to hire a standards and test specialist to fill any gaps on your core team. Often, spending a little money up-front can save thousands of dollars and weeks–or months–of time to pass each test the first time. In addition to contract engineering resources, many Nationally Recognized Test Laboratories (NRTL) will evaluate your first prototype and provide invaluable feedback to you. 

A few markets are highly-regulated and specialized. 

• Medical
• Aerospace
• Military
• Automotive
• Telecommunications

Agency and environmental standards can vary by geography.  So, it’s best to map-out and document which countries your product might ship to and to work with a NRTL specialist that can help you understand the standards that apply to your product. Usually, it’s more cost effective to run all tests concurrently for all geographies with the same test house, rather than test for USA compliance and later run other tests for Europe or Asia. This will also help you estimate the total cost of testing and allow you to make any adjustments in the test plan, as needed.

Product safety testing is a process ensures products comply with consumer product safety standards. Safety testing is a product that entails testing, inspecting, and certification. Safety testing keeps consumers, well, safe, and helps reduce death and injuries. Safety testing also protects businesses from legal risks like lawsuits. 

Manufacturing Test

When your goal is to deliver high-quality products that meet all of the specifications, the manufacturing test is indispensable. Unlike engineering or beta units, production systems are sold to actual customers and can have a huge positive or negative impact on your brand and reputation.  

So how much testing is required? This depends on the complexity of your product and how your production line and supply chain is setup.

At a minimum, you should test the final assembly and have a well defined test and pass-fail criteria. For complex products, this is risky because the amount of time to troubleshoot, rework, assemble and test can be huge.  

To reduce this risk, testing each sub-assembly before the final test is strongly recommended. This can be done as part of your standard manufacturing process or at your module supplier. Smaller components (e.g., pc-boards) can be tested at your supplier to confirm they meet specifications.  

Embrace a thoughtful validation strategy that tests product performance, long-term reliability, compliance to agency regulations, and safety standards. Save your business major headaches after your product launches. An investment in product validation is an investment in your success that is sure to pay big dividends over time. 

About the Author

Mike Freier is a Founder and Principal Consultant at the Product Management Consulting Group Inc.  He has 20+ years of experience with high-tech companies, including more than a decade in product management and product marketing.  He was a long-time board member and President of the Silicon Valley Product Management Association, served on the MIT-Stanford (VLAB) executive planning committee and spent several years mentoring start-up founders at the Cleantech Open.  Mike graduated from the University of Massachusetts at Amherst with a BSME degree