Taking on Unnecessary Project Risk

Guest Post by John Ayers (first posted on CERM ® RISK INSIGHTS – reposted here with permission)

This story is about risk at the enterprise program/project level.  It is an example of centering your design around a high-tech solution to a problem when a low risk proven technology would have more than sufficed.   Technical expertise ego got in the way of a practical low-cost approach resulting in unplanned schedule and cost growth.

Background

A defense company I worked for won a competitive contract to design, build, and deliver an expendable acoustic target for training sonar personnel at sea. It was common to use an actual submarine for the training but they were expensive and difficult to schedule frequently enough to be effective for the training program.  Hence the need for the expendable target. Expendable implies low cost and low risk but, in this case, to gain additional performance capability a bad decision (as was evident in hind sight) was made to use Lithium Thionyl Chloride batteries in lieu of the Alkaline batteries. Lithium batteries had several advantages: double the energy density of Alkaline batteries; long storage life, flat discharge rate preferred to achieve constant target speed; and wide operating temperature range. As we found out later, they also had several disadvantages: high cost; must have DOT (department of transportation approval to ship); and can be dangerous (have been known to explode under certain circumstances). But the die was cast at the beginning of the program when the decision was made to go with high performance batteries.

The Training Target

The training target comprised: batteries for power; 2 pressure switches to control motor turn on at depth; motor to propel target; an injection molded nose containing an acoustic device that emits acoustic signatures; electronics; a tail with horizontal elevator that move to make target dive or surface in water; and a body that connects the nose and tail and encases the electronics and batteries.

In operation, the target was deployed from the deck of the ship. As it sunk, it moved in a spiral direction. When it reached a predetermine pressure, the pressure switch was activated and turned the motor on causing the target to move upward towards the surface along the azimuth angle it was at when the motor turned on. When the target reached a certain depth near the surface of the water, the other pressure switch actuated and stopped the motor and the target spiraled down. This cycle repeated itself until the training exercise was over or the battery ran out. Results was a random walk that allows sonar personnel to track the target and sharpen their acoustic tracking skills. The target design was not complicated and the performance requirements were very achievable.

First Article Safety Tests

The design and build phase went well. A small number of targets (called First Article units) were built for functional and safety testing.  The functional tests went well but the safety tests did not. The safety tests were conducted at a government test facility located in Maryland.  This test site specialized in explosives and since the target contained potentially explosive Lithium batteries, they had to undergo and pass several tests.  The most difficult test comprised heating the target to high temperatures (in a bunker) and deliberately causing the Lithium batteries to explode, which they did.  The target had to vent the instant high-volume gases caused by the explosion to prevent flying debris which was a safety hazard. To achieve this goal, the target vents had to be large and high in number. Unfortunately, the target failed the first test and later (after modifications) failed the second test.  What should we do?  That was the question.

Battery Decision

It was pretty clear the Lithium batteries were a bad choice.  A decision was made to replace the Lithium batteries with the Alkaline type even though there would be a significant schedule and cost impact for the project. Fortunately, the contract included a large production order with several options that allowed project recovery over time.  The loss in performance by replacing the Lithium batteries turned out to be a non-problem because there was sufficient power to meet all of the contract requirements. The project continued to perform to plan with the Alkaline batteries. It turned out to be a very successful project for the Company.

Lesson Learned

1. Do not assume greater risk than necessary to meet the contractual requirements with reasonable margins.
2. Do not take on unproven, immature, or potential safety risk technology, components or features in your design. If required to do so, include budget and schedule time in the project baseline to adequately research and test these items beforehand.
3. Do not let your lead technical person drive the program. Technical folks focus on functionality. How can I make it more functional? Project managers focus on cost, schedule and meeting contract commitments. In the story above, I was the project manager and was swayed by the technical lead to use Lithium batteries when I was unsure about it. I learned on this project, I would not be swayed by a technical lead in the future unless I was very comfortable with it.
4. Always perform a risk/benefit analysis for a new approach or major design change. It was not done in the above story. If it was, I would think Lithium batteries would not have been used because the risks out weighed the benefits.