August 10th, 2009 - Stopping root decay
A nasty pricing typo in an a client’s ecommerce system caused me to write the following in an email. So I thought I’d go ahead and share some thinking about dealing with root causes:
Because ecommerce systems tend to be complex and the machinery that enables them is hidden from view, I’m always interested in ferreting out root causes to problems that crop up.
The reason? Other problems, associated with a given root cause, will, in all probability, show up at some decidedly inconvenient time in some wholly (Holy) unpredictable way. So, you might as well take the opportunity to deal with the underlying cause in advance since it has people’s attention anyway…
In particular, I’m interested in how an ecommerce team can make changes to processes, procedures, and protocols to:
- Prevent problems like this from happening,
- Capture them before they go live, and
- Minimize the effects and duration of them if/when they do.
The focus on process, procedure, and protocol is important because it gives people in the organization permission to work collectively against healthy “common enemies” of preventable errors, premeditated carelessness and worst of all, acratic action.
So, to the standard set of questions, I would add:
- When did the error actually happen?
- Why wasn’t it discovered before it went live?
- How long did it run before somebody figured it out?
- How was it discovered? and by whom?
- What process, procedure, or protocol change would have prevented it from happening in the first place?