I feel that many of my generation have lost touch with our intrinsic drive to make things. Instead we have replaced it with habits of consuming things others have made. And while there is that initial excitement at having bought something, it wears off quickly. But the thrill of making things on your own is different. It stays with you. There is a sense of pride and ownership about having learned something or having figured something out.

This past weekend I watched “the next list“, a program on CNN hosted by Dr. Sanjay Gupta. It profiled Dale Dougherty and his creation – the Maker Faire. At a time when everyone seems panicked about America losing its edge in creativity and innovation, Dale is igniting that fire single handedly. He is providing a venue for do-it-yourself-ers to show off their creations. “the next list” spot lighted a couple of makers and their creations at one such fair. I was amazed with what I saw.

Somewhere on our way to the 21st century we started to believe that only geniuses can create; that failure was not an option. We stopped making making things and exploring ideas for their own sake. It is time to reconnect with our humanity; get our hands dirty once again and start making things. America doesn’t have fear losing its leadership in innovation and creativity. But to maintain it we do need to replicate Maker Faire in all of our homes, schools and businesses.

A fundamental assumption in manufacturing is that if the inputs to the process remain constant e.g. you use exactly 6 cups of bread flour each time, and the process itself is constant e.g. the oven generates 350 F of heat every time, then each output of the process will be the same as the previous with no discernable difference. However, nothing is constant: there is natural variability in the quantity of the flour used; sometimes you might use as little as 5.5 cups; other times you might use as much as 6.5 cups. Even the oven periodically turns its heating mechanism on and off to provide a mean temperature of 350 F, but the actual temperature at any given instant is more likely than not to be above or below the mean. So in the physical world each output of the process i.e. each loaf of bread will be different from the previous.

The question then becomes is the loaf to loaf variability in the output, the result of the variability in the inputs and the process, noticeable by the consumer? Each characteristic of the output not only has a target value but also a range about the target that is considered acceptable. The bread may be okay if its crust is slightly more or less brown, but rejected if it is significantly dark (suggesting burnt) or light (suggesting underdone). What exactly are the limits of acceptability for each characteristic? That is decided through consumer research. Assuming, for our purposes that these limits are already specified, then if the measured value of a particular characteristic for a given loaf of bread falls within its upper specification limit and lower specification limit, it is considered acceptable.

During process validation the process is kept constant i.e. step sequence, parameter settings, etc. are fixed, while its inputs are varied between their extreme possible conditions. The thought is if the output of the process subjected to such extreme conditions of its inputs is within acceptable limits, then the output of the process with normal conditions of inputs will also be acceptable. The intent of this exercise is to demonstrate the robustness of the process to the natural variations in its inputs.

The design of experiments provides an efficient way to simultaneously vary every input between its extremes. For the bread making process in this example, there are 6 inputs: amount of bread flour, salt, vegetable oil, active dry yeast, white sugar and water. If we assume that each of these inputs will vary from their specified quantity as shown in the table below, then we can construct a two level six factor experiment for the process validation study.

For the purposes of this discussion, let’s assume that the product’s design fulfills the requirements of its intended use. This will allow us to focus on just its manufacturing process. How can you show that an output of the manufacturing process meets the requirements that define its fitness for use? One mechanism is to inspect and test every manufactured unit. And, so long as such activities do not destroy the manufactured unit in the process, it is a perfectly acceptable method to show its fitness for use. This sort of check is referred to as product verification.

However, inspection and testing of certain performance characteristics of a manufactured unit do destroy the unit in the process. Verifying such characteristics of each manufactured unit would not leave any units for use or sale. To address this issue, we have to look at data from samples of manufactured units, viewed through the lens of statistical theory, to draw conclusions about their overall population. This is the basis of process validation.

As long as the distribution of data points describing a particular performance characteristic of the product, collected from samples of manufactured units, falls within the limits that define the performance requirements for that particular product characteristic, we can be confident that the rest of the population of manufactured product meets those performance requirements as well. The theory of statistics provides us with a mechanism by which to quantitatively express the degree of our confidence that untested units will perform as expected.

An implicit point made in my assertion is that the manufacturing process is subjected to the range of variability present in its inputs. Each input to the manufacturing process has a distribution that describes its center and variation. These inputs interact with the manufacturing process and with each other as they are transformed into the output with its own distribution. However, the shape, location and spread of the output distribution is only revealed after significant data has been collected over time. And, it is the boundaries of this distribution when compared against requirements that demonstrates whether the manufacturing process can consistently produce units that are fit for use.

Companies do not have unlimited time or money to collect such data by conducting large numbers of manufacturing runs. And, such a large data set isn’t necessary either. Subjecting the manufacturing process to extreme values of each input will yield output values that represent the boundaries of the manufactured product. It is reasonable to expect that if the inputs are within their extreme values, the output will be within its boundary values. If the boundary values of the output are within the limits that define performance requirements, we can rest assured that the manufacturing process will produce units that are fit for the intended purpose. And, this manufacturing process can then be said to have been validated to produce the particular part.

A final thought: manufacturing processes have several inputs. It is not efficient to vary them one at a time. In fact, varying them one at a time doesn’t give you the complete picture of how they interact with the manufacturing process or with each other. Running controlled experiments that are properly designed can paint a more full picture. The science of the design of experiments should be the tool of choice when validating a process.

My aim is to march through the article, quoting the author and pointing out the misinformation. Hopefully this will help you understand process validation correctly or at least prevent you from learning the wrong things.

Because it is a repeatable process, laser welding can be statistically proven and easily validated.

Nothing about the laser welding process suggests that it is inherently a repeatable process. Like any process, if the inputs to the laser welding process vary, so will its output. That is why you need to perform a process validation: to demonstrate that expected variation in inputs to the process will still yield an output that meets performance requirements. There is nothing easy about this effort. It must be properly planned, executed with greatest care, and the resultant data analyzed with the objectivity of statistical tools. In doing so, nothing is proved statistically. Statistical analysis merely details the probabilities of occurrence. Those probabilities must then be evaluated in the context of business goals as either acceptable or not.

In a perfect world, you’d have the time to validate 100% of your samples.

It is obvious that the author does not understand the concept of sampling or sample size nor the difference between verification and validation. If you’re “validating 100% of your samples” i.e. inspecting every piece, you are performing verification and you’re not sampling.

When your company is faced with a validation issue, you’ve got a host of problems to deal with, from embarrassing to expensive.

Seriously? Embarrassment is the problem? Expense of the fallout of a validation issue is the problem? How about patient safety? Where does that fall in the spectrum of problems? Failure to validate a process that cannot be verified means you have no idea whether your product will perform as expected or whether it will fail at the most inopportune time e.g. in the middle of surgery. Just as the primary duty of a physician is to do no harm, the primary motivation of a medical device manufacturer should be to ensure that their product will perform as expected. That assurance is partly obtained through performing a manufacturing process validation and partly through continuing quality control activities. Embarrassment can be overcome and reputations can be rebuilt. Patients’ lives remain forever changed.

In the best-case scenario, your customers will require validation as part of the complete manufacturing process and will audit it closely.

If Tegra Medical truly believes this, then its customers need to reevaluate their relationship with the company. It is not your customers’ responsibility to ensure you are manufacturing good product. That responsibility is entirely yours. It should be your company’s culture to ensure that production is done properly such that your product performs as expected. So, the best-case scenario is that regardless of customer requirements you should perform process validation as part of your quality assurance activities.

When processes or parts are especially complex, validation provides a way to help control them. It enables real-time monitoring and process adjustments so you can improve processes statistically and evaluate your performance daily.

What nonsense! Process validation has nothing to do with the complexity of a part. The benefit of process validation is identical whether you are manufacturing a simple part or a complex one: assurance that it will function as expected.

Just what exactly is the author referring to when saying “validation provides a way to help control them.” Control what? If we’re talking about controlling process parameters, then quality control tools such as statistical process control (SPC) and run-rules are necessary.

Validation does not “enable real-time monitoring and process adjustments”. More importantly, when you’re conducting a process validation, process parameters should not be adjusted at all or you will contaminate the result you’re trying to validate. Process parameters’ operating windows should be established during process design; not during process validation.

The Quality System Regulation (QSR) known as 21 CFR Part 820 and ISO 13485:2003 require that validation include installation qualification (IQ), operational qualification (OQ), and process qualification (PQ).

No, they don’t. Neither 21 CFR Part 820 nor ISO 13485:2003 require that validations include IQ, OQ, and PQ. The regulations state:

Sec. 820.75 Process validation.

(a) Where the results of a process cannot be fully verified by subsequent inspection and test, the process shall be validated with a high degree of assurance and approved according to established procedures. The validation activities and results, including the date and signature of the individual(s) approving the validation and where appropriate the major equipment validated, shall be documented.

(b) Each manufacturer shall establish and maintain procedures for monitoring and control of process parameters for validated processes to ensure that the specified requirements continue to be met.

(1) Each manufacturer shall ensure that validated processes are performed by qualified individual(s).

(2) For validated processes, the monitoring and control methods and data, the date performed, and, where appropriate, the individual(s) performing the process or the major equipment used shall be documented.

(c) When changes or process deviations occur, the manufacturer shall review and evaluate the process and perform revalidation where appropriate. These activities shall be documented.

ISO 13485:2003 states:

7.5.2 Validation of processes for production and service provision

7.5.2.1 General requirements

The organization shall validate any processes for production and service provision where the resulting output cannot be verified by subsequent monitoring or measurement. This includes any processes where deficiencies become apparent only after the product is in use or the service has been delivered. Validation shall demonstrate the ability of these processes to achieve planned results. The organization shall establish arrangements for these processes including, as applicable

a) defined criteria for review and approval of the processes,

b) approval of equipment and qualification of personnel,

c) use of specific methods and procedures,

d) requirements for records (see 4.2.4), and

e) revalidation.

The organization shall establish documented procedures for the validation of the application of computer software (and changes to such software and/or its application) for production and service provision that affect the ability of the product to conform to specified requirements. Such software applications shall be validated prior to initial use.

Records of validation shall be maintained (see 4.2.4)

7.5.2.2 Particular requirements for sterile medical devices

The organization shall establish documented procedures for the validation of sterilization processes. Sterilization processes shall be validated prior to initial use. Records of validation of each sterilization process shall be maintained (see 4.2.4).

The Global Harmonization Task Force (GHTF) does recommends that validation activities be broken up into installation qualification, operational qualification and performance qualification. But, it is not required. And, yes, PQ stands for performance qualification; not process qualification as the author writes.

…all data are maintained in the company’s design history record (DHR)…

Obviously the author has confused a device history record (DHR) – a compilation of records containing the production history of a finished device – with a design history file (DHF) – a compilation of records which describes the design history of a finished device.

The entire section on laser welding IQ talks about equipment IQ. It does not address process IQ. An equipment qualification (equipment IQ, OQ, and PQ) is a portion of a process IQ which also includes, among other things, operator training for running the process, standard operating procedures for the process, a process risk analysis (FMEA), ensuring all process equipment are laid out properly, etc. A discussion of a proper process IQ is beyond the scope of this essay.

The author completely mischaracterizes the activity performed during process OQ. Suffice it to say that you shouldn’t be developing or designing your process (i.e. establishing operating windows) during validation. The process OQ is where you challenge the process by operating it at its inputs’ maximum and minimum values. These runs, performed most efficiently through the use of a properly designed experiment, should demonstrate that the process will yield an output that meets expectations even when the inputs are operating off their nominal values and at their extremes. Again, a discussion of proper process OQ is beyond the scope of this essay.

…process capability studies (known as gage repeatability and reproducibility (GR&R) or measurement systems evaluation…

Process capability studies are not GR&R studies. Process capability studies typically reflect the manufacturing process’s ability to make product within specification limits. GR&R studies are done on the measurement process, and do not have direct relationship to the manufacture of any given product.

This test uses a statistically significant sample plan such as a size of 60 parts based on a reliability and confidence level of 95%

Where does this sample size of 60 come from? For the life of me, I can’t figure it out.

The outcome of the test is torque or tensile data that shows with 100% accuracy how the material will hold up under different conditions.

There is no such thing as 100% certainty in the real world. Only degrees of confidence.

The author does a similar hack job in discussing laser welding PQ as he did with the IQ and OQ sections. Process performance qualification should be a final check of the process by running it at nominal levels of the process inputs. That is, during PQ you’re running production! Consider the initial batches as “risk production” if you will. The process performance here on out is monitored using statistical process control tools. Alas, a discussion of proper process PQ is also beyond the scope of this essay.

…run your laser welding parameters at the nominal condition three times in a row…

What is the statistical basis of running the process three times or whether those runs are consecutive? This is an industry myth that keeps being perpetuated over and over with no basis in fact.

I hope that I’ve successfully detailed where this article misinforms. It is throughout the entire body. Based on the type and scope of the misinformation, I can infer only that the author does not have a good understanding of process validation, how to perform it or what it is intended to achieve. Worse, his focus does not appear to be aligned with that of the FDA: patient safety. The fact that he is the vice president, quality assurance and regulatory affairs at Tegra Medical should give all of us pause. Additionally, we should all wonder about the vetting process MD+DI uses in deciding what to publish.

**

Correction: The gender of the author was corrected in the sentence “The author does a similar hack job in discussing laser welding PQ as she did with the IQ and OQ sections.” from “she” to “he”. The author is a man, not a woman. (11:10 AM, 25 Oct, 2011)

Consider a simple hypothetical example wherein I intend to make bread. The recipe calls for the bread to be baked in an oven at 400 F for 25 minutes. Applying this company’s point of view, the operational qualification of my oven would require only demonstrating that it functions at 400 F for 25 minutes. There is no need to validate that the oven is capable of functioning to its manufacturer’s claim of 170 F to 550 F. I believe this model for conducting an equipment operational qualification to be shortsighted at best and a total misunderstanding of what an operational qualification is at worst.

While such an approach fulfills the immediate need to demonstrate that my oven is capable of generating 400 F, such a limited test does not take into account my future potential needs. Perhaps better bread might be made by baking it at 350 F for 35 minutes. Or, I might want to make pizza – that requires my oven to operate at 550 F, the hottest it gets. Or, I might want to keep food warm in the oven at 170 F. An operational qualification of this type precludes me from using my oven at temperatures that haven’t been tested. I would have to repeat the operational qualification at other temperatures before use.

But, I believe that this approach to performing an operational qualification is actually more than just shortsighted. It is wrong. It is an incorrect interpretation of what a proper operational qualification is intended to demonstrate: that the equipment is capable of performing to its manufacturer’s claims. A proper operational qualification should be done such that the functionality of the equipment is evaluated at a minimum at the high & low settings of all its critical input controls. The design of experiments provides a statistically robust and cost efficient method to do this.

In an abstract sense an operational qualification is part of the receiving inspection process, albeit a complex one. And just as in the typical receiving inspection process, it is better to find defects or deficiencies in the equipment at this stage (even those that don’t directly impact our immediate application) than to discover them during production. Defects, like cracks, tend to migrate into the production operating zone over time.

At its most basic level an operational qualification has very little, if anything, to do with using the equipment for any given application. Operational qualification of my oven has very little to do with me using the oven to bake bread, make pizza or keep food warm. Validating that the equipment will perform as expected for a given application is done as part of a performance qualification (PQ) of the equipment. So, when my former employer performed an operational qualification of a tool at an application specific setting, it was actually performing a performance qualification; bypassing operational qualification.

When a proper operational qualification has been performed for a piece of equipment, it does not need to be repeated until the equipment undergoes some sort of preventive or breakdown maintenance when consumable or broken parts are replaced. As such, a company that intends to extend the use of a piece of equipment to multiple applications needs to only perform a performance qualification of the equipment for a given application’s settings before releasing the tool to run that new application.

It is disturbing when quality managers do not take the time to understand the purpose of a particular assurance activity. As you can imagine, this leads to wrong tests being run, wrong data being collected and wrong conclusions being drawn. The problem is compounded by their willingness to take shortcuts. Is getting to the wrong place faster better? It doesn’t make any sense to me why a company would not perform a check of the manufacturer’s claims when the effort required isn’t any greater and the long term benefits are significant.

The fundamental assumption that “the other side of innovation: SOLVING THE EXECUTION CHALLENGE” is based on is that your organization is attempting innovation initiatives beyond its current capabilities. Capabilities that you are not willing to develop internally. In other words you are attempting to climb Mt. Rainier, to use the authors’ opening example, when you are neither fit for the challenge nor possess the skills for it. Think about this for a second. Contemplate the likely outcome of such an attempt. If it doesn’t kill you it will most assuredly maim you leaving you worse off for having tried.

However, Vijay Govindarajan and Chris Trimble “see no reason why established organizations should be incapable of executing any innovation initiative”. So, what is the solution these authors dictate? After 10 years of field research at “[innovative companies] as diverse as Allstate, BMW, Harley-Davidson, IBM, Nucor, and Timberland”, they recommend that you “Build the Right Team” and “Run a Disciplined Experiment”. Let us understand something clearly: it doesn’t matter how many expert mountain guides you hire or how well you plan your expedition, if you are not fit, if you do not possess the necessary skills, it will fail… disastrously. So, shame on the authors for making a flawed assumption and then impelling organizations to attempt such challenges.

To be fair, they have made valid observations of several crucial shortcomings in organizations today:

• It is not an organization’s creativity and technology that falls short, it is its management’s capability: leaders just aren’t trained to drive innovation.
• Organizations vest most of their innovation efforts in defining opportunity and not much in executing it. This is tied to the previous point.
• As companies mature, they disengage from innovation efforts, relying on profit from increasing operational efficiency.

These are problems that organizations have to overcome internally through rigorous self reflection which leads to creating projects that (re)build the organization’s fitness and skills. They can thus expand their core competencies and cultivate deep domain knowledge necessary to address the total challenge of innovation. Not do it through hired outside experts or mergers and acquisitions.

Hugely successful innovation initiatives in recent memory such as Toyota’s Lexus or Scion, or Apple’s iPod, iPhone, iPad or iCloud have been internal to their respective companies. Neither Toyota nor Apple were first to market with these products. Each took its time to develop domain knowledge; stretching itself through multiple learning cycles before introducing class redefining products.

The authors underestimate and trivialize the value of such continuous improvement programs even when their own studies at Nucor and Deere & Company demonstrate successful outcomes. In fact they dismiss their own observations of successful innovation initiatives at many companies in favor of “What If?” scenarios. Never mind whether these hypothetical initiatives were ones these companies might have pursued but didn’t for lack of capability. They never say so.

The one example the authors cite where a transformative innovation initiative was required: the New York Times. But, this was necessary as the landscape of the entire newspaper industry was unexpectedly and fundamentally changed by an unaccounted force: the internet. An extremely rare event that no book or expert can help to plan for.

In any case, given that the book’s fundamental premise is faulty, the structure built upon it is rickety at best. Don’t get me wrong, there are some good ideas, but they are either poorly communicated or poorly reasoned or both. The authors assume an unjustified authoritative tone that often patronize the reader on his/her complaint (constructed in the authors’ imagination). There is a smugness in the manner they criticize their partner companies’ approach to innovation. I wonder if any of them will have these researchers back to work with them again. And, they fail to properly attribute their learning cycle – Plan-Do-Study-Act – to its developers and advocates: Walter Shewhart and W Edward Deming.

My suggestion is you pass on reading this book. It teaches the wrong lessons. Learn the proper way to execute innovation initiatives by benchmarking the best in class. Two excellent reads are “The Toyota Way” & “The Elegant Solution”. Develop your organization’s management foundations and knowledge building efforts with “Out of the Crisis”. There is a tough climb ahead, but not an insurmountable one provided you start preparing for it now.

Pink’s model of motivation is elegant. Made up of three parts, it is simple yet captures key factors that drive motivation:

1. People have a desire for autonomy with respect to task, time, team and technique,
2. People want to gain mastery over what they do, and
3. People crave to be a part of something bigger than themselves.

It holds an intuitive appeal for me. I suspect that might be a function of the type of work I do and my experience doing it. For as long as I have been working I have been dissatisfied with the management style used by each of my employers. Among other things, I found them lacking in the way they chose to motivate. The studies Pink disccusses in the book helped me understand why some of my job functions went from ‘play’ to ‘work’. They also explain why, from time to time, they reverted back from ‘work’ to ‘play’. His model crystallized what I am looking for as a knowledge worker: An opportunity to choose what I do, when I do it, with whom I do it and how I do it. An opportunity to do it well and get better at it. And, an opportunity to do it in the cause of something greater than myself.

“Drive: The Surprising Truth About What Motivates Us” is a well written book. Pink doesn’t make the reader work at it. He weaves stories that hook a reader and pull him along without being wiser to it. It takes lots of skill and effort to pull that off. Pink builds his case for his model of motivation logically, never losing the reader in the process. He doesn’t just cite studies, he explains their findings. Then he takes the additional step to explain the implication of those findings as he works to incorporate them into his model. So, the net result is an extremely reader friendly book that informs and educates. I place this book in the same space as Jonah Leher‘s “How We Decide“, Dan Ariely‘s “Predictably Irrational: The Hidden Forces That Shape Our Decisions“, Ori Brafman‘s “Sway: The Irresistible Pull of Irrational Behavior“.

Footnotes:

• Check out a really wild animation of Dan Pink’s talk here.
• Check out Dan Pink’s TED talk here.
• Check out Dan Pink’s RSA talk here (related to the animation above).

Excepting in the presence of active research in a pure science, the applications of the science tend to drop into a deadly rut of unthinking routine, incapable of progress beyond a limited range predetermined by accomplishments of pure science, and are in constant danger of falling into the hands of people who do not really understand the tools that they are working with and who are out of touch with those that do…

– Harold Hotelling, Memorandum to the Government of India, 24 February 1940

That is the predicament I found myself in.

I was hired to support the company’s efforts to validate their processes in preparation for registering their manufacturing facility with the FDA. The FDA defines process validation as:

…establishing by objective evidence that a process consistently produces a result or product meeting its predetermined specifications.

– 21CFR820.3(z)(1)

While the definition is succinct, process validation is not a trivial task. And, contrary to the belief of management that is not literate in the quality system regulations or the subject of quality assurance, it is certainly not something you can “whip out”. It requires an understanding of the process in question – its key inputs and the key attributes of its output – coupled with an understanding of statistical principles such as the design of experiments (DOE) and analysis of variance (ANOVA) necessary to generate the objective evidence that will establish for the company and the FDA that the “process consistently produces a result meeting its predetermined specifications”. And, it also helps to have a proper plan that allows management to identify and allocate the resources required to successfully meet its objectives. At a minimum, a basic project plan should include a detailed checklist of actions items with clearly defined owners and due dates.

But, when management does not understand or trivializes these requirements, it makes decisions that endanger the best interests of the company. Unnecessary risk is assumed. Resources are wasted. Workers are put in a chaotic situation that is the primary source of much of their frustration and fear. So it should not be a surprise when “new” action items “pop up” in crunch time; when there is confusion around the ownership of a task, or when deadlines are missed and missed again causing tensions to flare. And, while these gaffes in project management might be overcome through working harder (translated as management by proclamation – “because I said so” – and long hours), there is no hope to compensate for poorly designed experiments with insufficiently identified process parameters through brute force. Without the right data there is no way to show the process’s capability or even that it is in statistical control.

So there it is – a fork in the road. A choice that we are all confronted with more often than we would like: follow the whack-a-mole tactics of a management team without a strategy or as Dr. Hotelling put it “people who do not really understand the tools that they are working with and who are out of touch with those that do”, or make a swift exit to focus on developing your personal knowledge and skills while searching for a better opportunity. As scary as it seems, the latter choice will always lead to a better outcome. At least that has been my experience.