A [Breakdown in] Validation of Quality


Recently MD+DI (Medical Device and Diagnostic Industry) published “A Validation of Quality”. I would like to think it was motivated by a desire to inform and educate the readers of the value of conducting proper process validations. But, instead the author, Jean Mattar, misinforms the readers and perpetuates validation mythology. And, it seems no one at MD+DI bothered to do a spelling or grammar check, much less fact check what they were publishing. This is particularly troublesome as many medical device industry professionals use such articles in MD+DI as reference when setting up their quality systems or performing various quality assurance activities like process validation.

My aim is to march through the article, quoting the author and pointing out the misinformation. Hopefully this will help you understand process validation correctly or at least prevent you from learning the wrong things.

Because it is a repeatable process, laser welding can be statistically proven and easily validated.

Nothing about the laser welding process suggests that it is inherently a repeatable process. Like any process, if the inputs to the laser welding process vary, so will its output. That is why you need to perform a process validation: to demonstrate that expected variation in inputs to the process will still yield an output that meets performance requirements. There is nothing easy about this effort. It must be properly planned, executed with greatest care, and the resultant data analyzed with the objectivity of statistical tools. In doing so, nothing is proved statistically. Statistical analysis merely details the probabilities of occurrence. Those probabilities must then be evaluated in the context of business goals as either acceptable or not.

In a perfect world, you’d have the time to validate 100% of your samples.

It is obvious that the author does not understand the concept of sampling or sample size nor the difference between verification and validation. If you’re “validating 100% of your samples” i.e. inspecting every piece, you are performing verification and you’re not sampling.

When your company is faced with a validation issue, you’ve got a host of problems to deal with, from embarrassing to expensive.

Seriously? Embarrassment is the problem? Expense of the fallout of a validation issue is the problem? How about patient safety? Where does that fall in the spectrum of problems? Failure to validate a process that cannot be verified means you have no idea whether your product will perform as expected or whether it will fail at the most inopportune time e.g. in the middle of surgery. Just as the primary duty of a physician is to do no harm, the primary motivation of a medical device manufacturer should be to ensure that their product will perform as expected. That assurance is partly obtained through performing a manufacturing process validation and partly through continuing quality control activities. Embarrassment can be overcome and reputations can be rebuilt. Patients’ lives remain forever changed.

In the best-case scenario, your customers will require validation as part of the complete manufacturing process and will audit it closely.

If Tegra Medical truly believes this, then its customers need to reevaluate their relationship with the company. It is not your customers’ responsibility to ensure you are manufacturing good product. That responsibility is entirely yours. It should be your company’s culture to ensure that production is done properly such that your product performs as expected. So, the best-case scenario is that regardless of customer requirements you should perform process validation as part of your quality assurance activities.

When processes or parts are especially complex, validation provides a way to help control them. It enables real-time monitoring and process adjustments so you can improve processes statistically and evaluate your performance daily.

What nonsense! Process validation has nothing to do with the complexity of a part. The benefit of process validation is identical whether you are manufacturing a simple part or a complex one: assurance that it will function as expected.

Just what exactly is the author referring to when saying “validation provides a way to help control them.” Control what? If we’re talking about controlling process parameters, then quality control tools such as statistical process control (SPC) and run-rules are necessary.

Validation does not “enable real-time monitoring and process adjustments”. More importantly, when you’re conducting a process validation, process parameters should not be adjusted at all or you will contaminate the result you’re trying to validate. Process parameters’ operating windows should be established during process design; not during process validation.

The Quality System Regulation (QSR) known as 21 CFR Part 820 and ISO 13485:2003 require that validation include installation qualification (IQ), operational qualification (OQ), and process qualification (PQ).

No, they don’t. Neither 21 CFR Part 820 nor ISO 13485:2003 require that validations include IQ, OQ, and PQ. The regulations state:

Sec. 820.75 Process validation.

(a) Where the results of a process cannot be fully verified by subsequent inspection and test, the process shall be validated with a high degree of assurance and approved according to established procedures. The validation activities and results, including the date and signature of the individual(s) approving the validation and where appropriate the major equipment validated, shall be documented.

(b) Each manufacturer shall establish and maintain procedures for monitoring and control of process parameters for validated processes to ensure that the specified requirements continue to be met.

(1) Each manufacturer shall ensure that validated processes are performed by qualified individual(s).

(2) For validated processes, the monitoring and control methods and data, the date performed, and, where appropriate, the individual(s) performing the process or the major equipment used shall be documented.

(c) When changes or process deviations occur, the manufacturer shall review and evaluate the process and perform revalidation where appropriate. These activities shall be documented.

ISO 13485:2003 states:

7.5.2 Validation of processes for production and service provision

7.5.2.1 General requirements

The organization shall validate any processes for production and service provision where the resulting output cannot be verified by subsequent monitoring or measurement. This includes any processes where deficiencies become apparent only after the product is in use or the service has been delivered. Validation shall demonstrate the ability of these processes to achieve planned results. The organization shall establish arrangements for these processes including, as applicable

a) defined criteria for review and approval of the processes,

b) approval of equipment and qualification of personnel,

c) use of specific methods and procedures,

d) requirements for records (see 4.2.4), and

e) revalidation.

The organization shall establish documented procedures for the validation of the application of computer software (and changes to such software and/or its application) for production and service provision that affect the ability of the product to conform to specified requirements. Such software applications shall be validated prior to initial use.

Records of validation shall be maintained (see 4.2.4)

7.5.2.2 Particular requirements for sterile medical devices

The organization shall establish documented procedures for the validation of sterilization processes. Sterilization processes shall be validated prior to initial use. Records of validation of each sterilization process shall be maintained (see 4.2.4).

The Global Harmonization Task Force (GHTF) does recommends that validation activities be broken up into installation qualification, operational qualification and performance qualification. But, it is not required. And, yes, PQ stands for performance qualification; not process qualification as the author writes.

…all data are maintained in the company’s design history record (DHR)…

Obviously the author has confused a device history record (DHR) – a compilation of records containing the production history of a finished device – with a design history file (DHF) – a compilation of records which describes the design history of a finished device.

The entire section on laser welding IQ talks about equipment IQ. It does not address process IQ. An equipment qualification (equipment IQ, OQ, and PQ) is a portion of a process IQ which also includes, among other things, operator training for running the process, standard operating procedures for the process, a process risk analysis (FMEA), ensuring all process equipment are laid out properly, etc. A discussion of a proper process IQ is beyond the scope of this essay.

The author completely mischaracterizes the activity performed during process OQ. Suffice it to say that you shouldn’t be developing or designing your process (i.e. establishing operating windows) during validation. The process OQ is where you challenge the process by operating it at its inputs’ maximum and minimum values. These runs, performed most efficiently through the use of a properly designed experiment, should demonstrate that the process will yield an output that meets expectations even when the inputs are operating off their nominal values and at their extremes. Again, a discussion of proper process OQ is beyond the scope of this essay.

…process capability studies (known as gage repeatability and reproducibility (GR&R) or measurement systems evaluation…

Process capability studies are not GR&R studies. Process capability studies typically reflect the manufacturing process’s ability to make product within specification limits. GR&R studies are done on the measurement process, and do not have direct relationship to the manufacture of any given product.

This test uses a statistically significant sample plan such as a size of 60 parts based on a reliability and confidence level of 95%

Where does this sample size of 60 come from? For the life of me, I can’t figure it out.

The outcome of the test is torque or tensile data that shows with 100% accuracy how the material will hold up under different conditions.

There is no such thing as 100% certainty in the real world. Only degrees of confidence.

The author does a similar hack job in discussing laser welding PQ as he did with the IQ and OQ sections. Process performance qualification should be a final check of the process by running it at nominal levels of the process inputs. That is, during PQ you’re running production! Consider the initial batches as “risk production” if you will. The process performance here on out is monitored using statistical process control tools. Alas, a discussion of proper process PQ is also beyond the scope of this essay.

…run your laser welding parameters at the nominal condition three times in a row…

What is the statistical basis of running the process three times or whether those runs are consecutive? This is an industry myth that keeps being perpetuated over and over with no basis in fact.

I hope that I’ve successfully detailed where this article misinforms. It is throughout the entire body. Based on the type and scope of the misinformation, I can infer only that the author does not have a good understanding of process validation, how to perform it or what it is intended to achieve. Worse, his focus does not appear to be aligned with that of the FDA: patient safety. The fact that he is the vice president, quality assurance and regulatory affairs at Tegra Medical should give all of us pause. Additionally, we should all wonder about the vetting process MD+DI uses in deciding what to publish.

 

**

Correction: The gender of the author was corrected in the sentence “The author does a similar hack job in discussing laser welding PQ as she did with the IQ and OQ sections.” from “she” to “he”. The author is a man, not a woman. (11:10 AM, 25 Oct, 2011)

About these ads

2 responses

  1. Dear Shrikant Kalegaonkar:

    First of all Jean is a he and not a she (not that this is important). What is more important is that you have taken upon yourself to assess what my qualifications are, when in fact you made numerous unprofessional statements and assumptions. The topic was supposed to be a top level guide and not to get into the details. As a professional and most importantly in respecting other professionals like you, I would have had the courtesy to pick up the phone and had a chat rather than posting such reply. If you had any knowledge of confidence and interval percentages, you would have know that at a 95% confidence level and 95% reliability level the required sample size would be 59. No where in the article a reference was made that GR&R are the same as process capability studies…..We did not want to get into the nitty gritty, however if you read the article carefully, you would have noticed that our OQ study is aimed around low, nominal and high parameter settings to allow us to understand the possible variation in our process regarding process inputs and what potentially the output may be…..than the PQ is run at nominal settings (you missed this completely as well). We did not mentioned what other things we do during the process validation/verification and during in-process because the article would have been too long, rather you decided to publish your response without obtaining further information. Respectfully, if I was you, I would not have taken the road you have taken, because as professionals and as human beings we should not make assumptions without knowing all the facts, and above all we should should respect one another. If you believe that you are better, more qualified and knowledgeable, you should have picked up the phone and had a discussion with me, at least that would have been the only right thing to do. Anyway, I still appreciate your feedback and time. Professionals teach and learn from one another.
    Respectfully,

    Jean (John) Mattar.

    1. Dear Mr. Mattar,

      Thank you for your comment. While I disagree with almost the entirety of it, I have chosen to post it in the spirit of openness and fairness.

      First a few comments on your suggestion that I was disrespectful and unprofessional in my post. It is true that my tone was acerbic, and intentionally so. It was motivated by my frustration at people in leadership roles, who I expect to know better, doing sloppy work and perpetuating false ideas. There is nothing disrespectful or unprofessional about pointing that out or questioning the mindset of the person proposing them. It is unreasonable to expect that such criticism be cleared through the author before being posted.

      Now on to the points you are making:

      a. “… Jean is a he and not a she…”

      The one instance where I referred to you as “she” was a typo that will be corrected.

      b. “… taken upon yourself to assess what my qualifications are, when in fact you made numerous unprofessional statements and assumptions.”

      My extrapolation of your qualifications are based on what you wrote and the position you hold as Vice President of Quality and Regulatory Affairs at Tegra Medical. It is my opinion that someone in your leadership role should have not just deep knowledge of quality standards and regulations, but that you use it to improve the state of industry. It is your responsibility to spread the right message. Instead, your article was sloppy & full of misinformation that, sadly, many will use as a reference to design, develop and implement validation programs in their own companies. So, while my critique seems harsh, I expect more from people in leadership roles.

      c. “The topic was supposed to be a top level guide and not to get into the details.”

      I had recognized that the article was an overview of process validation. That is what made your false points even more egregious. The fundamentals were wrong!

      d. “As a professional and most importantly in respecting other professionals like you, I would have had the courtesy to pick up the phone and had a chat rather than posting such reply. ”

      When MD+DI published your article, it became part of the public domain. As such, it was open to criticism from people like me. I do not feel there is anything unprofessional about responding to it before clearing my criticism with you.

      e. “If you had any knowledge of confidence and interval percentages, you would have know that at a 95% confidence level and 95% reliability level the required sample size would be 59.”

      I find your assumption that the reader of this “general overview” should have knowledge that 95% confidence & reliability equals 59 samples unreasonable. Nevertheless, I will take it as homework to verify your claim.

      f. “No where in the article a reference was made that GR&R are the same as process capability studies.”

      I refer you to the quote from your article: “…process capability studies (known as gage repeatability and reproducibility (GR&R) or measurement system evaluation)…”.

      It is under the “Laser Welding OQ” heading, 4th paragraph, 3rd line.

      g. “We did not want to get into the nitty gritty, however if you read the article carefully, you would have noticed that our OQ study is aimed around low, nominal and high parameter settings to allow us to understand the possible variation in our process regarding process inputs and what potentially the output may be”

      I read the article again, carefully as you suggested, and I still cannot find where you suggest that your OQ study is aimed at low, nominal and high parameter settings. Feel free to point me to the location in your article.

      h. “…than the PQ is run at nominal settings (you missed this completely as well).”

      No. I didn’t miss the point. In fact, I quoted you in my post as saying “…run your laser welding parameters at the nominal condition three times in a row…”. I was questioning your statement that the process be run three times in a row to qualify its performance.

      i. “We did not mentioned what other things we do during the process validation/verification and during in-process because the article would have been too long,…”

      The issue isn’t with the level of detail in your article. It is in the false points it makes.

      Mr. Mattar, I am sure you had good intentions when you wrote your piece. The worker on the line has good intentions when s/he is doing the work. But, good intentions are not a substitute for deep understanding of the material and attention to detail. You, and others in leadership roles like you, need to not just set the bar high for others, but live up to that bar yourself.

      Sincerely,
      Shrikant Kalegaonkar

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 26 other followers

%d bloggers like this: