Sample Size Determination

---

OK, we know you need to test this system of widgets to make sure they meet spec, but how many do I need to test? Always a rabbit hole isn’t it…? There’s the boss pressing the schedule in terms of both testing time and possibly the time to build those systems of widgets. She/he also isn’t missing the opportunity to remind me of the cost implications to both of those factors. Then there are the statisticians among us who are screaming “you can’t simply say you’ll test 30!!!”

Well, let’s take a moment to push schedules and budgets aside and engage with that statistician. Knowing any of us can at some point be called to the principal’s office (AKA an audit!), we better put a plan together and explain our decisions. Oh, and do the math. I promise!

What’s paramount in our sampling plans is that they are Risk-based, that they’re statistically based (where relevant), and that they can be defended in both Audits and Regulatory submissions such as 510(K)’s or PMA’s.

So let’s now bear down on recommending the best, most defensible sample size for testing, without the boss in the room. Some of the things we need to consider include whether the data is pass/fail or attribute (yes, there’s a power switch), or if it’s variable data. Variable data is data where you can calculate a mean and standard deviation, and this is preferred wherever possible as this type of data gives us more information. If for example we have a tensile strength specification we could just test to the specification and say all samples withstood 12 lbs, but it would be much more informative to say all samples had a mean of 14.5 +/- .2 lbs.

Another factor depends greatly on how the requirements are worded. Often times design inputs are created without this in mind, so sometimes some interpretation is needed. 

For example, is it important that the entire population be above a particular specification, or is it sufficient that a percentage of the population (e.g. 95%) meet or exceed the specification? We promised no math, but the implications here are enormous in terms of sample size. Note also that the FDA accepts a failure rate of one in a million regarding sterilization. In other words, they don’t expect perfection.

Ready for another wrinkle? Disposables are relatively straightforward, but what about durable equipment? If I need a sample size of 12, is that 12 actuations of one piece of equipment or do I need 12 separate devices? What if it’s something super expensive like an MRI machine? Whoops….let’s not talk costs, but it’s a great point.

What’s key to consider is where you think the variability will come from. Remember the reason we test more than one sample is because we expect variation. The key is to think about the source of the variability. (See? No math…). 

Further, let’s say you have that MRI machine and confess that the variability may come from machine to machine. What’s strongly recommended here is to evaluate sources of variation at subsystem and component levels. Verifying individual circuit boards or mechanism subassemblies ensures a robust foundation prior to getting to the more complex system level. This too would seemingly be much cheaper and easier. Then once at the system level, you only need to consider subassembly interactions as a source of variation. Auditors will look for this robust approach to V&V. Robust subsystem sampling will also empower you to justify a smaller sample size at the system level.

OK, got time for one more wrinkle? Consider your system of widgets as including disposables that plug into a durable machine. Are those 12 samples we need all disposables? Do I need 12 machines and 12 sets of disposables? Again let’s go back to the foundation of understanding the sources of variation. It may not be sufficient to test 12 machines and 12 disposable sets, because each machine will have only been evaluated once with a single disposable set. Carefully examining the durable/disposable interaction can be the key to answering this question.

Well, this all sounds like a rabbit hole indeed, but we can help guide you back to daylight! 

Please feel free to get in touch with us to discuss your applications, and how we might be able to help you. Whether it’s proactively helping your team build a strategy to support a submission, or if you need help with audit preparedness we can help. Lastly you might be dealing with audit findings, observations or office actions. 

Remediation is no fun, but we have a great deal of experience in moving you to positive outcomes.