Chris Haney (christopher.haney@forensus.com) is a Managing Director in the Richmond office of the Forensus Group, LLC in Richmond, VA. Chip Hancock (chancock@hancockdaniel.com) is an Attorney in the Richmond office of Hancock Daniel, PC in Glen Allen, VA.
Statistical sampling is routinely used in audits, investigations, and self-disclosure filings when seeking to reach conclusions about large volumes of data in a cost-effective manner. Compliance auditors and investigators also regularly use sampling analysis to monitor compliance objectives and to quantify potential repayment obligations. As organizations strive to minimize the cost and effort involved in these analyses, probe samples are an increasingly common tool used to improve efficiency and to reduce administrative burden.
A probe sample can be helpful in identifying risk and quickly evaluating whether a full statistical sampling is required. More importantly, when planned and implemented properly, a probe sample can be incorporated as part of future sampling analysis to eliminate duplicated efforts and to further minimize cost and effort. This article addresses the role of probe samples, their proper design and implementation, and considerations for developing valid and defensible analyses of healthcare claims.
Statistical sampling overview
It is worthwhile to understand the purpose and role of statistical sampling in the first place. Statistical sampling analysis is most commonly used when one seeks to infer useful information about a relatively large population of data without examining every unit in the population. Instead, sampling analysis examines only a subset of the population (i.e., a sample). As part of the analysis, estimation or extrapolation is the procedure by which measured characteristics of the sample yield estimates, inferentially, about the population from which the sample was drawn. The term “probability” or “statistical” sampling arises from the fact that the sample is selected in a manner that is predictable in terms of the laws of probability, which eliminates both conscious and unconscious selection biason the part of individuals who perform the sample selection. The sample must be obtained in a particular way (i.e., randomly) to be objective and defensible.
Statistical sampling has a variety of applications in the healthcare compliance context. For example, internal auditors routinely use sampling to render reasonable conclusions about a set of transactions or patient claims where a complete analysis of those items might otherwise require unrealistic expense or effort. Similarly, the Centers for Medicare & Medicaid Services (CMS) routinely uses sampling to conduct audits of payments, and corporate integrity agreements (CIAs) commonly include sampling provisions as part of periodic claim review procedures.
The use of statistical sampling in litigation is a growing field as well. Testifying experts routinely use sampling techniques to estimate commercial damages, overpayment amounts, and other characteristics of relevant data. Beyond compliance and litigation, sampling is also frequently used by business leaders to increase efficiency and improve quality in internal operations. Sampling may also be used in a variety of other business contexts, such as to help estimate inventories, to evaluate the rate of work output, to estimate the length of equipment life, or to perform a variety of other forms of operational research.
A “full” statistically valid sampling analysis may involve the selection of several hundred or even thousands of sample units (e.g., patients, claims) in order to achieve results within a specified degree of uncertainty, typically described in terms of confidence and precision. A full sampling may be simpler than reviewing 100% of the population, but the analysis may still be time-consuming and expensive, especially if the objectives of a particular audit are somewhat uncertain. For example, a hypothetical analyst who is auditing a healthcare system with 12 facilities and 45 providers may need to sample more than 300 patient records to conduct a “full” sampling analysis in order to estimate treatment coding or billing errors. This can be cost prohibitive, particularly for routine audits where the risk of errors or overpayments may be low.
Understanding a probe sample
Enter the probe sample, sometimes referred to as a pilot or exploratory sample. Unlike a full sample, probe samples are not typically employed to achieve specified levels of confidence and precision. Instead, they are generally used to determine an error rate, and thereby to indicate whether further analysis (i.e., a full sample) is warranted. Additionally, without the emphasis on specified levels of precision and confidence, probe samples can generally be much smaller in size than a full sample. For example, an analyst may select 30 sample units for a probe sample and then audit those 30 claims to calculate the percentage of claims possessing errors. If the percentage exceeds some established threshold, say 5% or 10%, the analyst could set aside the data for a full sampling. However, if the error rate does not exceed the threshold, the analyst might conclude the analysis, finding that insufficient errors exist to justify further investigation. This conclusion could save significant cost and effort, while also allowing the analyst to focus on other areas of greater risk in the organization. Still, individual errors identified through a probe sample should be appropriately addressed and remediated, even in cases where the overall error rate does not merit further analysis.
Although a smaller sample size is an obvious benefit of the probe sample, another advantage is the ability to include findings from the probe sample as part of a subsequent full sampling, if a follow-on analysis is necessary. The Department of Health and Human Services, Office of Inspector General (OIG) addresses this topic by allowing a probe sample’s findings to be included in the full sample if statistically appropriate.[1] This can save significant cost and effort during internal audit activities and compliance investigations by reusing probe sample findings to minimize further self-disclosure analysis or overpayment testing. For example, if an analyst determines a sample size of 200 claims to be necessary in the full sample, they might first rely on the results from the 50 claims reviewed in the probe sample, thereby reducing the claims needed for additional review to 150. The assumption here is that the probe sample is statistically appropriate for inclusion in the full sample.
Similarly, some OIG CIAs include claim review procedures that require a “discovery sample.”[2] Although discovery samples are technically distinct from probe samples (i.e., they are intended to detect at leastone error, rather than to estimate an overall error rate[3] ), both types are useful in estimating the performance of the total population. In fact, OIG’s Frequently Asked Questions (FAQs) refer to their discovery sample as if it were a probe sample, by calculating an error rate to determine if further action is required. Whether performing audits, self-disclosure analysis, or potential overpayment analysis, probe samples are regularly employed to minimize cost and administrative burden in healthcare matters.
How is a probe sample determined?
Much like a full statistical sampling, designing and executing a probe sample should be approached methodically to ensure the conclusions of the analysis are defensible and reliable. A variety of steps exist to properly design, execute, and interpret valid sampling analyses, but this article focuses on two of the most common questions for probe samples: (1) What size is sufficient? and (2) How do I choose the sample? Failure to properly address these questions can result in a probe sample that doesn’t achieve the objectives of the analysis, and thereby wastes time and resources.
What size is sufficient?
Although a sample size of 30 is commonly used as a rule of thumb for probe samples, that size may not be appropriate in certain situations. In fact, a single industry standard does not exist for all circumstances, and a sufficient sample size ultimately depends on the objectives of the analysis. With that said, a probe sample of 30–50 units is generally sufficient, and several relevant benchmarks support that range.
To begin, OIG explicitly stated that a probe sample must include at least 30 sample units in its original Self-Disclosure Protocol (SDP) published in 1998.[4] That guidance was later updated and the prescribed probe sample size was removed, but the updated SDP still permits the use of probe samples when “statistically appropriate.”[5] The CMS Program Integrity Manual (MPIM) indicates that probe samples may be used by its contractors, and it stipulates a probe sample size of “generally 20– 40” and also clarifies that “the [contractors] shall ensure that such a sample is large enough to provide confidence in the result, but small enough to limit administrative burden.”[6]
OIG also specifies a sample size of 50 units for discovery samples in its CIA FAQs.[7] Recall that a CIA discovery sample is functionally the same as a probe sample; therefore, OIG’s CIA guidance is another yardstick for probe sample size. Based on these benchmarks, a probe sample size of 30– 50 units is reasonable for healthcare audits, self-disclosure analysis, and CIA claim reviews, absent any superseding contractual or regulatory requirements.
How do I choose the sample?
Beyond the size of a probe sample, it is also worthwhile to understand the methods by which the probe sample is selected. Designing a probe sample with the mind-set that it may later need to be incorporated in a “full” sampling is imperative; therefore, the sample should be properly randomized and selected. Without this foresight, the probe sample may not be statistically appropriate for inclusion in future analysis. Samples obtained by any method other than random selection are generally considered to be “judgment” samples, which typically result from haphazard selection or by means of convenience (i.e., choosing charts from the top of a pile). The results of these judgment samples may be informative, but they are not appropriate for use in statistical sampling and may require analysts to duplicate efforts and, ultimately, sample greater numbers.
Too many analysts wait until the probe sample has been reviewed before considering whether the probe results can be re-used. By then, it is often too late to avoid duplication of sampling efforts. Analysts should employ appropriate sample design and planning considerations in advance to minimize wasted time and resources. Similarly, the use of statistical software such as RAT-STATS or a similar program can help to ensure the probe sample is properly randomized and selected.
Considerations for probe sample analyses
Beyond the technical nuance of sampling analysis, a variety of other considerations exist that can affect the ultimate utility of sampling conclusions. To begin with, an analyst should consider any internal procedures, contractual obligations, or statutory requirements that might affect the design or permissibility of their sampling analysis (e.g., prescribed sample sizes or the preclusion of certain sampling analysis), which might supersede otherwise prudent sampling plans.
Sufficient documentation of an analyst’s planning, design, and execution should also be maintained to provide a reasonable record of the work performed. Specifically, this documentation should memorialize any and all decision-making, calculations, and findings for each step of the sampling process. Without these details, it may be impossible to evaluate whether a particular analysis is valid or reliable and, subsequently, whether the conclusions of the analysis are meaningful. Providers and suppliers should also be mindful to retain all audit and refund documentation related to sample findings to ensure overpayments are not re-audited or reimbursed again in subsequent periods. Ultimately, sampling analysis is a single tool in an organization’s tool kit, and analysts should routinely evaluate the objectives of the organization to ensure the appropriate tools are in use.
Conclusion
Probe samples can be useful in reducing the amount of time and effort required in matters that might otherwise involve a full sampling analysis. To best utilize probe samples, they should be designed and selected in a manner that will allow them to be incorporated into a full analysis, if necessary. When properly executed, probe samples can be used to develop effective and efficient evidence in healthcare audits, investigations, self-disclosure, and CIA claim reviews.
Takeaways
-
Properly planned and executed probe samples can minimize cost and effort.
-
Probe samples are regularly used in healthcare audits, self-disclosures, and CIAs.
-
Properly designed probe samples can be included in “full” statistical samples.
-
Analysts should ensure probe samples are appropriately sized and selected.
-
Early planning can save significant time and expense for statistical sampling.
