October 7, 2019 in INFORMS News

Medicine: It may be precise but is it accurate?

An introduction to the real-world complexities that face clinical practice and drug development.

Michael Liebman

SHARE: PRINT ARTICLE:

https://doi.org/10.1287/orms.2019.05.04

Semantically, the terms “precision” and “accuracy” are closely linked. The definition of “precision” incorporates the concept of “accuracy.” Science, however, has effectively differentiated these concepts: “precision,” i.e., clustered around a specific value, invokes a local accuracy; “accuracy,” i.e., how far that local value may be from the true value, invokes a global accuracy. In archery, how tightly the arrows cluster around a single point reflects their precision, but how far that point is from the bullseye reflects their accuracy. In medicine, ideally, we would like to have both, but is that achievable, given the complexity of both the underlying biology and healthcare ecosystem?

Two questions we need to confront: Do we let technology define the problem or do we identify the real question and search for technology(ies) that can address it? When we reduce the system’s complexity to analyze it, do we lose critical characteristics/factors of the system, even if we can’t address them yet?

While the National Institutes of Health (NIH) defines precision medicine [1] as “[taking] into account individual variability in genes, environment and lifestyle for each person,” conventionally (except for the best practitioners) it has tended to focus on genomics to identify disease biomarkers, as risk factors, diagnostics and therapeutic targets. The technologic capabilities that enable this emphasis may have a negative impact, however, on attaining global accuracy, because disease is a process and not simply a state, and methodologies to assess dynamic processes differ from those that assess static ones.

It is critical in medicine to evaluate and close the gap between local and global accuracies to improve patient care and outcome and enhance efficiency and efficacy in healthcare. This requires elements of both systems- and design-thinking and is generally applicable to problems across many domains. The practice of medicine and development of both therapeutics and diagnostics have not evolved in lockstep in evolving from a reductionist approach.

**Figure 1:** Neither clinical practice nor pharmaceutical development can fully embrace
the complexity of the underlying biology and healthcare ecosystem.

While neither clinical practice nor pharmaceutical development is equipped to fully embrace this true complexity, it is real and presents challenges and limitations in our ability to manage the “whole patient.” This is highlighted by several observations:

Failures in drug development. The overall success rate for drug development, from early development to regulatory approval, is ~10 percent [2].

Failures in drug efficacy. A new study by the German Institute for Quality and Efficiency in Health Care declares that more than half of all new drugs do not add benefit [3].

Misdiagnosis in the clinic. The U.S. Institute of Medicine published the observation that the major misdiagnosis rate stands at 10 percent. This does not consider that most diseases are either complex disorders or syndromes and require further stratification to achieve accurate diagnosis [4].

Rare diseases. Although rare/orphan diseases are commonly enumerated at ~7,000, the application of precision medicine, particularly with inclusion of disease stratification, more likely yields ~30-40,000 rare diseases, e.g. cancers, using the NIH definition [5].

These observations, while seemingly independent, in reality are not. They are strongly interrelated and highlight the need to: 1) expand the scope of the system we analyze to meet the goal of improving human health and the practice of medicine (systems-thinking); and 2) address fundamental and frequently unspoken questions before applying new technologies (design-thinking).

To better understand how these observations are related, we can examine the background that has led to each of these observations, remembering that each one actually represents a set of complex processes, not a singular event.

Drug Development

The process of drug development. This process typically involves several steps: 1) identifying the cause and/or symptoms for a condition that suggests a potential target for modulation with a small molecule or biologic agent; 2) creating/testing/screening candidates for activity against the target; 3) optimization, using medicinal chemistry, to enhance potential activity, reduce potential side effects and improve intellectual property positioning; 4) testing in animal models to establish safety and possible dosing levels; 5) initiating multiphase clinical studies under regulatory guidance; 6) finalizing regulatory approval; 7) confirming reimbursement potential from insurers; 8) developing a marketing team to educate and sell product to physicians and patients; and 9) initiation of post-market surveillance.

Naturally this requires a diverse team with specialized skills to successfully execute it, along with significant time and resource commitment, and all this effort is constrained by the limitation of protection of the initial patent for the product [7].

The reality of drug development. The complexity of the process outlined above provides many points at which quantitative assessments might be made to highlight where challenges may exist and predict the overall probability for success. Some of these metrics include: 1) the rate of successful progression from Phase 1 to regulatory approval is between 10 and 13 percent [2]; 2) (bio)pharma’s investment in R&D in 2017 topped $71 billion in the United States and produced 56 novel drugs [7]; 3) there are more than 1,110 new drugs and vaccines in development for cancer alone [8]; 4) as of August 2019, there have been more than 300,000 clinical trials registered with the U.S. government since 2000, with ~31,000 in 2018 alone [9]; and 5) more than one-third of all clinical trials involve oncology drugs [10].

Drug Efficacy/Effectiveness in the Clinic

Evaluating drug efficacy and effectiveness. Drug efficacy is evaluated in controlled studies while drug effectiveness evaluates drug efficacy under real-world conditions. The FDA’s approval process has traditionally focused on evaluation of drug safety and efficacy for a specified use, i.e., labelled, patient population, although physicians retain the authority to prescribe drugs “off label.” The typical process moves the results of discovery research into preclinical development, where safety, efficacy and potential toxicity are evaluated in animal models. From there, the process includes: 1) filing an investigational new drug application, 2) clinical trial testing in three phases – safety, proof of concept and regulatory proof, 3) filing a new drug application, 4) final approval, and 5) post-market safety reporting.

This process can take 10–14 years on average, and thus it approaches the limit of patent protection in the United States, which is 20 years. The process can be accelerated in the case of receiving a “breakthrough therapy designation” by the FDA [11] or meeting other criteria that justify accelerated review and approval. The Orphan Drug Act [12] additionally provides for accelerated review and approval processes, as well as extended exclusivity against competition to promote development for diseases that might not otherwise be pursued commercially. Internationally, many countries regulate a new drug’s pricing based on evaluation of its efficacy against the current standard of care.

Reality in drug efficacy/effectiveness. Although efficacy and effectiveness are commonly viewed with respect to a drug’s performance for a specific patient, more correctly they should be evaluated in terms of responses in real-world patient populations. An assessment by the German Health Technology Assessment Agency (IQWiG) [3] revealed that more than half of new drugs entering the German healthcare system have not been shown to add benefit across multiple indications. In oncology, between 2011 and 2018, the percentage of patients eligible for checkpoint inhibitor drugs increased from 1.5 to 43.6 percent, but the percentage of responsive patients only increased from 0.1 to 12.5 percent [13]. In another comparison of drugs receiving accelerated approval versus normal processing by the FDA, 29 percent of new drug-indication pairs performed no better than existing treatment options with some performing worse, and for nonexpedited pathway drugs this rose to 41 percent [14]. This compares with only 20 percent of those drugs that achieved accelerated review. It should be noted that these averages can vary greatly across specific indications, as well as the metric used to evaluate efficacy or effectiveness, e.g., quality of life, disease-free survival, extended lifetime, etc.

Misdiagnosis and Missed Diagnosis

Process of clinical diagnosis. Disease diagnosis is defined as the identification of the nature of an illness or other problem by examination of the symptoms. Additionally, the clinician will also consider results of specific laboratory testing and imaging. Clinical guidelines exist for both diagnosis and treatment decisions and are commonly developed by consensus or evidence-based analysis carried out by specialist groups or within professional societies. Physicians are not bound by such guidelines and practice based on experience and with respect to the “standard of care.” Physicians will also incorporate knowledge of the patient’s history, e.g., electronic health record/electronic medical record (EHR/EMR), and their professional experience. Decisions for treatment, further testing or referral to specialists are made based on the diagnosis and with consideration of insurance company policies that cover reimbursement for allowable services related to the diagnosis.

Realities of clinical diagnoses. Essentially all diagnoses are comprised of both objective data, e.g., clinical tests, and subjective data, e.g., influence of patient. The Institute of Medicine reported “most people will experience at least one diagnostic error in their lifetime” [4]. In fact, “a conservative estimate found that 5 percent of U.S. adults who seek outpatient care each year experiences a diagnostic error.” Even more disconcerting according to the study, “Postmortem examination research spanning decades has shown that diagnostic errors account for 10 percent of patient deaths,” and “Medical reviews suggest that diagnostic errors account for 6 percent to 17 percent of hospital adverse events” [15]. Precision medicine may actually increase this error rate, as discussed below.

Rare Diseases

Defining a rare disease. In the United States, a rare disease is defined as a condition that affects fewer than 200,000 people, i.e., 64 affected individuals/100,000 [5]. This definition was created by Congress in the Orphan Drug Act of 1983. Rare diseases became synonymous with orphan diseases because drug companies were not interested in targeting them for drug development. Internationally, other countries use different numbers of affected individuals to define rare/orphan diseases (all per 100,000), e.g., Japan (39); Canada (38), Russia (19), China (76), United Kingdom (40) and Denmark (13) [16]. Many rare diseases are observed in children and are thought to be of specific genetic origin. Pharmaceutical companies receive access to accelerated approval processes and extended periods of exclusivity to encourage their efforts to benefit these small patient populations through the Orphan Drug Act.

Reality of rare diseases. The designation of a treatment for a specific condition involves petitioning the FDA for approval. Interestingly, seemingly complex diseases may not receive orphan disease designation, i.e., ovarian cancer, fallopian tube cancer and primary peritoneal cancer are considered one disease, but they may consider the WHO designation of five distinct forms of pulmonary hypertension. Multiple sclerosis, which has a prevalence of 90/100,000 and four specified subtypes, is listed as an orphan disease and has 23 drugs designated, with seven approved and three withdrawn [17]. The introduction and expanding access to precision medicine and molecular/clinical/genomic biomarkers will likely further resolve many currently complex disorders and position them for review in the orphan drug category.

These four observations are not independent! By analogy, one might observe the effects of an earthquake on a building, a roadway, in a house and interruption of transportation networks without recognizing that these were all brought about by a set of events that propagated effects through linked networks and processes. This is the same conundrum presented above about the difference between local accuracy (precision) and global accuracy. While we seem to be good at identifying and addressing local issues, we rarely focus on identifying and evaluating the global challenges, in part because they are complex and not easily resolved by developing and applying elegant technologies.

Following are four foundational issues to consider that can lead to designing and implementing approaches that move from local to global accuracy:

Disease is a process not a state. Disease is a process that evolves over time. A patient does not transition from a non-disease to disease status just because one/some of their clinical tests exceed threshold values. The path for disease progression is potentially nonlinear and temporally nonuniform. We define the “dimensionality” of a disease based on the clinical observations used to make the “diagnosis.” We cannot, however, guarantee that these are the most critical or best dimensions to measure, only that they are accessible and likely represent the standard of care. In the case of a chronic disease, the disease process also interacts with normal developmental changes over time, with both being impacted. In addition, risk develops over time, is not constant and may even be reversible.

FIGURE 2: Disease is a process that evolves over time.

Diagnostics reflect correlative behaviors, rarely causality. Naturally, it is unethical not to treat a patient following diagnosis of disease. As a result, we do not have data that accurately represents disease progression that is absent of the impact of therapeutic intervention. In addition to the issue of “dimensionality” described above, determination of how far along a patient has progressed (i.e., staging) and how quickly they are progressing (i.e., velocity) are required to accurately diagnose the disease.

By analogy, however, the Heisenberg Uncertainty Principle of Medicine states that staging and velocity cannot be measured simultaneously. And diagnostics represent measurable observations that are correlated with disease characteristics, not necessarily describing the cause, but rather the effect of the disease. It is not uncommon that patients with different disease pathways may appear the same because of the clinical parameters being measured and when in each patient they are being diagnosed. Meanwhile, two patients with exactly the same disease pathway, may appear different because they are at different stages of their respective diseases.

FIGURE 3: Ambiguity in diagnosis and staging.

Essentially all diseases are either complex disorders or syndromes. A good diagnostician will acknowledge that “every established disease has multiple subtypes” and each may require somewhat different treatment. In addition, there are many diseases whose identification is based on the correlated presentation of a subset of observable/measurable criteria. The former describes “complex diseases”; the latter define “syndromes.” This is analogous to a tree that has multiple branches that are not identical (complex disease) versus several groups of various branches where the assumption is that they come from the same tree (syndrome). Even diseases associated with specific genetic mutations, e.g., sickle cell disease, manifest differently among affected patients, likely resulting from a combination of other genomic differences and interactions with lifestyle, environment and previous clinical history. Coupling this with the complexity of the disease process, and limitations in available diagnostic testing (as noted above) extremely accurate diagnosis remains a fine art, not an absolute science.

All of this operates within the constraint of the payer environment that requires the assignment of diagnosis codes to justify and reimburse for services. Clinical diagnosis frequently requires a combination of “rule in” and “rule out” criteria in alignment with clinical guidelines. As previously noted, most clinicians operate by combining both objective and subjective observations to reach a diagnosis.

Obviously, attaining global accuracy is critical in optimizing patient outcome, cost management and both drug discovery and development. Attempts to apply analytical methods to extract diagnoses from patient records, e.g., EHRs, continue as artificial intelligence, machine learning and deep learning technologies are evolving, but a limiting characteristic that is frequently overlooked is that data in the EHR is typically inadequately annotated. By example, blood pressure is included as a quantitative measure, but the method of measurement, time of day, proximity to meals, etc., is not noted and the variation could be 20 points, thus impacting the ability to perform noncontext-dependent analytics [18].

**Figure 4:** Real world: co-morbidities and polypharmacy. *Source [19].*

Real-world patients are, at a minimum, complex. The complexity that exists in real-world patient populations is not reflected in those selected for clinical trial participation, although selected populations may be more homogeneous. This reflects the gap between local and global accuracy in medicine and drug development. Every patient is the product of a complex history involving genetics, lifestyle, environmental exposures and previous clinical history. This is particularly relevant because in many diseases, patients are not being cured but rather being managed long term, with medication, diet, exercise and monitoring.

Specifically, every patient will have a history of diseases that have been, and may continue to be, treated, may currently be active, or may exist but are not yet presenting symptoms. Just this reality can impact presentation of symptoms, response to diagnostic testing, resulting diagnosis, prescribed therapeutic intervention, response to treatment (both in mitigating disease and resulting in side effects) and susceptibility to additional diseases. The impact of social determinants of health is only just now being recognized and presents additional and unquantified but critical challenges.

In Summary

I have written this as an introduction to the real-world complexities that face clinical practice and drug development as critical components of the healthcare eco-system. There are others. The emphasis on healthcare was intended to bring the need to consider system- and process-based complexities to a personal level, i.e., one that affects everyone. Fundamentally, however, it is an example of a generalizable problem: the emphasis on the development and implementation of technologies that focus on the “known unknowns.” The resulting solutions can and do work in specific instances but do not generalize because of the “unknown unknowns.”

Mapping out the larger scope of the problem, i.e., enabling the identification of what the right question is that should be asked, not just what the technology can answer, will leave gaps and challenges but will more accurately and transparently represent the real-world complexity of the problem. By ignoring this reality, our efforts will continue to achieve local accuracy but only attain global accuracy through serendipity. This is critical to appropriately understanding the limitations of what we are achieving in healthcare and drug development, but also in essentially all complex problems that we address.