My son typically feels he has the upper hand of knowledge over me (on just about any topic) because of what he has “learned” from watching YouTube. In my fleeting attempts to guide him, I let him know there is a lot of information out there, and it’s not all correct. It can be difficult to weed through all of the noise in order to find reliably factual, and pertinent, information. The same holds true for filtering through medical research and scientific studies. Beyond dedicated research scientists, who regularly produce research articles, nowadays most physicians-in-training have a research requirement. So, even those not interested in publishing must produce something and put it out into the world. This creates a situation where there is an overwhelming amount of studies where the quality is not always good. In order to find good information from medical research, you have to know where to look, then understand how to distill the grain of truth it offers. While doing so, one should have more than a healthy dose of skepticism.
Medical research is an incredible resource for building knowledge and making clinical decisions. It provides the cornerstone for medical advancement and helps to bridge the gap between basic scientific principals and real-world applications. The interpretation of medical and scientific research is difficult, especially for the layman. Physicians and scientists have been trained to dissect research to determine the strengths, weakness and limitations of a study in order to help shape our viewpoints. We first have to understand that there are numerous types of studies. These can be in the realm of basic science or applied clinical research and further broken down into observational versus experimental. The latter often has more clinical relevance and typically begins as a proof of concept trial, followed by safety studies, and finally, clinical trials. There are several different study designs including case-control studies, cohort studies, cross-sectional studies, qualitative studies and randomized-controlled trials (RCTs). RCTs are considered the gold-standard for medical research. While there is a lot of value with this type of study, they are not the panacea of knowledge, nor are they intended to be so. RCTs are intended to lend insight into a very specific question.
For example, a headline may read “New breakthrough drug works for {insert disease here}”. The study may show that the treatment group had 3% improvement of an outcome measure, compared to controls, when studying one hundred 45-year-old white men over three months. So, there may be a statistical significance in one outcome measure, but this may not be clinically significant, or relevant. We can also only conclude that this result was observed in the particular demographic measured (45-year-old white men) and the result should not necessarily be applied to a broader population or timeframe. Studies are meant to SUGGEST relationships, correlations and POTENTIAL cause and effect. They are not meant to be further extrapolated.
There are several influencing factors that affect the studies merits and interpretation. How clean is the study? Are there a lot of unaccounted variables? Is there a lack of homogeneity (similarity in testing)? What are the outcome measures (surveys, biopsies, function, subjective feedback, MRI findings, etc.) and how clinically relevant are these? The strength of study is also influenced by the number of participants, the number of treatment centers involved, the expertise of the researchers, subject compliance and drop-out. Were the measuring tools validated? Is the study reproducible? What was the study length and how was longitudinal data collected? In terms of reliability, we must ask how are the data analyzed and interpreted? Are the data statistically significant, clinically significant, or both?
One of the biggest factors that influences a study is the concept of bias. This comes in different forms but is almost ubiquitously present in research studies. Some of the bias can come from the researchers. This includes financial incentives. There needs to be transparency of who funded the trial and what relationship the investigators have to the funding source. Sometimes we discover that data were omitted or manipulated based on the collection methods. Even more egregious is when data is overtly falsified. There is also bias in the interpretation of study results, both by the researcher and the clinician. Physicians may interpret results based on how they chose to view the data. This may result in statements and headlines that are misleading or harmful.
Here is an example:
CONCLUSION:
Arthroscopic SLAP repair provides a clinical and statistically significant improvement in shoulder outcomes.
We may be led to assume that shoulder labral tears do great with surgery, and certainly many surgeons use this to promote ongoing surgery. Further evaluation of the study reveals that success was measured by improvement in questionnaire the patients filled out, but most subjects had decreased range-of motion after the surgery and many could not return to their previous level of function. Also, subjects of “advanced age” (which was defined as > 36 years of age) had poorer outcomes. 37% of patients met failure criteria, and almost 30% needed repeat surgery. While most (63%) of subjects were happy with their results, there are certainly several factors to consider when we define “success” in this study. Both advocates and detractors of surgical repair could use this study to support their case and this highlights the potential bias in the interpretation and application into the clinical realm.
We also need to be careful of the headlines that a study may generate. Take for example:
“Parachute use did not reduce death or major traumatic injury when jumping from aircraft” (https://www.bmj.com/content/363/bmj.k5094)
The parachute study is a satirical example that tries to make a serious point of the limitations of even well-designed research, especially when viewed in a vacuum. The catchy headline is consistent with the results of the study. When the research jargon is simplified, we see that the only participants willing to participate were those who agreed to jump from an aircraft about ½ meter off the ground. There were no casualties in either the parachute, or control group, and thus the conclusions were made. We must think about the study design, methodology and data while at the same time using common sense and logic to understand its potential merits in the real world.
The same is true when we look at medical research. For example, (to the best of my knowledge) there are no randomized double-blind control trials looking at knee replacements. Institutional Review Boards would have a hard time approving a study where the control group had a sham operation and it would be impossible to “blind” the surgeons. For medical devices, the manufacturer first develops the device and is then obligated to ensure that the proper trials are done to determine safety and efficacy. The FDA then determines its safety. Finally, the physician implements its use in practice. This is a time-consuming and expensive process (often 100s of millions of dollars). Common sense dictates that a metal implant would help improve mechanics and reduce pain compared to a severely degenerated joint.
There are real barriers in the ability perform excellent studies in orthopedics and we certainly see this in the subset of orthobiologics (platelet-rich plasma, mesenchymal stem cells, etc). Orthobiologic research is a relatively new and rapidly evolving aspect of orthopedic and musculoskeletal medicine. From a basic science perspective, it attempts to show how we can influence healing in tissues that inherently have significant limitations for repair. From a clinical perspective, this research attempts to demonstrate improvements in function and quality of life using outcome measures including before/after imaging, surveys, functional measures and reported improvements from the subjects. Oftentimes, trials will have comparisons to other forms of recognized treatment including physical therapy, steroid injections and surgery. While there are dozens of level 1A orthobiologic studies, there are still limitations in the research. Some of the limitations include low participant numbers, heterogeneity in the products and techniques employed, variability in the target populations and treatment area. A significant barrier to high-quality research is due to the extraordinary costs involved in producing a large-scale high-quality study. There is no drug company behind your own body’s cells, unlike what we see with commercial drugs and devices. Similar to the knee replacement example, there is also difficulty in blinding the subjects and investigators in orthobiologic research. We must use our knowledge of biology, physiology and biomechanics when understanding the application and potential mechanisms involved when implementing orthobiologic research in clinical practice.
As you can see, understanding medical research is complex and nuanced. Studies can be interpreted in many ways and the results can be manipulated to support even opposing viewpoints.
The bottom-lines:
-There is no perfect study! There is no study that gives a definitive answer on an issue!
-Do not blindly trust headlines.
-Research can generally be found to support any viewpoint.
-Be mindful that research is important, but only gives a narrow snapshot perspective into an issue. Research is not meant to be generalized to broader applications.
-We must use the overall body of knowledge to make educated assumptions of potential outcomes.
-We must not forget logic and common sense when applying study results to clinical applications.
Research should be taken into consideration when discussing treatment options for patients, but the numerous variables in an individual’s medical situation requires us to be pragmatic in our application of knowledge. No one has all of the answers and we should all be humble in the fact that we probably know far less than we think we do. Medical professionals should always strive to understand more and use our knowledge to help guide our patients through the current research and how this may apply to their unique circumstance. This will hopefully empower patients to make sound evidence-based medical decisions.
References:
-
A prospective analysis of 179 type 2 superior labrum anterior and posterior repairs: outcomes and factors associated with success and failure. Provencher, et al. Am J Sports Med. 2013 Apr;41(4):880-6.
-
Parachute use to prevent death and major trauma when jumping from aircraft: randomized controlled trial. Yeh, et al. BMJ 2018
