How to read a study: Adapted from the Health Integration Guide, Course 13: Research to Results

Why start here? 

We’ve already started and we’re going to continue publishing relevant articles and summaries of research about CrossFit: what it does to the body, who it works for, where it falls short, and even ask the questions about what nobody has tested yet.  Some studies are going to look great and others will be mixed and at times there might be studies that the community will not love. We want to set a standard (you know we like to do that in CrossFit) so that we can be exposed to education and information and speak a common language around it. If we just post headlines with no shared vocabulary, two predictable things will happen: people who like a finding will treat one small study as proof while people who don’t like a finding will dismiss the whole article. Both of these reactions stem from the same gap of not knowing how much weight any single study is supposed to carry. 

In Course 13 of the Health Integration Guide we call this out with the hopes of providing a starting point for research literacy. The goal is to avoid the drastic swing between extremes: uncritical acceptance of studies and outright rejection of science. CrossFit is a fitness and health education company. We create the best trainers through education and the relentless commitment to uphold the standard. This is no different. 

Hierarchies to keep straight

When someone says “science says” multiple considerations are at play. Put them on a hierarchy, a pyramid if you will. One pyramid will be about the statement or claim, how settled is this idea?  And the other will be about the evidence, how strong is the study being used to support it? We really need both because a weak study can support a very settled idea and a strong study can test an idea that hasn't been tested or confirmed yet. 

Hierarchy One: how settled is the claim?

Four definitions to understand here. 

• Conjecture. Think of this like an educated guess. Somebody noticed something and proposed an explanation as to why or how this was the case. Nothing has been tested.  

  • Example: Group fitness classes of five or more people might help people stick with exercise longer than smaller groups or personal training. While this is a reasonable idea it’s simply conjecture worth nothing yet. 

• Hypothesis. Take that educated guess and formulate it into something that could actually be proven wrong. It’s going to specify what you want to measure and what result would kill the idea, 

  • Example: adults assigned to classes of 5 or more people will attend more class sessions in a 12 month period than those who are assigned to classes of fewer than 4 people.  With specifics like this, it can be tested. 

• Theory. In everyday speech, the word “theory” means a hunch. But in science “theory” isn’t just a hunch, it’s an explanation that has survived many independent attempts to break it and prove it wrong, and is supported by multiple lines of evidence. 

• Law. It’s reasonable to think that law sits higher or where theory goes when it gets promoted. But saying something is law is a different type of statement. A law is describing what reliably happens. The theory explains why it happens. Newton’s Law of Gravity is describing the pull between two masses. It isn’t explaining what gravity actually is. The theory around it does. So law is answering a different question. 

Thinking about what this means for us, in exercise science and nutrition, almost nothing reaches the level of law. But a handful of things reach theory level. For example the theory that progressive overload builds strength or energy balance determines the body’s mass over time. Keeping this in mind, a lot of what we’ll see is at the hypothesis level. It will be a specific testable claim with some evidence behind it but with open questions to be tested and answered, further validated by other groups who can repeat the finding. This would ultimately push the hypothesis into the theory category. 

Hierarchy two: how strong is the evidence?

Think of this pyramid in the sense that the higher a study sits the harder it will be to explain its results away by luck, bias, or the kind of person who chose to participate in the study. We’ll define the types of evidence from top to bottom. 

• Guidelines and expert Consensus. This is a panel of specialists who reads everything available and issues recommendations. We’re noting this as a strong position when it's built on the evidence below it. 

• Systematic reviews and meta-analyses. Someone gathered numerous studies on a question, assessed their quality and combined the results. The closest thing to “what the evidence as a whole says.” One well-run meta analysis can outweigh a dozen individual studies. 

•  Randomized controlled trials. Participants are assigned by chance to one group or another and then followed. There is a key point in the randomization process. It’s what let’s us say the intervention causes the difference compared to a study when the person volunteered for that group. Depending on the size of the study and how it was conducted, we can infer the degree of causality. Does this single study support the word “caused.” 

• Cohort studies. A group is followed overtime and we watch what happens. This is a useful approach for questions you can’t ethically or practically randomize. Here we can show strong associations but can’t fully rule out that something else about those people could explain the result. 

• Case-control studies. Here you start with people who already have the desired outcome and compare them backward against people who do not. Vulnerable to memory and selection problems. 

• Cross-sectional surveys. This is a snapshot. Everyone answers at one moment in time. We see this as genuinely useful for describing a population and generating a hypothesis. It’s crucial to note that it can’t establish what caused what, ever.  

• Case reports and case series. This is when one patient or a small handful of people are described in detail. Even though this is lower on the pyramid, it’s still evidence and it’s still valuable. This is where some of the most important research from CrossFit comes from. The stories from INSIDE the affiliate. 

• Animal and laboratory studies. Research that cites being done in rodent studies, cell cultures, or isolated tissues. For what it’s worth, a lot of research with supplements and recovery is done at this level. A compound that does something impressive in the petri dish in the lab has actually cleared a very low bar in relation to human evidence. Important to note that something that happens to mice doesn’t translate to what could happen in humans. But this level of research is important because it does generate hypotheses for future research. 

One more mention of a design type: the qualitative study. Think of this as research that collects observations and words rather than numbers. This is typically done through interviews, focus groups, and direct observation. Instead of measuring how much or how often, this type of research explores why people might behave the way they do or what an experience might be like for them. An important note is that a qualitative study does not sit on the evidence pyramid because it’s answering a different type of question. Findings are reported as patterns and themes. A qualitative study is quite useful in showing why people start, stay, or quit. 

Something to note, this hierarchy ranks study design, not study quality. You can have a poorly run randomized controlled trial and a well buttoned up cohort study. 

So what do the results actually tell us? 

Once a study is built and run, it produces data. The job at hand is working out what the data (results) actually means, and here is where statistics comes in. At a baseline level, statistics is the set of tools researchers use to make sense of the data that was collected from the study and to judge what this data might likely say about people in a broader sense. Since no study can test everyone, researchers are testing a sample and then using statistics to estimate how far the results can be trusted outside of the sample size. Statistics are not meant to tell us if a finding is true, it’s meant to help us know how confident we can be and how large the effect appears to be. 

One important question to ask of the data is did the study find out what it set to find out? A well-designed study states, before it begins, what it expects to happen and how it will measure it. By declaring a measure, the study is naming a primary outcome or the main result the study is aiming to test. You’ll also see that studies measure other things along the way and are often noted as secondary outcomes. Meaning researchers might have planned for these in advance, but did not design the study around testing. Exploratory outcomes are when things are examined without having a specific prediction and it’s usually because the data was already there. 

So in the results section of the study, did the researchers learn what they set out to learn? Or did they learn something else? When the primary outcome delivers then the study has done its job by answering the question it set out to answer in the specific way the study was designed to test. If the primary outcome is flat and the secondary outcome is the attributed headline then it’s important to note that the researchers found something it wasn’t looking for. Super valuable, but also means that a new hypothesis with a specific design for this question needs to be done and tested on purpose. And consider how many things are being measured in a study. The more a single study measures, the more likely it is that one will look impressive by chance.

Sample size is referring to the number of participants in a study. This helps shape how much the results can be trusted. A handful of specific or unusual participants can swing results in either direction. A small sample size can miss a real effect entirely or produce one that appears to be much larger than it really is. Having larger sample sizes can help settle the noise making their results more stable but don’t come without their own considerations. A large sample size with enough people might detect a tiny and unimportant difference. 

If the goal of a study is to help explain a level of certainty and trust around what it tested, then how do researchers handle uncertainty? Here’s where we look at statistical significance, meaning the results are unlikely to be explained by chance alone. Judging certainty is used by looking at p-value, and by convention in our case here, a result below 0.05 is called significant. A confidence interval give the range within which the true effect most plausibly sits. A narrow range means the study measured the effect in a precise manner. A wide range means that the study did not pin it down. The key thing to note is that neither one tells you whether the result is important. They tell you whether it is likely to be real. And even if a result falls short of significance, it’s not an automatic that nothing happened, but rather this study was too small to tell. 

One note on effect size. A result can be real and still be too small to matter. Effect size is describing how large a difference actually is. Not whether it's detectable. This number tells you if a finding would change anything for a person in the gym or a patient in the clinic. So sure, a result might be significant, not how big? 

If a study is reporting a change in the risk of something, such as injury, disease or death, there are two ways to express this change. Relative risk compares the two groups to each other. Absolute risk is the actual chance of the outcome happening to a person. The absolute difference is how much that change really changed. If you see a headline that says “50% relative risk reduction” and the risk drops from 2 in 1,000 people to 1 in 1,000 people, the headline is accurate and sounds like a major breakthrough. Looking for the absolute risk is the number that describes what changes for an individual. 

Are you meant to be a statistician to effectively read research, absolutely not. But carry these three questions with you: 

Did the study answer the question it set out to ask? 

Was the study large enough to trust the result and how precise was the estimate? 

Is the difference big enough to matter to a real person? In absolute terms? 

So where does CrossFit research sit? 

Most published CrossFit research sits in the bottom half of the design pyramid comprising cross-sectional surveys, small non-randomized trials, and retrospective data collection. There are few randomized controlled trials, and most aren’t large ones. To us what’s exciting is that we have the n=1 story of how CrossFit changed someone’s life. We actually have a lot of n=1 examples. We have a lot of people who want to keep putting out conjectures and test the hypothesis. People want to study CrossFit. It’s not without challenges since 12 month controlled trials in real facilities with credentialed coaches is expensive. But the evidence for CrossFit is real and it’s growing. And while most of it is at an early stage what that really means is that we get to say, “this is promising and here’s what it showed.” 

Here’s what we want from you: 

Be an active participant in the material. It’s so easy to just consume content these days. Engage, read, reflect. 

1. Read the method before the result. Survey, trial, or review? That one word tells you how much the finding can carry. Check it first, every time.

2. "Associated with" is not "causes." If a study is cross-sectional or observational, the arrow can point either way. Do people train more because they are resilient, or become resilient because they train? A snapshot cannot tell you. Most headlines pick the flattering direction and hope you do not notice.

3. Ask who was in the study. A result from 139 athletes training six days a week does not automatically transfer to a sedentary 55-year-old starting next Monday. Neither population is more important. They are different questions.

4. Check the size, then check the effect, then ask relative to what. Small studies swing wildly. "Statistically significant" only means the result probably is not random noise, and says nothing about whether the change is big enough to matter to a human being. And a percentage is meaningless without its baseline: a 50% reduction in a risk that starts at 2 in 1,000 is a very different thing from a 50% reduction in a risk that starts at 1 in 3.

5. Look for what was not measured. This is where most overclaiming happens. A study can improve every variable it measured and still say nothing about the outcome you actually care about, because it never measured that one.

6. One study is a data point, not a conclusion. The question is never "what did this study find." It is "what does this study add to what we already knew." A finding that replicates across different teams, countries, and designs is worth more than any single impressive result.