Scientific Research and our Article in the Guardian

It's easy to accuse cherrypicking when cherrypicking yourself

We recently wrote a pretty simple article for the Guardian in which we covered a few of the more popular supplements.

There was a bit of backlash, and a lot of accusations were made, including the implication that we shill for supplements. Worst of all, our objectivity was questioned.

Examine.com has never (and will never) recommend any supplement brand. We have five editors from diverse backgrounds who all vet the research we publish (a sixth who is a PhD RD will be announced next week). The entire website is a collaborative effort. And we’ve had a singular focus from day one: evidence-based analysis.

So when Tania Browne posted an article on the Guardian questioning our objectivity, we thought it would be prudent to look at her claims of our purported misrepresentation of the evidence:


Examine.com has an overwhelming amount of citations

The Guardian's article links to the relevant sections on examine.com, where there are over 700 PubMed references citing evidence. - Tania Browne

Yes, our fish oil page has 700+ references. It also has almost 22,000 words. It’s a big page, and thus has a lot of citations. This is not a case of citations to prove the point - it’s a case of proving the point with citations.

Unlike most other citation-based websites, we:

  1. Specifically link each claim to a citation. It’s not just a mess of citations at the bottom - they are specifically numbered for a reason.
  2. Link directly to the appropriate pubmed (or journal) link. Most websites just refer to the document name, which adds a very cumbersome step.
  3. Our Human Effect Matrix (HEM) specifically links to the relevant papers for each attribute. Clicking “See all studies” shows every relevant research paper, links to it, includes demographic information given, and has a summary on what the full text found.
We make it as easy as possible to find our citations for any claim we make, both in the HEM and in our Scientific Research section.


Not all studies are equal

So how do you know whether a scientific paper can be trusted? - Tania Browne

Good point. Which is why we specifically label every study listed in the HEM by its trial design: meta-analysis, double blind, other clinical trials that are not double blind (such as cohort studies), and observational studies with no intended intervention.

Still, there is a reason why the same outcome (“health goal” on our site) for one supplement can have multiple meta-analyses - they are not all created equally. For instance, different meta-analyses can have different inclusion criteria, or different statistical methods. We can cite studies - Should meta-analyses trump observational studies? + Systematic Reviews: Meta-analysis and its problems - that explore how meta-studies are not the end-all be-all of scientific research. It is not enough to read the abstract of a meta-study; you must read the full text.

This is why every single paper cited on Examine.com is subjected to rigorous scrutiny by our scientific team.

It is not hard to read a pubmed abstract and come to a conclusion that is contrary to what the authors found. Prudent research requires one to read the full text and not just the abstract.


Glucosamine - do you need it?

We’ll accept blame that our language was not precise enough. You can look at our own glucosamine page to see our more nuanced view. We should have clearly differentiated between knee osteoarthritis (a serious degenerative disease) and other forms of osteoarthritis and less serious joint pain.

For that we accept blame.

With that in mind, let’s look at the actual research findings and the accusations of cherry-picking.

You can see our collation of studies on glucosamine and osteoarthritis here. The single meta-analysis which Tania Browne references is already in our HEM, which we summarized as follows: “A network meta-analysis on glucosamine sulfate found statistically significant pain relief, but deemed that the clinical significance of this pain relief was minimal.”

Except, instead of citing one meta-analysis, we cite six. And our analysis of all of the research (not one meta-analysis but all six of them) finds the following:

There appears to be a small decrease in osteoarthritis symptoms associated with glucosamine (as sulfate, not hydrochloride) which is somewhat unreliable but consistently outperforms placebo on meta-analyses. The magnitude of reduction, however, is somewhat minor but still comparable to acetaminophen (paracetamol). At least three of the meta-analyses also performed funnel plots and found no evidence of publication bias.

And that was our recommendation. Dr. Spencer Nadolsky, dealing with patients who have osteoarthritis, recommends glucosamine instead of acetaminophen. We specifically stated in our recommendation that it is not very potent, that the sulfate version should be used, and that it be used instead of non-steroidal anti-inflammatory drugs (NSAIDs). Glucosamine’s safety profile is well established.

By no means did we mean to imply it was a panacea.

Again - we concur that we should have been more explicit that we were only talking about osteoarthritis, and that instead of saying “it may not be a miracle supplement” we should have stated “it will only provide marginal relief.” We were being pragmatic, but were not clear enough.

In a clinical setting in which a patient is suffering from osteoarthritis, glucosamine is found to be as effective as NSAIDs or paracetamol. This does not mean it is very effective, and it was our mistake in not being clear about that.


Protein

We have written extensively about protein on our website.

The recommended daily amount of protein for 19-50 year olds is 55.5g for men and 45g for women. But according to the British Nutrition Foundation, average protein intake is 88g for men and 64g for women. - Tania Browne

Again, this is where one must look deeper into the evidence instead of just looking at the quick conclusion.

The RDA of protein is based on the minimal protein intake required to ensure no bone or muscle loss. This recommendation is based on a bodyweight and lifestyle that is now outdated due to the obesity epidemic.

Tania Browne commented that Dr. Nadolsky likes to lift weights (how is that relevant?), as if his hobby influences his dietary recommendations. Our suggested protein goals are, in fact, much lower than what most bodybuilders consume. We've summarized all of the research on how much protein you need.

In 2010, a Consumer Reports survey of 15 brands in the US found that some protein drinks were contaminated with heavy metals that could reach harmful levels in the recommended three daily servings. - Tania Browne

Another example of looking at only a subset of the data, not all of the data. Consumer Reports did indeed find contaminants in protein powders. But if you look at other everyday foods, similar amounts of contaminants were found. And here is the FDA’s larger dataset.

With that in mind, the 2010 testing found three positives out of 15 samples, which is not robust enough to serve as conclusive evidence that protein powders are harmful. If we extend this logic, then a past meat recall due to bacterial contamination would be enough to “prove” that meat is harmful. Furthermore, we do not recommend any brands.

Lastly, Consumer Reports looked at three servings per day, something we did not recommend. The author’s wording is basic fearmongering, using a subset of data to make protein look far more dangerous than it actually is.

Furthermore, the NHS warns that excessive protein intake can lead to bone demineralisation and osteoporosis. - Tania Browne

It should be noted that this is from a special report and not an official position statement. The 2011 NHS special report does indeed mention that protein could play a role in bone demineralization, although this contradicts the positions of other organizations like Osteoporosis Canada.

In cases where there is a dispute in positions, we can defer to the studies conducted. We find in epidemiology that low to normal protein intake is predictive of bone loss and higher protein intakes above the RDI are protective relative to lower intakes. This has been observed in other studies as well. Elderly women with a high calcium intake consuming up to 72g of protein a day slowly improved their bone mineral density. In this same study, the group that only took between 16-50g of protein experienced the greatest bone loss.

The studies showing bone loss are essentially non-existent. They were logically extended from initial studies showing increased calcium elimination in the urine with protein intake, and while at this time it seemed logical to assume that protein caused bone loss, the theory just didn’t pan out in interventions. For those who want to read more on this topic, here is a free in-depth review from the American Journal of Clinical Nutrition about the interactions of protein, calcium, and bone health.

More fearmongering with no basis in evidence.

And a recent article in the journal Cell linked high intakes of animal protein to cancer and higher overall mortality in the under 65s. - Tania Browne

We actually analyzed this exact study. And just as we have repeatedly stated, one must read deeper than the abstract.

That very study she linked to only looked at people over the age of 50. It split people into two groups, and found that increased “protein” was beneficial for those older than 65. We state protein in quotation marks because all protein sources were treated equally - be it chicken breast or bacon-wrapped sausages. The NHS also agrees that said study did not find protein intake to be bad.

This ties into an important point - when it comes to epidemiologic studies, it is important that one read the full text and not simply rely on the abstract stating the basic conclusion, and instead focus on the nuances underlying the claim and to which specific demographics the claim can apply to. As someone studying epidemiology, I'm sure the author can agree with that

Protein is an essential macronutrient, and we only recommended protein powder for its convenience. Worries about it having negative effects at the dosage we recommended is unnecessary.


Multivitamins

We will admit that we lost some of our nuance in our recommendation. Our own page on do I need a multivitamin? states the following:

Multivitamins are very useful if you have a poor diet, but they lose much of their benefit if you have a good diet. Many people with good diets take multivitamins unnecessarily. Just supplement the nutrients you need instead.

And that is our entire ethos. Supplementation is done in a targeted manner, and only after your diet is taken into consideration. Identify your deficiencies, and either fix your diet (preferable) or supplement.

In fact, multivitamins might make things worse - Tania Browne

To be blunt, while multivitamins have pretty much no evidence of being beneficial, to say that they “make things worse” is another case of fearmongering.

If we look at only negative effects, excess vitamin E (400 IU) has been associated with an increase in mortality and excess beta-carotene, in smokers, has been associated with an increase in lung cancer. Concerns about multivitamins in a nonsmoker can be negated by simply reducing the vitamin E content. The authors of the meta-analysis that found an increased mortality risk stated that, since most of the studies were done on metabolically unwell people, they are uncertain if the information applies to healthy individuals. Another meta-analysis specifically examining multivitamins finds no impact on mortality.

Multivitamin benefit or detriment is not a clearly agreed upon topic among researchers. Studies within the large meta-analysis include different populations and have widely different results. To say they make things worse is incorrect.


Vitamin D

It has been interesting to see vitamin D go from hype to backlash. The Lancet recently published some studies, and you can see our analysis on their study on vitamin D and osteoporosis prevention. We then summarized vitamin D’s utility in our The Truth about Vitamin D analysis.

Worryingly, Nadolsky also advises 50 micrograms (2,000 IU) of vitamin D per day - Tania Browne

Yes, Dr. Nadolsky did indeed recommend 2,000 IU of vitamin D daily. However, why was this worrying? The lowest dose with any adverse (not toxic) effect that has even been noted was a nonsignificant increase in fat gain at 5,000IU.

2,000IU has never been associated with harm, and at times is associated with significantly more benefit than 800 IU. Higher doses (of 5,000 IU or more) have been associated with varying degrees of harm.

The only evidence the author brought forth was a statement of how our recommendation was apparently “twice the NHS recommended maximum intake of 25 micrograms” and how the recommended intake is 5mcg, which is actually wrong in many countries, as 5mcg is 200 IU (vitamin D3) whereas the RDI tends to fluctuate between 400-800 IU depending on the nation (for example, having been recently increased to 800 IU in Canada). The tolerable upper limit in the US is 4000 IU (100 mcg).

In fact, the Endocrinology Society Guidelines recently bumped up their vitamin D recommendations so that "getting 25(OH)D levels consistently above 30 ng/mL may require at least 1500 to 2000 IU/day of vitamin D. This recommendation was tasked by Dr. Holick, who discovered both 25(OH)D and calcitriol.

In short, “high doses” of vitamin D do cause adverse effects, but 2,000 IU is by no means a high dose. We even have a section on toxicity.

Taking too much vitamin D for a long time can cause calcium build up, damaging the kidneys and even softening bones - Tania Browne
This is true in theory, but only when excess vitamin D is ingested (which as we previously covered, is well above 2000 IU). In short, vitamin D increases calcium utilization and at high doses this occurs in many tissues, which may not actually be beneficial.

Furthermore, vitamin D is thought to have this adverse property because it is in too high a ratio relative to vitamin K. Vitamin K takes calcium out of tissues and reduces calcium buildup in arteries, a reason why the combination of supplements was recommended.

Excess vitamin D can indeed cause harm, but our recommendation of 2000 is not excessive. Any attempt at linking our recommendation to high-dosage vitamin D risks is an illogical leap.

Summary

The real science of dietary supplements shows that very few of us need them and most of us will do just fine by eating a healthy, balanced diet. - Tania Browne

We agree, and we have never claimed otherwise. However, there are situations when the diet is not balanced where supplements can be very helpful.

We also agree that our wording for glucosamine and multivitamins was poor. We should have been more nuanced.

With that said, we take our research very seriously. We are 100% independent and neutral, and we readily admit when we are wrong (have been before, and will be in the future). Yet in the case of our original article for the Guardian, properly looking at the actual evidence clearly validates the assertions we made.

Accusations of chicanery and cherry-picking simply do not hold up when the evidence is looked at in-depth. There is a reason why we have five different editors with advanced degrees in different backgrounds (with a sixth who is a PhD RD being announced next week). If anything, Tania Browne’s article engaged in fearmongering and amateur analysis that was not backed by the full body of evidence.

We focus on the entire body of evidence. Our goal is to objectively summarize the scientific research for supplement and nutrition issues, keeping in mind study quality and methodological issues, and provide consumers with the knowledge needed to make informed decisions.

NOTE: This article was vetted by all of our editors


Published By on

Want to know how to look past the headlines and get the real facts on research?

Here's your guide to knowing how research is abused and how to be better at discerning it

Your e-mail is safe with us. We don’t share personal data.