Mistakes by Examine.com

Known issues and how we’re tackling them

We’ve started compiling a list of things we’ve gotten wrong or done suboptimally since our founding in 2011, along with ways we’re trying to improve. This list will grow over time.

For what reason did we take this step? Well, we take transparency very seriously. That’s why we don’t sell supplements, have external ads on the website, or accept donations from food or supplement companies. We’re not health gurus who claim to always be right; we’re simply researchers who are trying to be objective. Research involves honestly identifying your mistakes and improving your methods.

Note that we were inspired to do this by a similar page on the website of Givewell, a nonprofit charity evaluator website (which we’re not affiliated with in any way).

Red: Still a long way to go

Yellow: Making good progress

Green: Mostly solved

Content mistakes

  • Not enough focus on chronic health conditions
    • Mistake: There’s nothing wrong with trying to improve one’s physique, and more muscle and less fat is great for health. We're always going to stay up to date on those topics. But the public health burden of chronic health conditions is massive and accelerating. Disease improvement can be, relatively, much more complex and difficult than physique improvement.
    • Improvement: In the past year, we’ve started including more information on foods, diets, and health conditions, while still doing updates on physique supplements. Don't worry, we'll still be keeping abreast of the latest muscle gain and fat loss evidence!
  • Confusing topics page
    • Mistake: Our topics page is a mix of relatable health outcomes (such as anxiety) and very specific health outcomes (such as fibrinogen and lipoprotein lipase activity).
    • Improvement: We’re working on a brand new categorization scheme that will allow you to easily browse health conditions, then drill into specific outcomes if you want. This is a major undertaking that we’re quite excited to push live.
  • Missing systematic entries on popular diets and food topics
    • Mistake: We have always had a ton of well-researched pages on supplements, but relatively few systematic pages on diets and foods. For example, we have several blog articles on intermittent fasting, but no master entry with all the research covered on a given specific fasting protocol (e.g., 16:8 or ADF).
    • Improvement: We made thorough pages on CBD and the ketogenic diet, two of our bigger gaps. We also made pages on certain popular health foods, such as apple cider vinegar. But we have a long way left to go, with many diets and foods left to systematically cover. We do provide information about some of these throughout the site and in our products, but not on systematic pages yet.
  • Not enough page updates and new pages
    • Mistake: We had more frequent new content in the first years of our existence. In the subsequent years, we spent more time and effort on products and internal organization due to financial pressures.
    • Improvement: We’ve started to produce fewer new products, focusing more on the free information on the site and updating existing products. This was made financially possible by a shift toward membership plans for Examine.com supporters who want more detailed access to our research. Additional membership benefits are forthcoming as well.
  • Gaps in micronutrient research
    • Mistake: Some nutrient pages had rather large gaps in the studies they covered. We’re also missing a few nutrients, such as certain trace minerals.
    • Improvement: We added studies to the iron and vitamin C pages in the past months, and are planning further updates over the next few months.
  • Specifying funding sources of studies
    • Mistake: Everybody wants to know who funds supplement and diet studies. And for good reason, since industry funding may introduce a conflict of interest. However, listing this for every study with industry funding would take eons to do, since we have so many studies on the website. It could also make articles more difficult to read, and can even mislead readers in a different way, as industry funding doesn’t inherently mean the study is flawed or biased, or that the results aren’t accurate. Categorically “unhealthy” foods such as colas can have biased studies, but so can foods with more mixed evidence, such as red meat, eggs, or soybean oil. All of these have studies funded by industry, government, or both. And non-industry-funded studies can also have bias, as there are several possible sources of bias other than the funder.
    • Improvement: We discussed having a line item on pages listing how many studies were industry funded, but that didn't provide enough context to be useful. We will likely try a couple of different methods to make the nuances of funding and bias apparent to readers. Once we decide on a route to take, we’ll have to go back and edit many, many pages in our archives.
  • Blood sugar supplement side effects
    • Mistake: The side effects of blood-sugar supplements are not always fully understood, due to a shortage of research. We didn’t always note that important fact, especially given that there is no guarantee that they won’t cause hypoglycemic events (especially when mixed with other supplements or medications).
    • Improvement: We’ve been adding this information to our supplement entries and _Blood Sugar_ Supplement Guide.
  • Missing page summaries
    • Mistake: Some pages are missing quick summaries (the 2–3 sentences at the very top of the page), while many are missing full summaries (the longer paragraphs below that).
    • Improvement: We’ve filled a few dozen missing ones, but still have quite a way to go.

Presentation mistakes

  • Using dense language
    • Mistake: The website is not always easy to read, especially in the nitty-gritty “Scientific research” sections. The blog posts are usually easier to read.
    • Improvement: We’ve started making summaries easier to read, but there’s a daunting amount of non-summary information to improve. We hope to make more money, hire more editors, and fix the remaining language.
  • Legibility and usability wasn't great
    • Mistake: Although we have an audience that skews a bit older, we never went through the site to ensure that people with impaired vision or those not as adept at web usage could easily use and navigate the site.
    • Improvement: We’re systematically updating the legibility of the graphics and increasing font sizes. We’re also updating the menus to make them more clear and structured. Later on, we may introduce a bot or quiz to more easily funnel people to the information they’re looking for. Also in the plans are a beginner’s guide to using Examine.com, as well as a beginner’s guide to nutrition.
  • The research digest wasn’t quite digestible enough
    • Mistake: Our research digest breaks down studies from every angle. We increasingly found that subscribers got overwhelmed.
    • Improvement: We did a year-long round of feedback, and made many changes to the digest to improve usability. The articles are easier to read, the digest is online, etc. It’s also been rebranded as “NERD” now, instead of “ERD”.
  • Site too slow
    • Mistake: Some of our pages contain a LOT of information. The site was loading too slowly.
    • Improvement: We made several changes to the backend system to improve site speed. We also split out the most in-depth information into separate pages, which improved page loading time.
  • Inconsistent terms around “recommending” supplements
    • Mistake: When listing dosages that were used in trials, we sometimes indirectly phrase it (e.g. “Trials typically used between X-Y mg/d”), and sometimes used more direct language like “Take X mg/d”).
    • Improvement: We’ve figured out most of the contexts where we want to use indirect vs direct language, so as to not mislead readers. There is a lot of language to update though, throughout the site.
  • Overly-long blog posts without enough structure
    • Mistake: We wrote several very long, very detailed blog posts in the past few years. While these interested some people, many readers’ eyes glazed over. These posts were surprisingly costly to research, write, and copyedit. They also didn't adhere to the structured format of supplement and diet entries on the site (with a summary, FAQ section, research matrix, etc).
    • Improvement: We’re focusing on systematic entries for supplements/diets/nutrients, rather than having more and more narrative blog posts. With these systematic entries, we can build up pages over time as we gauge interest in different topics. For example, we could cover the most important meta-analyses and randomized trials for a given diet, then cover further studies if the topic is of interest.
  • Late to the game for Instagram and other social platforms
    • Mistake: Because we don’t give flashy proclamations or controversial hot takes, preferring to present the research as it is, our social media presence is not that big (compared to the vast amount of information on the site). We also failed to focus on social media early on, even though that’s how many people get their information. Lastly, we didn’t always have consistent branding/colors/etc in social media images.
    • Improvement: We’re working with our graphic designers to figure out what to convey to social media users, and in what format.

Organizational mistakes

  • Lack of diversity
    • Mistake: Our team is mostly comprised of men in their 30s and 40s. Meanwhile, our readership is evenly split between men and women, and it includes many older folks. In order to better address health issues facing women and older people, it would help to have some researchers who are women and/or older. A team of men in their 30s and 40s is a bit more likely to focus on issues more relevant to their demographic and have less expertise on issues that impact women and older people more. One of our major goals is removing subconscious biases and blind spots, in order to be more accurate and more helpful to readers.
    • Improvement: In our last hiring round, we blinded the applications and test assignments for more objective evaluation, and both eventual hires ended up being male. In our current efforts, we’ve made it a point to also proactively find researcher applicants with interest and expertise in women’s health and healthy aging, as well as female and older researchers.
  • Relying on surveys rather than interviews
    • Mistake: We frequently surveyed readers to find out what interests them. Yet we did few in-depth interviews to find out more about the health journeys of different readers. Knowing more about the full scope of readers’ experiences should help us provide helpful information in the best possible formats.
    • Improvement: We’ve started doing more in-depth interviews and are also asking a small subset of dedicated readers targeted questions to help shape some of our decisions. We may also perform focus groups in the future.

Stay up to date on nutrition and supplementation

Enter your email address and we’ll keep you on top of the latest nutrition research, supplement myths, and more.

This document was last updated on February 27, 2020.