Mistakes by Examine.com

Known issues and how we’re tackling them. Updated in July 2020.

Here's a running list of things we’ve gotten wrong (or done suboptimally) since our founding in 2011. We update it regularly.

The purpose is to keep Examine.com transparent and accountable. That’s also why we don’t have external ads on the website, or accept funding from food or supplement companies. We're just a team of research nerds who strive to honestly identify our mistakes and then solve them. Below is our (ever-growing!) list of mistakes, color-coded for easy scanning.

Red: Still a long way to go

Yellow: Making good progress

Green: Mostly solved

Content mistakes

  • Not enough "outside the box" treatments (*New in July)
    • Mistake: If you have a chronic health condition that's somewhat rare, or difficult to treat or diagnose, then you know how frustrating it can be when traditional approaches don't work. People may dismiss you. The old medical saying goes "When you hear hoofbeats behind you, don't expect to see a zebra". So what if you're a zebra, and you don't fit into neat boxes? Examine.com can already be of some help, but we could do much, much better.
    • Improvement: Supplements are a common option when other treatments don't work, and we've covered hundreds of supplements fairly well. Yet, we don't cover some of the most important supplements for difficult conditions, nor do we cover non-supplement options like infrared saunas, meditation, niche therapeutic diets, etc. We plan to start delving into these kinds of "outside the box" treatments more in coming months. After all, who needs more help then those who have tried everything and are still doing poorly?
  • Not enough page updates and new pages (*Updated July 2020)
    • Mistake: We had more frequent new content in the first years of our existence. In the subsequent years, we spent more time and effort on products and internal organization due to financial pressures.
    • Improvement: We’ve addressed this issue in two ways, although we're on the road to solving it. First, we recently hired one new full-time researcher, and two part-time researchers. Second, we started to produce fewer new products, focusing more on the free information on the site and updating existing products. This was made financially possible by a shift toward membership plans for Examine.com supporters who want more detailed access to our research.
  • Not enough focus on chronic health conditions (*Updated July 2020)
    • Mistake: There’s nothing wrong with trying to improve one’s physique, and more muscle and less fat is great for health. We're always going to stay up to date on those topics. But the public health burden of chronic health conditions is massive and accelerating. Disease improvement can be, relatively, much more complex and difficult than physique improvement.
    • Improvement: We had many, many internal debates about how to approach this. Should we focus deeply on a small handful of common conditions, or expand to a few dozen conditions? Eventually we went down the middle road, and systematically compiled the roughly 25 conditions that fit one or more of these characteristics: highly-prevalent, amenable to improvement through diet and/or supplementation, or difficult to treat through traditional medical routes. Our researchers set up rigorous Pubmed search strings, and we now collect most of the relevants study on these conditions each month. They're fed into our Personalized Membership, which is how we fund our efforts to keep abreast of the latest research each month through these resource-intensive means.
  • Unorganized topic listing (*Updated February 2020)
    • Mistake: Our topics page is a mix of relatable health outcomes (such as anxiety) and very specific health outcomes (such as fibrinogen and lipoprotein lipase activity).
    • Improvement: We’re working on a brand new categorization scheme that will allow you to easily browse health conditions, then drill into specific outcomes if you want. This is a major undertaking that we’re quite excited to push live.
  • Missing systematic entries on popular diets and food topics (*Updated February 2020)
    • Mistake: We have always had a ton of well-researched pages on supplements, but relatively few systematic pages on diets and foods. For example, we have several blog articles on intermittent fasting, but no master entry with all the research covered on a given specific fasting protocol (e.g., 16:8 or ADF).
    • Improvement: We made thorough pages on CBD and the ketogenic diet, two of our bigger gaps. We also made pages on certain popular health foods, such as apple cider vinegar. But we have a long way left to go, with many diets and foods left to systematically cover. We do provide information about some of these throughout the site and in our products, but not on systematic pages yet.
  • Gaps in micronutrient research (*Updated February 2020)
    • Mistake: Some nutrient pages had rather large gaps in the studies they covered. We’re also missing a few nutrients, such as certain trace minerals.
    • Improvement: We added studies to the iron and vitamin C pages in the past months, and are planning further updates over the next few months.
  • Specifying funding sources of studies (*Updated February 2020)
    • Mistake: Everybody wants to know who funds supplement and diet studies. And for good reason, since industry funding may introduce a conflict of interest. However, listing this for every study with industry funding would take eons to do, since we have so many studies on the website. It could also make articles more difficult to read, and can even mislead readers in a different way, as industry funding doesn’t inherently mean the study is flawed or biased, or that the results aren’t accurate. Categorically “unhealthy” foods such as colas can have biased studies, but so can foods with more mixed evidence, such as red meat, eggs, or soybean oil. All of these have studies funded by industry, government, or both. And non-industry-funded studies can also have bias, as there are several possible sources of bias other than the funder.
    • Improvement: We discussed having a line item on pages listing how many studies were industry funded, but that didn't provide enough context to be useful. We will likely try a couple of different methods to make the nuances of funding and bias apparent to readers. Once we decide on a route to take, we’ll have to go back and edit many, many pages in our archives.
  • Blood sugar supplement side effects (*Updated February 2020)
    • Mistake: The side effects of blood-sugar supplements are not always fully understood, due to a shortage of research. We didn’t always note that important fact, especially given that there is no guarantee that they won’t cause hypoglycemic events (especially when mixed with other supplements or medications).
    • Improvement: We’ve been adding this information to our supplement entries and Blood Sugar Supplement Guide.
  • Missing page summaries (*Updated February 2020)
    • Mistake: Some pages are missing quick summaries (the 2–3 sentences at the very top of the page), while many are missing full summaries (the longer paragraphs below that).
    • Improvement: We’ve filled a few dozen missing ones, but still have quite a way to go.

Presentation mistakes

  • Virtually no audio or video content (*New in July 2020)
    • Mistake: Hunching over a computer and reading text all day can get boring. People like to learn about health topics through audio and video, yet we have a near total lack of audio and video on the site.
    • Improvement: This is a surprisingly tough one. When Examine.com does something, we try to do it well, and professionally. All of our resources have been devoted to existing issues, and we've only dabbled lightly in audio and video. For example, this old post on bagged lettuce safety. Our current plan is to test out a few different audio and video formats on social media, see which readers enjoy, and then standardize and perfect those formats.
  • Many of our products were PDF-only (*New in July 2020)
    • Mistake: PDFs have certain advantages. For example, they don't require an internet connection, they can provide information in a standardized and pretty layout. But they're hard to update, and most readers prefer to have information available through browser pages.
    • Improvement: We weren't sure if people would miss the pretty layouts of our PDFs, or value products less because they were available online. We tested one of our Supplement Guides as an online version, and readers loved it. So now the Supplement Guides, Nutrition Examination Research DigestNERD, and Personalized Membershipare all available online.
  • Using dense language (*Updated July 2020)
    • Mistake: The website is not always easy to read, especially in the nitty-gritty “Scientific research” sections. The blog posts are usually easier to read.
    • Improvement: We’ve started making summaries easier to read, but there’s a daunting amount of non-summary information to improve. We hope to make more money, hire more editors, and fix the remaining language. There's still too many other priorities to devote enough resources to simplifying existing pages. If you're a medical copyeditor/copywriter with training in science, who's looking for some work, and loves nutrition research, get in touch with us!
  • Legibility and usability wasn't great (*Updated February 2020)
    • Mistake: Although we have an audience that skews a bit older, we never went through the site to ensure that people with impaired vision or those not as adept at web usage could easily use and navigate the site.
    • Improvement: We’re systematically updating the legibility of the graphics and increasing font sizes. We’re also updating the menus to make them more clear and structured. Later on, we may introduce a bot or quiz to more easily funnel people to the information they’re looking for. Also in the plans are a beginner’s guide to using Examine.com, as well as a beginner’s guide to nutrition.
  • The research digest wasn’t quite digestible enough (*Updated February 2020)
    • Mistake: Our research digest breaks down studies from every angle. We increasingly found that subscribers got overwhelmed.
    • Improvement: We did a year-long round of feedback, and made many changes to the digest to improve usability. The articles are easier to read, the digest is online, etc. It’s also been rebranded as “NERD” now, instead of “ERD”.
  • Site too slow (*Updated February 2020)
    • Mistake: Some of our pages contain a LOT of information. The site was loading too slowly.
    • Improvement: We made several changes to the backend system to improve site speed. We also split out the most in-depth information into separate pages, which improved page loading time.
  • Inconsistent terms around “recommending” supplements (*Updated February 2020)
    • Mistake: When listing dosages that were used in trials, we sometimes indirectly phrase it (e.g. “Trials typically used between X-Y mg/d”), and sometimes used more direct language like “Take X mg/d”).
    • Improvement: We’ve figured out most of the contexts where we want to use indirect vs direct language, so as to not mislead readers. There is a lot of language to update though, throughout the site.
  • Overly-long blog posts without enough structure (*Updated February 2020)
    • Mistake: We wrote several very long, very detailed blog posts in the past few years. While these interested some people, many readers’ eyes glazed over. These posts were surprisingly costly to research, write, and copyedit. They also didn't adhere to the structured format of supplement and diet entries on the site (with a summary, FAQ section, research matrix, etc).
    • Improvement: We’re focusing on systematic entries for supplements/diets/nutrients, rather than having more and more narrative blog posts. With these systematic entries, we can build up pages over time as we gauge interest in different topics. For example, we could cover the most important meta-analyses and randomized trials for a given diet, then cover further studies if the topic is of interest.
  • Late to the game for Instagram and other social platforms (*Updated February 2020)
    • Mistake: Because we don’t give flashy proclamations or controversial hot takes, preferring to present the research as it is, our social media presence is not that big (compared to the vast amount of information on the site). We also failed to focus on social media early on, even though that’s how many people get their information. Lastly, we didn’t always have consistent branding/colors/etc in social media images.
    • Improvement: We’re working with our graphic designers to figure out what to convey to social media users, and in what format.

Organizational mistakes

  • Lack of diversity (*Updated July 2020)
    • Mistake: Our team is mostly comprised of men in their 30s and 40s. Meanwhile, our readership is evenly split between men and women, and it includes many older folks. In order to better address health issues facing women and older people, it would help to have some researchers who are women and/or older. A team of men in their 30s and 40s is a bit more likely to focus on issues more relevant to their demographic and have less expertise on issues that impact women and older people more. One of our major goals is removing subconscious biases and blind spots, in order to be more accurate and more helpful to readers.
    • Improvement: In our last hiring round, we blinded the applications and test assignments for more objective evaluation, and both eventual hires ended up being male. In our current efforts, we’ve made it a point to also proactively find researcher applicants with interest and expertise in women’s health and healthy aging, as well as female and older researchers. It's difficult to hire a late-career (typically older) life science researcher, due to salary expectations and relative scarcity of people looking for job with those characteristics, but we have added a later-career MD/PhD reviewer.
  • Relying on surveys rather than interviews (*Updated July 2020)
    • Mistake: We frequently surveyed readers to find out what interests them. Yet we did few in-depth interviews to find out more about the health journeys of different readers. Knowing more about the full scope of readers’ experiences should help us provide helpful information in the best possible formats.
    • Improvement: We’ve conducted many more in-depth interviews in the past year than we did in the previous years combined, to figure out how we can improve our content and reading experience. We're also asking a small subset of dedicated readers more frequent, targeted questions to help shape some of our decisions. This is to make sure we aren't limited by the perspectives of our small team. Lastly, we may also perform focus groups in the future.

Stay up to date on nutrition and supplementation

Enter your email address and we’ll keep you on top of the latest nutrition research, supplement myths, and more.

This document was last updated on February 27, 2020.