*Updated April 2021*
Here's a running list of things we’ve gotten wrong since our founding in 2011. We update it regularly.
The purpose is to keep Examine transparent and accountable. That’s also why we don’t have external ads on the website, or accept funding from food or supplement companies. Below is our (ever-growing!) list of mistakes, color-coded for easy scanning.
Red: Still a long way to go
Yellow: Making good progress
Green: Mostly solved
- Overwhelming website without clear enough structure (*Updated April 2021)
- Mistake: Examine has a LOT of information on it, and it can get confusing. For example, our topics page is a mix of relatable health outcomes (such as anxiety) and very specific health outcomes (such as fibrinogen and lipoprotein lipase activity).
- Improvement: We've been working diligently on this, and have made much progress! Long story short, the website will be structured around cause-and-effect. When readers go to Examine, they're typically interested in effects. Sometimes it's supplement or diet focused ("What are the benefits and harms of fish oil?"), sometimes it's condition focused ("What all could help my anxiety?"), and sometimes it's comparative ("How does krill oil compare to fish oil?"). We'll still have all the existing pages, but now we'll add pages to tie effects together more clearly: new pages on health conditions, new FAQs, and more. And we'll have a better search function and menu to make it easier to find the effects you're looking for!
- Not enough page updates and new pages
(*Updated April 2021)
- Mistake: We had more frequent new content in the first years of our existence. In the subsequent years, we spent more time and effort on products and internal organization due to financial pressures.
- Improvement: This is pretty much solved. We hired two new full-time researchers in the past few months, and are bringing on a few part-time researchers next week. This was made financially possible by a shift toward membership plans for Examine supporters who want more detailed access to our research. Due to these changes, we've been updating pages much more regularly, and have several new pages queued up as well.
- Missing systematic entries on popular diets and food topics (*Updated April 2021)
- Mistake: We have always had a ton of well-researched pages on supplements, but relatively few systematic pages on diets and foods. For example, we have several blog articles on intermittent fasting, but for a long time we lacked pages on specific fasting protocols (e.g., 16:8 or ADF).
- Improvement: Recently, we started rolling out pages on specific fasting protocols, including on 16:8 fasting and OMAD. Earlier, we made thorough pages on CBD and the ketogenic diet, two of our bigger gaps. We also made pages on certain popular health foods, such as apple cider vinegar. But we have a ways left to go, with many diets and foods left to systematically cover. We do provide information about some of these throughout the site and in our products, but not on systematic pages yet.
Not enough "outside the box" treatments (*Updated April 2021)
- Mistake: If you have a chronic health condition that's somewhat rare, or difficult to treat or diagnose, then you know how frustrating it can be when traditional approaches don't work. People may dismiss you. The old medical saying goes "When you hear hoofbeats behind you, don't expect to see a zebra". So what if you're a zebra, and you don't fit into neat boxes? Examine can already be of some help, but we could do much, much better.
- Improvement: Over the winter, this mistake went from "red" to "yellow". Hooray! We've been feeding new studies on outside the box treatments into our membership database, and as treatments reach a threshold number of studies, we'll start crafting pages for each one. The treatments range from meditation to "forest bathing" to psychedelics!
- Not enough focus on chronic health conditions
(*Updated April 2021)
- Mistake: There’s nothing wrong with trying to improve one’s physique, and more muscle and less fat is great for health. We're always going to stay up to date on those topics. But the public health burden of chronic health conditions is massive and accelerating. Disease improvement can be, relatively, much more complex and difficult than physique improvement.
- Improvement: We had many, many internal debates about how to approach this. Should we focus deeply on a small handful of common conditions, or expand to a few dozen conditions? Eventually we went down the middle road, and systematically compiled the roughly 60 conditions that fit one or more of these characteristics: highly-prevalent, amenable to improvement through diet and/or supplementation, or difficult to treat through traditional medical routes. Our researchers set up rigorous Pubmed search strings, and we now collect most of the relevants study on these conditions each month. They're fed into our Membership, which is how we fund our efforts to keep abreast of the latest research each month through these resource-intensive means.
- Gaps in micronutrient research (*Updated February 2020)
- Mistake: Some nutrient pages had rather large gaps in the studies they covered. We’re also missing a few nutrients, such as certain trace minerals.
- Improvement: We added studies to the iron and vitamin C pages in the past months, and are planning further updates over the next few months.
- Non standardizing how we list study funding sources (*Updated February 2020)
- Mistake: Everybody wants to know who funds supplement and diet studies. And for good reason, since industry funding may introduce a conflict of interest. However, listing this for every study with industry funding would take eons to do, since we have so many studies on the website. It could also make articles more difficult to read, and can even mislead readers in a different way, as industry funding doesn’t inherently mean the study is flawed or biased, or that the results aren’t accurate. Categorically “unhealthy” foods such as colas can have biased studies, but so can foods with more mixed evidence, such as red meat, eggs, or soybean oil. All of these have studies funded by industry, government, or both. And non-industry-funded studies can also have bias, as there are several possible sources of bias other than the funder.
- Improvement: We discussed having a line item on pages listing how many studies were industry funded, but that didn't provide enough context to be useful. We will likely try a couple of different methods to make the nuances of funding and bias apparent to readers. Once we decide on a route to take, we’ll have to go back and edit many, many pages in our archives.
- Blood sugar supplement side effects (*Updated February 2020)
- Mistake: The side effects of blood-sugar supplements are not always fully understood, due to a shortage of research. We didn’t always note that important fact, especially given that there is no guarantee that they won’t cause hypoglycemic events (especially when mixed with other supplements or medications).
- Improvement: We’ve been adding this information to our supplement entries and Blood Sugar Supplement Guide.
- Missing page summaries (*Updated February
- Mistake: Some pages are missing quick summaries (the 2–3 sentences at the very top of the page), while many are missing full summaries (the longer paragraphs below that).
- Improvement: We’ve filled a few dozen missing ones, but still have quite a way to go.
- Membership options were confusing (*New in
- Mistake: Our first subscription product was the Examine Research Digest, which came out in 2014, and did reaaallly deep dives into a few of the most important new studies each month. It used to be called ERD, then became NERD (Nutrition Examination Research Digest). In 2020, we came out with Examine Personalized, which gave quick summaries of lots of new studies in areas that the member was personally interested in (Hence the name!). NERD members automatically got Examine Personalized. In between, we also had another less-used membership option, which I won't even get into, since you can see how confusing the options already were.
- Improvement: Merge all the things! After many super-long conversations among team members to hash out details and projections for revenue, growth, and member satisfaction, we merged NERD and Personalized, and now it's just called the Examine Membership.
- Virtually no audio or video content (*Updated
- Mistake: Hunching over a computer and reading text all day can get boring. People like to learn about health topics through audio and video, yet we have a near total lack of audio and video on the site.
- Improvement: This is a surprisingly tough one. When Examine does something, we try to do it well, and professionally. All of our resources have been devoted to existing issues, and we've only dabbled lightly in audio and video. For example, this old post on bagged lettuce
safety. But we've made a reasonable plan for audio and video content, and are proceding slowly. We've tested out some audio on Instagram, and will have a Youtube channel hopefully some time this spring. We may even embed audio and/or video summaries into some of our webpages this year.
- Site too slow (*Updated April 2021)
- Mistake: Some of our pages contain a LOT of information. The site was loading too slowly.
- Improvement: Our tech team is currently doing a full rewrite of our code base. Whew! This is going to take a few months and cost some money, but it will result in faster load times for you and more future flexibility for us, as we add more features to the site.
- Using dense language (*Updated April
- Mistake: The website is not always easy to read, especially in the nitty-gritty “Scientific research” sections. The blog posts are usually easier to read.
- Improvement: We hired a full time medical editor earlier this year! Combined with our existing two copyeditors, we now have a solid plan for tackling this issue. Make no mistake, there’s still a daunting amount of information to improve. But we have a plan.
- Late to the game for Instagram and other social platforms (*Updated April 2021)
- Mistake: Because we don’t give flashy proclamations or controversial hot takes, preferring to present the research as it is, our social media presence is not that big (compared to the vast amount of information on the site). We also failed to focus on social media early on, even though that’s how many people get their information. Lastly, we didn’t always have consistent branding/colors/etc in social media images.
- Improvement: We're still not social media geniuses. Part of it is that we focus on substance over flashiness, and part is that we spend most of the day analyzing papers and writing about the analysis. But in the past few months, we’ve increasingly worked with our graphic designers to get more Instagram posts out, and started doing a bit of interactive Instagram content as well. We also may have a YouTube channel arriving in the coming weeks.
- Many of our products were PDF-only (*New in July 2020)
- Mistake: PDFs have certain advantages. For example, they don't require an internet connection, they can provide information in a standardized and pretty layout. But they're hard to update, and most readers prefer to have information available through browser pages.
- Improvement: We weren't sure if people would miss the pretty layouts of our PDFs, or value products less because they were available online. We tested one of our Supplement Guides as an online version, and readers loved it. So now, both our Supplement Guides and our Membership are available online.
- Legibility and usability wasn't great
(*Updated February 2020)
- Mistake: Although we have an audience that skews a bit older, we never went through the site to ensure that people with impaired vision or those not as adept at web usage could easily use and navigate the site.
- Improvement: We’re systematically updating the legibility of the graphics and increasing font sizes. We’re also updating the menus to make them more clear and structured. Later on, we may introduce a bot or quiz to more easily funnel people to the information they’re looking for. Also in the plans are a beginner’s guide to using Examine (Jul 2021: Done!), as well as a beginner’s guide to nutrition.
- The research digest wasn’t quite digestible enough (*Updated February 2020)
- Mistake: Our membership research digest breaks down studies from every angle. We increasingly found that subscribers got overwhelmed.
- Improvement: We did a year-long round of feedback, and made many changes to the digest to improve usability. The articles are easier to read, the digest is online, etc. It’s also been rolled into our overall site membership, so that members can get really deep dives of notable studies alongside quicker summaries of studies in health areas they choose to follow.
- Inconsistent terms around “recommending” supplements (*Updated February 2020)
- Mistake: When listing dosages that were used in trials, we sometimes indirectly phrase it (e.g. “Trials typically used between X-Y mg/d”), and sometimes used more direct language like “Take X mg/d”).
- Improvement: We’ve figured out most of the contexts where we want to use indirect vs direct language, so as to not mislead readers. There is a lot of language to update though, throughout the site.
- Overly-long blog posts without enough structure
(*Updated February 2020)
- Mistake: We wrote several very long, very detailed blog posts in the past few years. While these interested some people, many readers’ eyes glazed over. These posts were surprisingly costly to research, write, and copyedit. They also didn't adhere to the structured format of supplement and diet entries on the site (with a summary, FAQ section, research matrix, etc).
- Improvement: We’re focusing on systematic entries for supplements/diets/nutrients, rather than having more and more narrative blog posts. With these systematic entries, we can build up pages over time as we gauge interest in different topics. For example, we could cover the most important meta-analyses and randomized trials for a given diet, then cover further studies if the topic is of interest.
- Easily-preventable website errors (*New in April 2021)
- Mistake: We have an "errors" channel on Examine's Slack. Several times a week, team members notice errors and post them for fixing. Eventually, we wondered if a proactive error-checking system could help speed up corrections.
- Improvement: Six team members are assigned different aspects of Examine for monthly error screening. For example, checking if various types of pages display correctly on different devices, double-checking parts of the study database, checking if new readers and new members would encounter any errors or things that don't make sense, etc. We've been catching some errors, and will see how this plays out in the next few months.
- The mystery of customers double-purchasing products (*New in April 2021)
- Mistake: We noticed that a surprising number of customers had duplicate products listed in their purchases. We talked to some customers who had bought a product and then bought it again later, forgetting that it was already purchased. Apparently, this had been going on at a low level for years, and it eventually built up into many, many purchases.
- Improvement: Although it was costly, we notified these customers and refunded each of them for their duplicate purchases.
- Enforcing work/life balance policy (*New in April 2021)
- Mistake: While Examine has a pretty generous time-off policy, a combination of time-crunched projects and not enough staff lead to a few team members not taking enough time off.
- Improvement: If an Examine team member doesn't reguarly take some time off, they get a notice that they have to take some time off soon. We also hired more staff, making that policy logistically feasible.
- Lack of diversity (*Updated April 2021)
- Mistake: Our team is mostly comprised of men in their 30s and 40s. Meanwhile, our readership is evenly split between men and women, and it includes many older folks. In order to better address health issues facing women and older people, it would help to have some researchers who are women and/or older. A team of men in their 30s and 40s is a bit more likely to focus on issues more relevant to their demographic and have less expertise on issues that impact women and older people more. One of our major goals is removing subconscious biases and blind spots, in order to be more accurate and more helpful to readers.
- Improvement: In our last hiring round, we blinded the applications and test assignments for more objective evaluation, and ended up hiring a mix of men and women of different age ranges. We've also proactively contacted potential applicants from a wide variety of backgrounds. For example, PhDs, health enthusiasts without advanced degrees, people with deep knowledge of particular areas, people from different countries, etc.
- Relying on surveys rather than interviews
(*Updated July 2020)
- Mistake: We frequently surveyed readers to find out what interests them. Yet we did few in-depth interviews to find out more about the health journeys of different readers. Knowing more about the full scope of readers’ experiences should help us provide helpful information in the best possible formats.
- Improvement: We’ve conducted many more in-depth interviews in the past year than we did in the previous years combined, to figure out how we can improve our content and reading experience. We're also asking a small subset of dedicated readers more frequent, targeted questions to help shape some of our decisions. This is to make sure we aren't limited by the perspectives of our small team. Lastly, we may also perform focus groups in the future.