*Updated November 2021*
Here's a running list of things we’ve gotten wrong since our founding in 2011. We update it regularly.
The purpose is to keep Examine transparent and accountable. That’s also why we don’t have external ads on the website, or accept funding from food or supplement companies. Below is our (ever-growing!) list of mistakes, color-coded for easy scanning.
Red: Still a long way to go
Yellow: Making good progress
Green: Mostly solved
- We don’t have enough gut health content
(*Updated November 2021)
- Mistake: Everybody knows the outsized role of gut health in overall wellness. However, we waited quite a while to make our big content push. Part of this was due to other content being made, but part was due to nervousness. Gut health is complex, and a lot of shoddy information is thrown out on the web. We didn't want to contribute to that shoddiness.
- Improvement: We hired a researcher who exclusively researches gut health. This has made it much easier to organize our efforts around gut health topics, because she's aware of the latest findings and has a balanced perspective on a variety of gut issues. New pages have already been written, and are being released every month or two.
- Overwhelming website without clear enough structure (*Updated November 2021)
- Mistake: Examine has a LOT of information on it, and it can get confusing. For example, our topics page is a mix of relatable health outcomes (such as anxiety) and very specific health outcomes (such as fibrinogen and lipoprotein lipase activity).
- Improvement: We're a little over half-way done with our new website structure. We'll still have all the existing pages, but now we'll add pages to tie everything together more clearly: new category pages to tie together various health conditions, a new FAQ structure to answer your questions more efficiently, and more. And we'll have a much, much better search function and menu to make it easier to find the effects you're looking for!
- Not enough page updates and new pages
(*Updated April 2021)
- Mistake: We had more frequent new content in the first years of our existence. In the subsequent years, we spent more time and effort on products and internal organization due to financial pressures.
- Improvement: This is pretty much solved. We hired two new full-time researchers in the past few months, and are bringing on a few part-time researchers next week. This was made financially possible by a shift toward membership plans for Examine supporters who want more detailed access to our research. Due to these changes, we've been updating pages much more regularly, and have several new pages queued up as well.
- Missing systematic entries on popular diets and food topics (*Updated April 2021)
- Mistake: We have always had a ton of well-researched pages on supplements, but relatively few systematic pages on diets and foods. For example, we have several blog articles on intermittent fasting, but for a long time we lacked pages on specific fasting protocols (e.g., 16:8 or ADF).
- Improvement: Recently, we started rolling out pages on specific fasting protocols, including on 16:8 fasting and OMAD. Earlier, we made thorough pages on CBD and the ketogenic diet, two of our bigger gaps. We also made pages on certain popular health foods, such as apple cider vinegar. But we have a ways left to go, with many diets and foods left to systematically cover. We do provide information about some of these throughout the site and in our products, but not on systematic pages yet.
Not enough "outside the box" treatments (*Updated November 2021)
- Mistake: If you have a chronic health condition that's somewhat rare, or difficult to treat or diagnose, then you know how frustrating it can be when traditional approaches don't work. People may dismiss you. The old medical saying goes "When you hear hoofbeats behind you, don't expect to see a zebra". So what if you're a zebra, and you don't fit into neat boxes? Examine can already be of some help, but we could do much, much better.
- Improvement: Over 2021, this mistake went from "red" to "yellow". Hooray! But it's still at yellow. Booooo. We've been feeding new studies on outside the box treatments into our membership database, and as treatments reach a threshold number of studies, we'll start crafting pages for each one. The treatments range from meditation to "forest bathing" to psychedelics! We're also planning a page covering trials on video games (with various trials looking at either benefits and detriments).
- Not enough focus on chronic health conditions
(*Updated April 2021)
- Mistake: There’s nothing wrong with trying to improve one’s physique, and more muscle and less fat is great for health. We're always going to stay up to date on those topics. But the public health burden of chronic health conditions is massive and accelerating. Disease improvement can be, relatively, much more complex and difficult than physique improvement.
- Improvement: We had many, many internal debates about how to approach this. Should we focus deeply on a small handful of common conditions, or expand to a few dozen conditions? Eventually we went down the middle road, and systematically compiled the roughly 60 conditions that fit one or more of these characteristics: highly-prevalent, amenable to improvement through diet and/or supplementation, or difficult to treat through traditional medical routes. Our researchers set up rigorous Pubmed search strings, and we now collect most of the relevants study on these conditions each month. They're fed into our Membership, which is how we fund our efforts to keep abreast of the latest research each month through these resource-intensive means.
- Gaps in micronutrient research (*Updated November 2021)
- Mistake: Some nutrient pages had rather large gaps in the studies they covered. We’re also missing a few nutrients, such as certain trace minerals.
- Improvement: We added studies to the iron and vitamin C pages a few months ago. While we planned further updates over the next few months, other topics such as gut health and herbal supplements were more in need of updates, so micronutrient updates were shifted further back in the queue.
- Not standardizing how we list study funding sources (*Updated February 2020)
- Mistake: Everybody wants to know who funds supplement and diet studies. And for good reason, since industry funding may introduce a conflict of interest. However, listing this for every study with industry funding would take eons to do, since we have so many studies on the website. It could also make articles more difficult to read, and can even mislead readers in a different way, as industry funding doesn’t inherently mean the study is flawed or biased, or that the results aren’t accurate. Categorically “unhealthy” foods such as colas can have biased studies, but so can foods with more mixed evidence, such as red meat, eggs, or soybean oil. All of these have studies funded by industry, government, or both. And non-industry-funded studies can also have bias, as there are several possible sources of bias other than the funder.
- Improvement: We discussed having a line item on pages listing how many studies were industry funded, but that didn't provide enough context to be useful. We will likely try a couple of different methods to make the nuances of funding and bias apparent to readers. Once we decide on a route to take, we’ll have to go back and edit many, many pages in our archives.
- Blood sugar supplement side effects (*Updated February 2020)
- Mistake: The side effects of blood-sugar supplements are not always fully understood, due to a shortage of research. We didn’t always note that important fact, especially given that there is no guarantee that they won’t cause hypoglycemic events (especially when mixed with other supplements or medications).
- Improvement: We’ve been adding this information to our supplement entries and Blood Sugar Supplement Guide.
- Missing page summaries (*Updated November
- Mistake: Some pages are missing quick summaries (the 2–3 sentences at the very top of the page), while many are missing full summaries (the longer paragraphs below that).
- Improvement: We’ve filled a bunch of missing ones, but still have quite a way to go. Complicating matters is that we recently decided to somewhat standardize the format (length, details, etc) of many of the quick summaries.
- The Examine website looks quite dated
(*Updated November 2021)
- Mistake: The website is functional but man oh man does it look dated! There's not much white space, we don't use illustrations, we don't make use of collapsible information well enough, yadda yadda.
- Improvement: A complete overhaul and redesign is coming in a few months! We sent out a detailed request-for-proposal to a bunch of different design firms, found several great candidates, and then hired our first legit UX/UI firm to help us redesign the site. They're doing a full site and user experience audit as I type this!
- Membership options were confusing (*New in
- Mistake: Our first subscription product was the Examine Research Digest, which came out in 2014, and did reaaallly deep dives into a few of the most important new studies each month. It used to be called ERD, then became NERD (Nutrition Examination Research Digest). In 2020, we came out with Examine Personalized, which gave quick summaries of lots of new studies in areas that the member was personally interested in (Hence the name!). NERD members automatically got Examine Personalized. In between, we also had another less-used membership option, which I won't even get into, since you can see how confusing the options already were.
- Improvement: Merge all the things! After many super-long conversations among team members to hash out details and projections for revenue, growth, and member satisfaction, we merged NERD and Personalized, and now it's just called the Examine Membership.
- Virtually no audio or video content (*Updated
- Mistake: Hunching over a computer and reading text all day can get boring. People like to learn about health topics through audio and video, yet we have a near total lack of audio and video on the site.
- Improvement: This is a surprisingly tough one. When Examine does something, we try to do it well, and professionally. All of our resources have been devoted to existing issues, and we've only dabbled lightly in audio and video. For example, this old post on bagged lettuce
safety. But we've made a reasonable plan for audio and video content, and are proceding slowly. We've tested out some audio on Instagram, and will have a Youtube channel hopefully some time in the next few months. We may even embed audio and/or video summaries into some of our webpages next year. Perhaps most exciting is our podcast that's planned for early 2022!
- Study summaries didn't provide an at-a-glance takeaway.
(*Updated November 2021)
- Mistake: When we started writing our 200+ study summaries each month in the summer of 2020, the summaries were quite short. But as they became longer, it was harder for readers to get all the way through them without having a quick takeaway provided.
- Improvement: This one is pretty simple. We added a summary blurb right at the top, so you can scan the takeaways much more quickly.
- Site was too slow (*Updated November 2021)
- Mistake: Some of our pages contain a LOT of information. The site was loading too slowly.
- Improvement: Our tech team is more than halfway done doing a full rewrite of our code base. Whew! This has taken a long time and cost quite a bit of time and money, but it will result in faster load times for you and more future flexibility for us, as we add more features to the site.
- We haven’t experimented with interactive graphics. (*New in November 2021)
- Mistake: We love websites that have interactive graphics. Especially when they're packed with data! But we never took the time to brainstorm these graphics for Examine.
- Improvement: This is on hold. We thought of a few potentially helpful interctive graphics, but we're in the process of redesigning the entire site and don't have the time to make these fairly work-intensive interactive graphics at the moment.
- Using dense language (*Updated November
- Mistake: The website is not always easy to read, especially in the nitty-gritty “Scientific research” sections. The blog posts are usually easier to read.
- Improvement: We hired a new full time medical editor in the beginning of 2021, plus an additional part time copyeditor in the fall. Combined with our existing two copyeditors, we now have a solid plan for tackling this issue. Make no mistake, there’s still a daunting amount of information to improve. But we have a plan.
- Late to the game for Instagram and other social platforms (*Updated April 2021)
- Mistake: Because we don’t give flashy proclamations or controversial hot takes, preferring to present the research as it is, our social media presence is not that big (compared to the vast amount of information on the site). We also failed to focus on social media early on, even though that’s how many people get their information. Lastly, we didn’t always have consistent branding/colors/etc in social media images.
- Improvement: We're still not social media geniuses. Part of it is that we focus on substance over flashiness, and part is that we spend most of the day analyzing papers and writing about the analysis. But in the past few months, we’ve increasingly worked with our graphic designers to get more Instagram posts out, and started doing a bit of interactive Instagram content as well. We also may have a YouTube channel arriving in the coming weeks.
- Many of our products were PDF-only (*New in July 2020)
- Mistake: PDFs have certain advantages. For example, they don't require an internet connection, they can provide information in a standardized and pretty layout. But they're hard to update, and most readers prefer to have information available through browser pages.
- Improvement: We weren't sure if people would miss the pretty layouts of our PDFs, or value products less because they were available online. We tested one of our Supplement Guides as an online version, and readers loved it. So now, both our Supplement Guides and our Membership are available online.
- Legibility and usability wasn't great
(*Updated February 2020)
- Mistake: Although we have an audience that skews a bit older, we never went through the site to ensure that people with impaired vision or those not as adept at web usage could easily use and navigate the site.
- Improvement: We’re systematically updating the legibility of the graphics and increasing font sizes. We’re also updating the menus to make them more clear and structured. Later on, we may introduce a bot or quiz to more easily funnel people to the information they’re looking for. Also in the plans are a beginner’s guide to using Examine (Jul 2021: Done!), as well as a beginner’s guide to nutrition.
- The research digest wasn’t quite digestible enough (*Updated November 2021)
- Mistake: Our membership research digest breaks down studies from every angle. We increasingly found that subscribers got overwhelmed.
- Improvement: We did a year-long round of feedback, and made many changes to the digest to improve usability. The articles are easier to read, the digest is online, etc. Eventually, we completely rolled the research digest into our monthly study summaries. Now they're very digestible, and link out to pages on the site with more permanant non-study-specific information.
- Inconsistent terms around “recommending” supplements (*Updated February 2020)
- Mistake: When listing dosages that were used in trials, we sometimes indirectly phrase it (e.g. “Trials typically used between X-Y mg/d”), and sometimes used more direct language like “Take X mg/d”).
- Improvement: We’ve figured out most of the contexts where we want to use indirect vs direct language, so as to not mislead readers. There is a lot of language to update though, throughout the site.
- Overly-long blog posts without enough structure
(*Updated February 2020)
- Mistake: We wrote several very long, very detailed blog posts in the past few years. While these interested some people, many readers’ eyes glazed over. These posts were surprisingly costly to research, write, and copyedit. They also didn't adhere to the structured format of supplement and diet entries on the site (with a summary, FAQ section, research matrix, etc).
- Improvement: We’re focusing on systematic entries for supplements/diets/nutrients, rather than having more and more narrative blog posts. With these systematic entries, we can build up pages over time as we gauge interest in different topics. For example, we could cover the most important meta-analyses and randomized trials for a given diet, then cover further studies if the topic is of interest.
- We don't convey who's working at Examine very well (*New in November 2021)
- Mistake: We have some very cool, very smart researchers at Examine. But when you read Examine or subscribe to our emails, you don't hear much about them. A personal connection is missing.
- Improvement: We have a podcast in the works. Oh yes, a podcast. But while that's brewing, we have some other ideas for letting you know who exactly we are. Stay tuned as some of these ideas become reality. Not everyone cares to have a more personal connection to the websites they read, but we're small enough that it might be nice to know who's doing the research you're reading!
- Easily-preventable website errors (*New in April 2021)
- Mistake: We have an "errors" channel on Examine's Slack. Several times a week, team members notice errors and post them for fixing. Eventually, we wondered if a proactive error-checking system could help speed up corrections.
- Improvement: Six team members are assigned different aspects of Examine for monthly error screening. For example, checking if various types of pages display correctly on different devices, double-checking parts of the study database, checking if new readers and new members would encounter any errors or things that don't make sense, etc. We've been catching some errors, and will see how this plays out in the next few months.
- The mystery of customers double-purchasing products (*New in April 2021)
- Mistake: We noticed that a surprising number of customers had duplicate products listed in their purchases. We talked to some customers who had bought a product and then bought it again later, forgetting that it was already purchased. Apparently, this had been going on at a low level for years, and it eventually built up into many, many purchases.
- Improvement: Although it was costly, we notified these customers and refunded each of them for their duplicate purchases.
- Enforcing work/life balance policy (*New in April 2021)
- Mistake: While Examine has a pretty generous time-off policy, a combination of time-crunched projects and not enough staff lead to a few team members not taking enough time off.
- Improvement: If an Examine team member doesn't reguarly take some time off, they get a notice that they have to take some time off soon. We also hired more staff, making that policy logistically feasible.
- Lack of diversity (*Updated April 2021)
- Mistake: Our team is mostly comprised of men in their 30s and 40s. Meanwhile, our readership is evenly split between men and women, and it includes many older folks. In order to better address health issues facing women and older people, it would help to have some researchers who are women and/or older. A team of men in their 30s and 40s is a bit more likely to focus on issues more relevant to their demographic and have less expertise on issues that impact women and older people more. One of our major goals is removing subconscious biases and blind spots, in order to be more accurate and more helpful to readers.
- Improvement: In our last hiring round, we blinded the applications and test assignments for more objective evaluation, and ended up hiring a mix of men and women of different age ranges. We've also proactively contacted potential applicants from a wide variety of backgrounds. For example, PhDs, health enthusiasts without advanced degrees, people with deep knowledge of particular areas, people from different countries, etc.
- Relying on surveys rather than interviews
(*Updated July 2020)
- Mistake: We frequently surveyed readers to find out what interests them. Yet we did few in-depth interviews to find out more about the health journeys of different readers. Knowing more about the full scope of readers’ experiences should help us provide helpful information in the best possible formats.
- Improvement: We’ve conducted many more in-depth interviews in the past year than we did in the previous years combined, to figure out how we can improve our content and reading experience. We're also asking a small subset of dedicated readers more frequent, targeted questions to help shape some of our decisions. This is to make sure we aren't limited by the perspectives of our small team. Lastly, we may also perform focus groups in the future.