Once a wise man said: “Be always wary of scientific studies trumpeted by mainstream news outlets as groundbreaking. Once the smoke settles down, the study in question is rarely groundbreaking, but rather limited with a lot of caveats”. If I have to summarize this study, that would be within the lines of the wise man. Despite what news outlets have been selling, this is a study that has its own merits, but its methodological limitations and caveats outweighs the novelty and significance. Especially considering the publication occurred in Scientific Reports, the response of Nature Publishing Group to the open-access model, this is an extra layer of concern as Scientific Reports prestige and quality has suffered major setbacks in the last few years due to papers retracted for blatant scientific misconducts that should have been spotted by reviewers.
About the authors: We have three authors. The first author seems to be a postdoc, as she apparently graduated from the same program from another lab. The second author maybe an undergrad, although a faculty with the same family name is listed. Finally the senior author is a faculty with an expertise (based on the publication record) on gastrointestinal (GI) tract physiology and pathophysiology. However, none of them seems to have a history of publication in any field of neurosciences. This is an important point, because it explains a lot of methodological flaws that anyone with neural stem cell biology (and brain development) could pick easily.
1. Introduction and hypothesis: The authors are basically using the rationale of changes in metabolomics observed in ASD patients and reported by several studies. In these studies, there are some indications that certain patients on the spectrum (especially those qualified as severely disabled) display an impaired GI function, in particular something we could qualify as something similar to inflammatory bowel diseases. There are studies suggesting that such GI condition is associated in a changes in the gut microbiota, yet with a fairly low resolution (we are able to document changes in a family of bacteria, but not able to pinpoint to the level of Genus species yet). In particular, there is a study that was recently published in Cell Stem Cells (very high impact factor journal) that highlighted changes in mice behavior and gene expression profile of several genes associated to autism following the fecal transplant from patient on the spectrum considered severe (https://www.cell.com/cell/fulltext/S0092-8674(19)30502-1).
In this study, the authors speculate that certain metabolites biosynthesized and/or bio transformed by these class of bacteria are contribution to the symptoms. In particular, the authors consider acetate (AC, CH3-COOH) , propionate (PPA, CH3-CH2-COOH) and butyrate (BA, CH3-CH2-CH2-COOH) as potential culprits, citing studies showing an elevated levels of these small chain fatty acids (SCFA) in fecal cultures of ASD patients compared to control patients. The authors also cite two rare genetic diseases such as neonatal propionic acidemia (PA) and propionyl coA carboxylase (PCC). PA will be very useful to us, because it will help us set what we would consider a pathological level of propionic acid (PPA) in blood. Yet comes the most speculative, and I would say the “jumping the shark” moment of the paper. The authors assume that since processed foods are rich in PPA, such amount of PPA can lead to the development of ASD in the fetal brain during pregnancy. That’s a lot of speculations with little or no layers of evidence. We have here the authors trying to make a statement four to five steps too far from the existing literature for several reasons (and also lakcing the literature backing them up) that be identified as the following:
1. Where is the literature providing evidence that these SCFA cross the GI tract, at which extent (bioavailability studies)?
2. What are the levels of PPA (and other SCFA) in the plasma/serum levels of neurotypic patients versus patients on the spectrum? For patients suffering from PA, I have found an old study referring to serum PPA as high as 0.337-1.35mM, with a normal level about 0.00337mM (https://www.jpeds.com/article/S0022-3476(81)80004-2/pdf)
3. How much of PPA can cross the blood-brain barrier? This is an important question to answer. We can try to build on the analogy of the SCFA to ketone bodies (acetoacetate, beta-hydroxybutyrate) that are formed when someone is fasting or forced into a ketogenic diet. Considering that patients suffering from GLUT1 Deficiency Syndrome are showing improvement when put on a ketogenic diet (with BHB levels around 2-3mM), we can speculate these compounds can cross the BBB readily.
4. How much of PPA can cross the placental barrier? I don’t have any clue either.
These talking points are important because it will determine if the experimental design of a study is sound or deeply flawed, which eventually will set the quality of the paper and the robustness of the conclusion made. That’s something I should mention by now, is that both the authors and the news outlets have been very fond of superlatives and trying to sell that paper at much higher level that it is meant to. Not only it is scientifically inadequate to make extraordinary conclusions without highly robust data to support them, but also it is nowadays dangerous to do so as such studies will be used by scammers, charlatans and other snakeoil salesmen to promote their supplements claimed to “cure autism” and use these type of studies to claim their products is supported by science.
2. Materials and methods: The authors used neural stem cells (NSCs) derived from fetal tissues (obtained from Life Technologies/Thermofisher) and maintained in a classical medium formulation aimed to maintain these NSCs in their pluripotency stage. The cells were passaged no more than three times, according to the author. This is important, as NSCs/NPCs passaging over time will coax them towards the astrocytes lineage compared to neurons (in terms of development, neurons appear earlier and mature earlier than astrocytes).
Now there is something intriguing. The authors claimed they looked at PPA and BA at concentrations ranging from 0.1mM (that would be physiological), 0.5, 1 and 2mM concentration. Considering such treatment would reproduce the fetal brain, we have to factor in what is the amount capable to cross the three barriers: GI barrier of the mother, the placental barrier and finally the BBB. However, the authors never showed what happened to these concentrations except the 2mM which is limit deadly (remember? This is the one that is detected in newborns having the rare genetic mutation). Are the authors trying to model the effect of PPA to model such diseases or are they assuming that the amount of PPA in processed found will be high enough to put the pregnant women into a severe metabolic acidosis? I don’t know but that like a red flag.
The differentiation of these NSCs into neurons/astrocytes were left to occur in a fairly random fashion, as the authors use the same medium used for NSCs maintenance but without growth factors. I think this is an important issue here as we may have a significant variability in terms of yield between passages in particular when it comes to neurons/astrocytes ratio (personal communication with Clive Svendsen).
Otherwise, nothing else really fancy and classical techniques found in any neurosciences studies: Immunocytochemistry, neurite outgrowth, qPCR…
3.1. Figure 1: I am a bit perplexed from what I see and from what I get when it comes to quantifications. The first issue I have is the lack of scale bar. A scale bar tells you how many pixels equals to a length. For example, a 512×512 pixel image may indicate you that 100 pixels equals to 50 micrometers. Here you have to trust that the experimenter did not fudge the data, crop the pictures and really show you a 10x magnification.
For simplicity, I will focus on the Day 10. My concern is about the health of the neurospheres in some of the groups, in particular the BHB-treated group. You can see in controls; we have nice rosette-shaped neurospheres with a dark core reflecting a dense. In contrast, look at the BHB treated group. These neurospheres are small, frail and lack the morphology observed in control. I wonder if BHB at this concentration is showing signs of toxicity? If yes, the authors were not concerned at all by this issue. And this is something concerning. If BHB is neurotoxic, so how can we make a conclusion to BHB as inhibitor. There are ways to show the viability of these neurospheres: Hoechst staining, propidium iodide staining, Fluorojade staining……..Because of this important issue, I will not consider the BHB treatment as valid.
If you compare the data shown in Fig.1B and 1C compared to the quantification made in Fig.1A (and shown in the bottom), you can see we have a certain discrepancy here. I would skip the issue in the y-axis labelling (the correct symbol for micrometers is µ (mu) not n(nu)), but compare the 10 days timepoint to the data we actually obtain from the Fig.1A. In scientific publications, you have to be sure that what your representative blot/micrograph picture shows matches your quantitative analysis. In other words, what I see in the micrograph pictures in Fig.1A should be reflected in Fig.1B and 1C.
Then explain me why the differences in diameter reported is not as different between my quantitation (using ImageJ internal functions) is different from the one displayed. How do the authors justify the use of SEM instead of SD, except for making the graph look nicer (you can see the data actually suggest a much more variability that likely undermine the statistical difference)? How does the authors explain the 3x difference in neurosphere counts between me and them? Did they crop the pictures? If so they should have accounted for, and highlights the importance of having a scale bar in micrographs pictures and normalizing such data to a surface area (e.g. pixels2, µm2, mm2….)
3.2. Figure2: These are immunofluorescence pictures of plated neurospheres allowed to differentiate by their own on Matrigel-coated plates (Cultrex). The pictures are okay, although not very convincing for some and certainly not suitable for quantification. The use of flow cytometer is definitively a go-to when it comes to assessing cell populations.
We are also here having a mixed results and missing important cell markers. First, the authors should have performed a nestin staining, as nestin is one of the markers present in NSCs/NPCs. Second, the use of GFAP as astrocytes marker has to be taken with a big grain of salt. I am not sure experts in the field would have let this fly with just one marker. GFAP can be expressed by NSCs and NPCs. Showing at least two markers per cell lineages (NeuN/bIII tubulin for neurons, S100B/GLAST1 for astrocytes) would have been much more convincing. bIII tubulin antibody (in particular the one used in this paper) is known to have a very strong non-specific staining. A good bIII tubulin would show nice neurites. I have attached a picture of an iPSC line developed by Sigma-Aldrich. You can use it to compare so you can see what a good bIII tubulin and a good GFAP staining should look like in NPC-derived astrocytes. Here we have just some blobs (that indicates a possible non-specific immunoreactivity) that dangerously overlap with GFAP. Technically, you cannot have a neuron that express both GFAP or bIII tubulin. It is either or but not both. R&D Systems has a nice interactive map that shows you the different cell markers expressed by the neural lineage as its differentiate into neurons and astrocytes here: https://www.rndsystems.com/pathways/neural-stem-cell-differentiation-pathways-lineage-specific-markers
What I am supposed to do with that?
3.3. Figure 3: The authors looked at both GFAP and bIII tubulin at mRNA levels (PCR) and protein levels (by ELISA). I would have personally put the PCR data first, followed by the ELISA data. The PCR data was normalized to GAPDH and the DeltaDeltaCt method was used, which is good, the authors also have represented the apparent bIII tubulin or GFAP/GAPDH ratio, which is good. However, I am more skeptical on the ELISA data. The reason why? The data is represented as micrograms of protein/microliters of cell extract. I am skeptical why the authors did not run a Western-blot analysis for these two housekeeping genes, since you expect a lot of proteins being expressed. The authors also forgot to mention if they have diluted the samples or just added the crude extract at is. This is important because you can easily blunt the accuracy of your ELISA. LSBio is honestly a cheapskate when it comes to showing a standard curve, unlike more established ELISA kits manufacturer such as R&D Systems or Abcam, that will show you their standard curves and tell you the coefficient of variation in them. The maximum concentration of the standard curve is 1000pg/mL or 1ng/mL, with a detection range of 15.63-1000pg/mL and a sensitivity of 9.38pg/mL. The authors reported concentrations for GFAP was 0.8-3 pg/ml according to their graphs. Something is wrong here, and we have at least 2 reviewers that completely miss that. Are the authors telling me that they were able to detect GFAP and bIII tubulin below the sensitivity level (9.38 and 313pg/mL respectively)? Give me a break! I would also have advised the authors to normalize their concentrations into something meaningful like mg of proteins. It is easy to take a fraction of the cell lysate and measure the total protein concentration by BCA. I ask my students whenever they use an ELISA for quantifying a cellular protein to normalize their amount detected (pg/mL) to a total protein concentration (mg/mL) which allows us to normalize the data. Failure to do this normalization is like showing a Western-blot without a proper loading control (e.g. actin, GAPDH….).
3.4. Figure 4: In this figure, the authors are trying to show the expression of GPR41 (aka free fatty acid receptor 3 or FFAR3) in those cells. Honestly, this is my breaking point of tolerance. First, the authors underwent some cherry-picking of the data, showing you only the PPA treatment in astrocytes (where is the BA treatment? Where are the BHB treatments?) and only the BA treatment in neurons (where is PPA? Where are BHB?).
I am also very skeptical that what the authors call astrocytes are really astrocytes looking. What we see in Fig.4A looks very similar to 4B: very thin cytoplasmic projections looking like neurites. Only neurons form neurites in cultures. Astrocytes have more a flat-shaped feature, sometimes a bit fusiform like shape. Again, the GPR41 protein expression is really up when you have tons of PPA given (mM and more). How come this went through peer-review unabated and have at least 2 reviewers did not notice this gross conundrum in the data?
4. Rest of the figures and conclusion: I can go further with this paper. It was looking very interesting and promising, but the lack of expertise from the authors quickly percolated into loose and inconclusive data. This is the kind of paper you wish the authors would seek feedback from across the street, from some faculty with a neuroscience background and give them an honest feedback to make this paper good and scientifically sound. What we have indeed is a half-baked study, served as the next big thing since sliced bread. Not only the data is far from convincing of the claims made my authors (I would probably accept as a possible model for modelling PA or PCC, but this paper IS ABSOLUTELY NOT SHOWING THAT PPA IS CAUSING AUTISM for several reasons below:
1. It does not account for the PPA levels found in normal persons, even less provide a study showing PPA levels in people eating processed foods (if such dietary habit even lead to such outcome).
2. It does not consider that in order to be valid the authors have to show that you have a 100% bioavailability of PPA across the GI barrier, the placental barrier and the BBB, which are not reported or cited by the authors in any credible form.
3. It does not account that the levels used as so ridiculously high that a pregnant mother would deal with a possibly deadly metabolic acidosis.
4. It also ignored that BHB was showing signs of neurotoxicity.
5. There is a worrisome pattern of data cherry-picking, with groups popping in and out intermittently, sometimes even in a complacent manner. This is a no-no and an unacceptable behavior that has no place in any respected peer-reviewed journals. Why did the reviewers overlooked that issue?
6. There are several inconsistencies in the data, especially whether the axis labels are botched or if the authors really provided measurements that were nornally impossible to reach (below detection limit).
This paper should at least had a “major revision” to fill the gaps. Yet, it went through at least 2 reviewers and none of them were able to see the obvious methodological flaws. As a reviewer for Sci Rep on a seldom basis, I am very concerned about the quality of review provided by the journal in the recent years, especially in light of series of retraction. Conclusions? The news outlets have been trying to sell an overhyped paper that does not live much under scrutiny. This is just “same old, same old” when it comes to journalistic reporting on science (trying to fudge it as groundbreaking), but also opens a dangerous precedent. I will bet that within 12 months, there will be some quack doctors and snake oilsalesmen that will claim they can cure autism by selling you supplements aimed to reduce the PPA or by selling you a dietary fad book, claiming it will cure your child autism by dietary restriction. I guess the keto diet will soon join the casein-free/gluten-free diet as outdated and have another fad being served as dietary torture to children on the spectrum.