The first (hosted by the Finnish Center for Artificial Intelligence) was on Monte Carlo Variational Inference.

This talk tries to summarize recent work on

• Importance-weighted VI (IWVI)
• How to generalize to {antithetic, stratified, quasi-Monte Carlo} VI through dividing and coupling.
• How to generalize to (marginal) particle filter VI.
• How to generalize even further to (uncorrected) Hamiltonian VI.

The second talk (for the Measure Transport, Diffusion Processes and Sampling Workshop organized by the Flatiron Institute) was on Diffusion-based variational inference.

(Apologies for the stupid joke.)

The goal of this talk was to try to understand just exactly how diffusion-based VI (where you sort of “put HMC inside of VI” ) is related to all these diffusion models that everyone is talking about.

The third talk (for the Conference on Approximate Inference in Theory and Practice organized by ESSEC) was on Convergence Guarantees for Variational Inference:

This summarizes progress on a problem I’ve been working on for the last 6 years, namely is black-box VI guaranteed to converge (ever)? We finally have some guarantees, but there’s a lot still to be done.

What is the length of the Mississippi river?

The second group gets these questions:

Is the Mississippi river greater or less than 70 miles long?

What is the length of the Mississippi river?

The third group gets these:

Is the Mississippi river greater or less than 2000 miles long?

What is the length of the Mississippi river?

When Jacowitz and Kahneman tried this in 1995, they found that the groups had median estimates of 800 miles, 300 miles, and 1500 miles, respectively.

In theory at least, the initial greater / less questions provide no information, and so shouldn’t change the final estimates people give. But they do change them, a lot.

This effect is known as anchoring. It’s widely discussed, but the replication crisis has made me paranoid cautious about trusting anything that hasn’t been replicated. So I tried to find anchoring studies with replications.

Anchoring for estimation

A seminal paper from 1995 is Measures of Anchoring in Estimation Tasks by Jacowitz and Kahneman. They created a list of 15 quantities to estimate and took 156 U.C. Berkeley students who were taking an intro to psychology course. They had three questionnaires, one—for calibration—simply asked them to estimate everything. The other two questionnaires first asked if each quantity was more or less than a certain anchor and then asked for an estimate of the quantity.

The results were that anchoring had a massive effect:

Here, the left column shows the median among people who weren’t given an anchor, the next two columns show what numbers were used as low/high anchors, and the last two columns show the median responses among people given low/high anchors.

But does it replicate?

The Many Labs project is an effort from a team of psychologists around the world. In their first paper (Klein et al., 2014) they tried to replicate 13 different well-known effects, including anchoring. They had a large sample of 5000 people from around the world and gave them four estimation problems.

There was one small difference from the original study. Rather than being asked if the population of Chicago was greater or less than the anchor, participants were simply informed of the truth.

And what are the results? You have to dig to find the raw data, but I was able to put it together from https://osf.io/tjx3f/ and https://osf.io/y36m8/.

The effect is real. It replicates.

Effect size mystery

There is a mystery, though. In the Many Labs paper, almost all the replications find smaller effect sizes than the original papers, but anchoring stands out as having a much larger effect size:

In the additional material on anchoring, they speculate about what might be causing the effect to be larger. This is all pretty strange, because here are the raw numbers in the original paper compared to those in the replication:

In every case, the effect is smaller, not larger.

What’s going on? After puzzling over this, I think it comes down to the definition of an “effect size”. It’s totally possible to have a smaller effect (in the normal English-language sense) whilst having a larger effect size (in the statistical sense).

One possibility is that it’s an artifact of how the effect size is calculated. They report, “The Anchoring original effect size is a mean point-biserial correlation computed across 15 different questions in a test-retest design, whereas the present replication adopted a between-subjects design with random assignments.”

Another possibility is that it comes down to the definition. Remember, the effect size is a technical term: The difference in means divided by the standard deviation of the full population. The above table shows that the difference in means is smaller in the replication. So why is the effect size larger? Well, it could be that even though the difference of means is smaller, the standard deviation in the replication is much smaller, meaning the overall ratio is larger.

I can’t confirm which of these explanations is right. This document seems to include standard deviations for the replication, but the original study doesn’t seem to list them, so I’m not sure.

In any case, I don’t think it changes the takeaways:

1. Anchoring for estimation tasks is real, and robust, and has large effects.
2. Be careful when looking at summary statistics. The further something is from raw data, the easier it is to misinterpret.

Bayesian view

Anchoring is real, but someone determined to see the entire world in Bayesian terms might chafe at the term “bias”. This ostensibly-not-me person might argue:

“You say anchoring is a ‘cognitive bias’ because being asked ‘Is the Mississippi river greater or less than 70 miles long?’ is a question, and thus contains no information, and thus shouldn’t change anything. But are you sure about that?”

Let me give an analogy. In a recent homework assignment, I asked students to find the minima of a bunch of mathematical functions, and at the end gave this clarification:

“In some cases, the argmin might not be unique. If so, you can just give any minima.”

Some students asked me, “I’m finding all unique minima. What am I doing wrong?” I tried to wriggle out of this with, “Well I never promised there had to be some function without a unique minima, so that’s not necessarily a problem…” but my students weren’t fooled. My supposedly zero-information hint did in fact contain information, and they knew it.

Or say you’re abducted by aliens. Upon waking up in their ship, they say, “Quick question, Earthling, do you think we first built the Hivemind Drive on our homeworld more or less than 40 million years ago?” After you answer, they then ask, “OK, now please estimate the true age of the Drive.” Being aware of anchoring, wouldn’t you still take the 40 million number into account?

So sure, in this experiment as performed, the anchors made estimates worse. But that only happened because the anchors were extremely loose. If the experiment were repeated with anchors that were closer to the true values, they would surely make the estimates better. It’s not at all clear that being “biased” is a problem in general.

Arguably, what’s happening is simply that people had unrealistic expectations of how experimenters would choose the anchor values. And can we blame them? The true height of Mt. Everest is 29k feet. Isn’t it a little perverse to ask if it is greater than 2k feet? In Bayesian terms, maybe it was very unlikely to choose such misleading anchors according to people’s expectations of how anchors would be selected.

Here’s another clue that people might be doing Bayesian updating: The anchoring effect is larger for quantities people know less about. Anchoring has a huge effect when estimating the number of babies born per day (which most people have no idea about) but a small effect on the distance from San Francisco to New York (where people have some information). That’s exactly what Bayesian inference would predict: If your prior is more concentrated, the likelihood will have less influence on your posterior.

I’m sure this has been suggested many times in the past, but to my surprise, I wasn’t able to find any specific reference. The closest thing I came across was a much more complex argument that anchoring stems from bounded rationality or limited computational resources.

Irrelevant anchors

Some people have suggested that even irrelevant numbers could act as anchors and change what numbers people choose. The most well-known example is a 2003 paper by Ariely, Loewenstein, and Prelec. They took 55 MBA students and showed them descriptions of six different products—a cordless keyboard, a bottle of wine, etc. Students were asked two questions:

1. To get the product, would you pay X dollars? (Where X was the last two digits of the student’s social security number.)
2. What is the maximum amount you would pay for the product?

Ideally, to make sure people were honest, you’d have them actually pay for the products. For whatever reason, that wasn’t done. Instead, there was a complex mechanism where depending on randomness and the choices people, people might get the product. This mechanism created an incentive to be honest about how much they valued things.

It’s unclear exactly how the experimenters got the social security numbers. Most likely subjects just wrote them down before the first question.

This experiment found a strong relationship between social security numbers and the numbers students gave for the second question.

The effect was huge and statistically significant for every product.

But does it replicate?

Some replications look good. In 2010, Bergman et al. tried to replicate this study using a very similar design. They are very clear about how their anchoring works: Students first write down the last two digits of their social security number (in Swedish Krona) and then state if they would pay that much. These results were fairly similar to the original study.

In 2017, Li et al. tried to replicate this result by having college students at Nankai University (near Beijing) write the last two digits of their phone number as an anchor, so anchors ranged from ¥0 to ¥99 (around $15). The purpose of this experiment was to test if the strength of the anchoring effect could be modulated by tDCS (applying a current to the head via electrodes, yes really). But ignoring that test, here is what they found for the 30 people in the control group—or, more precisely, the 30 people who had just undergone a sham version of tDCS with no actual stimulation:

Although the p-values are all above 0.05, this still looks like a positive relationship for chocolate (Ferrero Rocher) and maybe the novel (One Hundred Years of Solitude) but not for the wine. That could be related to the fact that the true value of the wine (¥150) was more than the largest possible anchor value of ¥99.

But seriously, does it replicate?

Other replications that don’t look so good. In 2012, Fudenberg, Levine, and Maniadis did one using 79 students from UCLA. One difference was that, rather than using social security numbers, the replication rolled two dice (in front of the participants) to create anchor values. A second difference was that participants in the replication were giving numbers they would sell the product at, rather than values they would buy them at. (With another complex mechanism.) They found nothing.

Another experiment was done by Ioannidis and Offerman in 2020. This had three phases. First, people were “given” a bottle of wine (on their screen) and were asked if they would sell the wine for some value between €0 and €9.9, chosen randomly by rolling two dice. Second, people participated in an auction for the wine with other participants. Finally, they were asked what value they would accept in lieu of the wine.

Their goal was to test if the auction phase would reduce the anchoring effect. Their results were that—umm—there was no anchoring effect, and so the market didn’t have anything to reduce.

Here’s how I score the replications:

• Bergman et al. (2010): successful
• Li et al. (2017): half-successful
• Fudenberg et al. (2012): failed
• Ioannidis and Offerman (2020): failed

What to make of this? One theory is that it depends on if people were choosing values to “buy” or “sell” the good at. (Or rather, mechanistic proxies for buying and selling.)

Another theory, suggested by Ioannidis and Offerman, is that it depends on how the anchor is chosen. The original study and the successful replication both used social security numbers. The half-successful replication used phone numbers instead. Both of the failed replications used dice, clearly demonstrated in front of the participants to be random.

Why would this matter? Well, Chapman and Johnson (1999; Experiment 3) did a related experiment with anchors and social security numbers. They found a correlation, but then also took the brave step of asking the participants what they thought was going on. Around 34% of participants said they thought the anchor values were informative and that that they thought the experimenters wanted them to be influenced by the anchor values! So perhaps anchoring on irrelevant information has an effect because participants for some reason think it is relevant.

I’m not sure if I believe this. For one thing, how could social security numbers be informative? It doesn’t make sense. For another, 34% of people doesn’t seem large enough to produce the reported effects. And third, when Chapman and Johnson broke things down, they found that among people who said anchor values were informative, the correlation was four times weaker than among those who didn’t say that!

Takeaways

So do people really anchor on irrelevant information?

When I first came across Fudenberg et al.’s failed replication, I assumed the effect wasn’t real. But if that were true, how can we explain one and a half successful replications? It’s very puzzling.

If the effect is real, it’s fragile. Either it depends on people mistakenly thinking the irrelevant information is relevant, or it depends on people wanting to please the experimenter, or it depends on if people are “buying” or “selling” something, or it depends on some other unclear details. Very small changes to how the experiment is done (like using dice instead of social security numbers) seem to make the effect vanish.

As things currently stand, I don’t think there’s any real-world scenario in which we can be confident that people will anchor on irrelevant information (other than, perhaps, exactly repeating Ariely et al.’s experiment). Hopefully future work can clarify this.

“With 90% confidence, the true fraction is between 26.9% and 54.2%.”

But what does this really mean? We’ve talked about confidence sets before—read that post first if you find this one too difficult. Now let’s talk about intervals.

Confidence intervals

Years from now, all Generation β does is sit around meditating on probability theory and reading Ars Conjectandi. You work at a carnival where one day your boss says, “Our new ride is now so popular that we only have capacity for 10% of guests. I’ve got some bent coins of different colors, and I want you to create a guessing game to ration access to the ride.

Here’s how the game is supposed to work:

1. The guest will pick one coin, flip it, and announce the outcome.
2. Based on that outcome, you guess some set of colors.
3. The true color of the coin is revealed. If it’s not in the set of colors, the guest can go on the ride.

It’s essential that no matter what the guests do, only 10% of them can win the game. But otherwise, you’d like to guess as few colors as possible to better impress the players.

You’re given five bent coins, which your boss has CT-scanned and run exhaustive simulations to find the true probability each will come up heads.

Your first idea for a game is the obvious one: Flip it, and try to guess the color based on the outcome of heads or tails. While you could do that it would be extremely boring.

Then, you have another idea. Why not flip the coin twice? Take the red coin. It’s easy to compute the probability of getting different numbers of heads:

• 0 heads: (0.9)² = 0.81 (The probability of rolling tails twice in a row.)
• 1 head: 0.1 × 0.9 + 0.9 × 0.1 = 0.18. (The probability of either rolling heads × tails plus the probability of rolling tails × heads.)
• 2 heads: (0.1)² = 0.01 (The probability of rolling heads twice in a row.)

Continuing this way, you make a table of the probability of getting a total number of heads after two coin flips for each of the coins:

You could make a game based on two coinflips, but why not make things even more interesting? Why not spice things up even more by flipping the coin, say, 5 times. You do a little bit of research, and you discover that the probability of getting a total of tot-heads heads after doing num-flips flips of a coin with a bias of prob is called a Binomial, namely Binomial(tot-heads | num-flips, prob). For example, the probabilities we calculated above are Binomial(0 | 2, 0.1)=0.81, Binomial(1 | 2, 0.1)=0.18, and Binomial(1 | 2, 0.1)=0.01. If num-flips is larger than 2 the math gets more complicated, but who cares? You find some code that can compute Binomial probabilities, and you use it to create the following table of the probability of getting each total number of heads after 5 coinflips:

This seems to make sense: The most likely outcome for the red coin is all tails, since the red coin is rarely heads. The most likely outcomes for the blue coin are nearly evenly distributed, while the most likely outcome for the white coin is all heads.

Now, what colors should you guess for each outcome? Again, you need to make sure that, no matter what color the guest chooses, you will include that color with 90% probability. This is equivalent to covering .9 of the probability from each row. You decide to go about this in a greedy way. For each row, you add entries from largest to smallest until you get a total that’s above 0.9. If you do that, you get this result:

This corresponds to the following confidence sets:

Remember what we stressed last time: When we get 4 heads and say we are “90% confident” the color is blue, yellow, or white, we don’t mean “90% probability”, we just mean that your guessing procedure will work for 90% of guests. After seeing 4 heads, the probability of given colors is—according to the worldview of confidence intervals—meaningless because it’s a fixed quantity. And even if you’re willing to talk about probabilities in such situations, the probability could be much higher or lower than 90%.

It occurs to you that you can visualize this as a heatmap, with lighter colors representing higher probabilities. The entries in the following figure are laid out in the same way as the above table. Remember that red has a .1 probability of .being heads so it is in the first row, green has a .3 probability so it’s in the second row, etc.

You can visualize the confidence sets by drawing an outline around the coin/outcome pairs that are included in your strategy.

Things go well for a while, but then your boss comes around again and says “People want more of a challenge!” You’re given 19 coins with each of the probabilities .05, .10, .15, …, .95 and told to increase the number of coin flips from 5 to 40.

At this point, it would be tedious to look at tables of numbers, but you can still visualize things:

You can use the same greedy strategy of including elements from each row until you get a sum of 0.9. If you do that, this is what you end up covering:

Notice: For any given outcome, the set of coins that you include are always next to each other. This happens just because for each coin, there’s a single mode of probability around a given outcome, and the location of this mode changes smoothly as the bias of the coin changes. This is why we can talk about confidence intervals rather than confidence sets: The math happens to work out in such a way that the included coins are always next to each other.

Finally, your boss suggests one last change. The game should work this way:

1. Each guest is given a soft-metal coin, which they can bend into whatever shape they want.
2. That coin is flipped 40 times, and the outcome is announced.
3. You need to guess some interval that hopefully contains the true bias of the coin.
4. The coin is CT-scanned, and the carnival’s compute cluster finds the true bias. If it’s not in the interval you guessed, the guest can go on the ride.

Thinking about how to address this came, it occurs to you that you can make figures in the same way with any number of coins, and if you use a fine enough grid, you will cover all possibilities. The following figure shows what you get if you use the same process with 1001 coins ranging from 0.000, 0.001, 0.002, …, 1.000.

Now, remember, where we started: We tested 40 cells and found that 16 of them had changed, and an online calculator told us that with 90% confidence the true fraction was between 26.9% and 54.2%.

To understand where these numbers come from, just take this figure and put a vertical line at # heads = 16:

What’s included is all the coins with biases between .269 and .542. That’s why the confidence interval for 16 out of 40 is 26.9% to 54.2%.

With 90% confidence, the true fraction is between 26.9% and 54.2%.

“With 70% confidence, the true color is either green or white.”

“We have a procedure that maps dice rolls to sets of colors. We’ve designed the procedure so that, if we roll any of the dice millions of times and compute the corresponding sets of colors, at least 70% of those sets will contain the true color. For the dice roll we observed in this particular instance our procedure maps the outcome to {green, white}.”

“With 70% confidence, the true color is nothing.”

Note: There’s a follow up post with more confidence games.

The replication crisis in psychology started with a bang with Daryl Bem’s 2011 paper Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect. In this paper, Bem performed experiments that appeared to prove the existence of extrasensory perception (ESP). For example, subjects could guess (better than chance) where erotic images would appear on a screen. There were thousands of subjects. The paper used simple, standard statistics methods, and found significance for eight of nine experiments. Bem is a respected researcher, who no one suspects of anything underhanded.

The problem, of course, is that ESP isn’t real.

The psychology community was shaken. A smart person had carefully followed standard research practices and proved something impossible. This was ominous. Bem’s paper was only questioned because the result was impossible. How many other false results were lurking undetected in the literature?

Today, we’d say he had wandered the garden of forking paths: If you look at enough datasets from enough angles, you’ll always find significance eventually. It’s easy to do this even if you have the best intentions.

Many psychologists saw the Bem affair as a crisis and quickly moved to assess the damage. In 2014, an entire issue of Social Psychology was dedicated to registered replications. Each paper was both a replication (a repeat of a previous experiment) and pre-registered (the research plan and statistical analysis were peer-reviewed before the experiment was done).

So how well did existing results replicate? In their introduction to the issue, Nosek and Lakens are rather coy:

This special issue contains several replications of textbook studies, sometimes with surprising results.

Awesome, but which ones were surprising, how surprising were they, and in what way?

I wanted a number, a summary, something I could use as a heuristic. So, I read all the papers and rated every experiment in every paper as either a full success (★), a success with a much smaller effect size (☆), or a failure (✗). I used the summary from the authors if they gave one, which they often didn’t, probably because people are squeamish about yelling my famous colleague’s Science paper is bogus too loudly. Then, for each effect, I averaged the scores for all the experiments using ✗=0, ☆=2, and ★=4.

So, let’s play a game.

I’ll give a short description of the effects that were tested. You can predict how well each replicated by choosing an integer between 0 (failure) and 4 (full success). You can score yourself manually, or there’s a form you can use below.

(Or, if you’re the impatient type, feel free to jump to the summary.)

The effects

The first few effects are textbook results. ()

(1) The primacy of warmth is the idea that when forming impressions of others, the "warmth" of someone’s personality is more important than competence. This claim goes back to Solomon Asch in 1946 where people were given lists of various attributes for people like "intelligent, skillful, industrious, warm, determined, practical, cautious" and asked which was most important.

(2) Deviation-rejection is the idea that if someone stubbornly holds an opinion contrary to a group consensus will ultimately be rejected from that group. Schachter (1951) ran focus groups to discuss what to do with a juvenile delinquent. Most people are lenient, but a confederate in the group always supported a harsh punishment. This confederate was talked to less over time, and chosen less for future groups.

(3) The Romeo and Juliet effect is the idea that parental opposition to a relationship will increase the love and commitment in that relationship. This was first suggested by Driscoll et al. (1972), who found that parental interference was correlated with love and trust in relationships and that, over time, increases in interference were correlated with an increase in love.

(4) Single exposure musical conditioning was investigated by Gorn (1982) by randomly showing people either a blue or beige pen, while playing either music they liked or disliked. Later, people could choose a pen of either color to keep. This experiment found that 79% of people who heard music they liked chose the pen shown on screen, while only 30% of people who hear music they disliked did the same.

The next few effects concerned papers where the literature is somewhat in conflict or where there are moderators with unclear impact. ()

(5) Stereotype priming was investigated by Blair and Banaji (1996) who found that briefly flashing a stereotypically "male" or "female" word in front of someone changed how long it took to discriminate between male or female first names. Additionally, Banaji and Hardin (1996) found an effect even if the discrimination was unrelated to the primes: "Male" words made participants faster to recognize male names even if the task was to discriminate city names vs. first names.

(6) Stereotype susceptibility (or stereotype threat) is the idea that cognitive performance is influenced by stereotypes. Shih et al (1999). took Asian-American women and had them take a math test, while being primed to either think about being women or about being Asian. The Asian-primed group 54% of the questions right while the female-primed group got only 43% right. However, these results came from a small sample, and results in real-world applications have been mixed.

(7) Sex differences in distress from infidelity. A common idea in evolutionary psychology is that males are more upset by sexual infidelity than females. Shackelford et al. (2004) looked at a population with a mean age of 20 years asked if people would be more distressed if their partner had passionate sex with someone else or formed a deep emotional attachment. 76% of young men chose sex, as opposed to 32% of women. In an older population with a mean age of 67 years, 68% of men chose sex, as opposed to 49% of women.

(8) Psychological distance and moral judgment is the idea that people make more intense moral judgments (positive or negative) for more psychologically distant acts. Eval et al. (2008) had participants make judgments about people who committed various acts (incest, eating a dead pet, donating to charity) varying if it happened in the near vs. distant future. People judged those in the distant future around 1.5 points more harshly on a -5 to +5 scale. However, Gong and Medin (2012) performed a similar experiment and came to the opposite conclusion.

Finally, there were recent studies that had attracted a lot of attention. ()

(9) Does cleanliness influence moral judgments? Schnall et al. (2008) found that people make less severe moral judgments when they feel clean. They primed people with words and then asked people to judge a set of moral dilemmas, e.g. trolley problems or faking information on a resume. They found that people primed with "clean" words judged people around 1 point less harshly than those primed with neutral words, on a 7-point scale. In a second experiment, people saw a clip from Trainspotting with a man using an unclean toilet and were asked to wash their hands (or not) before being given the same moral dilemmas. Those who washed their hands judged people around 0.7 points less harshly.

(10) Does physical warmth promote interpersonal warmth? Williams and Barg (2008) gave participants either a cold pack or a heat pack to evaluate after which they could choose a gift either for themselves or for a friend. Those given the heat pack were around 3.5x more likely to choose a gift for a friend.

(11) Moral licensing is the idea that someone who does something virtuous will later feel justified to do something morally questionable. Sachdeva et al. (2009) had participants write a short story about themselves using words that were either positive words ("caring", "generous") or negative words ("disloyal", "greedy"). Afterward, they asked how much they would be willing to donate to charity. Those who used the positive words donated a mean of $1.07, while those who used negative words donated $5.30.

(12) Can superstition improve performance? Damish et al. (2010) gave subjects a golf ball and asked them to attempt a 100cm putt. Subjects told the ball had been lucky so far were able to make 65% of putts, as opposed to 48% of controls.

(13) Is moral behavior connected to brightness? Banerjee et al. (2012) asked participants to describe something they did in the past that was either ethical or unethical, then asked them how bright their room was. Those in the ethical group described their room as around 0.6 points brighter on a 7-point scale. They also judged various light-emitting products (lamps, candles, flashlights) as around 2-point more desirable on a 7-point scale.

Scorecard

To play the game fill out this Google form quiz thing. f

Or, you can score yourself. In the following table, record your absolute error for each effect. (If you guessed 1 and the correct answer was 4, you get 3 points.) Then add all these errors up.

The best possible score is 0, while the worst is 41.

Here are the results for each effect, ranging from ○○○○ (failure) to ●●●● (full replication).

Effect	Type	Result
Primacy of warmth		○○○○
Deviation-rejection		●●○○
Romeo and Juliet effect		○○○○
Single exposure musical conditioning		●●●●
Stereotype priming		●○○○
Stereotype susceptibility		●○○○
Sex differences in distress from infidelity		●○○○
Psychological distance and moral judgment		●●○○
Cleanliness and moral judgments		○○○○
Physical warmth and interpersonal warmth		○○○○
Moral Licensing		○○○○
Superstition and performance		○○○○
Moral behavior and brightness		○○○○

What to make of this?

Obviously, the replication rate is strikingly low. Only one effect replicated at the original effect size, and even this has some caveats (below). The average replication score was 0.8 / 4. And one effect (Romeo and Juliet) deserves a negative score: It didn’t just fail to replicate, there was a large effect in the opposite direction.

Also striking is that all of the new hot effects () failed—not one experiment in one paper replicated even at a smaller effect size. Strangely, all the conflicted papers () replicated at least a little bit.

I don’t think things are nearly as bad as this table suggests. For one thing, if you were a psychologist, what would make you perform a registered replication? Personally, I’d be motivated by something like this:

I’d do it because I thought some famous effect wasn’t real. So this is a highly non-random sample. The correct conclusion is less psychology is all wrong and more when psychologists think an effect is bogus, they’re usually right.

Second, these results are mostly social psychology. You, dear reader, are probably not a social psychologist, but rather someone interested in the broad art of predicting stuff, which is the domain of cognitive psychology. (At least, unless you’re predicting how people interact in groups.) The Open Science Collaboration in 2015 tried to reproduce a bunch of recent papers in social and cognitive psychology. They found that around 28% of social effects subjectively "replicated", as opposed to around 53% of cognitive effects. They also found that replication effect sizes were much closer to the original effect sizes for cognitive psychology (a ratio of around 0.56 as compared to 0.31 in social psychology).

The replications

Here are more details about what happened in each of the replications, and justifications of the ratings I gave. (Click to expand.) If you’re the author of one of these replications and I got anything wrong, I’d love to hear from you.