May 11, 2011 -
http://www.thepsychologist.org.uk/blog/blogpost.cfm?threadid=1984&catid=48Psi study highlights replication problems
A team of British psychologists have completed a negative, three-fold replication of a study that caused a splash last year because it appeared to show classic psychological effects working backwards in time. However, the Journal of Personality and Social Psychology (JPSP), which published last year's positive result by Daryl Bem, has refused publication of the replication without even sending it out for peer review. The situation highlights the risk of a publication bias existing in psychology (and other sciences), whereby negative results are far less likely than positive findings to see the light of day.
Stuart Ritchie (University of Edinburgh), Richard Wiseman (University of Hertfordshire) and Christopher French (Goldsmiths College, University of London) focused on the ninth and final of Bem's experiments - the one that produced the largest psi effect. Students completed a memory test and showed superior recall for those words that they went on to study afterwards. Bem deliberately chose this simple method of testing standard effects backwards, so that replications of any observed psi effects would be straightforward for other labs to perform.
Using Bem's own computer programme and stats methods, Ritchie and his colleagues replicated his memory experiment three times, once each at their respective universities, with three groups of 50 participants. They found no backward psychological effects - the likelihood of a word being recalled was not related to whether or not time was spent studying it later on.
Chris French told The Psychologist that their work had also been rejected without peer review by Psychological Science and Science Brevia, raising serious issues about the replication process in psychology. 'Psychologists, along with scientists in general, often proclaim that replication is the cornerstone of science. But the truth is that straight replications are rare,' French said. 'We appreciate that the "top" journals receive far more submissions than they can accept, but we feel that JPSP have a certain moral obligation to send replication studies out for proper peer review given the highly controversial nature of Bem's paper, the inevitably high level of media coverage it received, and the fact that it explicitly contained an appeal to psychologists to attempt replications.'
Professor Eliot Smith, the editor of JPSP (Attitudes and Social Cognition section) told us that the journal has a long-standing policy of not publishing simple replications. 'This policy is not new and is not unique to this journal,' he said. 'The policy applies whether the replication is successful or unsuccessful; indeed, I have rejected a paper reporting a successful replication of Bem's work [as well as the negative replication by Ritchie et al].' Smith added that it would be impractical to suspend the journal's long-standing policy precisely because of the media attention that Bem's work had attracted. 'We would be flooded with such manuscripts and would not have page space for anything else,' he said.
Ritchie and his colleagues plan to resubmit elsewhere. 'We are glad that at least this episode has opened up the debate on the proper place of replication in the scientific literature,' French said. 'Ironically, parapsychology journals are much more open to publishing replications, failed or otherwise - but then they have very little impact because the mainstream science media simply don't pick up on them!'
- A trial registry for attempts to replicate Bem's study is at
www.richardwiseman.com/BemReplications.shtml 
