tag:blogger.com,1999:blog-987850932434001559.post1071328463083448764..comments2024-03-28T05:57:40.089+01:00Comments on The 20% Statistician: Why a meta-analysis of 90 precognition studies does not provide convincing evidence of a true effectDaniel Lakenshttp://www.blogger.com/profile/18143834258497875354noreply@blogger.comBlogger19125tag:blogger.com,1999:blog-987850932434001559.post-82184749152461124782021-08-02T07:59:52.359+02:002021-08-02T07:59:52.359+02:00This comment has been removed by a blog administrator.Sajjad Ahmedhttps://www.blogger.com/profile/08974601470220195378noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-26258538909601380122017-04-10T14:43:59.547+02:002017-04-10T14:43:59.547+02:00The Lakens 2014 reference link to SSRN is dead. I...The Lakens 2014 reference link to SSRN is dead. Is it possible to post a working link? Thanks!Adilhttps://www.blogger.com/profile/02372447704684232187noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-56843360254267370162015-11-08T20:39:15.444+01:002015-11-08T20:39:15.444+01:00This comment has been removed by the author.George Williamshttps://www.blogger.com/profile/04591902834514798796noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-61801669467814438422015-11-08T20:37:44.758+01:002015-11-08T20:37:44.758+01:00Hi Daniel, I have two questions:
1. In your criti...Hi Daniel, I have two questions:<br /><br />1. In your critique, you argue (regarding publication bias) that: “PET-PEESE meta-regression seems to be the best test to correct effect size estimates for publication bias we currently have.” Do you have a citation for that claim?<br /><br />The authors (Bem et al.) do use several measures (based on correlation between effect size and sample size) to explore the possibility of publication bias. The only one where they don’t get a significant effect is the PET technique, very similar to the one you choose for your analysis. However, the authors note that “Sterne & Egger (2005) (upon which PET is based) themselves caution, however, this procedure cannot assign a causal mechanism, such as selection bias, to the correlation between study size and effect size, and they urge the use of the more noncommittal term “small-study effect.” So the literature suggests we should use caution in how we interpret this particular measure. <br /><br /><br />2. I’m puzzled by your claim “there are no unpublished studies by Dr Bem in his own meta-analysis.” In Table 1, which summarizes the studies included in the meta-analysis you are critiquing, I count 38 lines (separate experiments) that were not peer reviewed. <br /><br />Thanks,<br /><br />George<br />George Williamshttps://www.blogger.com/profile/04591902834514798796noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-35531014328589502922015-11-06T08:07:54.923+01:002015-11-06T08:07:54.923+01:00Convincing ideas to the statistics report almost c...Convincing ideas to the statistics report almost concerning more values in it which used to influence more values that you really want to opt for your statistical report, so this is all been so important to get ready for your purpose. <a href="http://www.statisticaldataanalysis.net/our-statistical-analysis-and-data-mining-services/stata-statistical-data-analysis/" rel="nofollow">stata statistical analysis</a>Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-13124717462380896122015-11-06T06:53:35.795+01:002015-11-06T06:53:35.795+01:00Very good one. Very good one. Mary Hhttps://www.blogger.com/profile/02818395570305641255noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-88913194408326832602015-06-13T20:52:39.492+02:002015-06-13T20:52:39.492+02:00This comment has been removed by a blog administrator.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-49412759744406268422015-04-07T12:44:54.662+02:002015-04-07T12:44:54.662+02:00Great post!
I have a topical suggestion below...
...Great post!<br />I have a topical suggestion below...<br /><br />First: Let's just not talk about physics for a while as psychologists ok? <br /><br />[quickly puts on an -access all areas- philosopher badge]<br />"... you needed to dismiss the light waved coming from your electronical device which allow you to read this. There's no good theory of light (Waves? Particles?) "<br /><br />I think what Bem et al. have produced warrants the folowing note to be included with almost all modern electronic devices: "made possible by Quantum Physics, the ultimate empirically accurate scientific theory about light and matter that we will not give up just because some psychologists can't properly apply the scientific method and misinterpret everything that we have achieved in less than a century since 1930s when the Quantum Formalism was first postulated -what have you guys been doing all that time... still figuring out what that Fisher character went on about in 1925?- and if you think the theory is false pleas refrain from using all electronic devices or you will be fined."<br /><br />If you can use the fundamental knowledge about the world posited by a scientific theory to actually build things that would have been impossible to build without it... the theory must be on to something.<br /><br />About detecting small effects: The S.Q.U.I.D. sensors in a MEG scanner are: Superconducting Quantum Interference Devices. The best are able to detect magnetic fields of 10-18 Tesla (refrigerator magnets produce 10-2 T). Flash memory makes use of quantum tunneling to erase data, every light emitting diode display has some quantum physics going on, especially of course the QuantumDot technology. Oh, and then there's that old CD player which uses a LASER. which is based on a quantum phenomenon described by Einstein.<br /><br />The 'explanation' of the wave-like and particle-like behavior of light and matter IS Quantum Physics (more precisely, the Quantum Formalism of 1925-1935). The quantum revolution started with Einstein's explanation of the photo-electric effect in 1905, for which he won the Nobel Prize in 1921(not for relativity). In the Quantum Formalism the duality of Wave and Particle descriptions (the Schrödinger picture and Heisenberg picture) are two sides of the same structural coin and are equivalent for all intents and purposes. Oversimplifying: The difference lies in wether the distribution parameters (density-matrix) or the covariates (system observables) are considered time-varying. There is also an interaction picture in which both are time-varying: http://en.wikipedia.org/wiki/Heisenberg_picture#Summary_comparison_of_evolution_in_all_pictures <br /><br /><br /><br />Back to common ground...<br />The need to have independent replications conducted by different labs is being accepted... slowly (this has btw always been the ultimate test in physics needed to get consensus on a discovery)<br /><br />I think there's a real need for independent Meta-Analyses, conducted by teams who have no stakes in the topic analysed. There are other examples of Meta-gone-wrong by researchers testing their own beef, crop, pudding and what not.Anonymoushttps://www.blogger.com/profile/01414244802603249395noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-31682114563763881232015-04-06T19:29:26.284+02:002015-04-06T19:29:26.284+02:00Daniel, I want to commend you for taking the time ...Daniel, I want to commend you for taking the time and providing the technical expertise to evaluate this meta-analysis. Since your posting focuses on technical details of the meta-analysis, it may inspire readers to think that meta-analysis can compensate for weak methodology and solve a scientific controversy like this. I want to emphasize your comment that the way forward is pre-registered, well-powered confirmatory research. If the experimenters actually understand and control a real phenomenon as they claim, 80% or more of pre-registered confirmatory studies should provide significant evidence for an effect—as compared to the 20% to 33% found in this and other meta-analyses in parapsychology. <br /><br />Meta-analyses of mostly nonsignificant, underpowered, unregistered studies attempt to use post hoc analyses of observational data to compensate for methodological weaknesses in the original studies (“synthesis-generated evidence” rather than “study-generated evidence”--discussion and references at http://jeksite.org/psi/jp13a.pdf). That strategy has not been and cannot be expected to be effective for resolving scientific controversies. Confirmatory studies are needed that eliminate the methodological weaknesses. As you noted, some parapsychological researchers are making efforts to do this type of research.<br /><br />I have not delved into the technical details of this meta-analysis, but I did look into the distribution of effect sizes for Bem’s original paper. I came to the conclusion that a meaningful evaluation of effect sizes could not be done because the experiments used different tasks and had different numbers of trials per subject for the different tasks. The effective sample size for a study depends on both the number of subjects and number of trials per subject. I do not see any reason to expect a fixed effect size for the different experiments (as if they were replications of one experiment), and am skeptical of efforts (pro or con) to interpret funnel plots and related evaluations that combine the different types of experiments. <br />Jim Kennedyhttp://independent.academia.edu/JamesEKennedynoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-86015290690522197472015-04-04T19:59:51.482+02:002015-04-04T19:59:51.482+02:00Alex, we could just test H0: theta>0.6 where th...Alex, we could just test H0: theta>0.6 where theta is the success probability and theta= 0.5 is random performance. But I really prefer to know theta and its CI. if it's 0.7>theta>0.6 I would take notice. If it's 0.9>theta>0.7 I want to try that experiment in our lab, and if it's 0.9<theta I'm running to betting kiosk right away.matushttp://simkovic.github.io/noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-86953994785795144702015-04-04T19:47:53.878+02:002015-04-04T19:47:53.878+02:00Daniel,
I didn't say that "we needed to d...Daniel,<br />I didn't say that "we needed to dismiss everything modern physics can't explain". I said we should focus on reliably big effect sizes - such as found in physics. I also explicitly used the qualifier "currently". If there is some quantum magic involved in precognition, physics is in excellent position to uncover it.<br />"People who believe in precognition are not convinced by your argument, and use NHST to convince the general public"<br />... and they so far fail miserably (minus perhaps few eso-freak), so maybe they should start heeding my argument, which would help them better persuade the public. Btw. the soviets and americans studied psi extensively during the cold war, but gave up at some point. Why? No, not because they figured out that H0 is correct. What they figured out is that even if psi exists it's so weak that it doesn't impact our lives and there is no practical use for it.matushttp://simkovic.github.io/noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-63468868870392709152015-04-04T18:29:32.559+02:002015-04-04T18:29:32.559+02:00Matus, if we needed to dismiss everything modern p...Matus, if we needed to dismiss everything modern physics can't explain, you needed to dismiss the light waved coming from your electronical device which allow you to read this. There's no good theory of light (Waves? Particles?) but it is empirically demonstrable. People who believe in precognition are not convinced by your argument, and use NHST to convince the general public there might be something there. Yes, we can ignore it. Or we can show they are not using NHST correctly and draw conclusions based on a biased literature.Daniel Lakenshttps://www.blogger.com/profile/18143834258497875354noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-52311067677074691972015-04-04T18:13:49.392+02:002015-04-04T18:13:49.392+02:00What would you say to using a null interval instea...What would you say to using a null interval instead of a point null for hypothesis testing? Say it was uniform spanning +/-10^-6 or some other small interval. Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-86632963334388246622015-04-04T18:03:47.325+02:002015-04-04T18:03:47.325+02:00Yeah, discard BFs with the rest of hypothesis test...Yeah, discard BFs with the rest of hypothesis testing. Just do parameter estimation. The problem of precognition is that the effects are so tiny that we can discard them based on their their magnitude and do not need to care whether they are reliably tiny or just non-existent. Instead we can focus on research that demonstrates reliably big effects - for instance the modern-day physics, which currently tells us that precognition can't exist.matushttp://simkovic.github.io/noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-48545945876539804032015-04-04T17:46:54.299+02:002015-04-04T17:46:54.299+02:00My criticisms are not yet focussed on the raw data...My criticisms are not yet focussed on the raw data - this is purely based on the meta-analysis, and in some cases the data used to calculate the effect size. If we'd look at the raw data, I'm sure more criticisms would pop up.Daniel Lakenshttps://www.blogger.com/profile/18143834258497875354noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-50079158712593824922015-04-04T17:45:25.140+02:002015-04-04T17:45:25.140+02:00Bayes Factors lead to the same conclusion. And unl...Bayes Factors lead to the same conclusion. And unless you know how to model an effect without any theoretical prediction, you are either left with rejecting precognition outright (which would make you vunerable for the criticism of not being scientific) or testing the H0. Unless you have an alternative?Daniel Lakenshttps://www.blogger.com/profile/18143834258497875354noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-74679885315532623732015-04-04T17:25:16.243+02:002015-04-04T17:25:16.243+02:00The relevant quantity in each study will not be RT...The relevant quantity in each study will not be RT but some comparison between RTs such as difference or quotient.matushttp://simkovic.github.io/noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-91657812626883389822015-04-04T17:21:26.663+02:002015-04-04T17:21:26.663+02:00Muhaha Daniel, give up! Hypothesis testing does no...Muhaha Daniel, give up! Hypothesis testing does not work. H0 is always false and even the crazy precognition studies show this!matushttp://simkovic.github.io/noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-74428028660084316572015-04-04T16:43:40.687+02:002015-04-04T16:43:40.687+02:00Hi Daniel,
we all know RT are skewed - in those 9...Hi Daniel,<br /><br />we all know RT are skewed - in those 90 studies how many used mean(RT)? and what is the impact on p values? if I compare mean(RT) with median(RT) and/or mean(log(RT)), the mean(RT) can give very different results .. Dr Cyril Pernethttps://www.blogger.com/profile/11198127125335091943noreply@blogger.com