tag:blogger.com,1999:blog-987850932434001559.post3524536609091537335..comments2017-02-24T02:45:43.789-08:00Comments on The 20% Statistician: Observed power, and what to do if your editor asks for post-hoc power analysesDaniel Lakensnoreply@blogger.comBlogger4125tag:blogger.com,1999:blog-987850932434001559.post-13897246898035216432015-01-30T05:28:34.698-08:002015-01-30T05:28:34.698-08:00The way the effect size was calculated by Rosentha...The way the effect size was calculated by Rosenthal has been discredited (well it may be harsh to say "discredited"). See: http://www.researchgate.net/publication/232494334_Is_psychological_research_really_as_good_as_medical_research_Effect_size_comparisons_between_psychology_and_medicine Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-89521999624798140912014-12-20T11:18:47.294-08:002014-12-20T11:18:47.294-08:00Hi Roger, could you elaborate a little on "di...Hi Roger, could you elaborate a little on "discredited examples like Rosenthal's aspirin study"? I am aware of the paper you're referring to but not clear on what is discredited.Jake Westfallhttp://jakewestfall.orgnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-29978691964706169152014-12-19T16:33:38.320-08:002014-12-19T16:33:38.320-08:00You are right to focus attention on the effect siz...You are right to focus attention on the effect size rather than power as an argument for "proving the null" (even if only suggestively); but we have a long way to go in agreeing what effect size is indeed too small to matter, when discredited examples like Rosenthal's aspirin study keep circulating.<br /><br />I would go further and say that a priori power analysis is useful for a direct replication; has only suggestive value for a conceptual replication; and is near-useless for the 90% (?) of published research that rests on finding a novel effect, even one such as a moderation by context that may include an incidental replication (the power needed for the interaction will have little to do with that needed for the main effect).Roger G-Shttp://www.blogger.com/profile/08594440701279968693noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-5675240266049411092014-12-19T13:19:11.493-08:002014-12-19T13:19:11.493-08:001. I would like to add that post-hoc power for a s...1. I would like to add that post-hoc power for a single statistical test is useless. However, post-hoc power provides valuable information for a set of independent statistical tests. I would not trust an article that reports 10 significant results when the median power is 60%. Even if median power is 60% and only 60% of results are significant, post-hoc power is informative. It would suggest that researchers have only a 60% chance to get a significant effect in a replication study and should increase power to make a replication effort worthwhile. <br /><br />2. The skewed distribution of observed power for true power unequal .50 was discussed in Yuan and Maxwell (2005) http://scholar.google.ca/scholar_url?url=http://irt.com.ne.kr/data/on_the_pos-ences.pdf&hl=en&sa=X&scisig=AAGBfm1W3wVWuS3MecdWMU0dEkoVal4U3A&oi=scholarr&ei=wZOUVNuXD67IsASZ-YKIDQ&ved=0CB0QgAMoADAA<br /><br />3. It was also discussed in Schimmack (2012) as a problem in the averaging of observed power as an estimate of true power and was the reason why the replicability index uses the median to estimate true (median) power of a set of studies. http://r-index.org/uploads/3/5/6/7/3567479/introduction_to_the_r-index__14-12-01.pdf <br /><br /><br />drreplicablehttp://replicationindex.wordpress.com/noreply@blogger.com