tag:blogger.com,1999:blog-987850932434001559.comments2017-05-26T08:07:34.942-07:00The 20% StatisticianDaniel Lakensnoreply@blogger.comBlogger834125tag:blogger.com,1999:blog-987850932434001559.post-36002272227702761202017-05-26T08:07:34.942-07:002017-05-26T08:07:34.942-07:00An interesting theory. I like your thinking. I'...An interesting theory. I like your thinking. I'm currently working on a project in college. And a million questions have raised on this topic in my head . Thank you for answering many of them, Daniel.<br /><br />Bianca | <a href="https://mastergrades.com/essay-writing-service-uk" rel="nofollow">https://mastergrades.com/</a>Bianca Kendellhttp://www.blogger.com/profile/18169495269964199627noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-52737353200309434272017-05-26T08:01:29.318-07:002017-05-26T08:01:29.318-07:00thanksthanksAdriana Ryleehttp://www.blogger.com/profile/16945480422986632992noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-91210393735985232832017-05-25T08:21:52.616-07:002017-05-25T08:21:52.616-07:00Thanks for this article!
I have a question about ...Thanks for this article!<br /><br />I have a question about this package. This package provides a way to conduct a equivalence test between two groups using a t-test. I was wondering, since my data has a non-parametric distribution, is it also possible to conduct a two-one-sided test using a Mann-Whitney U test?<br /><br />I hope you can help me.<br /><br />Kind regards, Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-37985252483369267442017-05-17T12:52:16.455-07:002017-05-17T12:52:16.455-07:00I don't what your metric for scientific qualit...I don't what your metric for scientific quality is. If it is a single figure of merit metric, you can rank order science blog posts and science journals. I assert that there will be a range, mostly overlapping, range of quality. I opine that the worst blog is worse than the worst journal and that the best journal is better than the best blog, but not by much in either case.Martyhttp://fogey.com/contemplating/noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-48671645485256566892017-05-17T06:24:23.633-07:002017-05-17T06:24:23.633-07:00No, and I don't expect you to, thank you very ...No, and I don't expect you to, thank you very much for your advice! I have to get into doing simulations more, I find that really interesting.<br /><br />But yes, I'll ask the methods people at my department and see if they can help. It's strange, on the one hand reporting effect sizes, CIs etc. is more or less standard now, and on the other many people are advocating using mixed models in psychology, but somehow the two groups don't seem to intersect much.Benjaminhttp://www.uni-muenster.de/PsyIPBE/aejucks/personen/bbrummernhenrich.htmlnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-21826568451952437972017-05-17T01:38:34.571-07:002017-05-17T01:38:34.571-07:00Hi, glad to hear you like the Coursera course. But...Hi, glad to hear you like the Coursera course. But I agree with you, we need more practical 'how-to' articles. I don't have any references, nor simulation code - I have 2 colleagues who do the simulations (I rarely use linear mixed models even though I probably should). I'd ask around at you uni. If you can't find anyone, send me an email, but I'm hesitant to ask for the time of my colleagues....Daniel Lakenshttp://www.blogger.com/profile/18143834258497875354noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-73984524088671492162017-05-17T01:35:37.505-07:002017-05-17T01:35:37.505-07:00Thank you for your swift answer! I am using linear...Thank you for your swift answer! I am using linear mixed models (via lme4/lmerTest in R) because I wanted to include covariates and there are multiple measurements per condition. But the experimental design is purely within subjects.<br /><br />Do you have any code you could share for doing those simulations and/or a blog post or even article you could link to and that I could reference in the manuscript?<br /><br />By the way, I'm about halfway through your Coursera course and enjoying it (and learning!) a lot. So thank you for that!Benjaminhttp://www.uni-muenster.de/PsyIPBE/aejucks/personen/bbrummernhenrich.htmlnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-30495713334496643582017-05-16T08:24:54.068-07:002017-05-16T08:24:54.068-07:00Do you have a within design, or a mixed models des...Do you have a within design, or a mixed models design? For a paired t-test, use the formula above. For mixed models, we always just simulate data for power analysis. Daniel Lakenshttp://www.blogger.com/profile/18143834258497875354noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-17160193938877345582017-05-16T08:19:22.254-07:002017-05-16T08:19:22.254-07:00I am trying to do a power analysis for a new exper...I am trying to do a power analysis for a new experiment (varying 2 factors in a 2x2 within-subjects design) that is based on a pilot study. If I understand your post correctly I can compute the required sample size by first determining the sample size that would be needed in a pure between-subjects design (e.g., by using G*Power) and then plugging the resulting value as well as the correlation between the repeated measurements into the formula at the top of your post.<br /><br />Now I've run into a problem: I don't know how to get a standardized effect from that pilot study. Calculating standardized effect sizes from within subjects designs, and especially mixed models, which I've used in this case, seems to be a very contested issue. The most enlightening discussions (for me at least) that I've seen have been in the comments to a post on Jeff Rouder's blog (http://jeffrouder.blogspot.com/2016/03/the-effect-size-puzzler-answer.html), which in turn referenced a post from Jake Westfall (http://jakewestfall.org/blog/index.php/2016/03/25/five-different-cohens-d-statistics-for-within-subject-designs/).<br /><br />My outcome variables are Likert scales so I don't think the unstandardized effect sizes are very meaningful in themselves. What would you recommend, is there a (reasonably simple) standardized effect size to get from within-subject designs to 1. plug into G*Power, and 2. report for meta-analytic purposes?Benjaminhttp://www.uni-muenster.de/PsyIPBE/aejucks/personen/bbrummernhenrich.htmlnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-73836656823704951222017-05-16T04:55:49.068-07:002017-05-16T04:55:49.068-07:00Alright, thanks for your quick response!Alright, thanks for your quick response!Eva H.http://www.blogger.com/profile/12969969791334802390noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-3428518619333929902017-05-16T04:55:23.384-07:002017-05-16T04:55:23.384-07:00This comment has been removed by the author.Eva H.http://www.blogger.com/profile/12969969791334802390noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-12531255082551209892017-05-15T04:39:46.074-07:002017-05-15T04:39:46.074-07:00Hi, no, there are no ANOVA functions yet. If it be...Hi, no, there are no ANOVA functions yet. If it becomes popular, I might work out functions for it, or (even better) someone else might want to do that work.Daniel Lakenshttp://www.blogger.com/profile/18143834258497875354noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-8150688370089638832017-05-15T03:41:45.300-07:002017-05-15T03:41:45.300-07:00Hi Daniel, is there any way to control for covaria...Hi Daniel, is there any way to control for covariates using this package?Eva H.http://www.blogger.com/profile/12969969791334802390noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-91874191005280210742017-05-12T21:45:15.454-07:002017-05-12T21:45:15.454-07:00Since you asked about where it's equivocal, be...Since you asked about where it's equivocal, between pop ES and sample ES, here's one: you say "true effect size is 0 (or the null-hypothesis is true), and when d = 0.5."<br />Here d = .5 appears to speak of the pop ES. On your graph it's the observed.<br />Another: Your first figure shows d = .5 & also that d = .3, the first I take it is a pop, the second a sample.<br /><br />A separate issue I have with using these standardized pop d's is that it seems you're allowed to do the analysis without knowing the standard deviation. Is that so?<br />Deborah Mayohttp://www.blogger.com/profile/06527423269272136310noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-58682973597291985172017-05-12T00:26:38.344-07:002017-05-12T00:26:38.344-07:00Should you ever run out of ideas for blog posts, I...Should you ever run out of ideas for blog posts, I think one where you detail how you or your collaborators arrived at an effect size or SESOI would make for interesting reading. My sense is that power analyses are often based on canned effect sizes with little regard to the specifics of the study (theory and design), so it would be useful to see some more sophisticated approaches to specifying ESs.Janhttp://www.blogger.com/profile/17765078332699225416noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-32072014023562826412017-05-11T17:07:44.306-07:002017-05-11T17:07:44.306-07:00I find this equivocating between observed ES and p...I find this equivocating between observed ES and population ES. This is very common in psych, and it would really help if you labelled which you have in mind whenever used. Cohen had a subscript s for the observed ES. (I use difference for the observed, and discrepancy for the parametric effect size). <br />To take the simple one-sample test of a Normal mean : Ho: mu< 0 vs H1: mu > 0, the cut-off for rejection at the 025 level is a sample mean M of 1.96SE. Are you saying the pop effect size of interest is this cut-off, 1.96 SE? That would be to take, as the pop ES of interest, one against which the test has 50% power. I'm not saying that would be bad, I'm just trying to figure out your equivocal use of effect size. <br /><br />Deborah Mayohttp://www.blogger.com/profile/06527423269272136310noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-66138917542398954212017-05-11T16:06:10.198-07:002017-05-11T16:06:10.198-07:00Daniel, I'm a biostatistician, but I occasiona...Daniel, I'm a biostatistician, but I occasionally consult for social scientists. I disagree that 99% power is inefficient. We are interested not just in detecting an effect, but in obtaining a reasonably precise estimate of the effect size. The flip side of high power is narrow confidence intervals. <br /><br />As to sequential analysis, I agree that in many psych experiments the approach is useful. However, sequential analysis would not be practical in most studies I've been involved with. For example, if we need patients to be under treatment for three months, then, for many logistical reasons as well as financial ones, we really need the study to terminate after three months. JThttp://jt512.dyndns.orgnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-44988107811042988132017-05-11T14:46:34.519-07:002017-05-11T14:46:34.519-07:00JT - it's good you don't work at our depar...JT - it's good you don't work at our department! Our ethics department would not be easily convinced by designing studies with 99% power - it's wasteful, and our resources can be spent more efficiently! You should really do sequential analyses (see Lakens, 2014, for an introduction (you are anonymous, so I don't know what you know about stats, but if you've never learned about sequential analyses, you should!).Daniel Lakenshttp://www.blogger.com/profile/18143834258497875354noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-91677351586236964822017-05-11T14:39:00.106-07:002017-05-11T14:39:00.106-07:00Jan, although your last paragraph was presumably a...Jan, although your last paragraph was presumably aimed at Daniel, rather than me, when applying for funding, I always base my proposed sample size on having 90% power to detect the smallest effect size of interest. This usually winds up giving me 99% power or more to detect the hypothesized true effect size.JThttp://jt512.dyndns.orgnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-28192754531159117322017-05-11T14:33:45.235-07:002017-05-11T14:33:45.235-07:00Hi - current practice in power analysis is not ver...Hi - current practice in power analysis is not very state of the art in psychology, and the problem is, smallest effect sizes of interest do not exist, are almost never used or specified. That's why I wrote this post - to bootstrap a SESOI, building from a practice people use (power analysis).<br /><br />We've had this practice of 90% power for about 2 years. I could give examples - very often people in our group specify a SESOI, or they look at a pilot study, and then use a more conservative estimate in a power analysis. If there is large uncertainty, we recommend sequential analyses. Daniel Lakenshttp://www.blogger.com/profile/18143834258497875354noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-68202111742274276872017-05-11T14:11:47.640-07:002017-05-11T14:11:47.640-07:00Thanks! Changed (and I knew that - last minute add...Thanks! Changed (and I knew that - last minute addition I didn't think through! Thanks for correcting me!).Daniel Lakenshttp://www.blogger.com/profile/18143834258497875354noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-87746869619151402492017-05-11T13:50:01.420-07:002017-05-11T13:50:01.420-07:00Hi Daniël! Interesting post. Just a detail: I thin...Hi Daniël! Interesting post. Just a detail: I think Simonsohn (2015) did not suggest to set the smallest effect size of interest to 33% of the effect size in the original study, as you write. He suggested to set the smallest effect size of interest so that the original experiment had 33% power to reject the null if this ES was true. This smallest ES of interest thus does not depend on the found effect size of the original study: it only depends on the sample size. For instance, for n=20 per cell in a two cells design, the effect size would be d=0.5, because this gives 33% power. Your approach is that the smallest ES is the effect size that gives 50% power in the original study. It makes a difference, but I think your approach is, in the end, quite close to Simonsohn's approach.Auréliennoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-74818807746666154812017-05-11T12:28:52.755-07:002017-05-11T12:28:52.755-07:00That's what I was thinking - the d = 0.3, i.e....That's what I was thinking - the d = 0.3, i.e. what you call the smallest effect size of interest, is just the sample effect size, whereas the d = 0.5 you based the power calculation on is a postulated population effect size. If you have a smallest effect size of interest, wouldn't you want to treat it as a postulated population effect size and base your power computation on that?<br /><br />Incidentally, your departmental policy sounds interesting (swing for 90% power). Do you have any worked out examples, i.e., of your colleagues identifying the effect size that they're investigating etc.?Janhttp://www.blogger.com/profile/17765078332699225416noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-30753854226247224052017-05-11T11:31:24.438-07:002017-05-11T11:31:24.438-07:00If experimental psychologists are basing their sam...If experimental psychologists are basing their sample size requirements on the effect size they expect to observe, then they are making a mistake, because, for one thing, their experiment will be underpowered to detect a smaller effect size that they would still consider scientifically of interest. Sample size planning should always be based on the smallest effect size of interest.JThttp://jt512.dyndns.orgnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-63263577636275718012017-05-08T07:31:58.802-07:002017-05-08T07:31:58.802-07:00It's not the best approach - you want to use e...It's not the best approach - you want to use equivalence testing. I explain why in detail here: https://osf.io/preprints/psyarxiv/97gpc/Daniel Lakenshttp://www.blogger.com/profile/18143834258497875354noreply@blogger.com