tag:blogger.com,1999:blog-987850932434001559.post4681941233549222118..comments2022-09-24T19:05:35.512+02:00Comments on The 20% Statistician: Always use Welch's t-test instead of Student's t-testDaniel Lakenshttp://www.blogger.com/profile/18143834258497875354noreply@blogger.comBlogger30125tag:blogger.com,1999:blog-987850932434001559.post-63593112442373471002021-09-07T06:13:04.805+02:002021-09-07T06:13:04.805+02:00Thanks so much for the informative blog.Thanks so much for the informative blog.Anonymoushttps://www.blogger.com/profile/09441597549860805094noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-57320623022519515072021-07-08T12:59:30.607+02:002021-07-08T12:59:30.607+02:00This comment has been removed by a blog administrator.Memorial Healthhttps://www.blogger.com/profile/04546170349787873096noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-26461453412585137682020-12-16T08:29:45.875+01:002020-12-16T08:29:45.875+01:00This comment has been removed by a blog administrator.miawrihttps://www.blogger.com/profile/15519227137487411654noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-779960210518294262020-05-12T23:54:56.072+02:002020-05-12T23:54:56.072+02:00There's also a Welch's version of ANOVA. T...There's also a Welch's version of ANOVA. This blog post provides a short discussion: https://statisticsbyjim.com/anova/welchs-anova-compared-to-classic-one-way-anova/ Alistair Cullumhttps://www.blogger.com/profile/16193690419324782781noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-4249556460061666972020-04-01T00:33:25.827+02:002020-04-01T00:33:25.827+02:00OK, now I'm getting nervous. The F statistic ...OK, now I'm getting nervous. The F statistic in one-way ANOVA uses the MSE as its denominator, and that's just pooled variance on steroids. Should we re-think the F test with more carefully designed factors? TheRandomTexanhttps://www.blogger.com/profile/07562250696288742894noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-8404641570241575742017-10-20T13:36:19.052+02:002017-10-20T13:36:19.052+02:00How to kiss the other party the most impressed.
...<br />How to kiss the other party the most impressed.<br /> <a href="https://www.hiso-casino.com/" rel="nofollow">คาสิโนออนไลน์</a> Betting Online Betting Sites It is part of the online players. Get into online betting. It is a popular game of all kinds of gambling games. It very good Can receive news More news. Love show To impress people who love themselves. Whether it is a woman or a man. There are always different ways to express love. Some people choose to say "love" to express their feelings to the other party to know. But some people choose to act in body language, express themselves more deeply and show more love.<br /><br /> Gently kisses the forehead for bright love. I used to say fans. Everything in life is happy and pink. The man kisses her forehead gently. It shows the love and affection he has for women. Women will feel warmth and safest. For the first kiss should show a love that is not very much. Most of the love is more expressive.<br /><br /> Kissing is a way of expressing love without mistakes. But do not be blatant or do it in a crowded place. It should be done in a private place where only you and your loved ones. Making an impression of kissing is not difficult. It is believed that the power of love, whether it is a kiss, it is always happy for online casinos. The exciting game has given us a chance to diversify our services. Interesting game It is another game that will advance to the online betting service that has continuous betting. Games to Employees to Thai Society of Games <a href="https://www.hiso-casino.com/" rel="nofollow">คาสิโนออนไลน์</a>คาสิโนออนไลน์ Gclub sbobethttps://www.blogger.com/profile/17778159157555939498noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-43551346252237288942017-03-22T17:14:14.818+01:002017-03-22T17:14:14.818+01:00Hey, out of curiosity, what about in cases where y...Hey, out of curiosity, what about in cases where you are using an ANOVA with either one or multiple predictors? Kyle Morrisseyhttp://dogsbody.psych.mun.ca/rcdmc/Site/Kyle_Morrissey.htmlnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-83559159126723583832017-02-01T14:39:16.649+01:002017-02-01T14:39:16.649+01:00Thank you for this valuable information, it is rea...Thank you for this valuable information, it is really useful. Anonymoushttps://www.blogger.com/profile/01204900184465837271noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-59170705488265236352016-05-28T11:25:08.590+02:002016-05-28T11:25:08.590+02:00This comment has been removed by a blog administrator.Anonymoushttps://www.blogger.com/profile/04571828795230778384noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-57240004794691983522016-05-09T16:21:48.982+02:002016-05-09T16:21:48.982+02:00Yes! Statistics posts with references and simulati...Yes! Statistics posts with references and simulations! Thanks! <br /><br />In you example, the data is normal, the variances are different but the means are the same. For my data on 4 or 6 different genotype groups, the data is not normal, the mean and variances are different between groups. I have chosen to log transform my data then perform Welch's test followed by Games-Howell post hoc test. Is it correct to transform data in such a way before carrying out a welch test or can you not say without more information on the dataset?<br /><br />It is likely that my study has further problems, as the types of genetic crosses we do we give 4 or 6 genotypes where 1/8th of a population has the mutant genotype we wished to study, so low n and varied sample sizes are inevitable. As you say 'Student's t-test is more powerful when variances and sample sizes are unequal and the larger group has the smaller variance' but it affects the type I error rate, I am confused as to what whether Welch's would be the best option for me in such a case. Would you recommend a non-parametric test such as Kruskal-Wallis instead? But Kruskal-Wallis assumes the same shaped distrubution as far as I know, so would again not be correct for my data I beleive.<br /><br />I was a bit worried about having to explain Welch's test in my Viva to an older generation of scientists, especially as a young medical statician had no idea what I was talking about recently. But I understand it much better now despite still having confusions about my own data!<br /><br />Thanks<br /><br />DeeAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-18246174946174656442015-12-21T12:25:23.475+01:002015-12-21T12:25:23.475+01:00At first it seems to be quite a difficult story wh...At first it seems to be quite a difficult story which will help in solving my problem but you are really very good things to clear the concepts and also it seems to be quite easy to understand now. <a href="http://www.spsshelp.org/how-to-run-a-manova-in-spss/" rel="nofollow">how to run a manova in spss</a>Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-82703495367789317512015-12-21T11:08:48.884+01:002015-12-21T11:08:48.884+01:00This comment has been removed by the author.Sharda Hinkelhttps://www.blogger.com/profile/03049412878821578827noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-49145893609337767942015-01-29T18:29:46.403+01:002015-01-29T18:29:46.403+01:00Depends on your H0. It's true if your H0 says ...Depends on your H0. It's true if your H0 says 'mu 1 = mu 2' (i.e. two populations with the same mean), not if it says 'x1. and x2. are drawn from the same population'. If it's the latter H0 you're interested in (a decent choice given randomisation), you could actually make the case that the t-test (as well as the Mann-Whitney or permutation tests) outperforms the Welch test in detecting that the two populations are indeed different.Janhttps://www.blogger.com/profile/17765078332699225416noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-70744475134741515882015-01-29T18:00:17.651+01:002015-01-29T18:00:17.651+01:00This comment has been removed by the author.Janhttps://www.blogger.com/profile/17765078332699225416noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-52194743592440481922015-01-29T17:32:33.131+01:002015-01-29T17:32:33.131+01:00I think these are good questions, that require dat...I think these are good questions, that require data. Setting differences between means to 0, but assuming differences in variance is the only way to examine the Type 1 error rate of a test, but does it happen in practice. I think it might, depending on the field you work with, but it's really an empirical question. <br /><br /><br /><br />Daniel Lakenshttps://www.blogger.com/profile/18143834258497875354noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-44419472057431662532015-01-29T17:13:12.054+01:002015-01-29T17:13:12.054+01:00"The R code below examines the Type 1 error r..."The R code below examines the Type 1 error rate of a hypothetical study where 38 participants were assigned to condition X, and 22 participants were assigned to condition Y. The mean score on some DV in both groups is the same (e.g., 0), so there is no effect, but the standard deviations between groups differ, with the SD in condition X being 1.11, and the SD in condition Y being 1.84."<br /><br />What kind of experimental manipulation would lead to identical means but affect variance? And wouldn't the upshot of such a randomised experiment have to be that there <i>was</i> an effect - just not in terms of the mean.<br /><br />I wholeheartedly agree that Levene's test (or test for normality or <a href="http://janhove.github.io/silly%20significance%20tests/2014/09/26/balance-tests/" rel="nofollow">covariate balance</a> for that matter) are overused in randomised experiments, and I also think that the Welch test is a better default than the normal t test in <i>non-randomised</i> experiments.<br />But doesn't randomisation allow us to boldly assume that 'null hypothesis = no effect' comprises both 'no mean shift' <i>and></i> 'no change in variance', i.e. the assumption that both groups were drawn from <i>the same population</i> (whatever it may be)? (And use permutation tests while we're at it.)Janhttps://www.blogger.com/profile/17765078332699225416noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-54676300650552733882015-01-28T01:15:06.287+01:002015-01-28T01:15:06.287+01:001-Thanks for answering :) i think I'm with you...1-Thanks for answering :) i think I'm with you on the glass delta, and I'll be interested to hear about different robust effect size measures when you get around to writing about them. <br /><br />3-But what do we do if they disagree again once we replicate? Replicate again? But it's true they agree most of the time! <br /><br />4-that's true, and it's worthwhile ;)Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-62908588788514160402015-01-27T17:31:33.356+01:002015-01-27T17:31:33.356+01:00Daniel; thanks for your extensive answers.Daniel; thanks for your extensive answers.Joost de Winterhttps://sites.google.com/site/jcfdewinter/noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-79020145014961478982015-01-27T17:23:25.466+01:002015-01-27T17:23:25.466+01:00Hi Joost,
I provide one example, and then referen...Hi Joost,<br /><br />I provide one example, and then reference an extensive literature that has examined this issue in detail, with a vast amount of different values, in hundreds of simulations, This is not my idea, it is not debated, I'm just explaining it. The references are there, so if you care enough about this topic to run simulations, read the literature I am summarizing. <br /><br />There is a very specific set of effect size/sample size combination where Student's t-test has a little (but not enough to make it worthwhile) more power. This is discussed in the literature, and since you cannot be certain you have equal variances in tiny samples, you should always report Welch's t-test if you do science in the real world. I honestly don't care which specific combination you an come up with while running simulations in R where power values are a tiny bit in the advntage of Student's t. <br /><br />Your latest example has pointed out a situation where I need to add 2 participants to the smallest group to compensate for the difference in power, and you are already entering the domain of underpowered studies (which I hope you are not recommending as good practice). If you want to perform even more underpowered studies, you can boost the difference between Student's t-test and Welch's t-test even more. But this all does not change the very simple fact that you should always report Welch's t-test (or, if data are not normal, robust statistics, such as Yuen's method, an adaptation on Welch's method using trimmed means and windsorized variances - but I'm leaving that for a future post).<br /><br />Daniel Lakenshttps://www.blogger.com/profile/18143834258497875354noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-87078397473941463092015-01-27T16:56:36.712+01:002015-01-27T16:56:36.712+01:00Dear Daniel,
1) One could try other combinations ...Dear Daniel,<br /><br />1) One could try other combinations using larger sample sizes (e.g., n1=100, n2=10, sd1=sd2=1), and this also shows that the t test can have a considerable power advantage (85% vs. 78% for a mean shift of 1). Anyway, I am unclear why you are saying that (n1=30, n2=5) is less informative than (n1=38, n2=22). Sometimes researchers are facing small sample sizes (and large effects). William Gosset developed the t test exactly because he was wanted to obtain valid results for small samples. <br /><br />2) What I meant is NOT that unequal variances are rare IF you have normal distributions. What I meant is that unequal variances are usually accompanied by non-normal distributions. That is, unequal variances usually arise for some reason, such as a floor or ceiling measurement artefact. It would be interesting to explore how the Welch and t test compare in situations other than perfectly normal distributions. <br />For example, suppose one assumes a 5-point Likert scale, with the following density distribution: Totally disagree = 0%, Slightly disagree = 1%, Neutral = 3%, Slightly agree = 6%, Strongly agree = 90%, that is a highly skewed distribution. Also assume n1=22, n2=100. Now sample the two vectors from the same population. The Type I error rate is now 13% for the Welch test, and 4% for the t test. Of course, this is an extreme situation, but it nicely illustrates that the Welch test can break down completely if the distributional assumptions are violated. <br /><br />3) I am also not sure whether the burden of proof should be on me now, while you are the one claiming that the Welch test should “always” be used. I just think you are making quite a generalization by saying that the Welch test is always preferred, because your conclusion seems to be based on 1 simulation using 1 set of parameters and 1 type of distribution. Counterexamples are easily found….<br /><br />Cheers, Joost<br />Joost de Winterhttps://sites.google.com/site/jcfdewinter/noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-59842833258617166712015-01-27T11:21:40.496+01:002015-01-27T11:21:40.496+01:00Finally, in your example, as in the previous examp...Finally, in your example, as in the previous example, the power difference is mitigated by running one additional participant (so 6 instead of 5). Unless you have something with practical relevence to report, I'm sticking with my recommendation to alway report Welch's t-test.Daniel Lakenshttps://www.blogger.com/profile/18143834258497875354noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-42763917640773992372015-01-27T11:09:53.086+01:002015-01-27T11:09:53.086+01:00Also, please provide those empirical references fo...Also, please provide those empirical references for your earlier statement that unequal variances are rare in normal distributions. Thanks.Daniel Lakenshttps://www.blogger.com/profile/18143834258497875354noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-155032506549717422015-01-27T11:08:31.092+01:002015-01-27T11:08:31.092+01:00Joost, you do seem to love ridiculously small samp...Joost, you do seem to love ridiculously small sample sies despite my previous sttement that doing tatistics on such samples is not telling you more than flipping a coin. A type 1 error rate of 3% when you want it to be 5% means the test is performing poorly - so your example is demonstrating that Welch's test is performing better, but just don't realize it.Daniel Lakenshttps://www.blogger.com/profile/18143834258497875354noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-11965597877481161702015-01-27T09:55:18.943+01:002015-01-27T09:55:18.943+01:00Dear Daniel,
It is possible to devise more counte...Dear Daniel,<br /><br />It is possible to devise more counterexamples, even with normal distributions.<br /><br />For example, with n1 = 30, n2 = 5, sd1 = 1.2, sd2 = 1, I get a Type II error rate of 5% for the t test and 10% for the Welch test (for detecting a mean shift of 2). The Type I error rate is 3% for the t test and 6% for the Welch test. So we now have a situation where the Welch t test performs twice as poorly as the t test regarding both power and false positives.<br /><br />Best regards, JoostJoost de Winterhttps://sites.google.com/site/jcfdewinter/noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-45889072517598398962015-01-27T07:48:34.479+01:002015-01-27T07:48:34.479+01:00Hi Joost, yes, in tiny samples, power differences ...Hi Joost, yes, in tiny samples, power differences become more pronounced than 4%. But in your example (and as I note) adding one participant to the n=2 sample will mitigate the power differences (at least in one simulation I just ran). So if the difference in power can be solved by adding a single participant, I consider it practically unimportant. It's something people who like to run simulations might care about, but it is irrelevant for the researchers actually using statistics. <br /><br />Another reason I did not consider these situations is because doing inferential statistics on such small sample sizes is not very useful (http://daniellakens.blogspot.nl/2014/11/evaluating-estimation-accuracy-with.html). <br /><br />More importantly, because your Levene's test will have such low power you will never be able to convince yourself variances are equal - I don't think you can provide a reference for your claim that unequal variances do not occur a lot in normal distributions. As I explain, the assumption is, in it's extreme form, practically untennable, and with different variances, Welch's test should be used. <br /><br />But feel free to provide empirical support that variances are often equal, that would be very interesting. Daniel Lakenshttps://www.blogger.com/profile/18143834258497875354noreply@blogger.com