tag:blogger.com,1999:blog-987850932434001559.comments2018-03-22T09:20:20.050-07:00The 20% StatisticianDaniel Lakensnoreply@blogger.comBlogger925125tag:blogger.com,1999:blog-987850932434001559.post-26698327224875536912018-02-14T01:03:47.533-08:002018-02-14T01:03:47.533-08:00Sorry about that - blogger is spammed like crazy, ...Sorry about that - blogger is spammed like crazy, it is basically impossible to allow and moderate comments the last months, so they get caught in crappy spam filters or buried between spam. Daniel Lakenshttps://www.blogger.com/profile/18143834258497875354noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-83774370769074276922018-02-13T17:06:19.951-08:002018-02-13T17:06:19.951-08:00Wow thawt waѕ odd. I јust wrote an very long comme...Wow thawt waѕ odd. I јust wrote an very long comment bᥙt aftｅr I clicked submit mү comment diԀn't <br />snow up. Grrrr... welⅼ I'm not writing ɑll that over agаin. Аnyhow, just wantеd to sау <br />wonderful blog!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-43044643016817171542018-02-10T00:39:33.577-08:002018-02-10T00:39:33.577-08:00I did not. Please provide the quote you are referr...I did not. Please provide the quote you are referring to. Then we can discuss what I meant. Daniel Lakenshttps://www.blogger.com/profile/18143834258497875354noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-64093017800053417922018-02-09T19:39:57.718-08:002018-02-09T19:39:57.718-08:00In your online class you just said the opposite of...In your online class you just said the opposite of this post Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-10433825668587333732018-02-01T04:15:26.087-08:002018-02-01T04:15:26.087-08:00"Goldin-Meadow stated that "it’s just to..."Goldin-Meadow stated that "it’s just too costly or unwieldy to generate hypotheses on one sample and test them on another when, for example, we’re conducting a large field study or testing hard-to-find participants" (2016). A similar comment is made by Tackett and McShane (2018) in their comment on ZELD: “Specifically, large-scale replications are typically only possible when data collection is fast and not particularly costly, and thus they are, practically speaking, constrained to certain domains of psychology (e.g., cognitive and social).”"<br /><br />I always find this reasoning fascinating. To me, it doesn't make much sense, because these "hard to find" or "costly" participants are apparently only "hard to find" or "costly" for 1 study. <br /><br />After all, chances are high that a next study will use the same "hard to find" or "costly" particpants but now for a different study for which participants and money magically appear all of a sudden.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-78358443533372305112018-01-30T03:05:29.626-08:002018-01-30T03:05:29.626-08:00"Do you think that would be useful informatio..."Do you think that would be useful information, and do you know if this has been done by someone already?"<br /><br />I still think this might be very useful, also see my (as of yet unpublished) possibly directly related comment on this blogpost, and on this one here about a new format for performing and publishing psychological science: http://daniellakens.blogspot.nl/2018/<br /><br />Sorry if i keep bothering you with the idea, but i haven't received any feedback on it, so i don't know if you think it might make (some) sense or not. <br /><br />Wouldn't it be funny if someone would give you ideas and thoughts for free, which could possibly be helpful in thinking about how to optimally perform research, just because you give them the opportunity to try and contribute by havind this blog? <br />You wouldn't even need a fancy grant (worth how much money exactly?!) for any of this stuff! Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-28209184925278585682018-01-25T04:30:22.035-08:002018-01-25T04:30:22.035-08:00"Tell us at which probability that H0 is true..."Tell us at which probability that H0 is true you think it is efficient enough to weed out false positives from the literature through direct replications. Show us, by pre-registering all your main analyses, that you are building on strong theories that allow you to make correct predictions with a 92% success rate, and that you therefore do not feel direct replications are the more efficient way to gain knowledge in your area."<br /><br />Hmm, not sure if i agree with (my interpretation of) this. I reason:<br /><br />1) it seems to me that it is impossible to determine/gauge the percentage of hypotheses that (will) turn out to be "correct"/"proven".<br /><br />2) more importantly, it seems to me that a) it doesn't matter what this percentage is (within reasonable boundaries), and b) it is not even desirable to try and determine/gauge this percentage, because i reason both a) and b) are irrelevant for building knowledge. <br /><br />What matters in my reasoning is amassing things like optimally gathered (and thus maximaly informational) data, and arguments/reasoning, which can both be used for things like theory-building and -testing. <br /><br />I am all for "(...) scientists explicitly thinking about the utility of the research they perform", but i reason it might make more sense to think about this for all research, not just replications. In fact, i reason that it is more important for "original" research, because i predict that (nearly) all else will follow automatically once things are done more optimally from the start. <br /><br />The bottom line to me is, many of the things that might be wrong in psychological science seem to me to be connected, and based on a few things that need to be improved. I reason the rest of the problems will solve themselves automatically. <br /><br />Here is an idea which tries to solve some of the basic things that might be connected, and which i reason will also help solve other issues. I called it after what i think is a summarization of the few basic problems which i reason can all easily be solved: “Science is dependent on scientists (old flawed model?) V scientistst are dependent on Science (new improved model?)”:<br /><br />http://andrewgelman.com/2017/12/17/stranger-than-fiction/#comment-628652<br /><br />I hope you will (also) focus on how to optimally perform research in general, and not just on replications. I reason when the former is done, the rest will follow automatically. Good luck/ all the best with your work on this important topic !! Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-20900046189430447162018-01-23T12:39:03.901-08:002018-01-23T12:39:03.901-08:00Nothing changes - if you don't assume equal va...Nothing changes - if you don't assume equal variances, you do a one-sided Welch's t-test. Daniel Lakenshttps://www.blogger.com/profile/18143834258497875354noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-1635530522637079282018-01-23T07:56:03.623-08:002018-01-23T07:56:03.623-08:00Thanks for blog post Daniel :) Just wondering abou...Thanks for blog post Daniel :) Just wondering about the unequal variance issue? <br />I am guessing that converting a two-way to one-way test, i would assume that a balanced set and equal variance will be assumed or not?Animesh Sharmahttps://www.blogger.com/profile/07485653184282867837noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-8211190453055754312018-01-19T00:49:59.862-08:002018-01-19T00:49:59.862-08:00Perhaps it would be easier for people to get into ...Perhaps it would be easier for people to get into justifying replications in this way if they were also in the habit of similarly justifying why they run their initial studies. I'm not convinced that this is always the case. "Because we have a grad student who thinks this is interesting and a participant pool who have quotas to meet" does not necessarily meet this criterion, I would suggest. Nick Brownhttps://www.blogger.com/profile/07481236547943428014noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-64714907435572458192018-01-17T08:27:56.175-08:002018-01-17T08:27:56.175-08:00Great post.Great post.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-5895486407750795842017-12-13T08:51:48.609-08:002017-12-13T08:51:48.609-08:00Yes, you can always calculate the effect size you ...Yes, you can always calculate the effect size you could detect with a certain level of power. But, there is never information that goes beyond the p-value. So, knowing how sensitive your design was is always good info to have, but it is difficult to use it as a way to draw inferences from data. Daniel Lakenshttps://www.blogger.com/profile/18143834258497875354noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-45413209655198769272017-12-13T08:23:02.769-08:002017-12-13T08:23:02.769-08:00I think there are some forms of post-hoc power ana...I think there are some forms of post-hoc power analyses that are appropriate.<br /><br />I agree wholeheartedly with everything you have said. Calculating the power of the study from the sample size (and SD) the alpha and the *observed* power is completely useless post-hoc.<br /><br />However, would it not be reasonable for the editor to ask for the following:<br /><br />In cases where a power/sample size calculation has not been performed in the original paper (perhaps in cases where group sizes are determined by other factors), would it not be suitable for the editor to ask for a calculation of the *detectable* difference. i.e put into the power calculation the alpha, sample size, a pre-agreed beta value and see what size of difference the study would have been able to fix.<br /><br />I understand that this method would closely align with confidence intervals. However, I think it will demonstrate under-powered studies with more impact. Particularly in non-inferiority trials that claim non-inferiority when that are massively under-powered. George Harveyhttps://twitter.com/_Harvsnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-49809971415804238602017-12-09T07:09:59.683-08:002017-12-09T07:09:59.683-08:00Possibly related to the previous post, and related...Possibly related to the previous post, and related to how to “optimally” perform research concerning resources, I thought of the following.<br /><br />Even though I s#ck at statistics, given my previous post, I wondered if it would be possible to try and gauge what the optimal no. of 1) participants per study and/or researcher, and 2) direct replications could be.<br /><br />Then I thought about all the “Registered Replication Reports”, and the figures of all the separate labs with their associated no. of pp, confidence intervals, effect sizes, and p-values.<br /><br />I wondered if it would be possible to use this information from all “Registered Replication Reports” performed thus far, to see how diagnostic it would be to randomly draw 3, or 4, or 5 etc. labs, and see how diagnostic/accurate their associated no. of pp, confidence intervals, effect sizes, and p-values are given the total results of all labs combined. <br /><br />This could be possibly be very interesting, and useful, information (it being “real” compared to possible additional simulations) concerning how to “optimally” perform research, and could perhaps also provide information concerning the format I described in the previous post, and what the “optimal” amount of “direct” replications could be for instance.<br /><br />Do you think that would be useful information, and do you know if this has been done by someone already? I am not smart enough to do that.<br /><br />Regardless: thank you for all your efforts in trying to help improve psychological science!<br />Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-433796537573697162017-12-06T12:13:28.655-08:002017-12-06T12:13:28.655-08:00Nice article! Thanks!!Nice article! Thanks!!MJ Alamhttps://www.blogger.com/profile/03969328667025688253noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-9438388789133113742017-12-05T04:16:06.427-08:002017-12-05T04:16:06.427-08:00Since you (like I) are happy to assume a point nul...Since you (like I) are happy to assume a point null hypothesis, I can't understand why you don't frame the argument in terms of the false positive risk.David Colquhounhttp://www.dcscience.net/noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-18613266022758177112017-11-14T23:09:22.611-08:002017-11-14T23:09:22.611-08:00The tail of the test follows from the theoretical ...The tail of the test follows from the theoretical preduction you are testing. It is unrelated to belief (you can test a hypothesis you don't believe).Daniel Lakenshttps://www.blogger.com/profile/18143834258497875354noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-72356087942580425262017-11-14T12:58:32.226-08:002017-11-14T12:58:32.226-08:00Nice post. I tried to make a similar argument by p...Nice post. I tried to make a similar argument by pointing out that many statistical analyses depend on exactly the same information from the data but can produce different answers because they ask different questions. Details at:<br />https://link.springer.com/article/10.3758%2Fs13428-016-0812-3<br />Greg Francishttp://www2.psych.purdue.edu/~gfrancis/home.htmlnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-70714842382132046972017-11-13T04:57:20.435-08:002017-11-13T04:57:20.435-08:00>> There is no room for the Statistician’s F...>> There is no room for the Statistician’s Fallacy in our education. I do not think it is useful to tell researchers what they want to know. <br /><br />I haven't heard of anyone doing this in his/her lectures.<br /><br />>> the idea that there is always a single thing ‘we want’ is not true<br /><br />Nor do I see this idea being presented in any of the quotes. In my reading, the quoted statements are descriptive - they tell us what researchers want to do based on what we saw them doing in the past. There are numerous studies that show that researchers interpret p value as probability of the hypothesis (eg Oakes,1986) and Cumming has done some research questioning students about their use and interpretation of p values and effect size. <br /><br />In addition, in some qoutes, the "we" might refer to the author of the manuscript and the statements about "what we want" refer to goals that were set earlier in the writing. I can certainly imagine this being the case for the qoute from Cumming. We should consider the context.<br /><br />>> But I think it’s a good idea to teach them about all the possible questions they can ask.<br /><br />Of course, this is a nice idea and hardly anyone would disagree. But how do we find all those possible questions and how and what should we teach students about them? My guess is that in the end the popular answer would be to provide introduction into (history of) philosophy of science and perhaps some (history of) epistemology, sprinkled with some insights from Meehl, Cohen and other gurus. I doubt this improves inferences since it does not provide a formal, transparent and replicable (and thus scientific) language for asking and discussing scientific questions and statements. <br /><br />As You possibly already learned from my other comments, I think such language already exists and should be taught. It's the language of modern causal inference. <br /><br />matushttp://simkovic.github.ionoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-31178462143686287902017-11-12T04:26:41.329-08:002017-11-12T04:26:41.329-08:00Bravo! Occasionally my brighter students manage t...Bravo! Occasionally my brighter students manage to "come up for air" after being submerged in probability distributions, hypothesis tests, and modeling techniques. Then they ask "What's it all for?" and "How do I know when to use what?" and other embarrassing questions which most textbooks carefully avoid. Now I need to collect case studies that prompt my students to think about those "possible questions to ask."therandomtexanhttps://therandomtexan.wordpress.com/noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-36621011307927791502017-11-11T12:25:24.870-08:002017-11-11T12:25:24.870-08:00On pointOn pointSolomon Kurzhttps://www.blogger.com/profile/17246017680368802724noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-5136223451846920672017-11-11T10:33:23.705-08:002017-11-11T10:33:23.705-08:00This is your best post so far Daniel. I look forwa...This is your best post so far Daniel. I look forward to the one directly following up your last sentence.Unknownhttps://www.blogger.com/profile/00227235335343168838noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-33429465880327696532017-11-11T06:54:06.426-08:002017-11-11T06:54:06.426-08:00As a Stats teacher, I don't tell my students w...As a Stats teacher, I don't tell my students what they want to know. I do, however, tell them what kind of questions they can (and can't) get answered whenever they apply a particular technique or model. As a consultant, I typically hear the person first (to know "what they want") and then I try to advice accordingly. It does happen that then I rephrase what one can achieve when using a specific method (kind of what you can conclude once you replace a parametric by a nonparametric ANOVA, or even better, once you ditch frequestism over bayesianism). I think it is a best advice to educate students to phrase their questions in such a way that a translation into a suitable statistical model becomes feasible (instead of just freely phrasing questions).Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-51093736202975738752017-11-07T21:31:15.875-08:002017-11-07T21:31:15.875-08:00Never heard about it, but why don't you just u...Never heard about it, but why don't you just use the soreadsheet that comes with my 2017 article? And R is SUPER easy if you just want to use TOST. Like a simple calculator.Daniel Lakenshttps://www.blogger.com/profile/18143834258497875354noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-24243660740750757612017-11-07T16:45:48.380-08:002017-11-07T16:45:48.380-08:00Have you had experience using XLSTAT 'add on&#...Have you had experience using XLSTAT 'add on' software for Excel to calculate TOST? Their online tutorial makes it look simple. I am unfamiliar with R and to save me learning it, I thought this might be useful for equivalence testing. Any thoughts? Many thanks in advance.Sue Raddhttps://www.blogger.com/profile/06377200583605331824noreply@blogger.com