tag:blogger.com,1999:blog-987850932434001559.post6488638343074805212..comments2023-09-17T16:45:23.263+02:00Comments on The 20% Statistician: Sample Size Planning: P-values or Precision?Daniel Lakenshttp://www.blogger.com/profile/18143834258497875354noreply@blogger.comBlogger5125tag:blogger.com,1999:blog-987850932434001559.post-87509243485733038562014-11-04T22:48:05.692+01:002014-11-04T22:48:05.692+01:00This is very helpful, Daniel! I really love the id...This is very helpful, Daniel! I really love the idea of using what we know (i.e., the effect size estimate from the data we already have) and using that to plan further data collection. What an ingenious idea. Thanks so much for taking my question/problem so seriously!Michael McCulloughhttp://www.social-science-evolving.comnoreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-57047043225980451472014-10-31T21:40:38.484+01:002014-10-31T21:40:38.484+01:00As a student I had to choose some applied field fo...As a student I had to choose some applied field for core study and I went for clinical neuropsychology. Much of the diagnosistic trianing was focused on how you interpret various measures. Aside from the training, if you go into practice as a part of the patient record you will be handed results of the various test. If you just spent time looking at the scores of the various tests you will necessarily gain an impression what the variability in these measures is and how various interventions affect them. It's no black magic.<br /><br />In research the relevant domain knowledge comes from prior research usaully the published one. There has been more than century of this. It's also the reason why each report has an introduction where you describe the prior research. Although nowadays the authors just provide literature review and then just go to do something else.<br /><br />As to your researcher's problem. It may be a real-life problem, but imo psychologists are solving many artificial problems in their real life, This one seems to be of that sort. I already told you in our email exchange that I don't care much about planning of sample size. If the estimates are imprecise, you can always collect more data. I can't suggest more detailed strategy since the tweet doesn't tell more about the research question. But in general if you say that it's exploratory research this suggests that the author does not have any hypothesis. So what use does he have for hypothesis testing if he doesnt have any hypothesis? Looks like one more artificial real-life problem there..<br />matushttp://simkovic.github.io/noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-43466150257476619152014-10-31T12:30:19.430+01:002014-10-31T12:30:19.430+01:00Hi Matus, I'm talking about how to solve a pra...Hi Matus, I'm talking about how to solve a practical problem a real-lilfe researcher had. How would you have solved this? <br /><br />I'm not sure I understand how your clinician 'knows' so many things. Especially any understanding of how to achieve a lower bound on an ES is not in line with the statistical understanding of any clinicians I know. You are betting a lot on relevant domain knowledge - but I doubt it exists, and you do not clearly explain where that should come from. <br /><br />In this example, the researcher was clearly doing novel research and was trying to convince himself whether this was real, or not. So how should a sample size for such exploratory research be planned?Daniel Lakenshttps://www.blogger.com/profile/18143834258497875354noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-45022811749421261422014-10-30T22:44:00.904+01:002014-10-30T22:44:00.904+01:00Once again, Daniel, you present your very limited ...Once again, Daniel, you present your very limited idea of what statistics offers to researchers. Your argument is all about decisions. But what about people like me who don't make any decisions? If I develop a new therapy and test it, I then report that BDI decreases by x%, 95%CI [a,b]. Every idiot understands what this means (well, maybe minus the APA standard for reporting CI). A clinician further knows whether this decrease is notable. This depends on what alternative therapies there are to choose from. It is left to him/her to decide whether he wants to use this therapy. In comparison what does he gain if I tell him that my therapy significantly decreases the BDI score with p=0.023? I decide to discard a practically irrelevant and arbitrary hypothesis (H0, e.g. no change in BDI score) based on practically irrelevant and arbitrary cut off (e.g. alpha=0.05). Or what does your magical intuition for p-values somehow tell you whether a therapy with p=0.023 applying? I don't have such an intuition and I certainly doubt that any therapist has. Maybe you could finally disclose where and how these intuitions are obtained. You speak about them lot on this blog...<br /><br />The same goes for precision. For instance, a clinician may know that the lower bound a of the ES is not difficult to achieve with the available methods - some of which are less costly than the proposed therapy. But maybe the mean estimate x is a notable decrease such that he asks for more data which would increase the precision of the estimate and tell us whether the mean indeed around x or the result was a fluke and the mean estimate is more like a. Again this kind of reasoning is beyond what p and sample size can tell you. <br /><br />As to your graph, if you would tell us what in what units you was measured experts could use their domain relevant knowledge to decide whether the precision is good or should be improved.matushttp://simkovic.github.io/noreply@blogger.comtag:blogger.com,1999:blog-987850932434001559.post-44994276883511470962014-10-30T21:32:54.576+01:002014-10-30T21:32:54.576+01:00Nice post! Another way of thinking about this - &q...Nice post! Another way of thinking about this - "what do I want to compare this measurement to?" If your answer is "the null" then the p value and conventional power analysis is a good idea. If your answer is "another set of conditions that might plausibly differ by the following amount," then it may make more sense to think about the precision of the CI. They are the same thing, but one privileges testing against the null, while the other is more compatible with model-based thinking about actual numerical predictions across different conditions. Michael Frankhttps://www.blogger.com/profile/00681533046507717821noreply@blogger.com