This week I finished all of the one sample and two sample (continuous dependent variable) tests. Moving on, I just spent a few hours diving into ANOVA, ANCOVA, MANOVA, and MANCOVA analyses in all of their parametric vs non-parametric, repeated measures vs independent, covariate vs no covariate, multiple dependent variables vs single dependent variable, and multiple independent variables vs a single independent variable complexity. Yes, the complexity is real. My organization is looking something like this:
This is getting pretty complex. It is hard to understand and/or explain why a particular method should or shouldn’t be used. It is also hard to remember or document how to run each of these analyses, explain it in the right way, and interpret the results appropriately.
I started wondering why all of these methods are still used when they can almost (or maybe exclusively) be replaced with different forms of regression. If I can boil down 11 methods into 1 method, isn’t that a win for everybody? So I’m taking a step back to ponder how I want to proceed.
This also makes me consider alternative methods of analyzing data, such as bayesian approaches or bootstrapping/simulation. I wonder how many frequently used statistical methods I could replace (get similar or better interpretable results) with bayesian methods, regression, bootstrapping, and/or simulation. If I can, then maybe the whole pick your StatsTest is for naught. Instead, I just need to enable people to run complicated bayesian, bootstrapped, simulation-based, regression analyses in all cases!
In conclusion, I’m still thinking there is merit to StatsTest.com for a subset of commonly used statistical methods, but things get complicated very quickly, and Choose Your StatsTest won’t do a very good job of helping people here because it doesn’t do you any good to know what method you need to use if you don’t understand it at all and it is very complicated to run correctly. So I think the path forward is to pare down the table above by removing the multiple dependent variable analyses. That leaves me with 7 commonly used methods which seems like a decent path forward.