Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use multivariate as default for follow-up tests RM ANOVA #319

Merged
merged 14 commits into from
Mar 12, 2024

Conversation

JohnnyDoorn
Copy link
Contributor

@JohnnyDoorn JohnnyDoorn commented Feb 12, 2024

While setting this to "false" is in line with SPSS, the multivariate option is generally considered to be more appropriate since it handles deviations from sphericity better. See also the thread here, with a discussion between the authors of the emmeans and afex packages.

A caveat regarding the use of emmeans concerns the assumption of sphericity for ANOVAs including within-subjects/repeated-measures factors (with more than two levels). The current default for follow-up tests uses a univariate model (model = "univariate" in the call to emmeans), which does not adequately control for violations of sphericity. This may result in anti-conservative tests and contrasts somewhat with the default ANOVA table which reports results based on the Greenhousse-Geisser correction. An alternative is to use a multivariate model (model = "multivariate" in the call to emmeans) which should handle violations of sphericity better. The default will likely change to multivariate tests in one of the next versions of the package.

While we have added this option for a while now, I think it's best to make this the default option. I tweaked the unit tests such that those still use the univariate model, although I could also update them such that they use the multivariate tests, but then the results themselves need to be updated.

Fix jasp-stats/jasp-issues#1221
Fix jasp-stats/jasp-issues#1642

@vandenman
Copy link
Contributor

The reason for the change in default makes sense. That being said,

although I could also update them such that they use the multivariate tests, but then the results themselves need to be updated.

Please add (at least) one new test with options$multivariateModelFollowup <- TRUE 🙂

@JohnnyDoorn
Copy link
Contributor Author

@vandenman
Note on the changed unit test:

  • The Field results for follow-up tests did not match the actual results in the book, but now they do (they are based on the unpooled SE's now, but now I also still included the results based on the pooled SE's)
  • Updates to the CI's of Cohen's d, since now these also are affected by whether the pooled error term is used
  • I still had some errors in the ordinal restrictions (not sure why), and with the bootstrapped marginal means (which are not in Field), but unsure whether this has to do with differences in rng - curious what the github unit tests look like

@vandenman
Copy link
Contributor

There are some unit test failures, for example,

Failure ('test-anovarepeatedmeasures.R:1107:3'): Ordinal restrictions: Between factor marginal means table results match
The `test` argument provided to `expect_equal_tables()` is empty. Likely reason: the last run of jaspTools exited with a fatalError:

This analysis terminated unexpectedly.   Error in length(options[['restrictedMarginalMeanTerms']]) = 0 || options[['restrictedMarginalMeanTerms']] = : 'length = 2' in coercion to 'logical(1)'  

are these unrelated? If so, it would probably be good to fix these in a separate PR (I can take a look at that).

@JohnnyDoorn
Copy link
Contributor Author

Curious to see if the marginal means for RM ANOVA are now fixed (hopefully for all, but maybe only for mac?), since it passes locally.
Restricted analyses now run, but still has some issues for an(c)ova (also in simple main effects?!) but will look into those later.

@JohnnyDoorn
Copy link
Contributor Author

@vandenman I ended up removing all gorica fixes, will put them in a separate PR. For now the unit tests are only failing on some marginal means things.. shall we merge this?

@vandenman vandenman merged commit 3a1d832 into jasp-stats:master Mar 12, 2024
2 of 5 checks passed
@JohnnyDoorn JohnnyDoorn deleted the useMultivariateFollowUp branch May 21, 2024 12:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants