Weighting Scheme Comparison
Two different analyses were performed to ensure that WLS or IRLS gives you the right inferential results.
This function computes the type 1 error rate for a single channel (120 trials * 100 frames) using random ERP or Gaussian data - repeated 10000 times. The average (‘cell-wise’) type 1 error rate is at the nominal level (5%) for OLS. Results also show that IRLS are a little lenient, with a small but significant increase of ~0.005% (i.e. an error rate of ~0.055 = lower p-values in figures below), while WLS are conservative for simulated ERP (~0.04 = higher p-values in figures below) and lenient with purely Gaussian data (~0.065). This behaviour of WLS is caused by the PCP method that optimizes weights based on distances across time, except that with simulated Gaussian data there is no autocorrelation and the PCA returns a much higher number of ‘relevant’ dimensions, leading to a meaningless feature reduction and thus meaningless trial distances and weights.
Figure 1. 10000 Monte Carlo simulations for a regression design (1 channel 120 trials * 100 frames)
Figure 2. 10000 Monte Carlo simulations for an ANOVA design (1 channel 120 trials * 100 frames)
Figure 3. 10000 Monte Carlo simulations for an ANCOVA design (1 channel 120 trials * 100 frames)
Figure 4. 10000 Monte Carlo simulations for the covariate of an ANCOVA design (1 channel 120 trials * 100 frames)
This functions takes H0 limo generated folder in and check the type 1 error cell-wise of the data in, returning avg and binomial ci, and making figures of density and convergence rate. Running this function on Wakeman and Henson (2015) data for OLS, WLS, and IRLS showed again that weighting schemes are a little conservative when using theoretical p-values. Importantly, using the maximum F-values under the null and cluster-mass (sum of F values above p-value threshold) provides good control of type 1 Familly-Wise Error Rates, mandatory when testing the whole data space. This also allowed to confirm that subject-wise, 800 bootstraps are enough, as previously observed.
Figure 3. Type 1 error using OLS for H0 Wakeman and Henson 2015 data. Individual mean values ranged from 0.036 to 0.062 for maximum statistics (across subject average 0.0507) and 0.038 to 0.068 for spatial-temporal clustering (across subject average 0.0496).
Figure 4. Type 1 error using WLS for H0 Wakeman and Henson 2015 data. Individual mean values ranged from 0.039 to 0.069 for maximum statistics (across subject average 0.053) and 0.044 to 0.07 for spatial-temporal clustering (across subject average 0.051).