Fix MSRV: Remove use of `Vec::retain_mut` #857

Job	Run time
Test CUDA support	1m 14s
Test (stable)	1m 4s
Test (beta)	48s
Test (nightly)	43s
MSRV (ubuntu-latest)	4m 40s
Test (linux32)	59s
MSRV (windows-latest)	5m 1s
Test (macos)	1m 31s
Test (aarch64-ios)	1m 18s
Rustfmt	12s
Test (win32)	1m 38s
Test (win64)	1m 30s
Test (mingw32)	1m 54s
Test (mingw64)	1m 27s
Test (windows-2019)	2m 22s
Test (windows-aarch64, windows-latest, stable, aarch64-pc-windows-msvc, --no-run)	2m 48s
	29m 9s

Provide feedback