Add a CI check to ensure we pass base Ruby

Add a CI check to ensure we pass base Ruby #3126