Race condition on blocksToDelete

[danielblando]

Describe the bug
Running cortex image with race detection it was found a condition when deleting blocks from a user tsdb and creating new tsdb. This seems to be an ok scenario as we can be cleaning tsdb while receiving data for a new tenant, but we should eliminate the race condition

To Reproduce
Steps to reproduce the behavior:

1. Start Cortex (v.1.18) with -race enabled
2. Perform Operations Write while waiting for delete loop to be executed

Expected behavior
No race condition detected

Environment:

• Infrastructure: kubernets
• Deployment tool: helm

Additional Context
@yeya24 initial test data

WARNING: DATA RACE
Read at 0x00c005626100 by goroutine 5919:
  golang.a2z.com/Cortex/vendor/github.com/cortexproject/cortex/pkg/ingester.(*userTSDB).blocksToDelete()
      /local/home/benye/workplace/Cortex/build/Cortex/Cortex-1.0/AL2_x86_64/DEV.STD.PTHREAD/build/gopath/src/golang.a2z.com/Cortex/vendor/github.com/cortexproject/cortex/pkg/ingester/ingester.go:510 +0x58
  golang.a2z.com/Cortex/vendor/github.com/cortexproject/cortex/pkg/ingester.(*userTSDB).blocksToDelete-fm()
      <autogenerated>:1 +0x51
  golang.a2z.com/Cortex/vendor/github.com/prometheus/prometheus/tsdb.(*DB).reloadBlocks()
      /local/home/benye/workplace/Cortex/build/Cortex/Cortex-1.0/AL2_x86_64/DEV.STD.PTHREAD/build/gopath/src/golang.a2z.com/Cortex/vendor/github.com/prometheus/prometheus/tsdb/db.go:1558 +0x310
  golang.a2z.com/Cortex/vendor/github.com/prometheus/prometheus/tsdb.(*DB).run()
      /local/home/benye/workplace/Cortex/build/Cortex/Cortex-1.0/AL2_x86_64/DEV.STD.PTHREAD/build/gopath/src/golang.a2z.com/Cortex/vendor/github.com/prometheus/prometheus/tsdb/db.go:1078 +0x771
  golang.a2z.com/Cortex/vendor/github.com/prometheus/prometheus/tsdb.open.gowrap1()
      /local/home/benye/workplace/Cortex/build/Cortex/Cortex-1.0/AL2_x86_64/DEV.STD.PTHREAD/build/gopath/src/golang.a2z.com/Cortex/vendor/github.com/prometheus/prometheus/tsdb/db.go:1022 +0x4f

Previous write at 0x00c005626100 by goroutine 2314:
  golang.a2z.com/Cortex/vendor/github.com/cortexproject/cortex/pkg/ingester.(*Ingester).createTSDB()
      /local/home/benye/workplace/Cortex/build/Cortex/Cortex-1.0/AL2_x86_64/DEV.STD.PTHREAD/build/gopath/src/golang.a2z.com/Cortex/vendor/github.com/cortexproject/cortex/pkg/ingester/ingester.go:2346 +0x2415
  golang.a2z.com/Cortex/vendor/github.com/cortexproject/cortex/pkg/ingester.(*Ingester).openExistingTSDB.func1()
      /local/home/benye/workplace/Cortex/build/Cortex/Cortex-1.0/AL2_x86_64/DEV.STD.PTHREAD/build/gopath/src/golang.a2z.com/Cortex/vendor/github.com/cortexproject/cortex/pkg/ingester/ingester.go:2456 +0x3e4
  golang.a2z.com/Cortex/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1()
      /local/home/benye/workplace/Cortex/build/Cortex/Cortex-1.0/AL2_x86_64/DEV.STD.PTHREAD/build/gopath/src/golang.a2z.com/Cortex/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x91