Latency for http cache higher than expected

[anarazel]

Hi,

I was looking at using the http cache as a secondary ccache cache. Mainly with the goal of sharing cache state between multiple tasks - hard to do otherwise without each task overwriting cache state otherwise. The results were a bit underwhelming so far - mostly due to the high latency when accessing the cache:

Log excerpt for a cache miss case:
[2022-03-20T21:17:12.178867 19169] No 549cob1s4q8tepm2scqn2sffpm904c3dk in http://127.0.0.1:12321/ccache2 (40.64 ms)
[2022-03-20T21:17:12.253862 19169] No 7108io3fin8bfnsbaee120hus3go26mc6 in http://127.0.0.1:12321/ccache2 (29.00 ms)
[2022-03-20T21:17:12.392228 19169] Stored 7108io3fin8bfnsbaee120hus3go26mc6 in http://127.0.0.1:12321/ccache2 (57.49 ms)
[2022-03-20T21:17:12.450723 19169] Stored 549cob1s4q8tepm2scqn2sffpm904c3dk in http://127.0.0.1:12321/ccache2 (57.45 ms)

This is with keepalive enabled, FWIW (which explains why the second miss is faster).

Is it that roughly the expected speed for http cache accesses?

For experimentation I enable ccache logging and uploaded it as an artifact...

empty cache run:
https://cirrus-ci.com/task/4791740338012160

full cache run:
https://cirrus-ci.com/task/5084119632707584

Greetings,

Andres

[anarazel]

Oh, one more question: What's the size of the http cache and when will it be cleaned up?

[fkorotkov]

Second run was 2:30 https://cirrus-ci.com/task/6132684358418432. I guess there was some weird issue with the first one but anyways there is not gains found.

[anarazel]

Phew ;).

I think I'll give up on the plan to use the http cache for ccache - with cache entries mapping 1:1 to GCS/S3/etc I think the inherent latency (and likely cost) is just going to be too high. Before your reply, I hadn't realized that that's what the http cache maps to. I actually thought it'd just be served by a nearby instance and stored on instance storage or such (with in-memory caching).

[fkorotkov]

Maybe will be interesting to check if having a micro instance that is part of the free tier on GCP and have redis running there. It doesn't look like your overall ccache size is big (seems just 50Mb). Then you'll have an internal instance address that you can use from within your tasks.

[anarazel]

Under the hood cache instruction is using the HTTP cache and therefore the retention period is the same and equal to 90 days. For managed by us tasks the size limit of a cache entry is 5Gb and there is no limit if you use your own infra.

Small clarification question: I assume that the 5GB are for caches, logs, explicit artifacts together? And that if the 5GB are exceeded the oldest cache/log/artifact is deleted?

Would be nice to document the 5GB on https://cirrus-ci.org/faq/#are-there-any-limits - I guess I'll open a ticket?

Maybe will be interesting to check if having a micro instance that is part of the free tier on GCP and have redis running there.

Yea, I wondered about doing something like that. It'd be a bit of pain to deal with auth etc, given that one goal is to have random people be able to clone the repo and run CI tasks. But solvable.

It doesn't look like your overall ccache size is big (seems just 50Mb). Then you'll have an internal instance address that you can use from within your tasks.

The cache is bigger than that normally, I just made it artificially smaller to test the http caching...

[JeremyRand]

Small clarification question: I assume that the 5GB are for caches, logs, explicit artifacts together? And that if the 5GB are exceeded the oldest cache/log/artifact is deleted?

Also curious about this clarification.