Improve throughput of utask_main scheduling #4461

jonathanmetzman · 2024-11-28T00:40:46Z

Don't defer the tasks, they've already been deferred.
Improve the scheduling time by 20% by batching calls so that we only need to read YAML and query the database once, per queue read.

…g filing (#4415) ### Motivation As per Chrome request, it is desirable to know how long it takes for an issue to be opened, from the moment a testcase is created. Part of #4271

…#4414) ### Motivation Chrome folks need to know how long on average a fuzzer takes to generate a testcase. This PR implements that. Part of #4271

### Motivation Cumulative distribution metrics from the monitoring initiative were incorrectly set to use the fixed width bucketer, and/or width=0.05 and max_buckets=20. This caused percentile metrics to cap at 1, which was wrong behavior. This PR attempts to fix that by moving them all to Geometric Bucketer, without the aforementioned limits. It also reverts #4429 , since it apparently broke triage.py in chrome. Part of #4271

Fetching the testcase from the datastore has been wrongly moved before the call to `_update_testcase`. This leads to us holding an outdated version of the testcase.

…4437) Some parts of CF will chdir, breaking relative paths.

### Motivation Chrome folks want to know what build revision is being used in fuzz task. This PR implements that.

1. Don't unnecessarily list features about blobs other than names 2. Memoize leak blacklist stuff. 3. Properly skip cleanup.

Previously we tried to handle it by finding the fuzz targets in the list. But we still failed because it wasn't in the db. Save it to the db to solve this problem.

Getting credentials is pretty slow since it launches a gcloud process. We should just cache credentials. Supposedly the library itself handles refreshes. We are also caching in storage.py so there's proof it's safe.

We want tworkers to skip the function, not non-tworkers.

It's important for speed, probably safe on tworkers, and it will be very easy to notice if things go wrong since the preprocess queue will pile up. This PR also fixes a bad bug where the max_pool_size was not obeyed, but in practice since this code never runs on machines with more than 2 cores, it's unlikely to matter. Partial undoes #4430

Make some of the helper functions for batch task creation run on multiple tasks at once, that way we don't need to parse YAML or query the database as often. Also, don't defer utask_mains. Deferring/delaying already happened in preprocess step.

1. Don't defer the tasks, they've already been deferred. 2. Improve the scheduling time by 20% by batching calls so that we only need to read YAML and query the database once, per queue read. --------- Co-authored-by: Vitor Guidi <[email protected]> Co-authored-by: Ali HIJAZI <[email protected]>

vitorguidi and others added 15 commits November 25, 2024 16:20

[Monitoring] Adding metric to track time from testcase creation to bu…

28eb374

…g filing (#4415) ### Motivation As per Chrome request, it is desirable to know how long it takes for an issue to be opened, from the moment a testcase is created. Part of #4271

[Monitoring] Adding a blackbox fuzzer testcase generation time metric (…

dc41f5a

…#4414) ### Motivation Chrome folks need to know how long on average a fuzzer takes to generate a testcase. This PR implements that. Part of #4271

Fix Analyze Task (#4441)

26fa37a

Fetching the testcase from the datastore has been wrongly moved before the call to `_update_testcase`. This leads to us holding an outdated version of the testcase.

[NFC] Use an absolute path for configs when running butler scripts. (#…

18a9070

…4437) Some parts of CF will chdir, breaking relative paths.

[Monitoring] Adding build revision metric (#4428)

413f9cc

### Motivation Chrome folks want to know what build revision is being used in fuzz task. This PR implements that.

Do safe preprocess optimizations. (#4444)

1ff092d

1. Don't unnecessarily list features about blobs other than names 2. Memoize leak blacklist stuff. 3. Properly skip cleanup.

Handle new fuzz targets in split builds properly (#4436)

933a999

Previously we tried to handle it by finding the fuzz targets in the list. But we still failed because it wasn't in the db. Save it to the db to solve this problem.

Cache credentials (#4446)

9cea321

Getting credentials is pretty slow since it launches a gcloud process. We should just cache credentials. Supposedly the library itself handles refreshes. We are also caching in storage.py so there's proof it's safe.

Remove accidental not (#4448)

e83c7f4

We want tworkers to skip the function, not non-tworkers.

Reduce utask_main scheduler time by 20%

3989a15

Make some of the helper functions for batch task creation run on multiple tasks at once, that way we don't need to parse YAML or query the database as often. Also, don't defer utask_mains. Deferring/delaying already happened in preprocess step.

Merge

7e6fb2f

fix

49001da

Fix

a1a9a3d

jonathanmetzman changed the title ~~Schedulespeed~~ Improve throughput of utask_main scheduling Nov 28, 2024

jonathanmetzman requested a review from oliverchang November 28, 2024 00:49

fix tests

8299cfd

jonathanmetzman merged commit 144bcf1 into oss-fuzz Nov 28, 2024
3 checks passed

jonathanmetzman deleted the schedulespeed branch November 28, 2024 01:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve throughput of utask_main scheduling #4461

Improve throughput of utask_main scheduling #4461

jonathanmetzman commented Nov 28, 2024 •

edited

Loading

Improve throughput of utask_main scheduling #4461

Improve throughput of utask_main scheduling #4461

Conversation

jonathanmetzman commented Nov 28, 2024 • edited Loading

jonathanmetzman commented Nov 28, 2024 •

edited

Loading