[dv, dvsim] Add job queue/lic wait timeout in addition to run timeout #24817
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
venkatk-ot
force-pushed
the
nc_updates
branch
from
3be4254 to
b589300
Compare
|
Assigning to @luismarques to forward assign. @rswarbrick is away. |
rswarbrick
reviewed
Oct 21, 2024
venkatk-ot
force-pushed
the
nc_updates
branch
5 times, most recently
from
9ee2375 to
0418975
Compare
rswarbrick
reviewed
Oct 23, 2024
venkatk-ot
force-pushed
the
nc_updates
branch
from
0418975 to
a1a7f74
Compare
rswarbrick
reviewed
Oct 28, 2024
The timeout_mins specified in the test list json file or on dvsim command line is intended to be a timeout on the wall time elapsed after the job has started running.It is not intended to count in the time spent in waiting for a compute node and/or a license to become available. Despite best efforts to specify license resource strings, the availability of a compute node doesn't always guarantee the availability of a license as this scheme only works when all jobs from all projects running on the cluster adhere to this license requesting scheme which sadly can't be enforced in a fool proof manner. In the proposed scheme, the file size as a proxy to determine if the job has really started running. While this is not one hundred percent accurate, it prevents false timeouts on jobs/tests that haven't even had a chance to start running because of unavailability of compute or license resources. The file size threshold to determine whether the job has started running has been chosen after carefully analyzing log files of jobs executed successfuly and jobs that were killed while waiting Signed-off-by: Venkat Krishnan <[email protected]>
venkatk-ot
force-pushed
the
nc_updates
branch
from
a1a7f74 to
dd71989
Compare
rswarbrick
approved these changes
Oct 29, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The timeout_mins specified in the test list json file or on dvsim command
line is intended to be a timeout on the wall time elapsed after the job
has started running.It is not intended to count in the time spent in waiting for a compute node and/or a license to become available. Despite best efforts to specify license resource strings, the availability of a compute node doesn't always guarantee the availability of a license as this scheme only works when all jobs from all projects running on the cluster adhere to this license requesting scheme which sadly can't be enforced in a fool proof manner.
In the proposed scheme, the file size as a proxy to determine if the job has really started running. While this is not one hundred percent accurate, it prevents false timeouts on jobs/tests that haven't even had a chance to start running because of unavailability of compute or license
resources. The file size threshold to determine whether the job has
started running has been chosen after carefully analyzing log files of jobs executed successfuly and jobs that were killed while waiting