diff --git a/README.md b/README.md index ca96d2eb..e268fc41 100644 --- a/README.md +++ b/README.md @@ -175,7 +175,9 @@ You can exit the virtual environment simply by running `deactivate`. ### Step 4.1: Installing tools to access S3 bucket -The [`scripts/eessi-upload-to-staging`](https://github.com/EESSI/eessi-bot-software-layer/blob/main/scripts/eessi-upload-to-staging) script uploads a tarball and an associated metadata file to an S3 bucket. +The +[`scripts/eessi-upload-to-staging`](https://github.com/EESSI/eessi-bot-software-layer/blob/main/scripts/eessi-upload-to-staging) +script uploads an artefact and an associated metadata file to an S3 bucket. It needs two tools for this: * the `aws` command to actually upload the files; @@ -444,14 +446,17 @@ information about the result of the command that was run (can be empty). The `[deploycfg]` section defines settings for uploading built artefacts (tarballs). ``` -tarball_upload_script = PATH_TO_EESSI_BOT/scripts/eessi-upload-to-staging +artefact_upload_script = PATH_TO_EESSI_BOT/scripts/eessi-upload-to-staging ``` -`tarball_upload_script` provides the location for the script used for uploading built software packages to an S3 bucket. +`artefact_upload_script` provides the location for the script used for uploading built software packages to an S3 bucket. ``` endpoint_url = URL_TO_S3_SERVER ``` -`endpoint_url` provides an endpoint (URL) to a server hosting an S3 bucket. The server could be hosted by a commercial cloud provider like AWS or Azure, or running in a private environment, for example, using Minio. The bot uploads tarballs to the bucket which will be periodically scanned by the ingestion procedure at the Stratum 0 server. +`endpoint_url` provides an endpoint (URL) to a server hosting an S3 bucket. The +server could be hosted by a commercial cloud provider like AWS or Azure, or +running in a private environment, for example, using Minio. The bot uploads +artefacts to the bucket which will be periodically scanned by the ingestion procedure at the Stratum 0 server. ```ini @@ -466,7 +471,7 @@ bucket_name = { } ``` -`bucket_name` is the name of the bucket used for uploading of tarballs. +`bucket_name` is the name of the bucket used for uploading of artefacts. The bucket must be available on the default server (`https://${bucket_name}.s3.amazonaws.com`), or the one provided via `endpoint_url`. `bucket_name` can be specified as a string value to use the same bucket for all target repos, or it can be mapping from target repo id to bucket name. @@ -481,7 +486,7 @@ The `upload_policy` defines what policy is used for uploading built artefacts to |`upload_policy` value|Policy| |:--------|:--------------------------------| |`all`|Upload all artefacts (mulitple uploads of the same artefact possible).| -|`latest`|For each build target (prefix in tarball name `eessi-VERSION-{software,init,compat}-OS-ARCH)` only upload the latest built artefact.| +|`latest`|For each build target (prefix in artefact name `eessi-VERSION-{software,init,compat}-OS-ARCH)` only upload the latest built artefact.| |`once`|Only once upload any built artefact for the build target.| |`none`|Do not upload any built artefacts.| @@ -496,30 +501,30 @@ deployment), or a space delimited list of GitHub accounts. no_deploy_permission_comment = Label `bot:deploy` has been set by user `{deploy_labeler}`, but this person does not have permission to trigger deployments ``` This defines a message that is added to the status table in a PR comment -corresponding to a job whose tarball should have been uploaded (e.g., after +corresponding to a job whose artefact should have been uploaded (e.g., after setting the `bot:deploy` label). ``` metadata_prefix = LOCATION_WHERE_METADATA_FILE_GETS_DEPOSITED -tarball_prefix = LOCATION_WHERE_TARBALL_GETS_DEPOSITED +artefact_prefix = LOCATION_WHERE_TARBALL_GETS_DEPOSITED ``` These two settings are used to define where (which directory) in the S3 bucket -(see `bucket_name` above) the metadata file and the tarball will be stored. The +(see `bucket_name` above) the metadata file and the artefact will be stored. The value `LOCATION...` can be a string value to always use the same 'prefix' regardless of the target CVMFS repository, or can be a mapping of a target repository id (see also `repo_target_map` below) to a prefix. The prefix itself can use some (environment) variables that are set within -the upload script (see `tarball_upload_script` above). Currently those are: +the upload script (see `artefact_upload_script` above). Currently those are: * `'${github_repository}'` (which would be expanded to the full name of the GitHub repository, e.g., `EESSI/software-layer`), * `'${legacy_aws_path}'` (which expands to the legacy/old prefix being used for - storing tarballs/metadata files, the old prefix is + storing artefacts/metadata files, the old prefix is `EESSI_VERSION/TARBALL_TYPE/OS_TYPE/CPU_ARCHITECTURE/TIMESTAMP/`), _and_ * `'${pull_request_number}'` (which would be expanded to the number of the pull - request from which the tarball originates). + request from which the artefact originates). Note, it's important to single-quote (`'`) the variables as shown above, because they may likely not be defined when the bot calls the upload script. @@ -529,7 +534,7 @@ The list of supported variables can be shown by running **Examples:** ``` metadata_prefix = {"eessi.io-2023.06": "new/${github_repository}/${pull_request_number}"} -tarball_prefix = { +artefact_prefix = { "eessi-pilot-2023.06": "", "eessi.io-2023.06": "new/${github_repository}/${pull_request_number}" } @@ -656,46 +661,6 @@ running_job = job `{job_id}` is running #### `[finished_job_comments]` section The `[finished_job_comments]` section sets templates for messages about finished jobs. -``` -success = :grin: SUCCESS tarball `{tarball_name}` ({tarball_size} GiB) in job dir -``` -`success` specifies the message for a successful job that produced a tarball. - -``` -failure = :cry: FAILURE -``` -`failure` specifies the message for a failed job. - -``` -no_slurm_out = No slurm output `{slurm_out}` in job dir -``` -`no_slurm_out` specifies the message for missing Slurm output file. - -``` -slurm_out = Found slurm output `{slurm_out}` in job dir -``` -`slurm_out` specifies the message for found Slurm output file. - -``` -missing_modules = Slurm output lacks message "No missing modules!". -``` -`missing_modules` is used to signal the lack of a message that all modules were built. - -``` -no_tarball_message = Slurm output lacks message about created tarball. -``` -`no_tarball_message` is used to signal the lack of a message about a created tarball. - -``` -no_matching_tarball = No tarball matching `{tarball_pattern}` found in job dir. -``` -`no_matching_tarball` is used to signal a missing tarball. - -``` -multiple_tarballs = Found {num_tarballs} tarballs in job dir - only 1 matching `{tarball_pattern}` expected. -``` -`multiple_tarballs` is used to report that multiple tarballs have been found. - ``` job_result_unknown_fmt =
:shrug: UNKNOWN _(click triangle for details)_
``` diff --git a/app.cfg.example b/app.cfg.example index 3f9b3cf5..ae51ade6 100644 --- a/app.cfg.example +++ b/app.cfg.example @@ -127,7 +127,7 @@ no_build_permission_comment = Label `bot:build` has been set by user `{build_lab [deploycfg] # script for uploading built software packages -tarball_upload_script = PATH_TO_EESSI_BOT/scripts/eessi-upload-to-staging +artefact_upload_script = PATH_TO_EESSI_BOT/scripts/eessi-upload-to-staging # URL to S3/minio bucket # if attribute is set, bucket_base will be constructed as follows @@ -160,11 +160,11 @@ upload_policy = once # value can be a space delimited list of GH accounts deploy_permission = -# template for comment when user who set a label has no permission to trigger deploying tarballs +# template for comment when user who set a label has no permission to trigger deploying artefacts no_deploy_permission_comment = Label `bot:deploy` has been set by user `{deploy_labeler}`, but this person does not have permission to trigger deployments # settings for where (directory) in the S3 bucket to store the metadata file and -# the tarball +# the artefact # - Can be a string value to always use the same 'prefix' regardless of the target # CVMFS repository, or can be a mapping of a target repository id (see also # repo_target_map) to a prefix. @@ -173,17 +173,17 @@ no_deploy_permission_comment = Label `bot:deploy` has been set by user `{deploy_ # * 'github_repository' (which would be expanded to the full name of the GitHub # repository, e.g., 'EESSI/software-layer'), # * 'legacy_aws_path' (which expands to the legacy/old prefix being used for -# storing tarballs/metadata files) and +# storing artefacts/metadata files) and # * 'pull_request_number' (which would be expanded to the number of the pull -# request from which the tarball originates). +# request from which the artefact originates). # - The list of supported variables can be shown by running # `scripts/eessi-upload-to-staging --list-variables`. # - Examples: # metadata_prefix = {"eessi.io-2023.06": "new/${github_repository}/${pull_request_number}"} -# tarball_prefix = {"eessi-pilot-2023.06": "", "eessi.io-2023.06": "new/${github_repository}/${pull_request_number}"} +# artefact_prefix = {"eessi-pilot-2023.06": "", "eessi.io-2023.06": "new/${github_repository}/${pull_request_number}"} # If left empty, the old/legacy prefix is being used. metadata_prefix = -tarball_prefix = +artefact_prefix = [architecturetargets] @@ -247,14 +247,6 @@ running_job = job `{job_id}` is running [finished_job_comments] -success = :grin: SUCCESS tarball `{tarball_name}` ({tarball_size} GiB) in job dir -failure = :cry: FAILURE -no_slurm_out = No slurm output `{slurm_out}` in job dir -slurm_out = Found slurm output `{slurm_out}` in job dir -missing_modules = Slurm output lacks message "No missing modules!". -no_tarball_message = Slurm output lacks message about created tarball. -no_matching_tarball = No tarball matching `{tarball_pattern}` found in job dir. -multiple_tarballs = Found {num_tarballs} tarballs in job dir - only 1 matching `{tarball_pattern}` expected. job_result_unknown_fmt =
:shrug: UNKNOWN _(click triangle for detailed information)_
job_test_unknown_fmt =
:shrug: UNKNOWN _(click triangle for detailed information)_
diff --git a/eessi_bot_job_manager.py b/eessi_bot_job_manager.py index c01b4825..81d4f3f7 100644 --- a/eessi_bot_job_manager.py +++ b/eessi_bot_job_manager.py @@ -43,35 +43,23 @@ # Local application imports (anything from EESSI/eessi-bot-software-layer) from connections import github -from tools import config, run_cmd +from tools import config, job_metadata, run_cmd from tools.args import job_manager_parse -from tools.job_metadata import read_job_metadata_from_file, read_metadata_file from tools.pr_comments import get_submitted_job_comment, update_comment AWAITS_LAUNCH = "awaits_launch" -FAILURE = "failure" FINISHED_JOB_COMMENTS = "finished_job_comments" JOB_RESULT_COMMENT_DESCRIPTION = "comment_description" JOB_RESULT_UNKNOWN_FMT = "job_result_unknown_fmt" JOB_TEST_COMMENT_DESCRIPTION = "comment_description" JOB_TEST_UNKNOWN_FMT = "job_test_unknown_fmt" -MISSING_MODULES = "missing_modules" -MULTIPLE_TARBALLS = "multiple_tarballs" NEW_JOB_COMMENTS = "new_job_comments" -NO_MATCHING_TARBALL = "no_matching_tarball" -NO_SLURM_OUT = "no_slurm_out" -NO_TARBALL_MESSAGE = "no_tarball_message" RUNNING_JOB = "running_job" RUNNING_JOB_COMMENTS = "running_job_comments" -SLURM_OUT = "slurm_out" -SUCCESS = "success" REQUIRED_CONFIG = { - FINISHED_JOB_COMMENTS: [FAILURE, JOB_RESULT_UNKNOWN_FMT, MISSING_MODULES, - MULTIPLE_TARBALLS, NO_MATCHING_TARBALL, - NO_SLURM_OUT, NO_TARBALL_MESSAGE, SLURM_OUT, - SUCCESS], + FINISHED_JOB_COMMENTS: [JOB_RESULT_UNKNOWN_FMT, JOB_TEST_UNKNOWN_FMT], NEW_JOB_COMMENTS: [AWAITS_LAUNCH], RUNNING_JOB_COMMENTS: [RUNNING_JOB] } @@ -254,42 +242,6 @@ def determine_finished_jobs(self, known_jobs, current_jobs): return finished_jobs - def read_job_result(self, job_result_file_path): - """ - Read job result file and return the contents of the 'RESULT' section. - - Args: - job_result_file_path (string): path to job result file - - Returns: - (ConfigParser): instance of ConfigParser corresponding to the - 'RESULT' section or None - """ - # reuse function from module tools.job_metadata to read metadata file - result = read_metadata_file(job_result_file_path, self.logfile) - if result and "RESULT" in result: - return result["RESULT"] - else: - return None - - def read_job_test(self, job_test_file_path): - """ - Read job test file and return the contents of the 'TEST' section. - - Args: - job_test_file_path (string): path to job test file - - Returns: - (ConfigParser): instance of ConfigParser corresponding to the - 'TEST' section or None - """ - # reuse function from module tools.job_metadata to read metadata file - test = read_metadata_file(job_test_file_path, self.logfile) - if test and "TEST" in test: - return test["TEST"] - else: - return None - def process_new_job(self, new_job): """ Process a new job by verifying that it is a bot job and if so @@ -335,7 +287,9 @@ def process_new_job(self, new_job): # assuming that a bot job's working directory contains a metadata # file, its existence is used to check if the job belongs to the bot - metadata_pr = read_job_metadata_from_file(job_metadata_path, self.logfile) + metadata_pr = job_metadata.get_section_from_file(job_metadata_path, + job_metadata.JOB_PR_SECTION, + self.logfile) if metadata_pr is None: log(f"No metadata file found at {job_metadata_path} for job {job_id}, so skipping it", @@ -431,7 +385,9 @@ def process_running_jobs(self, running_job): job_metadata_path = os.path.join(job_dir, metadata_file) # check if metadata file exist - metadata_pr = read_job_metadata_from_file(job_metadata_path, self.logfile) + metadata_pr = job_metadata.get_section_from_file(job_metadata_path, + job_metadata.JOB_PR_SECTION, + self.logfile) if metadata_pr is None: raise Exception("Unable to find metadata file") @@ -525,11 +481,13 @@ def process_finished_job(self, finished_job): # check if _bot_jobJOBID.result exits job_result_file = f"_bot_job{job_id}.result" job_result_file_path = os.path.join(new_symlink, job_result_file) - job_results = self.read_job_result(job_result_file_path) + job_results = job_metadata.get_section_from_file(job_result_file_path, + job_metadata.JOB_RESULT_SECTION, + self.logfile) job_result_unknown_fmt = finished_job_comments_cfg[JOB_RESULT_UNKNOWN_FMT] # set fallback comment_description in case no result file was found - # (self.read_job_result returned None) + # (job_metadata.get_section_from_file returned None) comment_description = job_result_unknown_fmt.format(filename=job_result_file) if job_results: # get preformatted comment_description or use previously set default for unknown @@ -552,11 +510,13 @@ def process_finished_job(self, finished_job): # --> bot/test.sh and bot/check-test.sh scripts are run in job script used by bot for 'build' action job_test_file = f"_bot_job{job_id}.test" job_test_file_path = os.path.join(new_symlink, job_test_file) - job_tests = self.read_job_test(job_test_file_path) + job_tests = job_metadata.get_section_from_file(job_test_file_path, + job_metadata.JOB_TEST_SECTION, + self.logfile) job_test_unknown_fmt = finished_job_comments_cfg[JOB_TEST_UNKNOWN_FMT] # set fallback comment_description in case no test file was found - # (self.read_job_result returned None) + # (job_metadata.get_section_from_file returned None) comment_description = job_test_unknown_fmt.format(filename=job_test_file) if job_tests: # get preformatted comment_description or use previously set default for unknown @@ -576,7 +536,9 @@ def process_finished_job(self, finished_job): # obtain id of PR comment to be updated (from file '_bot_jobID.metadata') metadata_file = f"_bot_job{job_id}.metadata" job_metadata_path = os.path.join(new_symlink, metadata_file) - metadata_pr = read_job_metadata_from_file(job_metadata_path, self.logfile) + metadata_pr = job_metadata.get_section_from_file(job_metadata_path, + job_metadata.JOB_PR_SECTION, + self.logfile) if metadata_pr is None: raise Exception("Unable to find metadata file ... skip updating PR comment") diff --git a/scripts/eessi-upload-to-staging b/scripts/eessi-upload-to-staging index 45e52fbf..b5e4482d 100755 --- a/scripts/eessi-upload-to-staging +++ b/scripts/eessi-upload-to-staging @@ -38,7 +38,7 @@ function check_file_name function create_metadata_file { - _tarball=$1 + _artefact=$1 _url=$2 _repository=$3 _pull_request_number=$4 @@ -50,10 +50,10 @@ function create_metadata_file --arg un $(whoami) \ --arg ip $(curl -s https://checkip.amazonaws.com) \ --arg hn "$(hostname -f)" \ - --arg fn "$(basename ${_tarball})" \ - --arg sz "$(du -b "${_tarball}" | awk '{print $1}')" \ - --arg ct "$(date -r "${_tarball}")" \ - --arg sha256 "$(sha256sum "${_tarball}" | awk '{print $1}')" \ + --arg fn "$(basename ${_artefact})" \ + --arg sz "$(du -b "${_artefact}" | awk '{print $1}')" \ + --arg ct "$(date -r "${_artefact}")" \ + --arg sha256 "$(sha256sum "${_artefact}" | awk '{print $1}')" \ --arg url "${_url}" \ --arg repo "${_repository}" \ --arg pr "${_pull_request_number}" \ @@ -70,6 +70,11 @@ function create_metadata_file function display_help { echo "Usage: $0 [OPTIONS] " >&2 + echo " -a | --artefact-prefix PREFIX - a directory to which the artefact" >&2 + echo " shall be uploaded; BASH variable" >&2 + echo " expansion will be applied; arg '-l'" >&2 + echo " lists variables that are defined at" >&2 + echo " the time of expansion" >&2 echo " -e | --endpoint-url URL - endpoint url (needed for non AWS S3)" >&2 echo " -h | --help - display this usage information" >&2 echo " -i | --pr-comment-id - identifier of a PR comment; may be" >&2 @@ -88,11 +93,6 @@ function display_help echo " link the upload to a PR" >&2 echo " -r | --repository FULL_NAME - a repository name ACCOUNT/REPONAME;" >&2 echo " used to link the upload to a PR" >&2 - echo " -t | --tarball-prefix PREFIX - a directory to which the tarball" >&2 - echo " shall be uploaded; BASH variable" >&2 - echo " expansion will be applied; arg '-l'" >&2 - echo " lists variables that are defined at" >&2 - echo " the time of expansion" >&2 } if [[ $# -lt 1 ]]; then @@ -123,7 +123,7 @@ github_repository="EESSI/software-layer" # provided via options in the bot's config file app.cfg and/or command line argument metadata_prefix= -tarball_prefix= +artefact_prefix= # other variables legacy_aws_path= @@ -131,6 +131,10 @@ variables="github_repository legacy_aws_path pull_request_number" while [[ $# -gt 0 ]]; do case $1 in + -a|--artefact-prefix) + artefact_prefix="$2" + shift 2 + ;; -e|--endpoint-url) endpoint_url="$2" shift 2 @@ -167,10 +171,6 @@ while [[ $# -gt 0 ]]; do github_repository="$2" shift 2 ;; - -t|--tarball-prefix) - tarball_prefix="$2" - shift 2 - ;; -*|--*) echo "Error: Unknown option: $1" >&2 exit 1 @@ -204,17 +204,17 @@ for file in "$*"; do basefile=$( basename ${file} ) if check_file_name ${basefile}; then if tar tf "${file}" | head -n1 > /dev/null; then - # 'legacy_aws_path' might be used in tarball_prefix or metadata_prefix + # 'legacy_aws_path' might be used in artefact_prefix or metadata_prefix # its purpose is to support the old/legacy method to derive the location - # where to store the tarball and metadata file + # where to store the artefact and metadata file export legacy_aws_path=$(basename ${file} | tr -s '-' '/' \ | perl -pe 's/^eessi.//;' | perl -pe 's/\.tar\.gz$//;' ) - if [ -z ${tarball_prefix} ]; then + if [ -z ${artefact_prefix} ]; then aws_path=${legacy_aws_path} else export pull_request_number export github_repository - aws_path=$(envsubst <<< "${tarball_prefix}") + aws_path=$(envsubst <<< "${artefact_prefix}") fi aws_file=$(basename ${file}) echo "Creating metadata file" @@ -233,7 +233,7 @@ for file in "$*"; do cat ${metadata_file} echo Uploading to "${url}" - echo " store tarball at ${aws_path}/${aws_file}" + echo " store artefact at ${aws_path}/${aws_file}" upload_to_staging_bucket \ "${file}" \ "${bucket_name}" \ diff --git a/tasks/deploy.py b/tasks/deploy.py index 52575fb2..afd61662 100644 --- a/tasks/deploy.py +++ b/tasks/deploy.py @@ -28,10 +28,11 @@ from connections import github from tasks.build import CFG_DIRNAME, JOB_CFG_FILENAME, JOB_REPO_ID, JOB_REPOSITORY from tasks.build import get_build_env_cfg -from tools import config, pr_comments, run_cmd -from tools.job_metadata import read_job_metadata_from_file +from tools import config, job_metadata, pr_comments, run_cmd +ARTEFACT_PREFIX = "artefact_prefix" +ARTEFACT_UPLOAD_SCRIPT = "artefact_upload_script" BUCKET_NAME = "bucket_name" DEPLOYCFG = "deploycfg" DEPLOY_PERMISSION = "deploy_permission" @@ -39,8 +40,6 @@ JOBS_BASE_DIR = "jobs_base_dir" METADATA_PREFIX = "metadata_prefix" NO_DEPLOY_PERMISSION_COMMENT = "no_deploy_permission_comment" -TARBALL_PREFIX = "tarball_prefix" -TARBALL_UPLOAD_SCRIPT = "tarball_upload_script" UPLOAD_POLICY = "upload_policy" @@ -92,10 +91,10 @@ def determine_pr_comment_id(job_dir): """ # assumes that last part of job_dir encodes the job's id job_id = os.path.basename(os.path.normpath(job_dir)) - job_metadata_file = os.path.join(job_dir, f"_bot_job{job_id}.metadata") - job_metadata = read_job_metadata_from_file(job_metadata_file) - if job_metadata and "pr_comment_id" in job_metadata: - return int(job_metadata["pr_comment_id"]) + metadata_file = os.path.join(job_dir, f"_bot_job{job_id}.metadata") + metadata = job_metadata.get_section_from_file(metadata_file, job_metadata.JOB_PR_SECTION) + if metadata and "pr_comment_id" in metadata: + return int(metadata["pr_comment_id"]) else: return -1 @@ -121,84 +120,93 @@ def determine_slurm_out(job_dir): return slurm_out -def determine_eessi_tarballs(job_dir): +def determine_artefacts(job_dir): """ - Determine paths to EESSI software tarballs in a given job directory. + Determine paths to artefacts created by a job in a given job directory. Args: job_dir (string): working directory of the job Returns: - eessi_tarballs (list): list of paths to all tarballs in job_dir + (list): list of paths to all artefacts in job_dir """ - # determine all tarballs that are stored in the directory job_dir - # and whose name matches a certain pattern - tarball_pattern = "eessi-*software-*.tar.gz" - glob_str = os.path.join(job_dir, tarball_pattern) - eessi_tarballs = glob.glob(glob_str) + # determine all artefacts that are stored in the directory job_dir + # by using the _bot_jobSLURM_JOBID.result file in that job directory + job_id = job_metadata.determine_job_id_from_job_directory(job_dir) + if job_id == 0: + # could not determine job id, returning empty list of artefacts + return None + + job_result_file = f"_bot_job{job_id}.result" + job_result_file_path = os.path.join(job_dir, job_result_file) + job_result = job_metadata.get_section_from_file(job_result_file_path, job_metadata.JOB_RESULT_SECTION) - return eessi_tarballs + if job_result and job_metadata.JOB_RESULT_ARTEFACTS in job_result: + # transform multiline value into a list + artefacts_list = job_result[job_metadata.JOB_RESULT_ARTEFACTS].split('\n') + # drop elements of length zero + artefacts = [af for af in artefacts_list if len(af) > 0] + return artefacts + else: + return None -def check_build_status(slurm_out, eessi_tarballs): +def check_job_status(job_dir): """ Check status of the job in a given directory. Args: - slurm_out (string): path to job output file - eessi_tarballs (list): list of eessi tarballs found for job + job_dir (string): path to job directory Returns: (bool): True -> job succeeded, False -> job failed """ fn = sys._getframe().f_code.co_name - # TODO use _bot_job.result file to determine result status - # cases: - # (1) no result file --> add line with unknown status, found tarball xyz but no result file - # (2) result file && status = SUCCESS --> return True - # (3) result file && status = FAILURE --> return False - - # Function checks if all modules have been built and if a tarball has - # been created. - - # set some initial values - no_missing_modules = False - targz_created = False - - # check slurm out for the below strings - # ^No missing modules!$ --> all software successfully installed - # ^/eessi_bot_job/eessi-.*-software-.*.tar.gz created!$ --> - # tarball successfully created - if os.path.exists(slurm_out): - re_missing_modules = re.compile(".*No missing installations, party time!.*") - re_targz_created = re.compile("^/eessi_bot_job/eessi-.*-software-.*.tar.gz created!$") - outfile = open(slurm_out, "r") - for line in outfile: - if re_missing_modules.match(line): - # no missing modules - no_missing_modules = True - log(f"{fn}(): line '{line}' matches '.*No missing installations, party time!.*'") - if re_targz_created.match(line): - # tarball created - targz_created = True - log(f"{fn}(): line '{line}' matches '^/eessi_bot_job/eessi-.*-software-.*.tar.gz created!$'") - - log(f"{fn}(): found {len(eessi_tarballs)} tarballs for '{slurm_out}'") - - # we test results from the above check and if there is one tarball only - if no_missing_modules and targz_created and len(eessi_tarballs) == 1: - return True + # use _bot_job.result file to determine result status + # cases: + # (0) no job id --> return False + # (1) no result file --> return False + # (2) result file && status = SUCCESS --> return True + # (3) result file && status = FAILURE --> return False + + # case (0): no job id --> return False + job_id = job_metadata.determine_job_id_from_job_directory(job_dir) + if job_id == 0: + # could not determine job id, return False + log(f"{fn}(): could not determine job id from directory '{job_dir}'\n") + return False + + job_result_file = f"_bot_job{job_id}.result" + job_result_file_path = os.path.join(job_dir, job_result_file) + job_result = job_metadata.get_section_from_file(job_result_file_path, job_metadata.JOB_RESULT_SECTION) + + job_status = job_metadata.JOB_RESULT_FAILURE + if job_result and job_metadata.JOB_RESULT_STATUS in job_result: + job_status = job_result[job_metadata.JOB_RESULT_STATUS] + else: + # case (1): no result file or no status --> return False + log(f"{fn}(): no result file '{job_result_file_path}' or reading it failed\n") + return False - return False + log(f"{fn}(): job status is {job_status} (compare against {job_metadata.JOB_RESULT_SUCCESS})\n") + + if job_status == job_metadata.JOB_RESULT_SUCCESS: + # case (2): result file && status = SUCCESS --> return True + log(f"{fn}(): found status 'SUCCESS' from '{job_result_file_path}'\n") + return True + else: + # case (3): result file && status = FAILURE --> return False + log(f"{fn}(): found status 'FAILURE' from '{job_result_file_path}'\n") + return False -def update_pr_comment(tarball, repo_name, pr_number, pr_comment_id, state, msg): +def update_pr_comment(artefact, repo_name, pr_number, pr_comment_id, state, msg): """ - Update pull request comment for the given comment id or tarball name + Update pull request comment for the given comment id or artefact name Args: - tarball (string): name of tarball that is looked for in a PR comment + artefact (string): name of artefact that is looked for in a PR comment repo_name (string): name of the repository (USER_ORG/REPOSITORY) pr_number (int): pull request number state (string): value for state column to be used in update @@ -211,23 +219,23 @@ def update_pr_comment(tarball, repo_name, pr_number, pr_comment_id, state, msg): repo = gh.get_repo(repo_name) pull_request = repo.get_pull(pr_number) - issue_comment = pr_comments.determine_issue_comment(pull_request, pr_comment_id, tarball) + issue_comment = pr_comments.determine_issue_comment(pull_request, pr_comment_id, artefact) if issue_comment: dt = datetime.now(timezone.utc) comment_update = (f"\n|{dt.strftime('%b %d %X %Z %Y')}|{state}|" - f"transfer of `{tarball}` to S3 bucket {msg}|") + f"transfer of `{artefact}` to S3 bucket {msg}|") # append update to existing comment issue_comment.edit(issue_comment.body + comment_update) -def append_tarball_to_upload_log(tarball, job_dir): +def append_artefact_to_upload_log(artefact, job_dir): """ - Append tarball to upload log. + Append artefact to upload log. Args: - tarball (string): name of tarball that has been uploaded - job_dir (string): directory of the job that built the tarball + artefact (string): name of artefact that has been uploaded + job_dir (string): directory of the job that built the artefact Returns: None (implicitly) @@ -236,18 +244,19 @@ def append_tarball_to_upload_log(tarball, job_dir): pr_base_dir = os.path.dirname(job_dir) uploaded_txt = os.path.join(pr_base_dir, 'uploaded.txt') with open(uploaded_txt, "a") as upload_log: - job_plus_tarball = os.path.join(os.path.basename(job_dir), tarball) - upload_log.write(f"{job_plus_tarball}\n") + job_plus_artefact = os.path.join(os.path.basename(job_dir), artefact) + upload_log.write(f"{job_plus_artefact}\n") -def upload_tarball(job_dir, build_target, timestamp, repo_name, pr_number, pr_comment_id): +def upload_artefact(job_dir, payload, timestamp, repo_name, pr_number, pr_comment_id): """ - Upload built tarball to an S3 bucket. + Upload artefact to an S3 bucket. Args: job_dir (string): path to the job directory - build_target (string): eessi-VERSION-COMPONENT-OS-ARCH - timestamp (int): timestamp of the tarball + payload (string): can be any name describing the payload, e.g., for + EESSI it could have the format eessi-VERSION-COMPONENT-OS-ARCH + timestamp (int): timestamp of the artefact repo_name (string): repository of the pull request pr_number (int): number of the pull request pr_comment_id (int): id of the pull request comment @@ -257,18 +266,18 @@ def upload_tarball(job_dir, build_target, timestamp, repo_name, pr_number, pr_co """ funcname = sys._getframe().f_code.co_name - tarball = f"{build_target}-{timestamp}.tar.gz" - abs_path = os.path.join(job_dir, tarball) - log(f"{funcname}(): deploying build '{abs_path}'") + artefact = f"{payload}-{timestamp}.tar.gz" + abs_path = os.path.join(job_dir, artefact) + log(f"{funcname}(): uploading '{abs_path}'") # obtain config settings cfg = config.read_config() deploycfg = cfg[DEPLOYCFG] - tarball_upload_script = deploycfg.get(TARBALL_UPLOAD_SCRIPT) + artefact_upload_script = deploycfg.get(ARTEFACT_UPLOAD_SCRIPT) endpoint_url = deploycfg.get(ENDPOINT_URL) or '' bucket_spec = deploycfg.get(BUCKET_NAME) metadata_prefix = deploycfg.get(METADATA_PREFIX) - tarball_prefix = deploycfg.get(TARBALL_PREFIX) + artefact_prefix = deploycfg.get(ARTEFACT_PREFIX) # if bucket_spec value looks like a dict, try parsing it as such if bucket_spec.lstrip().startswith('{'): @@ -278,9 +287,9 @@ def upload_tarball(job_dir, build_target, timestamp, repo_name, pr_number, pr_co if metadata_prefix.lstrip().startswith('{'): metadata_prefix = json.loads(metadata_prefix) - # if tarball_prefix value looks like a dict, try parsing it as such - if tarball_prefix.lstrip().startswith('{'): - tarball_prefix = json.loads(tarball_prefix) + # if artefact_prefix value looks like a dict, try parsing it as such + if artefact_prefix.lstrip().startswith('{'): + artefact_prefix = json.loads(artefact_prefix) jobcfg_path = os.path.join(job_dir, CFG_DIRNAME, JOB_CFG_FILENAME) jobcfg = config.read_config(jobcfg_path) @@ -293,13 +302,13 @@ def upload_tarball(job_dir, build_target, timestamp, repo_name, pr_number, pr_co # bucket spec may be a mapping of target repo id to bucket name bucket_name = bucket_spec.get(target_repo_id) if bucket_name is None: - update_pr_comment(tarball, repo_name, pr_number, pr_comment_id, "not uploaded", + update_pr_comment(artefact, repo_name, pr_number, pr_comment_id, "not uploaded", f"failed (no bucket specified for {target_repo_id})") return else: log(f"Using bucket for {target_repo_id}: {bucket_name}") else: - update_pr_comment(tarball, repo_name, pr_number, pr_comment_id, "not uploaded", + update_pr_comment(artefact, repo_name, pr_number, pr_comment_id, "not uploaded", f"failed (incorrect bucket spec: {bucket_spec})") return @@ -310,31 +319,31 @@ def upload_tarball(job_dir, build_target, timestamp, repo_name, pr_number, pr_co # metadata prefix spec may be a mapping of target repo id to metadata prefix metadata_prefix_arg = metadata_prefix.get(target_repo_id) if metadata_prefix_arg is None: - update_pr_comment(tarball, repo_name, pr_number, pr_comment_id, "not uploaded", + update_pr_comment(artefact, repo_name, pr_number, pr_comment_id, "not uploaded", f"failed (no metadata prefix specified for {target_repo_id})") return else: log(f"Using metadata prefix for {target_repo_id}: {metadata_prefix_arg}") else: - update_pr_comment(tarball, repo_name, pr_number, pr_comment_id, "not uploaded", + update_pr_comment(artefact, repo_name, pr_number, pr_comment_id, "not uploaded", f"failed (incorrect metadata prefix spec: {metadata_prefix_arg})") return - if isinstance(tarball_prefix, str): - tarball_prefix_arg = tarball_prefix - log(f"Using specified tarball prefix: {tarball_prefix_arg}") - elif isinstance(tarball_prefix, dict): - # tarball prefix spec may be a mapping of target repo id to tarball prefix - tarball_prefix_arg = tarball_prefix.get(target_repo_id) - if tarball_prefix_arg is None: - update_pr_comment(tarball, repo_name, pr_number, pr_comment_id, "not uploaded", - f"failed (no tarball prefix specified for {target_repo_id})") + if isinstance(artefact_prefix, str): + artefact_prefix_arg = artefact_prefix + log(f"Using specified artefact prefix: {artefact_prefix_arg}") + elif isinstance(artefact_prefix, dict): + # artefact prefix spec may be a mapping of target repo id to artefact prefix + artefact_prefix_arg = artefact_prefix.get(target_repo_id) + if artefact_prefix_arg is None: + update_pr_comment(artefact, repo_name, pr_number, pr_comment_id, "not uploaded", + f"failed (no artefact prefix specified for {target_repo_id})") return else: - log(f"Using tarball prefix for {target_repo_id}: {tarball_prefix_arg}") + log(f"Using artefact prefix for {target_repo_id}: {artefact_prefix_arg}") else: - update_pr_comment(tarball, repo_name, pr_number, pr_comment_id, "not uploaded", - f"failed (incorrect tarball prefix spec: {tarball_prefix_arg})") + update_pr_comment(artefact, repo_name, pr_number, pr_comment_id, "not uploaded", + f"failed (incorrect artefact prefix spec: {artefact_prefix_arg})") return # run 'eessi-upload-to-staging {abs_path}' @@ -343,52 +352,53 @@ def upload_tarball(job_dir, build_target, timestamp, repo_name, pr_number, pr_co # bucket_name = 'eessi-staging' # if endpoint_url not set use EESSI S3 bucket # (2) run command - cmd_args = [tarball_upload_script, ] + cmd_args = [artefact_upload_script, ] + if len(artefact_prefix_arg) > 0: + cmd_args.extend(['--artefact-prefix', artefact_prefix_arg]) if len(bucket_name) > 0: cmd_args.extend(['--bucket-name', bucket_name]) if len(endpoint_url) > 0: cmd_args.extend(['--endpoint-url', endpoint_url]) if len(metadata_prefix_arg) > 0: cmd_args.extend(['--metadata-prefix', metadata_prefix_arg]) - cmd_args.extend(['--repository', repo_name]) - cmd_args.extend(['--pull-request-number', str(pr_number)]) cmd_args.extend(['--pr-comment-id', str(pr_comment_id)]) - if len(tarball_prefix_arg) > 0: - cmd_args.extend(['--tarball-prefix', tarball_prefix_arg]) + cmd_args.extend(['--pull-request-number', str(pr_number)]) + cmd_args.extend(['--repository', repo_name]) cmd_args.append(abs_path) upload_cmd = ' '.join(cmd_args) # run_cmd does all the logging we might need - out, err, ec = run_cmd(upload_cmd, 'Upload tarball to S3 bucket', raise_on_error=False) + out, err, ec = run_cmd(upload_cmd, 'Upload artefact to S3 bucket', raise_on_error=False) if ec == 0: # add file to 'job_dir/../uploaded.txt' - append_tarball_to_upload_log(tarball, job_dir) + append_artefact_to_upload_log(artefact, job_dir) # update pull request comment - update_pr_comment(tarball, repo_name, pr_number, pr_comment_id, "uploaded", + update_pr_comment(artefact, repo_name, pr_number, pr_comment_id, "uploaded", "succeeded") else: # update pull request comment - update_pr_comment(tarball, repo_name, pr_number, pr_comment_id, "not uploaded", + update_pr_comment(artefact, repo_name, pr_number, pr_comment_id, "not uploaded", "failed") -def uploaded_before(build_target, job_dir): +def uploaded_before(payload, job_dir): """ - Determines if a tarball for a job has been uploaded before. Function + Determines if an artefact for a job has been uploaded before. Function scans the log file named 'job_dir/../uploaded.txt' for the string - '.*build_target-.*.tar.gz'. + '.*{payload}-.*.tar.gz'. Args: - build_target (string): eessi-VERSION-COMPONENT-OS-ARCH + payload (string): can be any name describing the payload, e.g., for + EESSI it could have the format eessi-VERSION-COMPONENT-OS-ARCH job_dir (string): working directory of the job Returns: - (string): name of the first tarball found if any or None. + (string): name of the first artefact found if any or None. """ funcname = sys._getframe().f_code.co_name - log(f"{funcname}(): any previous uploads for {build_target}?") + log(f"{funcname}(): any previous uploads for {payload}?") pr_base_dir = os.path.dirname(job_dir) uploaded_txt = os.path.join(pr_base_dir, "uploaded.txt") @@ -396,13 +406,13 @@ def uploaded_before(build_target, job_dir): if os.path.exists(uploaded_txt): log(f"{funcname}(): upload log '{uploaded_txt}' exists") - re_string = f".*{build_target}-.*.tar.gz.*" - re_build_target = re.compile(re_string) + re_string = f".*{payload}-.*.tar.gz.*" + re_payload = re.compile(re_string) with open(uploaded_txt, "r") as uploaded_log: log(f"{funcname}(): scan log for pattern '{re_string}'") for line in uploaded_log: - if re_build_target.match(line): + if re_payload.match(line): log(f"{funcname}(): found earlier upload {line.strip()}") return line.strip() else: @@ -427,37 +437,34 @@ def determine_successful_jobs(job_dirs): successes = [] for job_dir in job_dirs: - slurm_out = determine_slurm_out(job_dir) - eessi_tarballs = determine_eessi_tarballs(job_dir) + artefacts = determine_artefacts(job_dir) pr_comment_id = determine_pr_comment_id(job_dir) - if check_build_status(slurm_out, eessi_tarballs): - log(f"{funcname}(): SUCCESSFUL build in '{job_dir}'") + if check_job_status(job_dir): + log(f"{funcname}(): SUCCESSFUL job in '{job_dir}'") successes.append({'job_dir': job_dir, - 'slurm_out': slurm_out, 'pr_comment_id': pr_comment_id, - 'eessi_tarballs': eessi_tarballs}) + 'artefacts': artefacts}) else: - log(f"{funcname}(): FAILED build in '{job_dir}'") + log(f"{funcname}(): FAILED job in '{job_dir}'") return successes -def determine_tarballs_to_deploy(successes, upload_policy): +def determine_artefacts_to_deploy(successes, upload_policy): """ - Determine tarballs to deploy depending on upload policy + Determine artefacts to deploy depending on upload policy Args: successes (list): list of dictionaries - {'job_dir':job_dir, 'slurm_out':slurm_out, 'eessi_tarballs':eessi_tarballs} + {'job_dir':job_dir, 'pr_comment_id':pr_comment_id, 'artefacts':artefacts} upload_policy (string): one of 'all', 'latest' or 'once' 'all': deploy all - 'latest': deploy only the last for each build target - 'once': deploy only latest if none for this build target has + 'latest': deploy only the last for each payload + 'once': deploy only latest if none for this payload has been deployed before Returns: - (dictionary): dictionary of dictionaries representing built tarballs to - be deployed + (dictionary): dictionary of dictionaries representing artefacts to be deployed """ funcname = sys._getframe().f_code.co_name @@ -465,52 +472,51 @@ def determine_tarballs_to_deploy(successes, upload_policy): to_be_deployed = {} for job in successes: - # all tarballs for successful job - tarballs = job["eessi_tarballs"] - log(f"{funcname}(): num tarballs {len(tarballs)}") + # all artefacts for successful job + artefacts = job["artefacts"] + log(f"{funcname}(): num artefacts {len(artefacts)}") - # full path to first tarball for successful job - # Note, only one tarball per job is expected. - tb0 = tarballs[0] - log(f"{funcname}(): path to 1st tarball: '{tb0}'") + # full path to first artefact for successful job + # Note, only one artefact per job is expected. + artefact = artefacts[0] + log(f"{funcname}(): path to 1st artefact: '{artefact}'") - # name of tarball file only - tb0_base = os.path.basename(tb0) - log(f"{funcname}(): tarball filename: '{tb0_base}'") + # name of artefact file only + artefact_base = os.path.basename(artefact) + log(f"{funcname}(): artefact filename: '{artefact_base}'") - # tarball name format: eessi-VERSION-COMPONENT-OS-ARCH-TIMESTAMP.tar.gz - # remove "-TIMESTAMP.tar.gz" - # build_target format: eessi-VERSION-COMPONENT-OS-ARCH - build_target = "-".join(tb0_base.split("-")[:-1]) - log(f"{funcname}(): tarball build target '{build_target}'") + # artefact name format: PAYLOAD-TIMESTAMP.tar.gz + # remove "-TIMESTAMP.tar.gz" (last element when splitting along '-') + payload = "-".join(artefact_base.split("-")[:-1]) + log(f"{funcname}(): artefact payload '{payload}'") # timestamp in the filename - timestamp = int(tb0_base.split("-")[-1][:-7]) - log(f"{funcname}(): tarball timestamp {timestamp}") + timestamp = int(artefact_base.split("-")[-1][:-7]) + log(f"{funcname}(): artefact timestamp {timestamp}") deploy = False if upload_policy == "all": deploy = True elif upload_policy == "latest": - if build_target in to_be_deployed: - if to_be_deployed[build_target]["timestamp"] < timestamp: + if payload in to_be_deployed: + if to_be_deployed[payload]["timestamp"] < timestamp: # current one will be replaced deploy = True else: deploy = True elif upload_policy == "once": - uploaded = uploaded_before(build_target, job["job_dir"]) + uploaded = uploaded_before(payload, job["job_dir"]) if uploaded is None: deploy = True else: indent_fname = f"{' '*len(funcname + '(): ')}" - log(f"{funcname}(): tarball for build target '{build_target}'\n" + log(f"{funcname}(): artefact for payload '{payload}'\n" f"{indent_fname}has been uploaded through '{uploaded}'") if deploy: - to_be_deployed[build_target] = {"job_dir": job["job_dir"], - "pr_comment_id": job["pr_comment_id"], - "timestamp": timestamp} + to_be_deployed[payload] = {"job_dir": job["job_dir"], + "pr_comment_id": job["pr_comment_id"], + "timestamp": timestamp} return to_be_deployed @@ -571,14 +577,13 @@ def deploy_built_artefacts(pr, event_info): # 3) for the successful ones, determine which to deploy depending on # the upload policy - to_be_deployed = determine_tarballs_to_deploy(successes, upload_policy) + to_be_deployed = determine_artefacts_to_deploy(successes, upload_policy) # 4) call function to deploy a single artefact per software subdir - # - update PR comments (look for comments with build-ts.tar.gz) repo_name = pr.base.repo.full_name - for target, job in to_be_deployed.items(): + for payload, job in to_be_deployed.items(): job_dir = job['job_dir'] timestamp = job['timestamp'] pr_comment_id = job['pr_comment_id'] - upload_tarball(job_dir, target, timestamp, repo_name, pr.number, pr_comment_id) + upload_artefact(job_dir, payload, timestamp, repo_name, pr.number, pr_comment_id) diff --git a/tests/test_app.cfg b/tests/test_app.cfg index f9634422..f940c1df 100644 --- a/tests/test_app.cfg +++ b/tests/test_app.cfg @@ -25,11 +25,3 @@ awaits_lauch = job awaits launch by Slurm scheduler running_job = job `{job_id}` is running [finished_job_comments] -success = :grin: SUCCESS tarball `{tarball_name}` ({tarball_size} GiB) in job dir -failure = :cry: FAILURE -no_slurm_out = No slurm output `{slurm_out}` in job dir -slurm_out = Found slurm output `{slurm_out}` in job dir -missing_modules = Slurm output lacks message "No missing modules!". -no_tarball_message = Slurm output lacks message about created tarball. -no_matching_tarball = No tarball matching `{tarball_pattern}` found in job dir. -multiple_tarballs = Found {num_tarballs} tarballs in job dir - only 1 matching `{tarball_pattern}` expected. diff --git a/tests/test_tools_job_metadata.py b/tests/test_tools_job_metadata.py index f5542c6f..0d788248 100644 --- a/tests/test_tools_job_metadata.py +++ b/tests/test_tools_job_metadata.py @@ -11,21 +11,21 @@ import os -from tools.job_metadata import read_job_metadata_from_file +from tools.job_metadata import get_section_from_file, JOB_PR_SECTION -def test_read_job_metadata_from_file(tmpdir): - logfile = os.path.join(tmpdir, 'test_read_job_metadata_from_file.log') +def test_get_section_from_file(tmpdir): + logfile = os.path.join(tmpdir, 'test_get_section_from_file.log') # if metadata file does not exist, we should get None as return value path = os.path.join(tmpdir, 'test.metadata') - assert read_job_metadata_from_file(path, logfile) is None + assert get_section_from_file(path, JOB_PR_SECTION, logfile) is None with open(path, 'w') as fp: fp.write('''[PR] repo=test pr_number=12345''') - metadata_pr = read_job_metadata_from_file(path, logfile) + metadata_pr = get_section_from_file(path, JOB_PR_SECTION, logfile) expected = { "repo": "test", "pr_number": "12345", diff --git a/tools/job_metadata.py b/tools/job_metadata.py index b8cd4f0d..e93218b0 100644 --- a/tools/job_metadata.py +++ b/tools/job_metadata.py @@ -21,6 +21,15 @@ # (none yet) +JOB_PR_SECTION = "PR" +JOB_RESULT_ARTEFACTS = "artefacts" +JOB_RESULT_FAILURE = "FAILURE" +JOB_RESULT_SECTION = "RESULT" +JOB_RESULT_STATUS = "status" +JOB_RESULT_SUCCESS = "SUCCESS" +JOB_TEST_SECTION = "TEST" + + def create_metadata_file(job, job_id, pr_comment): """ Create job metadata file in job working directory @@ -50,6 +59,54 @@ def create_metadata_file(job, job_id, pr_comment): log(f"{fn}(): created job metadata file {bot_jobfile_path}") +def determine_job_id_from_job_directory(job_directory, log_file=None): + """ + Determine job id from a job directory. + + Args: + job_directory (string): path to job directory + log_file (string): path to log file + + Returns: + (int): job id or 0 + """ + # job id could be found in + # - current directory name + # - part of a 'slurm-JOB_ID.out' file name + # - part of a '_bot_jobJOB_ID.metadata' file + # For now we just use the first alternative. + job_dir_basename = os.path.basename(job_directory) + from_dir_job_id = 0 + if job_dir_basename.replace('.', '', 1).isdigit(): + from_dir_job_id = int(job_dir_basename) + return from_dir_job_id + + +def get_section_from_file(filepath, section, log_file=None): + """ + Read filepath (ini/cfg format) and return contents of a section. + + Args: + filepath (string): path to a metadata file + section (string): name of the section to obtain contents for + log_file (string): path to log file + + Returns: + (ConfigParser): instance of ConfigParser corresponding to the section or None + """ + # reuse function from module tools.job_metadata to read metadata file + section_contents = None + metadata = read_metadata_file(filepath, log_file=log_file) + if metadata: + # get section + if section in metadata: + section_contents = metadata[section] + else: + section_contents = {} + + return section_contents + + def read_metadata_file(metadata_path, log_file=None): """ Read metadata file into ConfigParser instance @@ -80,28 +137,3 @@ def read_metadata_file(metadata_path, log_file=None): else: log(f"No metadata file found at {metadata_path}.", log_file) return None - - -def read_job_metadata_from_file(filepath, log_file=None): - """ - Read job metadata from file - - Args: - filepath (string): path to job metadata file - log_file (string): path to log file - - Returns: - job_metadata (dict): dictionary containing job metadata or None - """ - - metadata = read_metadata_file(filepath, log_file=log_file) - if metadata: - # get PR section - if "PR" in metadata: - metadata_pr = metadata["PR"] - else: - metadata_pr = {} - return metadata_pr - else: - log(f"Metadata file '{filepath}' does not exist or could not be read") - return None