Code

Containertask#62

Code

Open

jgolob wants to merge 101 commits intopharmbio:masterpharmbio/sciluigi:masterfrom jgolob:containertaskjgolob/sciluigi:containertaskCopy head branch name to clipboard

+1,983-75

Commits

Commits on Apr 9, 2018

• Added containertasks

  

  Jonathan Golob

  committed
  ff68e8f

• Added engine to containerinfo

  

  Jonathan Golob

  committed
  9dabb6d

• added singularity_slurm engine basics

  

  Jonathan Golob

  committed
  3c3e878

Commits on Apr 10, 2018

• Added bucket_command_wrapper.py to tools

  

  Jonathan Golob

  committed
  bcbb8f5

Commits on Apr 11, 2018

• Pre pull commit

  

  Jonathan Golob

  committed
  8e763da

• Working bucket_command_wrapper.py

  

  Jonathan Golob

  committed
  cccc0ac

Commits on Apr 12, 2018

• Changed BCW to allow repeated file commands and accept command as env variable

  

  Jonathan Golob

  committed
  f771ba0

• Added ContainerTarget

  

  Jonathan Golob

  committed
  52f0426

• Made filesytem path just path component of url

  

  Jonathan Golob

  committed
  0347653

• Added ContainerTaskInfo to base import

  

  Jonathan Golob

  committed
  9373740

• Made containertaskinfo inherit from taskinfo

  

  Jonathan Golob

  committed
  09e9032

Commits on Apr 13, 2018

• Partially completed AWS batch engine

  

  Jonathan Golob

  committed
  c2e630a

• working on aws batch

  

  Jonathan Golob

  committed
  652df08

Commits on Apr 16, 2018

• Working towards complete AWS-batch engine

  

  Jonathan Golob

  committed
  a594b5f

Commits on Apr 20, 2018

• Pre work

  

  Jonathan Golob

  committed
  c141ef8

Commits on Apr 25, 2018

• Working AWS batch engine with ability to upload / download from S3 as needed

  

  Jonathan Golob

  committed
  7f1ae5d

• Tidied up a bit of the logging for AWS-batch engine

  

  Jonathan Golob

  committed
  7802356

• Allowed for piped and && / || commands in both docker and aws_batch

  

  Jonathan Golob

  committed
  8ef016d

Commits on Apr 26, 2018

• Moved custom mounts to containerinfo

  

  Jonathan Golob

  committed
  21055c8

• Moved mounts to a containerinfo parameter, and implemented for AWS batch

  

  Jonathan Golob

  committed
  3041e88

• Switch from set to list

  

  Jonathan Golob

  committed
  814ac0a

• From string to bool for readonly

  

  Jonathan Golob

  committed
  f532674

• name for volume through uuid

  

  Jonathan Golob

  committed
  c3c54dd

Commits on Apr 27, 2018

• check not just registered job def, but active

  

  Jonathan Golob

  committed
  7e572c6

• String for name not class uuid

  

  Jonathan Golob

  committed
  23dcc64

• Fixed a few bugs, and moved over to the packaged bucket_container_wrapper

  

  Jonathan Golob

  committed
  554e4c8

• Fixed bug where no files are being uploaded to S3 temp

  

  Jonathan Golob

  committed
  1c62815

Commits on Apr 28, 2018

• Docker engine working, with switch to targets rather than paths to be faancy

  

  Jonathan Golob

  committed
  ab6a6cb

Commits on May 1, 2018

• basic engine for docker and aws_batch done

  

  Jonathan Golob

  committed
  ec3dcd6

• Effort to get basics of slurm_singularity engine working

  Show description for a860c34
  

  Jonathan Golob

  committed
  a860c34

• Fixed bug in containerinfo init

  

  Jonathan Golob

  committed
  06ba025

Commits on May 2, 2018

• Working on singularity_slurm code

  

  Jonathan Golob

  committed
  b212c62

• Generally working slurm-sciluigi version

  

  Jonathan Golob

  committed
  0160a00

• Pre pull commit

  

  Jonathan Golob

  committed
  771c5e4

• Merge branch 'containertask' of https://github.com/jgolob/sciluigi into containertask

  

  Jonathan Golob

  committed
  1e255f4

• Working singularity_slurm code

  

  Jonathan Golob

  committed
  d22e6d6

• Continued refinement of slurm_singularity engine

  

  Jonathan Golob

  committed
  56ab1bd

• Merge branch 'containertask' of https://github.com/jgolob/sciluigi into containertask

  

  Jonathan Golob

  committed
  feb7c22

Commits on May 3, 2018

• Catch ClientError for AWS Batch API calls

  

  Sam Minot

  committed
  eef05af

• Add MAX_BOTO_TRIES

  Show description for f617a29
  

  Sam Minot

  committed
  f617a29

• Merge branch 'catch_batch_client_exceptions' of https://github.com/jgolob/sciluigi into containertask

  

  Jonathan Golob

  committed
  8082c49

• Pre pull commit

  

  Jonathan Golob

  committed
  a7e1669

• Pre-pull commit

  

  Jonathan Golob

  committed
  b319c4d

• Switch boto_max_tries to a containerinfo variable

  

  Jonathan Golob

  committed
  8bcb241

• Merge branch 'catch_batch_client_exceptions' into containertask

  

  Jonathan Golob

  committed
  308c93b

• Fixed a minor bug when checking batch job status fails

  

  Jonathan Golob

  committed
  0fbf5da

Commits on May 10, 2018

• Add custom aws batch job name param

  

  Sam Minot

  committed
  329452f

• Change to "_prefix" to match functionality

  

  Sam Minot

  committed
  3e4f1fe

• Merge pull request #3 from jgolob/custom_aws_batch_job_name

  Show description for 9ae1efd
  jgolob

  authored
  9ae1efd

Commits on May 11, 2018

• Mild changes

  

  Jonathan Golob

  committed
  3820e9d

Commits on May 14, 2018

• Fixed bug involving need to make directories when batch -> local FS

  

  Jonathan Golob

  committed
  44dde95

Commits on May 17, 2018

• More logging for boto3 ClientError

  

  Sam Minot

  committed
  38942c9

• Merge pull request #4 from jgolob/containerTask_error_logging

  Show description for b177a1e
  jgolob

  authored
  b177a1e

Commits on May 25, 2018

• Parameterized aws batch job poll time.

  

  Jonathan Golob

  committed
  7bedecc

• Merge branch 'containertask' of https://github.com/jgolob/sciluigi into containertask

  

  Jonathan Golob

  committed
  376e837

• removed extra param

  

  Jonathan Golob

  committed
  699165b

• commonprefix -> commonpath to fix a bug

  

  Jonathan Golob

  committed
  27dfc98

• Fixed bug in pol vs poll

  

  Jonathan Golob

  committed
  611f53f

Commits on Jun 27, 2018

• Updated the readme to document how containers work now.

  

  Jonathan Golob

  committed
  637c576

• Added dependencies

  

  Jonathan Golob

  committed
  004dcd6

Commits on Aug 7, 2018

• Permissions changes. setup.py modified to better fit actual dependencies

  

  Jonathan Golob

  authored and

  Jonathan Golob

  committed
  352b63e

Commits on Aug 15, 2018

• Working version for singularity on PBS.

  

  Jonathan Golob

  committed
  5621e16

Commits on Aug 28, 2018

• Many changes to optimize with PBS-singularity

  

  Jonathan Golob

  committed
  8f08f62

Commits on Sep 5, 2018

• Fixes for PBS

  

  Jonathan Golob

  committed
  0ad11ac

• Merge branch 'pbs' into containertask

  

  Jonathan Golob

  authored and

  Jonathan Golob

  committed
  0da06e5

Commits on Sep 6, 2018

• Added example for ContainerTask as well as an example Dockerfile to make a container

  

  Jonathan Golob

  authored and

  Jonathan Golob

  committed
  feab1e6

Commits on Sep 18, 2018

• Working example plus new class to poll AWS

  

  Jonathan Golob

  authored and

  Jonathan Golob

  committed
  81e99c3

• Working with a few bits of fuss batch poller

  

  Jonathan Golob

  authored and

  Jonathan Golob

  committed
  d63f558

• Working consolidation of polling for job status on AWS

  

  Jonathan Golob

  authored and

  Jonathan Golob

  committed
  6e9d3bf

• Merge branch 'polling_aws' into containertask

  

  Jonathan Golob

  authored and

  Jonathan Golob

  committed
  87a40af

Commits on Sep 28, 2018

• Added ability to read containerinfo from a config file

  

  Jonathan Golob

  committed
  a6d288e

Commits on Nov 7, 2018

• Fix of singularity run commands

  

  Jonathan Golob

  committed
  8a1ace8

• Merge pull request #5 from pharmbio/master

  Show description for 9b12226
  jgolob

  authored
  9b12226

• Merge pull request #6 from jgolob/master

  Show description for c56fb20
  jgolob

  authored
  c56fb20

Commits on Nov 15, 2018

• Corrected loading of mounts from ini

  

  Jonathan Golob

  committed
  75d6b76

• Merge branch 'containertask' of https://github.com/jgolob/sciluigi into containertask

  

  Jonathan Golob

  committed
  ad3f563

Commits on Nov 19, 2018

• Reading of config from ini more robust

  

  Jonathan Golob

  committed
  906445c

• Fixed bug in readconfig

  

  Jonathan Golob

  committed
  4e03a24

Commits on Nov 20, 2018

• Some debugging of aws mounts

  

  Jonathan Golob

  committed
  438c91b

• Switch job def

  

  Jonathan Golob

  committed
  eef7eae

• Fixed loading of mounts from config

  

  Jonathan Golob

  committed
  6d906ff

Commits on Nov 21, 2018

• A bit more logging around task watcher

  

  Jonathan Golob

  committed
  99a477a

• Moved batch_task_watcher global holder to __init__.py

  

  Jonathan Golob

  committed
  e1ff25b

• Moved global

  

  Jonathan Golob

  committed
  94bf206

• Moved batch watcher to module global

  

  Jonathan Golob

  committed
  59269c0

Commits on Nov 27, 2018

• Change to loading of batch task watcher

  

  Jonathan Golob

  committed
  0dc340e

• Made import a bit more robust in broken AWS environments

  

  Jonathan Golob

  committed
  957fa21

• Switch to debug for task count

  

  Jonathan Golob

  committed
  59ac40e

• back to info for polling

  

  Jonathan Golob

  committed
  8b0e400

Commits on Nov 29, 2018

• Attempt to terminate running aws jobs on workflow exit

  

  Jonathan Golob

  committed
  613670e

• Added lock for singularity

  

  Jonathan Golob

  committed
  2010738

Commits on Feb 14, 2019

• Fixed a few bugs in the slurm engine

  

  Jonathan Golob

  committed
  0c0cdda

Commits on Feb 15, 2019

• Fixed a few bugs in slurm.

  Show description for b9e4348
  

  Jonathan Golob

  committed
  b9e4348

Commits on Feb 17, 2019

• SLURM changes

  

  Jonathan Golob

  committed
  0df2b08

Commits on Apr 19, 2019

• ugly fix for interface change in luigi

  

  Jonathan Golob

  committed
  dd6a7a7

Commits on Apr 23, 2019

• Added sample config. Bumped version to 2.0.0

  

  Jonathan Golob

  committed
  1c5d27a

Commits on Apr 26, 2019

• made easier defaults in config

  

  Jonathan Golob

  authored and

  Jonathan Golob

  committed
  b43a1d1

Commits on May 2, 2019

• Adjustments to singularity settings.

  

  Jonathan Golob

  committed
  79bb766

• Bumped version to 2.0.1

  

  Jonathan Golob

  committed
  d83ed1d

• pre pull

  

  Jonathan Golob

  committed
  2631917

• Merge branch 'containertask' of https://github.com/jgolob/sciluigi into containertask

  

  Jonathan Golob

  committed
  8a24da4