Code

Production Deploy#980

Code

Merged

davidsmejia merged 134 commits intomainAlexsLemonade/scpca-portal:mainfrom devAlexsLemonade/scpca-portal:devCopy head branch name to clipboardNov 22, 2024

+806-188

Commits

Commits on Sep 18, 2024

• add infrastructure/batch directory, add batch/roles.tf file which includes a batch service role with attached polices and an ecs_task role with attached policies

  avrohomgottlieb

  committed
  7b27ef1

• add batch/security.tf file with batch security group and db resource ingress rule

  avrohomgottlieb

  committed
  e14f579

Commits on Sep 19, 2024

• add batch/computed_environment.tf

  avrohomgottlieb

  committed
  3ff3441

Commits on Sep 25, 2024

• change compute_environment name

  avrohomgottlieb

  committed
  f073407

• update ecs_task role name

  avrohomgottlieb

  committed
  7b6b0d6

• add first pass at job_queue

  avrohomgottlieb

  committed
  ad8699f

Commits on Sep 26, 2024

• Merge pull request #889 from AlexsLemonade/feature/loader

  Show description for 1443cdb
  avrohomgottlieb

  authored
  1443cdb

• fix typo in job_queue, format batch terraform files

  avrohomgottlieb

  committed
  a08b3da

• update queue and compute_environment names

  avrohomgottlieb

  committed
  695004a

• first pass at batch job_definition

  avrohomgottlieb

  committed
  9d3adda

• update name property of job queue

  avrohomgottlieb

  committed
  8d49eb4

• update resourceRequirements to current version

  avrohomgottlieb

  committed
  9cc543d

Commits on Sep 27, 2024

• Merge pull request #899 from AlexsLemonade/avrohom/844-batch-computed-environment

  Show description for 93672a4
  avrohomgottlieb

  authored
  93672a4

• add ephemeralStorage and fargatePlatformConfiguration in job_definition container_properties

  avrohomgottlieb

  committed
  559c96f

• update dockerhub image name

  avrohomgottlieb

  committed
  5f328b3

• update command in job_definition::container_properties

  avrohomgottlieb

  committed
  8de8591

Commits on Oct 1, 2024

• Merge pull request #910 from AlexsLemonade/avrohom/900-add-batch-job-queue

  Show description for 2ac9ab8
  avrohomgottlieb

  authored
  2ac9ab8

• update command in job_definition

  Show description for a1142bb
  

  avrohomgottlieb

  and

  davidsmejia

  authored
  a1142bb

Commits on Oct 7, 2024

• remove aws provider version comment

  avrohomgottlieb

  committed
  16d247f

• Merge pull request #911 from AlexsLemonade/avrohom/901-add-batch-job-definition

  Show description for 89f0048
  avrohomgottlieb

  authored
  89f0048

Commits on Oct 9, 2024

• move aws provider from networking.tf to provider.tf

  avrohomgottlieb

  committed
  0a641b8

• update aws provider version to 5.12.0

  avrohomgottlieb

  committed
  8a1a67a

Commits on Oct 11, 2024

• Merge pull request #929 from AlexsLemonade/avrohom/918-aws-provider-in-terraform

  Show description for 0ab77b9
  avrohomgottlieb

  authored
  0ab77b9

Commits on Oct 13, 2024

• add batch/variables.tf file, include environment variable

  avrohomgottlieb

  committed
  1aa5b61

• update reference in job_definition, rename var from environment to batch_environment

  avrohomgottlieb

  committed
  da616ca

• move batch_image and batch_resource_requirement fields from job_definition to batch variables

  avrohomgottlieb

  committed
  9b29737

Commits on Oct 14, 2024

• move image and resource_requirement var values back into job_definition

  avrohomgottlieb

  committed
  7d0b4ba

• add retry_strategy job_definition::container_properties

  avrohomgottlieb

  committed
  a28f960

• move environment_variables from batch/variables.tf back into job_definition

  avrohomgottlieb

  committed
  b5a9f4e

• fix hidden computed_file query bug in Project::purge_computed_files

  avrohomgottlieb

  committed
  92a7849

• add Sample::purge_computed_files method

  avrohomgottlieb

  committed
  82ce51e

Commits on Oct 15, 2024

• add ComputedFile::purge, fix a few Project and Sample CF related name mistakes

  avrohomgottlieb

  committed
  b578d27

• add break up loader::_create_computed_file into loader::_create_computed_file_callback and loader::_create_computed_file

  avrohomgottlieb

  committed
  b9d8476

• add loader::generate_computed_file function

  avrohomgottlieb

  committed
  8df351d

• convert all args to kwargs in loader::generate_computed_file

  avrohomgottlieb

  committed
  19bd10e

• add generate_computed_file management command

  avrohomgottlieb

  committed
  77ae571

• add validation to generate_computed_file command

  avrohomgottlieb

  committed
  6a8c4f5

Commits on Oct 16, 2024

• Merge pull request #937 from AlexsLemonade/dev

  Show description for 262e2a4
  avrohomgottlieb

  authored
  262e2a4

• remove computed_file_name parameter from CF::get_project|sample_file

  avrohomgottlieb

  committed
  0bd2539

• improve readability of loader::generate_computed_file

  avrohomgottlieb

  committed
  4a30d4e

• remove return early statements, clean up logging statements

  avrohomgottlieb

  committed
  f2c04b3

• Merge branch 'feature/batch' into avrohom/915-add-generate-computed-files-command

  avrohomgottlieb

  committed
  99a1175

• add dispatch_to_batch command

  avrohomgottlieb

  committed
  a2bf77c

Commits on Oct 28, 2024

• improve logging statements, fix download config reference bug

  avrohomgottlieb

  committed
  d12761f

• improve code quality in projects query

  avrohomgottlieb

  committed
  5b10871

• enforce usage of computed_file property in project and sample model methods

  avrohomgottlieb

  committed
  b8bfdbc

• Merge pull request #933 from AlexsLemonade/avrohom/920-add-batch-envars

  Show description for 84cc335
  avrohomgottlieb

  authored
  84cc335

Commits on Oct 29, 2024

• Merge pull request #936 from AlexsLemonade/avrohom/915-add-generate-computed-files-command

  Show description for 5ab62f6
  avrohomgottlieb

  authored
  5ab62f6

Commits on Oct 31, 2024

• Add access to local public key for dev env in deploy script

  avrohomgottlieb

  committed
  4fb9a64

• Rename dockerhub_repo var to dockerhub_account throughout infrastructure files

  avrohomgottlieb

  committed
  2de4984

• update infrastructure readme with new dev stack instructions

  avrohomgottlieb

  committed
  29aeec0

• add public ssh key and 1password integrations to gitignore

  avrohomgottlieb

  committed
  e838b34

• fix typo in infrastructure readme

  avrohomgottlieb

  committed
  5d7ac2f

Commits on Nov 1, 2024

• Merge pull request #940 from AlexsLemonade/avrohom/939-update-deploy-script

  Show description for 0b6ec04
  avrohomgottlieb

  authored
  0b6ec04

Commits on Nov 3, 2024

• add batch module file, and batch/variables file in order to get resources and envars into the module

  avrohomgottlieb

  committed
  ce649a9

• update database depricated field name

  avrohomgottlieb

  committed
  701624a

• update reference to db field

  avrohomgottlieb

  committed
  b6978ff

• update references to resources passed in as vars, fix typos

  avrohomgottlieb

  committed
  19164c9

• fix all deprecation errors

  avrohomgottlieb

  committed
  d2c9155

Commits on Nov 4, 2024

• update aws_vpc and aws_cloudwatch_log_stream resource names to conform to naming conventions

  avrohomgottlieb

  committed
  cfde68e

• update ecs_task policy resource names to conform to naming conventions, add comments to batch/variables

  avrohomgottlieb

  committed
  d308bc5

• keep job_queue::compute_environments for now (will update to job_queue::compute_environment_order when we update the aws provider to version 5.40.0)

  avrohomgottlieb

  committed
  b2d3b70

• rename aws_vpc and aws_cloudwatch_log_stream resource names to original names (will conform to conventions in later PR)

  avrohomgottlieb

  committed
  73385ce

Commits on Nov 5, 2024

• add project property method all_samples_no_multiplexed_duplicates

  avrohomgottlieb

  committed
  aa891e2

• remove locks from ComputedFile::get_sample_file

  avrohomgottlieb

  committed
  6603b1c

• add ComputedFile::bulk_create_multiplexed_files

  avrohomgottlieb

  committed
  3af36ae

• update loader::_create_computed_file to account for new multiplexed sample approach

  avrohomgottlieb

  committed
  4af3ea1

• fix bugs in loader::_create_computed_file, ComputedFile::bulk_create_multiplexed_files, and Project::all_samples_no_multiplexed_duplicates

  avrohomgottlieb

  committed
  acf9744

• update multiplexed sample test in test_loader, along with expected values

  avrohomgottlieb

  committed
  5ae96dc

Commits on Nov 6, 2024

• update dispatch to batch caller with new project.sample query

  avrohomgottlieb

  committed
  38cbb1a

• clarify comments, update typos

  avrohomgottlieb

  committed
  c7461cc

Commits on Nov 7, 2024

• rename Project::all_samples_no_multipexed_duplicates to Project::samples_to_generate

  avrohomgottlieb

  committed
  dd82522

• rename ComputedFile::bulk_create_multiplexed_files to ComputedFile::get_multiplexed_files, implement real bulk creation of all multiplexed sample computed files at the same time

  avrohomgottlieb

  committed
  bee5265

Commits on Nov 8, 2024

• Merge pull request #945 from AlexsLemonade/avrohom/941-debug-batch-implementation

  Show description for 3afe296
  avrohomgottlieb

  authored
  3afe296

• clean up ComputedFile::get_multiplexed_computed_files by swapping out model_to_dict and building objects explicitly

  avrohomgottlieb

  committed
  7e50523

Commits on Nov 10, 2024

• add batch_job_role, add jobRoleArn to job_definition, tighten up batch s3 access, remove unnecessary s3 access for ecs_task_role

  avrohomgottlieb

  committed
  42bfd74

• wire up job queueand job definition access through terraform for job submission on dispatch_to_bath

  avrohomgottlieb

  committed
  2201241

• fix typos in dispatch_to_batch

  avrohomgottlieb

  committed
  d2c2712

• add batch_submit_job policy to api permissions

  avrohomgottlieb

  committed
  4ebee70

• allow public network access for batch jobs, fix incorrect reference

  avrohomgottlieb

  committed
  c033dff

• ensure access to AWS credential envars inside docker container

  avrohomgottlieb

  committed
  b0f76f0

• add load-metadata, generate-computed-files, and dispatch-to-batch to available commands dict in bin/sportal

  avrohomgottlieb

  committed
  2be5043

• fix json errors in new batch_job_role s3_access_policy

  avrohomgottlieb

  committed
  82e2239

Commits on Nov 13, 2024

• fix spatial input metadata file exclusion bug for batch

  avrohomgottlieb

  committed
  a0921dd

• Merge pull request #946 from AlexsLemonade/avrohom/943-fix-multiplexed-for-batch

  Show description for 6843730
  avrohomgottlieb

  authored
  6843730

• merge in updates from 743-fix-multiplexed-for-batch

  avrohomgottlieb

  committed
  73beeba

• remove dispatch_to_batch and mock-data from sportal

  avrohomgottlieb

  committed
  00c1fe3

• invert return logic in Library::get_data_file_paths

  avrohomgottlieb

  committed
  6b37c6c

• move envars accession out of app code and into config

  avrohomgottlieb

  committed
  a9e2d25

Commits on Nov 14, 2024

• move AWS_REGION from confing/prod to config/common

  avrohomgottlieb

  committed
  3e7425e

• Merge pull request #947 from AlexsLemonade/avrohom/debug-dispatch-to-batch

  Show description for 7e9bad3
  avrohomgottlieb

  authored
  7e9bad3

• Merge branch 'dev' into feature/batch

  avrohomgottlieb

  committed
  efefb62

Commits on Nov 15, 2024

• Merge pull request #953 from AlexsLemonade/avrohom/merge-dev-into-feature-batch

  Show description for 0c0c4e1
  avrohomgottlieb

  authored
  0c0c4e1

• Merge pull request #890 from AlexsLemonade/feature/batch

  Show description for 2191d6b
  avrohomgottlieb

  authored
  2191d6b

Commits on Nov 20, 2024

• Added corrected svg

  dvenprasad

  committed
  eb5bd3f

Commits on Nov 21, 2024

• upgrade GHA terraform version to 1.9.8, and actions/checkout to v4

  avrohomgottlieb

  committed
  de44ba4

• upgrade hashicorp/setup-terraform from v1 to v3

  avrohomgottlieb

  committed
  a4873ed

• Merge pull request #968 from AlexsLemonade/avrohom/964-address-terraform-aws-version-error

  Show description for 3473f7a
  avrohomgottlieb

  authored
  3473f7a

• downgrade to tf 0.13.0 in github gha workflows

  avrohomgottlieb

  committed
  41ea6cf

• Merge pull request #969 from AlexsLemonade/avrohom/downgrade-to-tf-0-13

  Show description for 8019958
  avrohomgottlieb

  authored
  8019958

• change aws provider version to 4.0.0

  avrohomgottlieb

  committed
  41120fa

• comment out batch to go around new feature usages while changing aws provider version

  avrohomgottlieb

  committed
  fe5260c

• comment out batch module output vars and add TODOs to uncomment later

  avrohomgottlieb

  committed
  28ee16f

• Merge pull request #970 from AlexsLemonade/avrohom/upgrade-aws-provider-version

  Show description for 600cf09
  davidsmejia

  authored
  600cf09

• downgrade aws provider version from 4.0.0 to 3.76.1

  avrohomgottlieb

  committed
  a7176c2

• Merge pull request #971 from AlexsLemonade/avrohom/downgrade-aws-version-to-3-76-1

  Show description for 5c8c446
  avrohomgottlieb

  authored
  5c8c446

Commits on Nov 22, 2024

• Merge pull request #967 from AlexsLemonade/deepa-fix-about-page-image-typo

  Show description for bdc5d94
  dvenprasad

  authored
  bdc5d94

• change aws_provider version back to 3.37.0, revert deprecation warning changes from feature/batch to be in accordance with old 3.37.0 implementation.

  avrohomgottlieb

  committed
  db8128b

• handle merge conflict

  avrohomgottlieb

  committed
  a2001d0

• comment out newer 5.12.0 version of aws resources

  avrohomgottlieb

  committed
  56c6432

• fix typo

  avrohomgottlieb

  committed
  ca9fe3d

• add upgrade flag to init_terraform script

  avrohomgottlieb

  committed
  1c2d784

• Update infrastructure/variables.tf

  Show description for e538730
  

  avrohomgottlieb

  and

  davidsmejia

  authored
  e538730

• Update infrastructure/api.tf

  Show description for 8d8a88f
  

  avrohomgottlieb

  and

  davidsmejia

  authored
  8d8a88f

• Update infrastructure/s3.tf

  Show description for 11105d6
  

  avrohomgottlieb

  and

  davidsmejia

  authored
  11105d6

• Update infrastructure/s3.tf

  Show description for d049c27
  

  avrohomgottlieb

  and

  davidsmejia

  authored
  d049c27

• Update infrastructure/s3.tf

  Show description for 3a0fc2f
  

  avrohomgottlieb

  and

  davidsmejia

  authored
  3a0fc2f

• Update infrastructure/networking.tf

  Show description for 127f372
  

  avrohomgottlieb

  and

  davidsmejia

  authored
  127f372

• Update infrastructure/database.tf

  Show description for 923d24f
  

  avrohomgottlieb

  and

  davidsmejia

  authored
  923d24f

• Merge pull request #972 from AlexsLemonade/avrohom/revert-tf-resources-aws-version-3-37-0-implementation

  Show description for bf43e39
  avrohomgottlieb

  authored
  bf43e39

• move aws provider version to provider block

  avrohomgottlieb

  committed
  6f7b666

• Merge pull request #973 from AlexsLemonade/avrohom/tf-move-version-to-provider-block

  Show description for b9bc9b0
  avrohomgottlieb

  authored
  b9bc9b0

• set tf og to trace in init_terraform script

  avrohomgottlieb

  committed
  a499cd9

• Merge pull request #974 from AlexsLemonade/avrohom/set-tf-log-to-trace

  Show description for 01d28a0
  avrohomgottlieb

  authored
  01d28a0

• update popen command with TF_LOG envar

  avrohomgottlieb

  committed
  e1929ce

• Merge pull request #975 from AlexsLemonade/avrohom/add-tf-log-to-popen

  Show description for d52df33
  avrohomgottlieb

  authored
  d52df33

• remove unnecessary double TF_LOG

  avrohomgottlieb

  committed
  4a21884

• Merge pull request #976 from AlexsLemonade/avrohom/remove-double-tf-logg

  Show description for c2cdfdf
  avrohomgottlieb

  authored
  c2cdfdf

• try different source and newer requirement version locking

  avrohomgottlieb

  committed
  696e578

• Merge pull request #977 from AlexsLemonade/avrohom/terraform-provider-version

  Show description for d7ff5f9
  avrohomgottlieb

  authored
  d7ff5f9

• remove default tags as they're already included in the provider block

  avrohomgottlieb

  committed
  29a934a

• Merge pull request #978 from AlexsLemonade/avrohom/fix-tags-and-acl-issue

  Show description for 46b2f20
  avrohomgottlieb

  authored
  46b2f20

• remove extra db tag

  avrohomgottlieb

  committed
  2e60cb7

• Merge pull request #979 from AlexsLemonade/avrohom/fix-extra-db-tag

  Show description for 59327f9
  avrohomgottlieb

  authored
  59327f9