Merge branch 'develop' of https://github.com/daneisburgh/invaas into …

…develop
daneisburgh · Jan 14, 2024 · ea9ef37 · ea9ef37
2 parents b3b423c + 42d9ee2
commit ea9ef37
Show file tree

Hide file tree

Showing 27 changed files with 9,334 additions and 2,464 deletions.
diff --git a/.github/workflows/release.yml → .github/workflows/deployment.yml b/.github/workflows/release.yml → .github/workflows/deployment.yml
@@ -1,10 +1,10 @@
-name: Release
+name: Deployment
 on:
-  push:
-    branches:
-      - "main"
+  release:
+    types: [released]
 jobs:
-  release-pipeline:
+  deployment:
+    name: Deployment
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v3
@@ -19,10 +19,6 @@ jobs:
         run: python -m pip install --upgrade pip
       - name: Install dependencies
         run: pip install -e ".[dev]"
-      - name: Run format check
-        run: black --diff --color --check .
-      - name: Run type check
-        run: mypy --install-types --non-interactive .
       - run: databricks bundle deploy
         working-directory: .
         env:

diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml
@@ -1,13 +1,13 @@
 name: Integration
-on:
-  push:
-    branches-ignore:
-      - main
+on: [push]
 jobs:
-  integration-pipeline:
+  integration:
+    name: Integration
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v3
+        with:
+          fetch-depth: 0
       - name: Set up Python
         uses: actions/setup-python@v4
         with:
@@ -22,3 +22,8 @@ jobs:
         run: black --diff --color --check .
       - name: Run type check
         run: mypy --install-types --non-interactive .
+      - name: Create release
+        if: github.ref == 'refs/heads/main'
+        env:
+          GH_TOKEN: ${{ secrets.GH_TOKEN }}
+        run: semantic-release version --changelog --commit --push --vcs-release --skip-build
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -0,0 +1,170 @@
+# CHANGELOG
+
+
+
+## v1.2.1 (2023-12-21)
+
+### Fix
+
+* fix: temporarily disable schwab task ([`acc7876`](https://github.com/daneisburgh/invaas/commit/acc7876d68b0d818ba1029677d62d0791f904bae))
+
+
+## v1.2.0 (2023-12-20)
+
+### Documentation
+
+* docs: update CHANGELOG ([`1367e54`](https://github.com/daneisburgh/invaas/commit/1367e545064b2f1269c6ad19079ca7a62d5bc73a))
+
+### Feature
+
+* feat: update deployment config target variables ([`ad416a2`](https://github.com/daneisburgh/invaas/commit/ad416a266f1943ed4e4fc9e69ec27061946f7337))
+
+* feat: update deployment config task variables ([`8a2b134`](https://github.com/daneisburgh/invaas/commit/8a2b134338524defc172fdaa1b93b2053f6f5474))
+
+* feat: update task app env variable ([`e117190`](https://github.com/daneisburgh/invaas/commit/e1171903cb2cb8588016421a6d12b455325b9f23))
+
+### Fix
+
+* fix: coinbase and schwab tasks ([`c913a61`](https://github.com/daneisburgh/invaas/commit/c913a615ee667a0760951381bd29bef0980ec5d1))
+
+### Refactor
+
+* refactor: update task logging ([`16a3019`](https://github.com/daneisburgh/invaas/commit/16a301999d3233733d029f31c72bf47b97f0597c))
+
+* refactor: coinbase and schwab notebooks ([`7ed9eba`](https://github.com/daneisburgh/invaas/commit/7ed9ebafdcfb0a5f49112738567c8368f99f0121))
+
+### Style
+
+* style: format files ([`5e267f0`](https://github.com/daneisburgh/invaas/commit/5e267f02ac590bad148f4836a514e870417ea0d2))
+
+
+## v1.1.2 (2023-12-11)
+
+### Fix
+
+* fix: update app environment variables name format ([`33e4a2b`](https://github.com/daneisburgh/invaas/commit/33e4a2bef51c4fc02a52ae1570b47534eca76834))
+
+
+## v1.1.1 (2023-12-11)
+
+### Documentation
+
+* docs: update CHANGELOG ([`1598b5c`](https://github.com/daneisburgh/invaas/commit/1598b5c38224e5ef9c0b0038ac973363d921e9db))
+
+* docs: update CHANGELOG ([`be0a526`](https://github.com/daneisburgh/invaas/commit/be0a526d9fb65728a8f62af2dc3fa1b238e3ab60))
+
+### Fix
+
+* fix: update workflow schedule ([`fe1f90e`](https://github.com/daneisburgh/invaas/commit/fe1f90ec7eaec962dd7762dc57a0402ecac4aec0))
+
+
+## v1.1.0 (2023-12-11)
+
+### Build
+
+* build: update databricks config
+
+- change workflow schedule
+- add environment variables
+- update notebook tasks ([`f17ae29`](https://github.com/daneisburgh/invaas/commit/f17ae293da8a7fb291983beaaaa87450408cee95))
+
+### Chore
+
+* chore: update pyproject config ([`2e3abdb`](https://github.com/daneisburgh/invaas/commit/2e3abdbb4863133f544ee021cd4626ebf64271a8))
+
+* chore: add semantic release and configuration ([`171e14f`](https://github.com/daneisburgh/invaas/commit/171e14f130579609cc98b7826bd47ba0a124ddf3))
+
+### Ci
+
+* ci: add checkout fetch depth ([`4e07a46`](https://github.com/daneisburgh/invaas/commit/4e07a46bd719f746d8be916d67cfeed595865525))
+
+* ci: add token for release ([`c3bea0e`](https://github.com/daneisburgh/invaas/commit/c3bea0e3b1a5dec79fb49ea31b634e59e1bdfb50))
+
+* ci: update integration and deployment workflows ([`8182951`](https://github.com/daneisburgh/invaas/commit/81829511b23bc3f65b2969abd5c12b6b069632a9))
+
+### Documentation
+
+* docs: update README ([`d572906`](https://github.com/daneisburgh/invaas/commit/d5729069a6fa2f4dde1026d9fbec03ad5e2ab7c5))
+
+### Feature
+
+* feat: add initial schwab task and notebook ([`65178bc`](https://github.com/daneisburgh/invaas/commit/65178bcde2c7ce3fd27493d741c8bb3e3cc93d3e))
+
+### Fix
+
+* fix: shuffle product ids ([`bb667d8`](https://github.com/daneisburgh/invaas/commit/bb667d88771db4f3bc57813681e61581d9cbd275))
+
+* fix: shuffle product ids ([`ba3588e`](https://github.com/daneisburgh/invaas/commit/ba3588e2aea2c0bfdc8faece34bd425dc17b7e54))
+
+### Refactor
+
+* refactor: setup find packages ([`469772e`](https://github.com/daneisburgh/invaas/commit/469772ea4b8b55b57414480b8c0484c148763f04))
+
+* refactor: coinbase task and notebook ([`2639dfc`](https://github.com/daneisburgh/invaas/commit/2639dfca9f56e840c88cfbfb9690dd51df533c79))
+
+### Unknown
+
+* Merge branch &#39;develop&#39; into feature/schwab ([`dc50fd7`](https://github.com/daneisburgh/invaas/commit/dc50fd75c2265d35baa10776bba888a131182ea6))
+
+
+## v1.0.0 (2023-12-06)
+
+### Chore
+
+* chore: update requirements ([`a67c794`](https://github.com/daneisburgh/invaas/commit/a67c79492b9feb6f3006f1e634787d2bd4c93768))
+
+* chore: format files ([`e4dba11`](https://github.com/daneisburgh/invaas/commit/e4dba11d55251bbd4671349ea9cd411751a226da))
+
+* chore: add notebook utils ([`03869ee`](https://github.com/daneisburgh/invaas/commit/03869ee4b2f1adf32ec4c2cdb6125bc200cba3c6))
+
+* chore: update backtesting notebooks ([`a29f37a`](https://github.com/daneisburgh/invaas/commit/a29f37adf5f1429c035777334881e4455253c6ff))
+
+* chore: add backtesting notebooks ([`f7c9f23`](https://github.com/daneisburgh/invaas/commit/f7c9f232e669480f0822fabbe3062201d4fd81e3))
+
+* chore: update gitignore ([`abfaaa5`](https://github.com/daneisburgh/invaas/commit/abfaaa58ffa5e772bebde7aaa58f8c8dcbfb54cd))
+
+* chore: update vscode settings ([`d59b206`](https://github.com/daneisburgh/invaas/commit/d59b206c78650689c23e2d4acb863c393d441d15))
+
+* chore: update devcontainer ([`ce5b975`](https://github.com/daneisburgh/invaas/commit/ce5b975456e70c584f22b11dd098c39f832836f2))
+
+### Ci
+
+* ci: update workflow job names ([`29c908d`](https://github.com/daneisburgh/invaas/commit/29c908d3ac03f329c8b9adbb6d7106ab6c43bedb))
+
+* ci: fix workflow python versions ([`9ac188a`](https://github.com/daneisburgh/invaas/commit/9ac188a04ee8abdf7c30838a24865cab1467c2b6))
+
+* ci: update databricks config ([`9e3bc37`](https://github.com/daneisburgh/invaas/commit/9e3bc378d4f7ff94c289119fa03ad2197c7bbde3))
+
+* ci: add integration and release workflows ([`6ba1f43`](https://github.com/daneisburgh/invaas/commit/6ba1f43234fca9d224f4d69d5ef4f04337f387c9))
+
+### Feature
+
+* feat: add cnn fear and greed index retrieval ([`0928ab8`](https://github.com/daneisburgh/invaas/commit/0928ab847fd2e5989710a014ec71f6ad663d966c))
+
+* feat: update job schedule and node type ([`0c01c94`](https://github.com/daneisburgh/invaas/commit/0c01c947acb3a37a66e21eea3ed8b458538f913f))
+
+* feat: use only fear greed index in main task class ([`067935c`](https://github.com/daneisburgh/invaas/commit/067935ca8147f0e1cc64f0aaf26f27ac2bd6f6a2))
+
+* feat: add scaler to next open change computation ([`d6c9ae3`](https://github.com/daneisburgh/invaas/commit/d6c9ae33ac2d369ca5cb44891efe898658e9bcff))
+
+* feat: add invaas notebook ([`b1cd8e1`](https://github.com/daneisburgh/invaas/commit/b1cd8e1b4bbc1117b1ca764eadf50f19f4f16742))
+
+### Fix
+
+* fix: update invaas notebook ([`2afb026`](https://github.com/daneisburgh/invaas/commit/2afb026fb69ed45bd737f3a1aeb099cc7daedb96))
+
+* fix: typing errors ([`9e15b86`](https://github.com/daneisburgh/invaas/commit/9e15b861e757e9163cc2872714d031ac4d1c564a))
+
+### Refactor
+
+* refactor: package structure ([`5c5bfce`](https://github.com/daneisburgh/invaas/commit/5c5bfce9772a71283f88d166ef6890a5144355e9))
+
+* refactor: invaas notebook init and main task usage ([`c85172e`](https://github.com/daneisburgh/invaas/commit/c85172e98e8574ef70992293b95e1dc92deae55d))
+
+* refactor: update coinbase client enums classes ([`09fa454`](https://github.com/daneisburgh/invaas/commit/09fa454c0b6dca5db9cb2850008c70576d6086ea))
+
+* refactor: main task ([`2a029dd`](https://github.com/daneisburgh/invaas/commit/2a029ddc02e56f672978e409649eabc5c2397f7a))
+
+### Unknown
+
+* first commit ([`23ff997`](https://github.com/daneisburgh/invaas/commit/23ff997814f30bf1d4a089cba9c89f70d68d7e5d))
diff --git a/README.md b/README.md
@@ -1,34 +1,44 @@
-# invaas
+# Investment as a Service
 
-The 'invaas' project was generated by using the default-python template.
+This is a Python project designed to run in Databricks to automatically buy and sell securities, including cryptocurrencies and stocks/ETFs, through different brokerages.
+
+[![Integration](https://github.com/daneisburgh/invaas/actions/workflows/integration.yml/badge.svg)](https://github.com/daneisburgh/invaas/actions/workflows/integration.yml)
+[![Deployment](https://github.com/daneisburgh/invaas/actions/workflows/deployment.yml/badge.svg)](https://github.com/daneisburgh/invaas/actions/workflows/deployment.yml)
+[![semantic-release](https://img.shields.io/badge/%20%20%F0%9F%93%A6%F0%9F%9A%80-semantic--release-e10079.svg)](https://github.com/semantic-release/semantic-release)
+[![Commitizen friendly](https://img.shields.io/badge/commitizen-friendly-brightgreen.svg)](http://commitizen.github.io/cz-cli/)
 
 ## Getting started
 
 1. Install the Databricks CLI from https://docs.databricks.com/dev-tools/cli/databricks-cli.html
 
 2. Authenticate to your Databricks workspace:
-    ```
-    $ databricks configure
-    ```
+
+   ```
+   $ databricks configure
+   ```
 
 3. To deploy a development copy of this project, type:
-    ```
-    $ databricks bundle deploy --target dev
-    ```
-    (Note that "dev" is the default target, so the `--target` parameter
-    is optional here.)
 
-    This deploys everything that's defined for this project.
-    For example, the default template would deploy a job called
-    `[dev yourname] invaas_job` to your workspace.
-    You can find that job by opening your workpace and clicking on **Workflows**.
+   ```
+   $ databricks bundle deploy --target dev
+   ```
+
+   (Note that "dev" is the default target, so the `--target` parameter
+   is optional here.)
+
+   This deploys everything that's defined for this project.
+   For example, the default template would deploy a job called
+   `[dev yourname] invaas_job` to your workspace.
+   You can find that job by opening your workpace and clicking on **Workflows**.
 
 4. Similarly, to deploy a production copy, type:
+
    ```
    $ databricks bundle deploy --target prod
    ```
 
 5. To run a job or pipeline, use the "run" comand:
+
    ```
    $ databricks bundle run
    ```

diff --git a/databricks.yml b/databricks.yml
@@ -1,23 +1,38 @@
 # This is a Databricks asset bundle definition for invaas.
 # See https://docs.databricks.com/dev-tools/bundles/index.html for documentation.
+variables:
+  app_name:
+    default: invaas
+  email:
+    default: [email protected]
+  schedule_pause_status:
+    default: "UNPAUSED"
+  root_path:
+    default: "/Workspace/Shared/.bundle/prod/invaas/files"
 bundle:
-  name: invaas
+  name: ${var.app_name}
 resources:
   jobs:
     invaas:
-      name: invaas
+      name: ${var.app_name}
+      timeout_seconds: 900
+      max_concurrent_runs: 1
       schedule:
-        quartz_cron_expression: "44 55 * * * ?"
+        quartz_cron_expression: "0 0 0 * * ?"
         timezone_id: UTC
+        pause_status: ${var.schedule_pause_status}
       job_clusters:
         - job_cluster_key: job_cluster
           new_cluster:
             spark_version: 13.3.x-scala2.12
-            node_type_id: Standard_F4s_v2
+            node_type_id: Standard_F4s
             data_security_mode: LEGACY_SINGLE_USER_STANDARD
             spark_conf:
               spark.databricks.cluster.profile: singleNode
               spark.master: "local[*,4]"
+            spark_env_vars:
+              APP_ENV: ${bundle.target}
+              APP_ROOT_PATH: ${var.root_path}
             custom_tags:
               ResourceClass: SingleNode
             azure_attributes:
@@ -26,17 +41,33 @@ resources:
               spot_bid_max_price: -1
       email_notifications:
         on_failure:
-          - [email protected]
+          - ${var.email}
       tasks:
-        - task_key: notebook_task
+        - task_key: coinbase
           job_cluster_key: job_cluster
           notebook_task:
-            notebook_path: ./notebooks/invaas-notebook.ipynb
+            notebook_path: "invaas/coinbase/coinbase_notebook.ipynb"
+        # - task_key: schwab
+        #   depends_on:
+        #     - task_key: coinbase
+        #   job_cluster_key: job_cluster
+        #   notebook_task:
+        #     notebook_path: "invaas/schwab/schwab_notebook.ipynb"
 targets:
+  dev:
+    mode: development
+    default: true
+    run_as:
+      user_name: ${var.email}
+    variables:
+      schedule_pause_status: "PAUSED"
+      root_path: "/Workspace/Shared/.bundle/dev/${var.email}/invaas/files"
+    workspace:
+      host: https://adb-6718246255739942.2.azuredatabricks.net
   prod:
     mode: production
+    run_as:
+      user_name: ${var.email}
     workspace:
       host: https://adb-6718246255739942.2.azuredatabricks.net
       root_path: /Shared/.bundle/prod/invaas
-    run_as:
-      user_name: [email protected]
diff --git a/invaas/__init__.py b/invaas/__init__.py
@@ -1 +1 @@
-__version__ = "1.0.0"
+__version__ = "1.2.1"
diff --git a/invaas/coinbase/__init__.py b/invaas/coinbase/__init__.py
diff --git a/invaas/coinbase_auth.py → invaas/coinbase/coinbase_auth.py b/invaas/coinbase_auth.py → invaas/coinbase/coinbase_auth.py