feat: revamp to new v2 with better implementation (#23)

* chore: remove old implementations/deps

* ref: update dependencies

* feat: upgrade ci/cd version

* ref(docs): new app config template

* new gitignore

* feat: new helper pkg

* feat: new config pkg

* feat: new logger pkg

* ref: move dir scanner to pkg

* feat: new provider contract/interface w helper

* feat: new worker w first version of implementation

* feat: add support Google Drive as cloud provider

* fix(docs): should use fake path as template for app config

* update LICENSE

* feat: implement all packages to v2

* docs: update to reflect new version/changes

* ci/cd: add custom go releaser config

.github/workflows/ci.yml

@@ -2,14 +2,14 @@ on: workflow_call
 
 jobs:
   build-and-test:
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-22.04
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v3
 
     - name: Set up Go
-      uses: actions/setup-go@v2
+      uses: actions/setup-go@v3
       with:
-        go-version: 1.18
+        go-version: 1.19
 
     - name: Build
       run: go build -v ./...

.github/workflows/release.yml

@@ -3,23 +3,23 @@ on: workflow_call
 jobs:
   release:
     name: Compile and Release Binary
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     steps:
       - name: Checkout
-        uses: actions/checkout@v2
+        uses: actions/checkout@v3
         with:
           fetch-depth: 0
 
       - name: Set up Go
-        uses: actions/setup-go@v2
+        uses: actions/setup-go@v3
         with:
-          go-version: 1.18
+          go-version: 1.19
 
       - name: Run GoReleaser
         uses: goreleaser/goreleaser-action@v2
         with:
           distribution: goreleaser
           version: latest
-          args: release --rm-dist
+          args: release -f .goreleaser.yml --rm-dist
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.gitignore

@@ -1,3 +1,2 @@
-.idea/
-bin/
-app-config.yml
+# app config
+app.yml

.goreleaser.yml

@@ -0,0 +1,16 @@
+builds:
+  - goos:
+      - linux
+      - windows
+      - darwin
+    goarch:
+      - amd64
+      - arm64
+archives:
+  - format_overrides:
+      - goos: windows
+        format: zip
+    files:
+      - LICENSE
+      - README.md
+      - app.yml.example

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2022 #MDanial (NzK)
+Copyright (c) 2023 M Danial (NzK)
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -1,61 +1,57 @@
 # Cronjob App for Upload to Cloud Provider
-Little app that upload files in any readable local directories to folders in Cloud Provider. Mainly used conjunction
-with [cron-backup](https://github.com/mdanialr/cron-backup).
-App [cron-backup](https://github.com/mdanialr/cron-backup) for archiving local files and database, then this app
-for uploading them to Cloud Provider.
+CLI app to upload files in any readable local directories to Cloud Provider.
+
+Mainly used in conjunction with [cron-backup](https://github.com/mdanialr/cron-backup).
+The [cron-backup](https://github.com/mdanialr/cron-backup) is for archiving local files and database dump, then this app is for uploading them to Cloud Provider.
 
 # Features
 * Upload any readable local directories as many as possible. (_make sure your cloud provider's capacity is sufficient_).
 * Automatically create folders (**that doesn't exist yet**) in cloud provider based on config file.
-* Option to delete files in cloud provider that exceed the maximum number of days.
+* Option to set the maximum number of days (retain) delete files in cloud provider that exceed the maximum number of days (retain).
+* Option to set the number of worker for upload and or delete job.
+* Option to set the chunk size (in byte) of the file when uploading.
 
 # How to Use
 1. Download the latest binary file from Releases.
-2. Make directory where the download and extracted binary file will reside. We will use bin directory as example.
-```bash
-mkdir bin
-```
-3. Make sure the binary file is executable.
-```bash
-chmod u+x bin/cron-upload
-```
-4. Create configuration file.
-```bash
-touch app-config.yml
-```
-5. Fill in config file as needed. You can check app-config.yml.example in this repo for reference.
-6. Prepare required files. See below.
-7. Execute the binary file from directory where this config file exist, otherwise you will get error config file is not found.
-8. Execute with `-refresh` params first to get refresh token then, exchange it with access token with `-init` params.
-```bash
-./bin/cron-upload -refresh -drive
-./bin/cron-upload -init -drive
-./bin/cron-upload -drive
-```
-9. Check logs file for any error. Maybe required fields are empty, etc.
-10. __Repeat__ step __no.8__ for every 6 days. Because `refresh token` lifetime is just 7 days so make sure to renew them before get expired.
-11. (optional but recommended) Create a cronjob to run this app.
-> Example
-```bash
-@daily cd /full/path/to/cron-upload && ./bin/cron-upload -drive
-```
+2. Extract the downloaded binary file and make sure it's executable.
+    ```bash
+    tar -xzf cron-upload....tar.gz
+    chmod u+x cron-upload
+    ```
+3. Create configuration file from the template.
+    ```bash
+    cp app.yml.example app.yml
+    ```
+4. Edit the app config file as needed. You can check the template for explanation of each field. 
+5. Try to execute and check if there is any error in the app config file.
+   ```bash
+   ./cron-upload -test
+   ```
+6. Check the logs file for any error. Maybe failed to upload or delete files, etc.
+7. Create a cronjob to run this app. (*optional but recommended*) 
+
+    **Example**:
+    ```bash
+    @daily cd /full/path/to/cron-upload && ./cron-upload -log file
+    ```
 
-# Prepare Required Files (Google Drive)
-1. Create OAuth client with 'Desktop Client'.
-2. Download credential.json file.
-3. Write the path where credential.json file reside to app-config in `provider.cred` segment.
-4. Make sure credential.json file **readable** & **accessible** by this app.
-5. You're good to go.
+# Supported Cloud Provider
+Currently only support Google Drive as the cloud provider. 
+## Google Drive
+1. Create Google Service Account and download the credential in json format. You can follow this awesome [tutorial](https://www.labnol.org/google-api-service-account-220404),
+   but following until the [#4](https://www.labnol.org/google-api-service-account-220404#4-share-a-drive-folder) step will be sufficient. Use the shared folder's name as `root` in app config file.
+2. Put the full file path where the downloaded credential is to the app config, like so.
+    ```yml
+    provider:
+      name: drive
+      cred: /full/path/to/credential.json
+    ```
 
 # Arguments
-* `-refresh`: if used with provider argument (e.g. `-drive`), renew or init refresh token.
-* `-init`: if used with provider argument (e.g. `-drive`), retrieve token for authentication against Google Drive provider.
-* `-drive`: do the upload job using Google Drive provider.
-
-# Under the Hood
-1. `./bin/cron-upload -refresh -drive`. this will exchange credential.json file for authorization code and create json file that defined in app-config file that contain refresh token before exchange it for access token.
-2. `./bin/cron-upload -init -drive`. this will exchange refresh token for access token and create ...token.json file that contain access token.
-3. `./bin/cron-upload -drive`. this will do the upload job sequentially and automatically renew access token in ...token.json file if expired.
+* `-path`: set where to find the config file. Default is set to current directory where the binary file is run.
+* `-log`: set where to write the log. Default is set to stdout.
+  You can change it to `-log file` to write the log to file in the directory that you set in config file.
+* `-test`: run all sort of tests such as, validations for the config file, try to create folder, upload & delete files to cloud provider and also check if there is any error.
 
 # License
 This project is licensed under the **MIT License** - see the [LICENSE](LICENSE "LICENSE") file for details.

app.yml.example

@@ -0,0 +1,15 @@
+root: Backup VPS # required. the name of the root folder in the cloud provider
+log: /full/path/to/log/cron-upload/ # log directory where the log file will be written
+retain: 30 # duration in minutes to decide whether the data in the cloud should be deleted. default is set to 6 days (8640)
+worker: 2 # positive number of worker that will be spawned. default is set to 2
+chunk: # the chunk size of the uploaded files. default set to 0 which means there is no chunk size and upload a file at once
+provider:
+  name: drive # the name of the provider. currently only support 'drive' for Google Drive
+  cred: /full/path/to/credential.json # the full path of credential for the chosen provider in json format
+upload:
+  - name: some-app-name # required. unique. directory name in the provider, will be created if not exist yet
+    path: /full/path/to/app/dir # required. the app directory that will be archived
+    retain: 4 # default is set to follow retain field in the root
+  - name: some-another-app-name
+    path: /full/path/to/another/app/dir
+    retain:

go.mod

@@ -1,27 +1,45 @@
 module github.com/mdanialr/cron-upload
 
-go 1.18
+go 1.19
 
 require (
-	github.com/stretchr/testify v1.7.1
-	golang.org/x/oauth2 v0.0.0-20220411215720-9780585627b5
-	google.golang.org/api v0.77.0
-	gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c
+	github.com/spf13/viper v1.14.0
+	github.com/stretchr/testify v1.8.1
+	golang.org/x/exp v0.0.0-20221230185412-738e83a70c30
+	golang.org/x/oauth2 v0.0.0-20221014153046-6fdb5e3db783
+	google.golang.org/api v0.107.0
 )
 
 require (
-	cloud.google.com/go/compute v1.6.0 // indirect
-	github.com/davecgh/go-spew v1.1.0 // indirect
-	github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e // indirect
+	cloud.google.com/go/compute v1.14.0 // indirect
+	cloud.google.com/go/compute/metadata v0.2.3 // indirect
+	github.com/davecgh/go-spew v1.1.1 // indirect
+	github.com/fsnotify/fsnotify v1.6.0 // indirect
+	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
 	github.com/golang/protobuf v1.5.2 // indirect
-	github.com/googleapis/gax-go/v2 v2.3.0 // indirect
+	github.com/google/uuid v1.3.0 // indirect
+	github.com/googleapis/enterprise-certificate-proxy v0.2.1 // indirect
+	github.com/googleapis/gax-go/v2 v2.7.0 // indirect
+	github.com/hashicorp/hcl v1.0.0 // indirect
+	github.com/magiconair/properties v1.8.6 // indirect
+	github.com/mitchellh/mapstructure v1.5.0 // indirect
+	github.com/pelletier/go-toml v1.9.5 // indirect
+	github.com/pelletier/go-toml/v2 v2.0.5 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
-	go.opencensus.io v0.23.0 // indirect
-	golang.org/x/net v0.0.0-20220412020605-290c469a71a5 // indirect
-	golang.org/x/sys v0.0.0-20220412211240-33da011f77ad // indirect
-	golang.org/x/text v0.3.7 // indirect
+	github.com/spf13/afero v1.9.2 // indirect
+	github.com/spf13/cast v1.5.0 // indirect
+	github.com/spf13/jwalterweatherman v1.1.0 // indirect
+	github.com/spf13/pflag v1.0.5 // indirect
+	github.com/subosito/gotenv v1.4.1 // indirect
+	go.opencensus.io v0.24.0 // indirect
+	golang.org/x/net v0.0.0-20221014081412-f15817d10f9b // indirect
+	golang.org/x/sys v0.1.0 // indirect
+	golang.org/x/text v0.5.0 // indirect
 	google.golang.org/appengine v1.6.7 // indirect
-	google.golang.org/genproto v0.0.0-20220414192740-2d67ff6cf2b4 // indirect
-	google.golang.org/grpc v1.45.0 // indirect
-	google.golang.org/protobuf v1.28.0 // indirect
+	google.golang.org/genproto v0.0.0-20221227171554-f9683d7f8bef // indirect
+	google.golang.org/grpc v1.51.0 // indirect
+	google.golang.org/protobuf v1.28.1 // indirect
+	gopkg.in/ini.v1 v1.67.0 // indirect
+	gopkg.in/yaml.v2 v2.4.0 // indirect
+	gopkg.in/yaml.v3 v3.0.1 // indirect
 )