v3 (#468)

* fix(activities/video): use correct err when returning temporal error * Initial commit * bump tdl version * fix log output for exec functions * feat: breakup tasks into sub-packages * run check videos with task * delete existing chat render if exists * feat(queue): start task route and function * fix live chat render * fix * use video variable for live chat convert output * move more schedules to tasks * remove git from docker ignore so the hash can be used during builds * add .git folder in docker build for commit hash * fix(tasks/chat): move live chat * ref(platform): use standard structs and interface rather than generics * fix: use pointer in platform * update live channel check to use new twitch platform interface * fix(tasks): allow no watched channel to not fail * auth updates * move jwks to tasks; clean up jwt flow * remove temporal workflows * move emotes and badges to platform * allow minimal thumbnail task to fail * Potential fix for incorrectly parsed emotes when a message is offset due to unicode characters. (#467) * Update tdl.go Updated for scenarios where a live comment may be offset due to the presence of unicode characters. * Apply suggestions from code review Applied Zibbp's suggestions. Thank you! Co-authored-by: Isaac <[email protected]> --------- Co-authored-by: Isaac <[email protected]> * video_dir and temp_dir migration functions * refactor twitch gql * 1 minute heartbeat * update twitch routes to use platform * support getting chapters and muted segments in platform 'GetVideo' * save chapters and muted segments in database * move almost everything out of internal/twitch * update tasks route to run new tasks * bump package versions * refactor dockerfile so it is a single for for multiple arches * fix gosu * start refactoring handler tests * improve docker build workflow * Initial commit * bump tdl version * fix log output for exec functions * feat: breakup tasks into sub-packages * run check videos with task * delete existing chat render if exists * feat(queue): start task route and function * fix live chat render * fix * use video variable for live chat convert output * move more schedules to tasks * remove git from docker ignore so the hash can be used during builds * add .git folder in docker build for commit hash * fix(tasks/chat): move live chat * ref(platform): use standard structs and interface rather than generics * fix: use pointer in platform * update live channel check to use new twitch platform interface * fix(tasks): allow no watched channel to not fail * auth updates * move jwks to tasks; clean up jwt flow * remove temporal workflows * move emotes and badges to platform * allow minimal thumbnail task to fail * video_dir and temp_dir migration functions * refactor twitch gql * 1 minute heartbeat * update twitch routes to use platform * support getting chapters and muted segments in platform 'GetVideo' * save chapters and muted segments in database * move almost everything out of internal/twitch * update tasks route to run new tasks * bump package versions * refactor dockerfile so it is a single for for multiple arches * fix gosu * start refactoring handler tests * improve docker build workflow * fix tags * fix lint * auto latest tag and dev tag * add task to update stream id with vod id after archive finishes * feat: blocked_vods service * blocked video support * feat: optionally apply category restrictions to livestreams * fix blocked videos tests * refactor twitchMakeRequest to accept url.Values to resolve live channel check bug * fix storage template bug * support deleting temp vod files * include additional job types when cancelling archive task * river prometheus metrics * possible fix for chat render stderr output * test chat render log streaming * revert * temporarily push workflow image build * push image for testing * auth to ghcr * refactor config * use file template when saving info * possible fix for commit hash missing * use github commit sha * fix * test * test * return channels with playlist vods * include channels in queue request * use correct worker queue variables * allow customizing logs and temp config dir * update entrypoint to use directory env vars * dont need this * default videos to /data/videos * fix * handle migrating channel image paths * update nginx and compose for paths * save the channel after change * test fix for bad commit hash * fix hls archiving - create the temp hls directory during convert - clean up temp hls directory and converted mp4 on video move * update nginx config * bump tdl version * fix live stream recovery on worker crash * feat(vod): regenerate static thumbnail task and endpoint * create a structured json response * define task types * fix: do not render chat if watched channel has it disabled * implement new response on some playback routes * do not download chat if queue doesn't request it * feat: start writing integration tests * fix auth_test * fix config dir * test * move package init to custom init function * add twitch credentials in ci * remove temp push logic for beta * documentation update --------- Co-authored-by: Manuel <[email protected]>
Zibbp · Aug 11, 2024 · e03616e · e03616e
1 parent c513cfc
commit e03616e
Show file tree

Hide file tree

Showing 165 changed files with 15,405 additions and 10,088 deletions.
diff --git a/.devcontainer/Dockerfile b/.devcontainer/Dockerfile
@@ -1,21 +1,15 @@
 FROM mcr.microsoft.com/devcontainers/go:1
 
-ENV CHAT_DOWNLOADER_VER=0.2.8
-
 RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
   && apt-get -y install --no-install-recommends ffmpeg python3 python3-pip \
   && apt-get clean -y && rm -rf /var/lib/apt/lists/*
 
 RUN pip3 install --no-cache --upgrade --break-system-packages pip streamlink chat-downloader
 
-
 WORKDIR /tmp
 
 RUN wget https://github.com/rsms/inter/releases/download/v4.0-beta7/Inter-4.0-beta7.zip && unzip Inter-4.0-beta7.zip && mkdir -p /usr/share/fonts/opentype/inter/ && cp /tmp/Desktop/Inter-*.otf /usr/share/fonts/opentype/inter/ && fc-cache -f -v
 
-RUN wget https://github.com/lay295/TwitchDownloader/releases/download/1.54.3/TwitchDownloaderCLI-1.54.3-Linux-x64.zip && unzip TwitchDownloaderCLI-1.54.3-Linux-x64.zip && mv TwitchDownloaderCLI /usr/local/bin/ && chmod +x /usr/local/bin/TwitchDownloaderCLI && rm TwitchDownloaderCLI-1.54.3-Linux-x64.zip
-
-#RUN wget https://github.com/xenova/chat-downloader/archive/refs/tags/v${CHAT_DOWNLOADER_VER}.tar.gz
-#RUN tar -xvf v${CHAT_DOWNLOADER_VER}.tar.gz && cd chat-downloader-${CHAT_DOWNLOADER_VER} && python3 setup.py install && cd .. && rm -f v${CHAT_DOWNLOADER_VER}.tar.gz && rm -rf chat-downloader-${CHAT_DOWNLOADER_VER}
+RUN wget https://github.com/lay295/TwitchDownloader/releases/download/1.55.0/TwitchDownloaderCLI-1.55.0-Linux-x64.zip && unzip TwitchDownloaderCLI-1.55.0-Linux-x64.zip && mv TwitchDownloaderCLI /usr/local/bin/ && chmod +x /usr/local/bin/TwitchDownloaderCLI && rm TwitchDownloaderCLI-1.55.0-Linux-x64.zip
 
 RUN curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin v1.50.1
diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json
@@ -2,15 +2,17 @@
   "build": { "dockerfile": "Dockerfile" },
   "features": {
     "ghcr.io/jungaretti/features/make:1": {},
-    "ghcr.io/devcontainers/features/github-cli:1": {}
+    "ghcr.io/devcontainers/features/github-cli:1": {},
+    "ghcr.io/devcontainers/features/docker-in-docker:2": {}
   },
   "customizations": {
     "vscode": {
       "extensions": [
         "dbaeumer.vscode-eslint",
         "esbenp.prettier-vscode",
         "eamodio.gitlens",
-        "github.copilot"
+        "github.copilot",
+        "yzhang.markdown-all-in-one"
       ]
     }
   },
@@ -35,5 +37,5 @@
   ],
   "workspaceMount": "source=${localWorkspaceFolder},target=/workspace,type=bind,consistency=cached",
   "workspaceFolder": "/workspace",
-  "postAttachCommand": "go install github.com/joho/godotenv/cmd/godotenv@latest && go install github.com/cosmtrek/air@latest"
+  "postAttachCommand": "make dev_setup"
 }
diff --git a/.dockerignore b/.dockerignore
@@ -1,5 +1,4 @@
 .github
-.git
 dev
 tmp
 bin
diff --git a/.github/workflows/docker-publish.yml b/.github/workflows/docker-publish.yml
@@ -15,180 +15,49 @@ env:
   IMAGE_NAME: ${{ github.repository }}
 
 jobs:
-  build-push-amd64:
+  docker-build:
     runs-on: ubuntu-latest
-    permissions:
-      contents: read
-      packages: write
-      # This is used to complete the identity challenge
-      # with sigstore/fulcio when running outside of PRs.
-      id-token: write
-
     steps:
+      # Checkout the repo
       - name: Checkout repository
         uses: actions/checkout@v4
-
-      # Workaround: https://github.com/docker/build-push-action/issues/461
-      - name: Setup Docker buildx
-        uses: docker/[email protected]
-
-      # Login against a Docker registry except on PR
-      # https://github.com/docker/login-action
-      - name: Log into registry ${{ env.REGISTRY }}
-        if: github.event_name != 'pull_request'
-        uses: docker/[email protected]
         with:
-          registry: ${{ env.REGISTRY }}
-          username: ${{ github.actor }}
-          password: ${{ secrets.GITHUB_TOKEN }}
-
-      # Extract metadata (tags, labels) for Docker
-      # https://github.com/docker/metadata-action
-      - name: Extract Docker metadata
-        id: meta
-        uses: docker/[email protected]
-        with:
-          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
-
-      # Build and push Docker image with Buildx (don't push on PR)
-      # https://github.com/docker/build-push-action
-      - name: Build and push Docker image (amd64)
-        id: build-and-push-amd64
-        uses: docker/[email protected]
-        with:
-          context: .
-          push: ${{ github.event_name != 'pull_request' }}
-          tags: ${{ steps.meta.outputs.tags }}
-          provenance: false
-          # labels: ${{ steps.meta.outputs.labels }}
-          secrets: |
-            VERSION=${{ steps.meta.outputs.version }}
-          platforms: linux/amd64
-          file: Dockerfile
-          cache-from: type=gha,scope=${{ env.IMAGE_NAME }}
-          cache-to: type=gha,scope=${{ env.IMAGE_NAME }},mode=max
-
-  build-push-arm64:
-    # Do not run on PRs
-    if: github.event_name != 'pull_request'
-    runs-on: ubuntu-latest
-    permissions:
-      contents: read
-      packages: write
-      # This is used to complete the identity challenge
-      # with sigstore/fulcio when running outside of PRs.
-      id-token: write
-
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@v4
+          fetch-depth: 0
 
+      # Set up QEMU for Arm64
       - name: Set up QEMU
-        uses: docker/[email protected]
-        with:
-          platforms: arm64
+        uses: docker/setup-qemu-action@v3
 
-      # Workaround: https://github.com/docker/build-push-action/issues/461
-      - name: Setup Docker buildx
-        uses: docker/setup-buildx-action@v3.3.0
+      # Set up Docker Buildx
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
 
-      # Login against a Docker registry except on PR
-      # https://github.com/docker/login-action
+      # Login into GitHub Container Registry except on PR
       - name: Log into registry ${{ env.REGISTRY }}
         if: github.event_name != 'pull_request'
-        uses: docker/login-action@v3.1.0
+        uses: docker/login-action@v3
         with:
           registry: ${{ env.REGISTRY }}
           username: ${{ github.actor }}
           password: ${{ secrets.GITHUB_TOKEN }}
 
       # Extract metadata (tags, labels) for Docker
-      # https://github.com/docker/metadata-action
-      - name: Extract Docker metadata
+      - name: Extract Docker metadata (release)
         id: meta
-        uses: docker/metadata-action@v5.5.1
+        uses: docker/metadata-action@v5
         with:
           images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
+          flavor: |
+            latest=auto
+          tags: |
+            type=semver,pattern={{version}}
+            type=raw,value=dev
 
-      # Build and push Docker image with Buildx (don't push on PR)
-      # https://github.com/docker/build-push-action
-      - name: Build and push Docker image (arm64)
-        id: build-and-push-arm64
-        uses: docker/[email protected]
+      - name: Build and push
+        uses: docker/build-push-action@v6
         with:
-          context: .
+          platforms: linux/amd64,linux/arm64
           push: ${{ github.event_name != 'pull_request' }}
-          tags: ${{ steps.meta.outputs.tags }}-arm64
-          provenance: false
-          # labels: ${{ steps.meta.outputs.labels }}
-          secrets: |
-            VERSION=${{ steps.meta.outputs.version }}
-          platforms: linux/arm64
-          file: Dockerfile.aarch64
-          cache-from: type=gha,scope=${{ env.IMAGE_NAME }}
-          cache-to: type=gha,scope=${{ env.IMAGE_NAME }},mode=max
-
-  create-manifests:
-    # Do not run on PRs
-    if: github.event_name != 'pull_request'
-    runs-on: ubuntu-latest
-    needs: [build-push-amd64, build-push-arm64]
-    permissions:
-      contents: read
-      packages: write
-      id-token: write
-
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@v4
-
-      - name: Extract Docker metadata
-        id: meta
-        uses: docker/[email protected]
-        with:
-          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
-
-      - name: Log into registry ${{ env.REGISTRY }}
-        if: github.event_name != 'pull_request'
-        uses: docker/[email protected]
-        with:
-          registry: ${{ env.REGISTRY }}
-          username: ${{ github.actor }}
-          password: ${{ secrets.GITHUB_TOKEN }}
-
-      - name: Set repo name
-        run: |
-          echo "IMAGE_NAME=$(echo ${{ github.repository }} | tr '[:upper:]' '[:lower:]')" >> $GITHUB_ENV
-
-      # Create v* tag manifests and push
-      - name: Create ref tag manifest and push
-        if: startsWith(github.ref, 'refs/tags/v')
-        run: |
-          echo "Creating manifest for: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.ref_name }}"
-          docker manifest create \
-            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.ref_name }} \
-            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest \
-            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest-arm64
-          docker manifest push ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.ref_name }}
-
-      # Create latest tag manifests and push
-      - name: Create latest tag manifest and push
-        if: startsWith(github.ref, 'refs/tags/v')
-        run: |
-          echo "Creating manifest for: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.ref_name }}"
-          docker manifest create \
-            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest \
-            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest \
-            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest-arm64
-          docker manifest push ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest
-
-      # Create manifest and push
-      - name: Create manifest and push
-        # Run only on main branch push
-        if: github.ref == 'refs/heads/main'
-        run: |
-          docker manifest create \
-            ${{ steps.meta.outputs.tags }} \
-            --amend ${{ steps.meta.outputs.tags }} \
-            --amend ${{ steps.meta.outputs.tags }}-arm64
-          docker manifest push ${{ steps.meta.outputs.tags }}
+          tags: ${{ steps.meta.outputs.tags }}
+          context: .
+          dockerfile: ./Dockerfile
diff --git a/.github/workflows/go-test.yml b/.github/workflows/go-test.yml
@@ -22,11 +22,8 @@ jobs:
       - name: Install dependencies
         run: go mod download
 
-      - name: Create directories
-        run: sudo mkdir -p /vods && sudo chmod 777 /vods && sudo mkdir -p /logs && sudo chmod 777 /logs
-
       - name: Run Tests
-        run: go test -v ./...
         env:
           TWITCH_CLIENT_ID: ${{ secrets.TWITCH_CLIENT_ID }}
           TWITCH_CLIENT_SECRET: ${{ secrets.TWITCH_CLIENT_SECRET }}
+        run: make test
diff --git a/.gitignore b/.gitignore
@@ -20,7 +20,7 @@
 # Go workspace file
 go.work
 
-.env.dev
+.env
 
 /cmd/server/__debug_bin
 dev

diff --git a/.server.air.toml b/.server.air.toml
@@ -5,14 +5,14 @@ tmp_dir = "tmp"
 [build]
   args_bin = []
   bin = "./tmp/server"
-  cmd = "go build -o ./tmp/server ./cmd/server/main.go"
+  cmd = "make build_dev_server"
   delay = 1000
   exclude_dir = ["tmp", "dev"]
   exclude_file = []
   exclude_regex = ["_test.go"]
   exclude_unchanged = false
   follow_symlink = false
-  full_bin = "godotenv -f .env.dev ./tmp/server"
+  full_bin = "godotenv -f .env ./tmp/server"
   include_dir = []
   include_ext = ["go", "tpl", "tmpl", "html"]
   include_file = []
@@ -25,7 +25,7 @@ tmp_dir = "tmp"
   rerun = false
   rerun_delay = 500
   send_interrupt = false
-  stop_on_error = false
+  stop_on_error = true
 
 [color]
   app = ""

diff --git a/.vscode/launch.json b/.vscode/launch.json
@@ -16,15 +16,15 @@
       "request": "launch",
       "mode": "auto",
       "program": "${workspaceFolder}/cmd/server/main.go",
-      "envFile": "${workspaceFolder}/.env.dev"
+      "envFile": "${workspaceFolder}/.env"
     },
     {
       "name": "dev-worker",
       "type": "go",
       "request": "launch",
       "mode": "auto",
       "program": "${workspaceFolder}/cmd/worker/main.go",
-      "envFile": "${workspaceFolder}/.env.dev"
+      "envFile": "${workspaceFolder}/.env"
     }
   ]
 }
diff --git a/.worker.air.toml b/.worker.air.toml
@@ -5,14 +5,14 @@ tmp_dir = "tmp"
 [build]
   args_bin = []
   bin = "./tmp/worker"
-  cmd = "go build -o ./tmp/worker ./cmd/worker/main.go"
+  cmd = "make build_dev_worker"
   delay = 1000
   exclude_dir = ["tmp", "dev"]
   exclude_file = []
   exclude_regex = ["_test.go"]
   exclude_unchanged = false
   follow_symlink = false
-  full_bin = "godotenv -f .env.dev ./tmp/worker"
+  full_bin = "godotenv -f .env ./tmp/worker"
   include_dir = []
   include_ext = ["go", "tpl", "tmpl", "html"]
   include_file = []
@@ -25,7 +25,7 @@ tmp_dir = "tmp"
   rerun = false
   rerun_delay = 500
   send_interrupt = false
-  stop_on_error = false
+  stop_on_error = true
 
 [color]
   app = ""
-Original file line number
+Diff line change
@@ -1,5 +1,4 @@
     .github
-    .git
     dev
     tmp
     bin