Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MVP of self-hosting #1718

Merged
merged 131 commits into from
Nov 5, 2024
Merged
Show file tree
Hide file tree
Changes from 124 commits
Commits
Show all changes
131 commits
Select commit Hold shift + click to select a range
5af7552
Create a storage trait and implement for s3 and filesystem
ddimaria Jul 30, 2024
0a0ba6b
Implement storage when reading/writing in the file service
ddimaria Jul 30, 2024
e0f760c
Serve files from the file service
ddimaria Jul 30, 2024
26a5789
Abstract jwt and error functionality
ddimaria Jul 30, 2024
e7a0eea
Ignore storage contents
ddimaria Jul 30, 2024
b8c1fe4
Rename aws dir to storage
ddimaria Jul 30, 2024
550b484
Handle thumbnail uploads, start presigned URL work
ddimaria Aug 1, 2024
4d613db
Presigned URLs, cleanup, documentation, tests
ddimaria Aug 2, 2024
310c8ab
Cleanup
ddimaria Aug 2, 2024
afec88e
Merge remote-tracking branch 'origin/qa' into simple-file-storage
ddimaria Aug 5, 2024
8c93d43
Merge remote-tracking branch 'origin/qa' into simple-file-storage
ddimaria Aug 5, 2024
159e42a
Merge remote-tracking branch 'origin/qa' into simple-file-storage
ddimaria Aug 5, 2024
c384653
Fix jest test mock
ddimaria Aug 5, 2024
443ab19
Fix docker compose down command in quadratic-connection
ddimaria Aug 5, 2024
f6a3e6b
Fix quadratic-api test mocks
ddimaria Aug 5, 2024
82df198
Merge branch 'qa' into simple-file-storage
AyushAgrawal-A2 Aug 5, 2024
35ad668
Abstract auth functionality in API, switch using env var AUTH_TYPE
ddimaria Aug 6, 2024
9ba22f8
Full kratos integration into docker compose
ddimaria Aug 7, 2024
8b5ac52
Abstract auth0Client into a generic AuthClient implementation
ddimaria Aug 8, 2024
b7aed3d
add env var to example files
davidkircos Aug 12, 2024
19f7d63
Full login/logout flow with server integration
ddimaria Aug 13, 2024
1102a07
Add getUsersFromOryByEmail() in api + cleanup
ddimaria Aug 13, 2024
e33cc35
Remove unused
ddimaria Aug 13, 2024
ad1b2b2
Merge branch 'simple-file-storage' into self-hosting-auth-alternative
ddimaria Aug 13, 2024
a4c270a
Merge remote-tracking branch 'origin/qa' into simple-file-storage
ddimaria Aug 14, 2024
2297262
Merge remote-tracking branch 'origin/simple-file-storage' into self-h…
ddimaria Aug 14, 2024
30e7412
iMVP of self-hosting
ddimaria Aug 16, 2024
5f340fe
Merge branch 'qa' into simple-file-storage
AyushAgrawal-A2 Aug 18, 2024
770de01
Checkpoint after full setup, plus non-working TLS
ddimaria Aug 20, 2024
3f8976a
Route the client in self-hosting through caddy
ddimaria Aug 21, 2024
1d1d43c
Embed env vars in docker-compose
ddimaria Aug 21, 2024
298bc07
Create script that downloads self-hosting directory and starts docker
ddimaria Aug 21, 2024
8584061
Enforce self-hosting license
ddimaria Aug 23, 2024
73909aa
Handle licenses
ddimaria Aug 26, 2024
d8c11bf
Update copy of revoked license on the file view
ddimaria Aug 26, 2024
4599956
Ask user for license key in self-hosting and add value to docker-compose
ddimaria Aug 26, 2024
b00d61b
Merge remote-tracking branch 'origin/simple-file-storage' into self-h…
ddimaria Aug 26, 2024
a84179b
Treat invalid licenses like revoked licenses
ddimaria Aug 26, 2024
3e7c743
Don't cache errors or non-active licenses
ddimaria Aug 26, 2024
71457ed
Update copy on share dialog
ddimaria Aug 26, 2024
8bd7c9d
Replace docker compose license key before starting docker-compose
ddimaria Aug 26, 2024
2b53087
Separate apart init.sh and start.sh
ddimaria Aug 26, 2024
f1ef337
Include the url to the self-hosting admin in inti.sh
ddimaria Aug 26, 2024
bddbe98
Re-enable the cache
ddimaria Aug 26, 2024
4dc570a
test image push
davidkircos Aug 26, 2024
8b6d8ed
point towards dev
davidkircos Aug 26, 2024
3d19a2b
public
davidkircos Aug 26, 2024
d442f4e
create in us-east-1
davidkircos Aug 26, 2024
d695e85
use correct URL
davidkircos Aug 27, 2024
9a078dc
use public
davidkircos Aug 27, 2024
016874f
always tag latest as well
davidkircos Aug 27, 2024
03d8f40
change repo to staging
davidkircos Aug 27, 2024
661bc4e
create repo then describe it
davidkircos Aug 27, 2024
85b5f5e
add second tag
davidkircos Aug 27, 2024
2771463
Use ecr images in self-hosting docker-compose
ddimaria Aug 28, 2024
19454ac
Update client Dockerfile
ddimaria Aug 28, 2024
e48f821
Add self-hosting documentation, fix quadratic-api to conditionally load
ddimaria Aug 28, 2024
18e2d12
Revert docker-compose
ddimaria Aug 28, 2024
11e73b6
Add prisma target debian-openssl-3.0.x and add empty passwords
ddimaria Aug 28, 2024
22a8e9e
Run postinstall during the api build, better workaround for auth0/s3 …
ddimaria Aug 28, 2024
9a3b537
Hard code LICENSE_API_URI
ddimaria Aug 28, 2024
1688d1a
Fix axios headers
ddimaria Aug 28, 2024
2f4bb84
Fix typo in init.sh
ddimaria Aug 29, 2024
81f4594
Add more info to self-hosting README
ddimaria Aug 29, 2024
fb3ef81
Merge remote-tracking branch 'origin/qa' into self-hosting-setup
ddimaria Aug 29, 2024
c848618
Fix all broken things from the merge and beyond
ddimaria Aug 29, 2024
24067c8
Remove container_name from postgres and caddy
ddimaria Aug 29, 2024
e20a821
Improve init experience for self-hosting
ddimaria Aug 30, 2024
2fdb1ed
Update init.sh link
ddimaria Aug 30, 2024
a857530
Update init.sh link
ddimaria Aug 30, 2024
e8d4d5d
Migrate database on startup
ddimaria Aug 30, 2024
067b083
Migrate database on startup v2
ddimaria Aug 30, 2024
065f531
Redirect to login-result after kratos login
ddimaria Aug 30, 2024
b31bec3
Redirect to login-result after kratos registration
ddimaria Aug 30, 2024
6c4c62b
Use bash over sh in the init.sh script
ddimaria Sep 3, 2024
86e1cf6
Update new init.sh link
ddimaria Sep 3, 2024
c3e2461
Add Installing on Ubuntu instructions
ddimaria Sep 3, 2024
5ca28b9
Prefer 127.0.0.1 over host.docker.internal
ddimaria Sep 3, 2024
c6ebfd2
Prefer 0.0.0.0 over 127.0.0.1 for self hosting
ddimaria Sep 3, 2024
e425d5a
Add extra_hosts in docker compose for ubuntu
ddimaria Sep 3, 2024
50b8ed5
Remove trailing slash from LICENSE_API_URI
ddimaria Sep 3, 2024
c4b8f44
File perms logging
ddimaria Sep 3, 2024
fbf4b28
File perms logging v2
ddimaria Sep 3, 2024
de599cf
Improve the parseDomain() function
ddimaria Sep 4, 2024
3cc9298
Add CORS headers to nginx
ddimaria Sep 4, 2024
c0212b4
Add tests for parseDomain
ddimaria Sep 4, 2024
2cddc14
Hard-code kratos login and registration URLs
ddimaria Sep 4, 2024
e93082d
Update to mac friendly sed
ddimaria Sep 4, 2024
2b9f2ea
Prompt user for host name during initialization
ddimaria Sep 4, 2024
7a8b99c
Remove .bak files
ddimaria Sep 4, 2024
e208253
Update init.sh link in README
ddimaria Sep 4, 2024
bf35b7a
Update instructions to include Creating an EC2 Instance
ddimaria Sep 4, 2024
3951a55
Ensure login-redirect after ory registration
ddimaria Sep 4, 2024
21057a2
Allow command line arguments to be used in init.sh
ddimaria Sep 4, 2024
443179d
Update readme with new init link and ports to open
ddimaria Sep 4, 2024
d90f78c
Update readme requirement to include all ports
ddimaria Sep 4, 2024
0f44de8
Merge remote-tracking branch 'origin/qa' into self-hosting-setup
ddimaria Sep 13, 2024
b818ecc
Remove self-hosting directory (in favor of new self-hosting repo
ddimaria Sep 13, 2024
9b59b3f
Update URLs to self-hosting portal production domain
ddimaria Sep 13, 2024
8c9c9de
Fix clippy lints
ddimaria Sep 13, 2024
414a74e
Fix some api test failures
ddimaria Sep 13, 2024
ad9bcda
Mock licenseClient
ddimaria Sep 13, 2024
5e822a2
Fix quadratic-client lints
ddimaria Sep 13, 2024
22df915
Fix quadratic-api tests
ddimaria Sep 13, 2024
fa43dac
Create .env.test for quadratic-client to fix unit tests
ddimaria Sep 16, 2024
e3f8b8c
Prefer npm run compile --workspace=quadratic-shared in the client Doc…
ddimaria Sep 16, 2024
917b77e
Merge remote-tracking branch 'origin/qa' into self-hosting-setup
ddimaria Oct 8, 2024
8151a40
Add in bypass for quadratic license key
ddimaria Oct 23, 2024
e8cdd29
productionize image creation
davidkircos Oct 23, 2024
9e6fc6b
Address PR feedback
ddimaria Oct 23, 2024
1b45c8f
Merge remote-tracking branch 'origin/qa' into self-hosting-setup
ddimaria Oct 23, 2024
ef2d28c
Refresh user count when added or removed from a team
ddimaria Oct 24, 2024
d13358d
Fix lint issue related to different event emitter library
ddimaria Oct 24, 2024
398ee36
productionize
davidkircos Oct 24, 2024
353a2d7
Merge branch 'self-hosting-setup' of https://github.com/quadratichq/q…
davidkircos Oct 24, 2024
ab44c0f
Merge branch 'qa' into self-hosting-setup
ddimaria Oct 30, 2024
764324b
Update _dashboard.tsx, remove useTheme
ddimaria Oct 30, 2024
6fdcf3e
Fix import issue
ddimaria Oct 30, 2024
38ba7af
Try export-table option when building rust-client
ddimaria Oct 30, 2024
717ec59
Merge remote-tracking branch 'origin/qa' into self-hosting-setup
ddimaria Oct 30, 2024
b435723
Remove wasm target
ddimaria Oct 30, 2024
130dda6
Merge remote-tracking branch 'origin/qa' into self-hosting-setup
ddimaria Oct 31, 2024
19ac842
Remove export-tables option
ddimaria Oct 31, 2024
e65c3e6
Fix kratos creation bug
ddimaria Oct 31, 2024
f43d051
echo versions
davidkircos Nov 4, 2024
249e0c9
try
davidkircos Nov 4, 2024
bea8d4f
Respect the S3 storage type on the client
ddimaria Nov 4, 2024
3fa9673
Hide examples where VITE_STORAGE_TYPE === 'file-system'
ddimaria Nov 4, 2024
9062aef
Only show revoked message when the license is revoked, not exceeded
ddimaria Nov 4, 2024
7991c6b
Merge branch 'self-hosting-setup' of github.com:quadratichq/quadratic…
ddimaria Nov 4, 2024
7653c07
Move JWT initialization
ddimaria Nov 5, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions .github/workflows/production-publish-images.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
name: Build and Publish Images to ECR

on:
push:
branches:
- self-hosting-setup #remove
- main

concurrency:
group: production-publish-images

jobs:
publish_images:
runs-on: ubuntu-latest-8-cores
strategy:
matrix:
service: [multiplayer, files, connection, client, api]
steps:
- uses: actions/checkout@v4

- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-east-1

- name: Login to Amazon ECR Public
id: login-ecr
uses: aws-actions/amazon-ecr-login@v2
with:
registry-type: public

- name: Define repository name
id: repo-name
run: |
echo "REPO_NAME=quadratic-${{ matrix.service }}" >> $GITHUB_OUTPUT

- name: Create Public ECR Repository if not exists
id: create-ecr
env:
REPO_NAME: ${{ steps.repo-name.outputs.REPO_NAME }}
run: |
aws ecr-public create-repository --repository-name $REPO_NAME || true
REPO_INFO=$(aws ecr-public describe-repositories --repository-names $REPO_NAME)
ECR_URL=$(echo $REPO_INFO | jq -r '.repositories[0].repositoryUri')
echo "ECR_URL=$ECR_URL" >> $GITHUB_OUTPUT

- name: Read VERSION file
id: version
run: echo "VERSION=$(cat VERSION)" >> $GITHUB_OUTPUT

- name: Build, Tag, and Push Image to Amazon ECR Public
env:
ECR_URL: ${{ steps.create-ecr.outputs.ECR_URL }}
IMAGE_TAG: ${{ steps.version.outputs.VERSION }}
run: |
docker build -t $ECR_URL:$IMAGE_TAG -t $ECR_URL:latest -f quadratic-${{ matrix.service }}/Dockerfile .
docker push $ECR_URL:$IMAGE_TAG
docker push $ECR_URL:latest
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ quadratic-connection/target/
quadratic-core/target/
quadratic-core/tmp.txt
quadratic-files/target/
quadratic-files/storage
quadratic-multiplayer/target/
quadratic-multiplayer/updateAlertVersion.json

Expand All @@ -64,6 +65,7 @@ docker/mysql/data
docker/postgres/data
docker/redis/data
docker/static/html
docker/file-storage
docker/postgres-connection/data
docker/mysql-connection/data
docker/snowflake-connection/data
Expand Down
Loading
Loading