feat(openapi): generate merged spec (#497)

Because - SDKs like [tryAGI](https://github.com/tryAGI/Instill-AI) need a single OpenAPI definition. - Merging our definitions would make sense as the public API is exposed by a single service (`api-gateway`). Headers represent that service's interface rather than the backend ones. This commit - Generates a merged OpenAPI definition. - For now individual definitions are respected and used to feed openapi.instill.tech. This adds some duplication in the configuration (`/<service>/openapi.proto.templ` and `/openapiv2/conf.proto`). We need to decide if we want to use the merged OpenAPI instead (we'll lose the service as a [category](https://docs.readme.com/main/docs/openapi-categories-pages-subpages#tags-and-summaries) and the endpoints will be only grouped by tags. - Tags are respected as much as possible, just modified to avoid collisions. - Depending on whether we want to transition to this single OpenAPI for readme.com, we might consider group endpoints by service or merge some tags ([User, Organization] -> Namespace). --------- Co-authored-by: droplet-bot <[email protected]>
instill-ai · Oct 28, 2024 · 9d69b5c · 9d69b5c
1 parent 9c270d2
commit 9d69b5c
Show file tree

Hide file tree

Showing 29 changed files with 10,967 additions and 12,014 deletions.
diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md
@@ -7,13 +7,107 @@ We appreciate your contribution to this amazing project! Any form of engagement
 - roadmap suggestion
 - ...and so on!
 
-Please refer to the [community contributing section](https://github.com/instill-ai/community#contributing) for more details.
+## Introduction
 
-## Development and codebase contribution
+Before delving into the details to come up with your first PR, please
+familiarize yourself with the project structure of 🔮 [**Instill
+Core**](https://github.com/instill-ai/instill-core).
 
-Before delving into the details to come up with your first PR, please familiarise yourself with the project structure of [Instill Core](https://github.com/instill-ai/community#instill-core).
+Instill AI contracts are defined as [Protocol
+Buffers](https://protobuf.dev/). The proto files are the source from which
+different code is auto-generated (e.g., language SDKs, OpenAPI
+specification of the contracts).
 
-### Sending PRs
+### OpenAPI contracts
+
+All the public endpoints are exposed in a single service
+([`api-gateway`](https://github.com/instill-ai/api-gateway)]). These
+endpoints are documented [in OpenAPI V2
+format](../openapi/v2/service.swagger.yaml) and publicly available at
+[openapi.instill.tech](https://openapi.instill.tech/) through
+(readme.com)[https://readme.com/]. The OpenAPI specification is
+auto-generated via [`grpc-gateway`](https://grpc-ecosystem.github.io/grpc-gateway/)
+and it only reflects the protobuf specification.
+
+## Codebase contribution
+
+The Instill AI contracts follow most of the guidelines provided by [Google
+AIP](https://google.aip.dev/) but have made adjustments in certain areas
+based on our product experience and specific needs.
+
+Some of these conventions are checked at the CI workflows, though for some
+others we rely on the developer's awareness and good judgement. Please,
+use the following guidelines to align your contract updates with our
+conventions.
+
+### Field Behavior
+
+APIs **MUST** apply the `google.api.field_behavior` annotation on every
+field on a message or sub-message used in a request. Even `optional` fields
+must include this annotation in order to keep annotations consistent.
+
+This helps users to understand how an endpoint works. Annotations also
+[modify the generated OpenAPI
+spec](https://grpc-ecosystem.github.io/grpc-gateway/docs/mapping/customizing_openapi_output/#using-googleapifield_behavior)
+marking fields as `required` or `readonly`.
+
+### Method Tags
+
+Every **public** endpoint **MUST** have one (and only one) tag. This tag
+**MUST** also be defined in the [OpenAPI
+configuration](../openapi/v2/conf.proto) with a name and description. The
+tag can be added with `grpc-gateway`'s `tags` operation option:
+
+```proto
+// Get a pipeline
+//
+// Returns the details of a pipeline.
+rpc GetNamespacePipeline(GetNamespacePipelineRequest) returns (GetNamespacePipelineResponse) {
+    option (google.api.http) = {get: "/v1beta/namespaces/{namespace_id}/pipelines/{pipeline_id}"};
+    option (grpc.gateway.protoc_gen_openapiv2.options.openapiv2_operation) = {tags: "💧 VDP"};
+}
+```
+
+We tend to group our endpoints by service (i.e., use a single tag for all
+the endpoints in a service), though this isn't a hard requirement.
+
+[openapi.instill.tech](https://openapi.instill.tech) groups the endpoints
+under the first tag defined in their specification and supports exactly one
+tag. If more tags are defined, it will result in empty sections at the end
+of the sidebar.
+
+### Method Title & Description
+
+Every **public** method **MUST** have a comment with a title and a
+description, separated by a blank comment line. The description might have
+several paragraphs separated by blank lines. Try to document the endpoint
+in detail, including behaviors such as who can access a given resource of
+perform an operation, how to access the result of the operation or if the
+endpoint produces any effect in the system that might not be obvious.
+
+```proto
+// Update a pipeline
+//
+// Udpates a pipeline, accessing it by its resource name, which is defined by
+// the parent namespace and the ID of the pipeline. The authenticated namespace must be
+// the parent of the pipeline in order to modify it.
+//
+// In REST requests, only the supplied pipeline fields will be taken into
+// account when updating the resource.
+rpc UpdateNamespacePipeline(UpdateNamespacePipelineRequest) returns (UpdateNamespacePipelineResponse) {
+    option (google.api.http) = {
+        patch: "/v1beta/namespaces/{namespace_id}/pipelines/{pipeline_id}"
+        body: "pipeline"
+    };
+    option (grpc.gateway.protoc_gen_openapiv2.options.openapiv2_operation) = {tags: "💧 VDP"};
+}
+```
+
+[openapi.instill.tech](https://openapi.instill.tech) will render each part
+as the endpoint title (which can also used in the search engine) and the
+description in the endpoint's view.
+
+## Sending PRs
 
 Please take these general guidelines into consideration when you are sending a PR:
 

diff --git a/.github/workflows/buf-gen-openapi.yaml b/.github/workflows/buf-gen-openapi.yaml
@@ -4,11 +4,10 @@ on:
   pull_request:
     paths:
       - '**.proto'
-      - '**.proto.templ'
       - 'buf.gen.*.yaml'
       - 'Makefile'
       - 'scripts/generate-openapi-doc-info.sh'
-      - 'common/openapi/**'
+      - '.spectral.yaml'
 
 jobs:
   gen-buf-openapi:
@@ -28,32 +27,16 @@ jobs:
       - name: Generate OpenAPI definitions
         run: |
           make openapi
-      - name: Lint generated Core OpenAPI definitions
-        uses: readmeio/rdme@v8
+      - name: Lint generated OpenAPI definitions
+        uses: stoplightio/spectral-action@latest
         with:
-          rdme: openapi:validate openapiv2/core/service.swagger.yaml
-      - name: Lint generated VDP OpenAPI definitions
-        uses: readmeio/rdme@v8
-        with:
-          rdme: openapi:validate openapiv2/vdp/service.swagger.yaml
-      - name: Lint generated Model OpenAPI definitions
-        uses: readmeio/rdme@v8
-        with:
-          rdme: openapi:validate openapiv2/model/service.swagger.yaml
-      - name: Lint generated Artifact OpenAPI definitions
-        uses: readmeio/rdme@v8
-        with:
-          rdme: openapi:validate openapiv2/artifact/service.swagger.yaml
-      - name: Lint generated App OpenAPI definitions
-        uses: readmeio/rdme@v8
-        with:
-          rdme: openapi:validate openapiv2/app/service.swagger.yaml
+          file_glob: 'openapi/v2/*.swagger.yaml'
       - name: Commit and push
         run: |
           if [[ `git status --porcelain` ]]; then
             git fetch origin
             git checkout "${GITHUB_HEAD_REF}"
-            git add openapiv2
+            git add openapi/v2
             git commit -S -m "chore: auto-gen by protobufs" -m "triggered by commit: https://github.com/instill-ai/protobufs/commit/${GITHUB_SHA}"
             git push -u origin "${GITHUB_HEAD_REF}"
           fi
diff --git a/.github/workflows/sync-api-docs.yml b/.github/workflows/sync-api-docs.yml
@@ -8,7 +8,7 @@ on:
       # pushed to the `main` branch.
       - main
     paths:
-      - 'openapiv2/**'
+      - 'openapi/v2/**'
 
 jobs:
   sync-openapi-private-dev:
@@ -19,30 +19,10 @@ jobs:
       - name: Check out repo 📚
         uses: actions/checkout@v3
 
-      - name: Sync Core 🔮
+      - name: Sync OpenAPI docs 📜
         uses: readmeio/rdme@v8
         with:
-          rdme: openapi openapiv2/core/service.swagger.yaml --title "🔮 Core (PR ${{ github.ref_name }})" --key=${{ secrets.README_API_KEY }} --id=669fb9c72dd8c500184c652d
-
-      - name: Sync Model ⚗️
-        uses: readmeio/rdme@v8
-        with:
-          rdme: openapi openapiv2/model/service.swagger.yaml --title "⚗️ Model (PR ${{ github.ref_name }})"--key=${{ secrets.README_API_KEY }} --id=669fb9c72dd8c500184c652e
-
-      - name: Sync VDP 💧
-        uses: readmeio/rdme@v8
-        with:
-          rdme: openapi openapiv2/vdp/service.swagger.yaml --title "💧 VDP (PR ${{ github.ref_name }})"--key=${{ secrets.README_API_KEY }} --id=669fb9c72dd8c500184c652f
-
-      - name: Sync Artifact 💾
-        uses: readmeio/rdme@v8
-        with:
-          rdme: openapi openapiv2/artifact/service.swagger.yaml --title "💾 Artifact (PR ${{ github.ref_name }})" --key=${{ secrets.README_API_KEY }} --id=669fb9c72dd8c500184c6531
-
-      - name: Sync App 📱
-        uses: readmeio/rdme@v8
-        with:
-          rdme: openapi openapiv2/app/service.swagger.yaml --title "📱 App (PR ${{ github.ref_name }})" --key=${{ secrets.README_API_KEY }} --id=66f3f3781c34e9001ff23013
+          rdme: openapi openapi/v2/service.swagger.yaml --title "🔮 Instill AI API (PR ${{ github.ref_name }})" --key=${{ secrets.README_API_KEY }} --id=671f3ec63ce9be54bc25adb8
 
   sync-openapi-private-staging:
     name: Keep private (staging) docs in sync with `main`
@@ -58,38 +38,18 @@ jobs:
           # Needed in check-new-release to compare with the previous commit.
           fetch-depth: 0
 
-      - name: Sync Core 🔮
-        uses: readmeio/rdme@v8
-        with:
-          rdme: openapi openapiv2/core/service.swagger.yaml --key=${{ secrets.README_API_KEY }} --id=66f40c5c346c5d004a090996
-
-      - name: Sync Model ⚗️
-        uses: readmeio/rdme@v8
-        with:
-          rdme: openapi openapiv2/model/service.swagger.yaml --key=${{ secrets.README_API_KEY }} --id=66f40c5c346c5d004a090997
-
-      - name: Sync VDP 💧
-        uses: readmeio/rdme@v8
-        with:
-          rdme: openapi openapiv2/vdp/service.swagger.yaml --key=${{ secrets.README_API_KEY }} --id=66f40c5c346c5d004a090998
-
-      - name: Sync Artifact 💾
-        uses: readmeio/rdme@v8
-        with:
-          rdme: openapi openapiv2/artifact/service.swagger.yaml --key=${{ secrets.README_API_KEY }} --id=66f40c5c346c5d004a090999
-
-      - name: Sync App 📱
+      - name: Sync OpenAPI docs 📜
         uses: readmeio/rdme@v8
         with:
-          rdme: openapi openapiv2/app/service.swagger.yaml --key=${{ secrets.README_API_KEY }} --id=66f40c5c346c5d004a09099a
+          rdme: openapi openapi/v2/service.swagger.yaml --key=${{ secrets.README_API_KEY }} --id=671f3f971dc007003da17bbc
 
       - name: Check new release 🔍
         id: check-new-release
         run: |
           # If the version in the OpenAPI configuration has changed, extract
           # the old and new release versions (without the "v" prefix) to
           # variables.
-          version_file=common/openapi/v1beta/api_info.conf
+          version_file=openapi/v2/conf.proto
           capture_old='^-\s\+\<version:'
           capture_new='^+\s\+\<version:'
           extract_version='s/.*"v\(.*\)".*/\1/'

diff --git a/.spectral.yaml b/.spectral.yaml
@@ -0,0 +1,20 @@
+extends: ["spectral:oas"]
+rules:
+  # When uploading the specs to readme.com, this field will be used to render
+  # information about an endpoint.
+  operation-description: error
+  # Unused or missing tags produce an empty group in readme.com.
+  operation-tag-defined: error
+
+  # Exceptions
+
+  # [`grpc-gateway` issue 3058](https://github.com/grpc-ecosystem/grpc-gateway/issues/3058):
+  # Most tooling ignores keywords next to `$ref`. However, `grpc-gateway`
+  # produces such keywords.
+  no-$ref-siblings: off
+  # `grpc-gateway` generates types for nested types, but sometimes these are
+  # only used by requests (e.g. `ComponentDefinition.View`). Request structures
+  # are defined as part of the endpoint definition, so these nested types end
+  # up defined but not referenced.
+  oas2-unused-definition: off
+
diff --git a/Makefile b/Makefile
@@ -11,21 +11,16 @@ SED_IN_PLACE := sed -i
 ifeq (${OS_NAME}, darwin)
 	SED_IN_PLACE = gsed -i
 endif
+.PHONY: openapi
 openapi:
-	@# Inject common API configuration into each OpenAPI proto template.
-	./scripts/generate-openapi-doc-info.sh
-
-	@# Generate an OpenAPI definition for each directory at the root that
-	@# contains at least one public proto file.
-	find . -name '*public*proto' | cut -d'/' -f 2-2 | sort | uniq | xargs -I '{}' buf generate --template buf.gen.openapi.yaml --path {} --path common -o openapiv2/{}
-
-	@# grpc-ecosystem/openapiv2:v2.19.0 will define protobufNullValue as an
-	@# empty enum, which isn't allowed by the linter we use. This linter is
-	@# used before importing the definitions and exposing them in a public
-	@# server.
-	@#
-	@# For each file in openapiv2, remove empty enum declarations.
-	find openapiv2 -type f | xargs -I '{}' ${SED_IN_PLACE} '/^[[:space:]]*enum: \[\]/,+0d' {}
+	@# Generate a common OpenAPI file, representing the public interface
+	@# of Instill AI.
+	@echo '-> Generate OpenAPI specs'
+	@buf generate --template buf.gen.openapi.yaml --output openapi/v2
+	@echo \# This file is auto-generated. DO NOT EDIT. | cat - openapi/v2/service.swagger.yaml > openapi/v2/service.swagger.tmp.yaml
+	@mv openapi/v2/service.swagger.tmp.yaml openapi/v2/service.swagger.yaml
+.PHONY: openapi-lint
 openapi-lint:
-	@# Lint each file under openapiv2.
-	find openapiv2 -type f | xargs -I '{}' rdme openapi:validate {}
+	@# The spectral ruleset adds extra validation rules that allow us to
+	@# keep the documents consistent.
+	@spectral lint openapi/v2/service.swagger.yaml