Skip to content

Commit

Permalink
Merge branch 'main' into 30509-add-minio-resources
Browse files Browse the repository at this point in the history
  • Loading branch information
machariamuguku authored Apr 18, 2024
2 parents 62170d6 + 2ffa8e3 commit c62e1d5
Show file tree
Hide file tree
Showing 544 changed files with 15,683 additions and 8,458 deletions.
30 changes: 20 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,8 @@
We believe that only an **open-source** solution to data movement can cover the **long tail of data sources** while empowering data engineers to **customize existing connectors**. Our ultimate vision is to help you move data from any source to any destination. Airbyte already provides [300+ connectors](https://docs.airbyte.com/integrations/) for popular APIs, databases, data warehouses and data lakes.

Airbyte connectors can be implemented in any language and take the form of a Docker image that follows the [Airbyte specification](https://docs.airbyte.com/understanding-airbyte/airbyte-protocol/). You can create new connectors very fast with:
- The [low-code Connector Development Kit](https://docs.airbyte.com/connector-development/config-based/low-code-cdk-overview) (CDK) for API connectors ([demo](https://www.youtube.com/watch?v=i7VSL2bDvmw))
- The [Python CDK](https://docs.airbyte.com/connector-development/cdk-python/) ([tutorial](https://docs.airbyte.com/connector-development/tutorials/cdk-speedrun))
- The [low-code Connector Development Kit](https://docs.airbyte.com/connector-development/config-based/low-code-cdk-overview) (CDK) for API connectors ([demo](https://www.youtube.com/watch?v=i7VSL2bDvmw))
- The [Python CDK](https://docs.airbyte.com/connector-development/cdk-python/) ([tutorial](https://docs.airbyte.com/connector-development/tutorials/cdk-speedrun))

Airbyte has a built-in scheduler and uses [Temporal](https://airbyte.com/blog/scale-workflow-orchestration-with-temporal) to orchestrate jobs and ensure reliability at scale. Airbyte leverages [dbt](https://www.youtube.com/watch?v=saXwh6SpeHA) to normalize extracted data and can trigger custom transformations in SQL and dbt. You can also orchestrate Airbyte syncs with [Airflow](https://docs.airbyte.com/operator-guides/using-the-airflow-airbyte-operator), [Prefect](https://docs.airbyte.com/operator-guides/using-prefect-task), [Dagster](https://docs.airbyte.com/operator-guides/using-dagster-integration), or [Kestra](https://docs.airbyte.com/operator-guides/using-kestra-plugin/).

Expand All @@ -44,25 +44,35 @@ Explore our [demo app](https://demo.airbyte.io/).

### Run Airbyte locally

You can run Airbyte locally with Docker.
You can run Airbyte locally with `abctl`.

## Setup & launch Airbyte

- Install `Docker Desktop` \(see [instructions](https://docs.docker.com/desktop/install/mac-install/)\).
- After `Docker Desktop` is installed, you must enable `Kubernetes` \(see [instructions](https://docs.docker.com/desktop/kubernetes/)\).
- Download the latest version of `abctl` from the [releases page](https://github.com/airbytehq/abctl/releases) and run the following command:

```bash
git clone --depth 1 https://github.com/airbytehq/airbyte.git
cd airbyte
./run-ab-platform.sh
abctl local install
```

Login to the web app at [http://localhost:8000](http://localhost:8000) by entering the default credentials found in your .env file.
- Your browser should open to the Airbyte Application, if it does not visit [http://localhost](http://localhost)
- You will be asked for a username and password. By default, that's username `airbyte` and password `password`. You can set these values through command line flags or environment variables. For example, to set the username and password to `foo` and `bar` respectively, you can run the following command:

```
BASIC_AUTH_USERNAME=airbyte
BASIC_AUTH_PASSWORD=password
```bash
abctl local install --username foo --password bar

# Or as Environment Variables
ABCTL_LOCAL_INSTALL_PASSWORD=foo
ABCTL_LOCAL_INSTALL_USERNAME=bar
```

Follow web app UI instructions to set up a source, destination and connection to replicate data. Connections support the most popular sync modes: full refresh, incremental and change data capture for databases.

Read the [Airbyte docs](https://docs.airbyte.com).

The previous Docker Compose instructions are [here](https://docs.airbyte.com/deploying-airbyte/docker-compose).

### Manage Airbyte configurations with code

You can also programmatically manage sources, destinations, and connections with YAML files, [Octavia CLI](https://github.com/airbytehq/airbyte/tree/master/octavia-cli), and API.
Expand Down
52 changes: 26 additions & 26 deletions airbyte-analytics/build.gradle.kts
Original file line number Diff line number Diff line change
@@ -1,40 +1,40 @@
plugins {
id("io.airbyte.gradle.jvm.lib")
id("io.airbyte.gradle.publish")
id("org.jetbrains.kotlin.jvm")
id("org.jetbrains.kotlin.kapt")
id("io.airbyte.gradle.jvm.lib")
id("io.airbyte.gradle.publish")
id("org.jetbrains.kotlin.jvm")
id("org.jetbrains.kotlin.kapt")
}

dependencies {
kapt(platform(libs.micronaut.platform))
kapt(libs.bundles.micronaut.annotation.processor)
kapt(platform(libs.micronaut.platform))
kapt(libs.bundles.micronaut.annotation.processor)

api(libs.segment.java.analytics)
api(libs.micronaut.http)
api(libs.micronaut.cache.caffeine)
api(libs.bundles.micronaut.annotation)
api(libs.bundles.micronaut.kotlin)
api(libs.kotlin.logging)
api(project(":airbyte-commons"))
api(project(":airbyte-config:config-models"))
api(project(":airbyte-api"))
api(libs.segment.java.analytics)
api(libs.micronaut.http)
api(libs.micronaut.cache.caffeine)
api(libs.bundles.micronaut.annotation)
api(libs.bundles.micronaut.kotlin)
api(libs.kotlin.logging)
api(project(":airbyte-commons"))
api(project(":airbyte-config:config-models"))
api(project(":airbyte-api"))


testAnnotationProcessor(platform(libs.micronaut.platform))
testAnnotationProcessor(libs.bundles.micronaut.test.annotation.processor)
testImplementation(libs.bundles.junit)
testImplementation(libs.assertj.core)
testImplementation(libs.junit.pioneer)
testImplementation(libs.mockk)
testImplementation(libs.kotlin.test.runner.junit5)
testRuntimeOnly(libs.junit.jupiter.engine)
testAnnotationProcessor(platform(libs.micronaut.platform))
testAnnotationProcessor(libs.bundles.micronaut.test.annotation.processor)
testImplementation(libs.bundles.junit)
testImplementation(libs.assertj.core)
testImplementation(libs.junit.pioneer)
testImplementation(libs.mockk)
testImplementation(libs.kotlin.test.runner.junit5)
testRuntimeOnly(libs.junit.jupiter.engine)
}

// This is a workaround related to kaptBuild errors.
// TODO: this should be removed when we move to kotlin 1.9.20
// TODO: we should write tests
afterEvaluate {
tasks.named("kaptGenerateStubsTestKotlin") {
enabled = false
}
tasks.named("kaptGenerateStubsTestKotlin") {
enabled = false
}
}
16 changes: 8 additions & 8 deletions airbyte-api-server/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
ARG JDK_IMAGE=airbyte/airbyte-base-java-image:3.1.0
FROM ${JDK_IMAGE} AS server
ARG JDK_IMAGE=airbyte/airbyte-base-java-image:3.2.1

FROM scratch as builder
WORKDIR /app
ADD airbyte-app.tar /app

FROM ${JDK_IMAGE}
EXPOSE 8006 5005
ENV APPLICATION airbyte-api-server
ENV VERSION ${VERSION}

WORKDIR /app

# This is automatically unzipped by Docker
USER root
ADD airbyte-app.tar /app
RUN chown -R airbyte:airbyte /app
COPY --chown=airbyte:airbyte --from=builder /app /app
USER airbyte:airbyte

# wait for upstream dependencies to become available before starting server
Expand Down
143 changes: 74 additions & 69 deletions airbyte-api-server/build.gradle.kts
Original file line number Diff line number Diff line change
@@ -1,101 +1,106 @@
import java.util.Properties

plugins {
id("io.airbyte.gradle.jvm.app")
id("io.airbyte.gradle.jvm.lib")
id("io.airbyte.gradle.publish")
id("io.airbyte.gradle.docker")
kotlin("jvm")
kotlin("kapt")
id("io.airbyte.gradle.jvm.app")
id("io.airbyte.gradle.jvm.lib")
id("io.airbyte.gradle.publish")
id("io.airbyte.gradle.docker")
kotlin("jvm")
kotlin("kapt")
}

dependencies {
kapt(platform(libs.micronaut.platform))
kapt(libs.bundles.micronaut.annotation.processor)
kapt(libs.micronaut.jaxrs.processor)
kapt(platform(libs.micronaut.platform))
kapt(libs.bundles.micronaut.annotation.processor)
kapt(libs.micronaut.jaxrs.processor)

kaptTest(platform(libs.micronaut.platform))
kaptTest(libs.bundles.micronaut.test.annotation.processor)
kaptTest(libs.micronaut.jaxrs.processor)
kaptTest(platform(libs.micronaut.platform))
kaptTest(libs.bundles.micronaut.test.annotation.processor)
kaptTest(libs.micronaut.jaxrs.processor)

annotationProcessor(platform(libs.micronaut.platform))
annotationProcessor(libs.bundles.micronaut.annotation.processor)
annotationProcessor(libs.micronaut.jaxrs.processor)
annotationProcessor(platform(libs.micronaut.platform))
annotationProcessor(libs.bundles.micronaut.annotation.processor)
annotationProcessor(libs.micronaut.jaxrs.processor)

implementation(project(":airbyte-analytics"))
implementation(project(":airbyte-api"))
implementation(project(":airbyte-commons"))
implementation(project(":airbyte-config:config-models"))
implementation(platform(libs.micronaut.platform))
implementation(libs.cron.utils)
implementation(libs.log4j.slf4j2.impl)
implementation(libs.bundles.jackson)
implementation(libs.bundles.micronaut)
implementation(libs.bundles.micronaut.cache)
implementation(libs.bundles.micronaut.data.jdbc)
implementation(libs.bundles.micronaut.metrics)
implementation(libs.micronaut.jaxrs.server)
implementation(libs.micronaut.problem.json)
implementation(libs.micronaut.security)
implementation(libs.sentry.java)
implementation(libs.swagger.annotations)
implementation(libs.jakarta.ws.rs.api)
implementation(libs.airbyte.protocol)
implementation(project(":airbyte-analytics"))
implementation(project(":airbyte-api"))
implementation(project(":airbyte-commons"))
implementation(project(":airbyte-config:config-models"))
implementation(platform(libs.micronaut.platform))
implementation(libs.cron.utils)
implementation(libs.log4j.slf4j2.impl)
implementation(libs.bundles.jackson)
implementation(libs.bundles.micronaut)
implementation(libs.bundles.micronaut.cache)
implementation(libs.bundles.micronaut.data.jdbc)
implementation(libs.bundles.micronaut.metrics)
implementation(libs.micronaut.jaxrs.server)
implementation(libs.micronaut.problem.json)
implementation(libs.micronaut.security)
implementation(libs.sentry.java)
implementation(libs.swagger.annotations)
implementation(libs.jakarta.ws.rs.api)
implementation(libs.airbyte.protocol)

runtimeOnly(libs.javax.databind)
runtimeOnly(libs.snakeyaml)
runtimeOnly(libs.javax.databind)
runtimeOnly(libs.snakeyaml)

testAnnotationProcessor(platform(libs.micronaut.platform))
testAnnotationProcessor(libs.bundles.micronaut.test.annotation.processor)
testAnnotationProcessor(libs.micronaut.jaxrs.processor)
testAnnotationProcessor(platform(libs.micronaut.platform))
testAnnotationProcessor(libs.bundles.micronaut.test.annotation.processor)
testAnnotationProcessor(libs.micronaut.jaxrs.processor)

testImplementation(project(":airbyte-test-utils"))
testImplementation(libs.bundles.micronaut.test)
testImplementation(libs.postgresql)
testImplementation(libs.platform.testcontainers.postgresql)
testImplementation(libs.mockwebserver)
testImplementation(libs.mockito.inline)
testImplementation(project(":airbyte-test-utils"))
testImplementation(libs.bundles.micronaut.test)
testImplementation(libs.postgresql)
testImplementation(libs.platform.testcontainers.postgresql)
testImplementation(libs.mockwebserver)
testImplementation(libs.mockito.inline)
testImplementation(libs.mockk)
}

kapt {
correctErrorTypes = true
correctErrorTypes = true
}

val env = Properties().apply {
load(rootProject.file(".env.dev").inputStream())
load(rootProject.file(".env.dev").inputStream())
}

airbyte {
application {
mainClass = "io.airbyte.api.server.ApplicationKt"
defaultJvmArgs = listOf("-XX:+ExitOnOutOfMemoryError", "-XX:MaxRAMPercentage=75.0")
application {
mainClass = "io.airbyte.api.server.ApplicationKt"
defaultJvmArgs = listOf("-XX:+ExitOnOutOfMemoryError", "-XX:MaxRAMPercentage=75.0")

@Suppress("UNCHECKED_CAST")
localEnvVars.putAll(env.toMutableMap() as Map<String, String>)
localEnvVars.putAll(mapOf(
"AIRBYTE_ROLE" to (System.getenv("AIRBYTE_ROLE") ?: "undefined"),
"AIRBYTE_VERSION" to env["VERSION"].toString(),
"MICRONAUT_ENVIRONMENTS" to "control-plane",
"SERVICE_NAME" to project.name,
"TRACKING_STRATEGY" to env["TRACKING_STRATEGY"].toString(),
))
}
docker {
imageName = "airbyte-api-server"
}
@Suppress("UNCHECKED_CAST")
localEnvVars.putAll(env.toMutableMap() as Map<String, String>)
localEnvVars.putAll(
mapOf(
"AIRBYTE_ROLE" to (System.getenv("AIRBYTE_ROLE") ?: "undefined"),
"AIRBYTE_VERSION" to env["VERSION"].toString(),
"MICRONAUT_ENVIRONMENTS" to "control-plane",
"SERVICE_NAME" to project.name,
"TRACKING_STRATEGY" to env["TRACKING_STRATEGY"].toString(),
)
)
}
docker {
imageName = "airbyte-api-server"
}
}

tasks.named<Test>("test") {
environment(mapOf(
"AIRBYTE_VERSION" to env["VERSION"],
"MICRONAUT_ENVIRONMENTS" to "test",
"SERVICE_NAME" to project.name,
))
environment(
mapOf(
"AIRBYTE_VERSION" to env["VERSION"],
"MICRONAUT_ENVIRONMENTS" to "test",
"SERVICE_NAME" to project.name,
)
)
}

// Even though Kotlin is excluded on Spotbugs, this projects)
// still runs into spotbug issues. Working theory is that)
// generated code is being picked up. Disable as a short-term fix.)
tasks.named("spotbugsMain") {
enabled = false
enabled = false
}
Loading

0 comments on commit c62e1d5

Please sign in to comment.