tech-stack.md

Technical Stack

Airbyte Core Backend

• Java 17
• Framework: Jersey
• API: OAS3
• Databases: PostgreSQL
• Unit & E2E testing: JUnit 5
• Orchestration: Temporal

Connectors

Connectors can be written in any language. However the most common languages are:

• Python 3.9.0
• Java 17

Frontend

• Node.js 16
• TypeScript
• Web Framework/Library: React

Additional Tools

• CI/CD: GitHub Actions
• Containerization: Docker and Docker Compose
• Linter (Frontend): ESLint
• Formatter (Frontend): Prettier
• Formatter (Backend): Spotless

FAQ

Why do we write most destination/database connectors in Java?

JDBC makes writing reusable database connector frameworks fairly easy, saving us a lot of development time.

Why are most REST API connectors written in Python?

Most contributors felt comfortable writing in Python, so we created a Python CDK to accelerate this development. You can write a connector from scratch in any language as long as it follows the Airbyte Specification.

Why did we choose to build the server with Java?

Simply put, the team has more experience writing production Java code.

Why do we use Temporal for orchestration?

Temporal solves the two major hurdles that exist in orchestrating hundreds to thousands of jobs simultaneously: scaling state management and proper queue management. Temporal solves this by offering primitives that allow serialising the jobs' current runtime memory into a DB. Since a job's entire state is stored, it's trivial to recover from failures, and it's easy to determine if a job was assigned correctly.