Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DOCS-1113 Create ai accelerator ga documentation section #6292

Draft
wants to merge 39 commits into
base: develop
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
394f8d2
EPAS minor release - release notes
nidhibhammar Nov 14, 2024
ac2b4ee
Update product_docs/docs/epas/14/epas_rel_notes/index.mdx
nidhibhammar Nov 14, 2024
9f80042
Update product_docs/docs/epas/16/epas_rel_notes/epas16_5_0_rel_notes.mdx
nidhibhammar Nov 14, 2024
e8b3315
Minimal update
djw-m Nov 14, 2024
c154e18
Fixed max version
djw-m Nov 14, 2024
3024c3b
Edits to Docs 1104 Document missing optimized topology changes to tab…
ebgitelman Nov 14, 2024
c5499ee
First commit of relgen
djw-m Oct 25, 2024
0689b43
Various fixes, TPA supported
djw-m Oct 28, 2024
3f453ac
Various fixes to tune the notes
djw-m Oct 28, 2024
5ceabc2
Notes added on schemas.
djw-m Oct 28, 2024
1ce416b
Added update optional field, made impactSort case agnostic.
djw-m Oct 28, 2024
8214539
Fix intros in release pages, update TPA and PGD examples
djw-m Oct 29, 2024
6b1ee48
do something sensible if invoked without params in a CWD that isn't r…
josh-heyer Oct 29, 2024
c835e37
Expand details elements in PDF renders
josh-heyer Oct 30, 2024
2f38622
Expand details elements during printing
josh-heyer Oct 30, 2024
c48d035
Fix bad file reference for TPA meta.yml
josh-heyer Oct 30, 2024
1a7111f
Removed tpoa, enhanced for skipping drafts
djw-m Nov 14, 2024
50d0b5c
First commit
djw-m Nov 14, 2024
09b0f6b
Some changes for release notes
djw-m Nov 18, 2024
a9f11a9
Work on content
djw-m Nov 18, 2024
9c3e97b
First draft commit Not For Release
djw-m Nov 25, 2024
4312f7e
Merge branch 'develop' into DOCS-1113-create-ai-accelerator-ga-docume…
djw-m Nov 25, 2024
8ddd54c
Fixes for deploy
djw-m Nov 25, 2024
3b5619b
Fix redirects
djw-m Nov 25, 2024
a471dbb
Link fixups
djw-m Nov 25, 2024
aa9531c
Frontpage linkage
djw-m Nov 25, 2024
4c62c04
Merge branch 'develop' into DOCS-1113-create-ai-accelerator-ga-docume…
djw-m Nov 25, 2024
1e878c8
Merge branch 'develop' into DOCS-1113-create-ai-accelerator-ga-docume…
djw-m Nov 25, 2024
5fc92ee
Tidy front matter
djw-m Nov 25, 2024
e580503
models first commit
djw-m Nov 26, 2024
ff287cc
Daniel fixes
djw-m Nov 26, 2024
936efe6
Retriever updates
djw-m Nov 26, 2024
778c02c
PGVector page filled out
djw-m Nov 26, 2024
5692fad
Fix opener
djw-m Nov 27, 2024
5f26dea
Tighten naming
djw-m Nov 27, 2024
9dff819
Sub-edit review and fixes
djw-m Nov 28, 2024
ce42713
Release note fixes, updated retrievers reference
djw-m Nov 28, 2024
005498d
Added Using Models
djw-m Nov 28, 2024
c579b06
Expanded models section, fixed up retrievers reference.
djw-m Nov 28, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions advocacy_docs/edb-postgres-ai/ai-accelerator/index.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
---
title: "EDB Postgres AI - AI Accelerator"
navTitle: "AI Accelerator"
directoryDefaults:
product: "EDB Postgres AI"
iconName: BrainCircuit
indexCards: simple
description: "All about the EDB Postgres AI - AI Accelerator suite of tools including Pipelines and PGvector."
navigation:
- overview
- pipelines
redirects:
- /edb-postgres-ai/ai-ml/
---


36 changes: 36 additions & 0 deletions advocacy_docs/edb-postgres-ai/ai-accelerator/overview.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
---
title: "EDB Postgres AI - AI Accelerator - Overview"
navTitle: "Overview"
description: "Overview of the EDB Postgres AI AI Accelerator suite of tools including EDB PGvector and Pipelines."
---

## What is the AI Accelerator?

AI Accelerator simplifies AI data management by combining vector search from open source PGvector with Pipelines, an extension for AI data pipeline automation. It handles vector embedding operations for structured and unstructured data, simplifying similarity searches and natural language querying, and streamlines development of AI-powered applications by providing retrieval of AI-ready data in a familiar Postgres environment.

## Features

### Vector Toolkit

The EDB Pipelines extension enables automated vector embedding, storage, and retrieval workflows, and it comes preloaded with PGvector for seamless management of vector data in Postgres. This enables developers to build complex GenAI functionality using SQL commands in the familiar Postgres environment — with just 5 lines of code instead of 130+.

### Swappable Configurations

Easily switch between models, integrate multiple data modalities, and select from a variety of storage locations — in the cloud or on prem — enabling tailored performance and efficient data management that adapts to any need across your business.

### Automated Pipelines

Simplify data processing with Automated Pipelines for fetching data from Postgres or object storage, generating vector embeddings as new data is ingested, and triggering updates to embeddings when source data changes — meaning always-up-to-date data for query and retrieval without tedious maintenance.

### GenAI Query Engine

Leverage semantic search across text and images that’s 4.22X faster[^1] than purpose-built vector databases and an Intelligent Retriever that abstracts away the complexities of vector similarity calculations — transforming your Postgres database into a powerful GenAI search engine across multiple storage locations.

[^1]: [Why we replaced Pinecone with PGVector](https://www.confident-ai.com/blog/why-we-replaced-pinecone-with-pgvector) Confident AI blog post.

## Use cases

* **Sovereign AI**: Your controlled, adaptable AI Platform. Secure, flexible, and cost-effective where your data lives.
* **Virtual Expert**: Natural language AI agents (e.g. chatbots, copilots) to improve workforce and customer outcomes.
* **Cognitive AI**: Intelligent search, analysis, and recommendation across diverse data types.

38 changes: 38 additions & 0 deletions advocacy_docs/edb-postgres-ai/ai-accelerator/pgvector/index.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
---
title: "AI Accelerator - PGvector"
navTitle: "PGvector"
description: "PGvector is a Postgres extension that provides vector data types and functions to store and manipulate vector data."
deepToC: true
---

## What is PGvector

PGvector is an extension for Postgres that enables efficient storage and similarity search of high-dimensional vector data, commonly used for machine learning models, recommendation systems, and natural language processing applications.

### Part of the EDB Postgres AI platform

EDB Postgres AI with PGvector delivers a flexible solution for enterprise AI, integrating seamlessly with existing Postgres environments. It outperforms standalone vector databases with 4.22X faster queries and eliminates data silos via integrations with 18X cost-efficient object storage. This unified platform accelerates AI deployment, simplifies management, and ensures up to 99.999% availability, enabling businesses to innovate rapidly and future-proof their data infrastructure without disrupting current operations.

### Native Vector Data Type Support

PGvector on EDB Postgres AI enables storage of AI/ML model embeddings as first-class data types, allowing efficient indexing and querying of large volumes of AI data stored in object storage, seamlessly integrated with traditional relational data.

### Advanced 4.22X Vector Query

Extends standard SQL with vector-specific operators and functions, enabling complex queries that combine vector operations, relational data, and full SQL capabilities, going far beyond simple similarity searches to support sophisticated AI-driven applications.

### High-performance Indexing

With real-time indexing, storage, and querying of AI data, PGvector enables efficient vector similarity search on embeddings from various LLMs, while leveraging Postgres transactionality for consistent handling of mixed workloads.

### Integrated Vector Data Platform

PGvector unifies vector database capabilities with EDB Postgres AI's mature enterprise features, ensuring high availability, robust backup/recovery, strong security, and ACID data integrity, all within a single vendor solution for comprehensive data management and AI workloads.

## Installation

The extension is included with AI Accelerator's Pipelines and installed automatically when Pipelines is installed.

## Further information

For more information on the PGvector extension, see the [PGvector repository](https://github.com/pgvector/pgvector).
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
---
title: "Getting Started with Pipelines"
navTitle: "Getting Started"
description: "How to get started with AI Accelerator Pipelines."
---

## Where to Start

The best place to start is with the [Pipelines Overview](/edb-postgres-ai/ai-accelerator/pipelines/overview) to get an understanding of what Pipelines is and how it works.

## Installation

Pipelines is included with the EDB Postgres AI - AI Accelerator suite of tools. To install Pipelines, follow the instructions in the [AI Accelerator Installation Guide](/edb-postgres-ai/ai-accelerator/pipelines/installing).

## Using Pipelines

Once you have Pipelines installed, you can start using it to work with your AI data.

###TODO

Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
---
title: "AI Accelerator - Pipelines"
navTitle: "Pipelines"
indexCards: simple
iconName: BrainCircuit
description: "Installing and using AI Accelerator Pipelines with the aidb and PGFS extensions."
navigation:
- "#Introducing"
- overview
- installing
- gettingstarted
- "#Components"
- models
- retrievers
- pgfs
- "#Resources"
- reference
- rel_notes
- licenses
---

As part of the EDB Postgres AI platform, Pipelines abstracts away the complexity of working with AI data. It transforms Postgres into a powerful platform for AI data management, as it combines vector search from PGvector with automation for complex AI workflows.
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
---
title: "Completing and verifying the extension installation"
navTitle: "Completing the installation"
description: "Completing and verifying the installation of the AI Database and File System extensions."
---

### Installing the AI Database extension

The AI Database extension is an extension that provides a set of functions to run AI/ML models in the database. The extension is installed using the `CREATE EXTENSION` command.

```sql
ebd=# CREATE EXTENSION aidb CASCADE;
NOTICE: installing required extension "vector"
CREATE EXTENSION
edb=#
```

### Installing the File System extension

The File System extension is an extension that provides a set of functions to interact with the file system from within the database. The extension is installed using the `CREATE EXTENSION` command.

```sql
edb=# create extension pgfs;
CREATE EXTENSION
```

### Validating the installation

You can check the extensions have been installed by running the `\dx` command in `psql`.

```sql
edb=# \dx
__OUTPUT__
List of installed extensions
Name | Version | Schema | Description
------------------+---------+------------+------------------------------------------------------------
aidb | 1.0.1 | aidb | aidb: makes it easy to build AI applications with postgres
pgfs | 1.0.2 | pgfs | pgfs: enables access to filesystem-like storage locations
vector | 0.7.4 | public | vector data type and ivfflat and hnsw access methods
```

Typically, there will be other extensions listed in this view. The `aidb`, `pgfs`, and `vector` extensions should be listed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
---
title: "Installing AI Accelerator Pipelines"
navTitle: "Installing"
description: "How to install AI Accelerator Pipelines."
navigation:
- packages
- complete
---

Pipelines is delivered as a set of extensions. Depending on how you are deploying Pipelines, these extensions may be installed by your deployment platform (such as EDB Cloud Service) or if you deploy your own Postgres server, you will need to install them manually.

- [Manually installing pipelines packages](packages)

Once the packages are installed, you can [complete the installation](complete) by activating the extensions within Postgres.

Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
---
title: "AI Accelerator - Pipelines"
navTitle: "Installing packages"
description: "How to install the aidb and PGFS extensions from packages."
prevNext: true
---

## Installing the extension packages

There are three extension packages that are required to use the AI Accelerator Pipelines:

- `aidb` - The AI Database extension
- `pgfs` - The PGFS (Postgres File System) extension
- `vector` - The PGvector extension

### Prerequisite

This guide assumes you have a PostgreSQL server running. You can install AI Accelerator Pipelines on a Community PostgreSQL server, an EDB Postgres Advanced Server, or an EDB Postgres Extended server. Consult the installation guides for those products if you need help setting up a server. When you install the server, make a note of the package name used as it will be needed to install the AI Accelerator Pipelines packages.

- For Community PostgreSQL, the package name is `postgresql-<version>`.
- For EDB Postgres Advanced Server, the package name is `edb-as<version>` followed by `-server`.
- For EDB Postgres Extended, the package name is `edb-postgresextended-<version>` followed by `-server`.

Once you have established the package name, you can install the server using the package manager for your operating system.

### Installing the packages

Once the server is running, there will be a number of packages that need to be installed. The packages are available in the EDB repository, and can be installed using the appropriate package manager for your operating system. You will need to use either:

- `apt` for Debian/Ubuntu based systems with .deb packages
- `dnf` for Red Hat based systems with .rpm packages

Depending on your operating system. The packages are:

#### For Community PostgreSQL
djw-m marked this conversation as resolved.
Show resolved Hide resolved

- `edb-pg<version>-aidb` - The AI Database extension
- `edb-pg<version>-pgfs` - The File System extension

and for the Vector extension either:

- rpm-based systems: `edb-pg<version>-pgvector0`
- deb-based systems: `edb-pg<version>-pgvector-0`

#### For EDB Postgres Advanced Server

- `edb-as<version>-aidb` - The AI Database extension
- `edb-as<version>-pgfs` - The File System extension

and for the Vector extension either:

- rpm-based systems: `edb-as<version>-pgvector0`
- deb-based systems: `edb-as<version>-pgvector-0`

#### For EDB Postgres Extended

- `edb-pgextended-<version>-aidb` - The AI Database extension
- `edb-pgextended-<version>-pgfs` - The File System extension

and for the Vector extension either:

- rpm-based systems: `edb-postgresextended<version>-pgvector0`
- deb-based systems: `edb-postgresextended<version>-pgvector-0`


!!! Note Example
Installing the packages on Ubuntu 22.04, with EDB Postgres Advanced Server 16 would look like this:

```shell
sudo apt-get install edb-as16-aidb edb-as16-pgfs edb-as16-pgvector-0
```

With the packages installed, you can now connect to the database and [complete the installation the extensions in Postgres](complete).

!!!
Loading