Merge branch 'CLI/fern-sdk' of https://github.com/devgenix/agenta int…

…o cli/fern-sdk

LICENSE

@@ -1,6 +1,6 @@
 The MIT License
 
-Copyright (c) Mahmoud Mabrouk and Akrem Abayed
+Copyright (c) Agentatech UG (haftungsbeschränkt)
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
@@ -18,4 +18,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-THE SOFTWARE.
+THE SOFTWARE.

agenta-backend/agenta_backend/main.py

@@ -45,8 +45,7 @@ async def lifespan(application: FastAPI, cache=True):
     yield
 
 
-# app = FastAPI(lifespan=lifespan)
-app = FastAPI()
+app = FastAPI(lifespan=lifespan)
 
 allow_headers = ["Content-Type"]
 

docs/images/problems.drawio

@@ -0,0 +1,46 @@
+<mxfile host="app.diagrams.net" modified="2023-12-15T17:10:08.570Z" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:120.0) Gecko/20100101 Firefox/120.0" etag="LOpdpmdsNK03oSl0pV7P" version="22.0.0" type="device">
+  <diagram name="Page-1" id="3e8102d1-ef87-2e61-34e1-82d9a586fe2e">
+    <mxGraphModel dx="1269" dy="1592" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1100" pageHeight="850" background="none" math="0" shadow="0">
+      <root>
+        <mxCell id="0" />
+        <mxCell id="1" parent="0" />
+        <mxCell id="70666756c84bb93c-1" value="" style="ellipse;whiteSpace=wrap;html=1;rounded=0;shadow=0;comic=0;strokeWidth=1;fontFamily=Verdana;fontSize=14;fillColor=#2AAD7E;fontColor=#ffffff;strokeColor=#005700;" parent="1" vertex="1">
+          <mxGeometry x="280" y="110" width="500" height="500" as="geometry" />
+        </mxCell>
+        <mxCell id="70666756c84bb93c-5" value="&lt;div&gt;Team&lt;/div&gt;&lt;div&gt;Collaboration&lt;br&gt;&lt;/div&gt;" style="shadow=0;dashed=0;html=1;strokeColor=none;fillColor=#ffffff;labelPosition=center;verticalLabelPosition=bottom;verticalAlign=top;shape=mxgraph.office.users.users;rounded=0;comic=0;fontFamily=Verdana;fontSize=14;fontColor=#FFFFFF;" parent="1" vertex="1">
+          <mxGeometry x="681" y="257.8" width="56" height="50" as="geometry" />
+        </mxCell>
+        <mxCell id="70666756c84bb93c-13" value="Versioning" style="shadow=0;dashed=0;html=1;strokeColor=none;shape=mxgraph.mscae.general.continuous_cycle;fillColor=#ffffff;rounded=0;comic=0;fontFamily=Verdana;fontSize=14;fontColor=#FFFFFF;verticalAlign=top;labelPosition=center;verticalLabelPosition=bottom;align=center;" parent="1" vertex="1">
+          <mxGeometry x="600" y="150" width="60" height="60" as="geometry" />
+        </mxCell>
+        <mxCell id="JDmTQxFqgsX-q80Pc-4e-5" value="&lt;div&gt;Prompt&lt;/div&gt;&lt;div&gt;Management&lt;br&gt;&lt;/div&gt;" style="sketch=0;pointerEvents=1;shadow=0;dashed=0;html=1;labelPosition=center;verticalLabelPosition=bottom;verticalAlign=top;outlineConnect=0;align=center;shape=mxgraph.office.concepts.writing_pen;fontFamily=Verdana;fontSize=14;fontColor=#FFFFFF;strokeColor=#FFFFFF;" vertex="1" parent="1">
+          <mxGeometry x="477.19" y="130" width="54" height="54" as="geometry" />
+        </mxCell>
+        <mxCell id="JDmTQxFqgsX-q80Pc-4e-9" value="&lt;div style=&quot;font-size: 14px;&quot;&gt;&lt;font style=&quot;font-size: 14px; color: rgb(255, 255, 255);&quot;&gt;&lt;span style=&quot;font-family: Verdana;&quot;&gt;Debugging&lt;/span&gt;&lt;/font&gt;&lt;/div&gt;" style="sketch=0;outlineConnect=0;dashed=0;verticalLabelPosition=bottom;verticalAlign=top;align=center;html=1;fontSize=12;fontStyle=0;aspect=fixed;pointerEvents=1;shape=mxgraph.aws4.iot_thing_freertos_device;" vertex="1" parent="1">
+          <mxGeometry x="667" y="390" width="70" height="70" as="geometry" />
+        </mxCell>
+        <mxCell id="JDmTQxFqgsX-q80Pc-4e-10" value="&lt;div style=&quot;font-size: 14px;&quot;&gt;&lt;font style=&quot;font-family: Verdana; font-size: 14px; color: rgb(255, 255, 255);&quot;&gt;Monitoring and&lt;/font&gt;&lt;/div&gt;&lt;div style=&quot;font-size: 14px;&quot;&gt;&lt;font style=&quot;font-family: Verdana; font-size: 14px; color: rgb(255, 255, 255);&quot;&gt;Observability&lt;br&gt;&lt;/font&gt;&lt;/div&gt;" style="verticalLabelPosition=bottom;sketch=0;html=1;verticalAlign=top;align=center;points=[[0,0.5,0],[0.125,0.25,0],[0.25,0,0],[0.5,0,0],[0.75,0,0],[0.875,0.25,0],[1,0.5,0],[0.875,0.75,0],[0.75,1,0],[0.5,1,0],[0.125,0.75,0]];pointerEvents=1;shape=mxgraph.cisco_safe.compositeIcon;bgIcon=mxgraph.cisco_safe.design.blank_device;resIcon=mxgraph.cisco_safe.design.monitoring;strokeColor=#2AAD7E;" vertex="1" parent="1">
+          <mxGeometry x="310" y="332.5" width="60" height="55" as="geometry" />
+        </mxCell>
+        <mxCell id="JDmTQxFqgsX-q80Pc-4e-11" value="&lt;font style=&quot;font-size: 14px; font-family: Verdana;&quot;&gt;CI/CD&lt;/font&gt;" style="verticalLabelPosition=bottom;sketch=0;html=1;verticalAlign=top;pointerEvents=1;align=center;shape=mxgraph.cisco_safe.security_icons.virtual_deployment;fontColor=#FFFFFF;" vertex="1" parent="1">
+          <mxGeometry x="370" y="460" width="60" height="54.5" as="geometry" />
+        </mxCell>
+        <mxCell id="JDmTQxFqgsX-q80Pc-4e-13" value="&lt;font style=&quot;font-size: 14px; font-family: Verdana; color: rgb(255, 255, 255);&quot;&gt;Evaluation&lt;/font&gt;" style="sketch=0;outlineConnect=0;dashed=0;verticalLabelPosition=bottom;verticalAlign=top;align=center;html=1;fontSize=12;fontStyle=0;aspect=fixed;pointerEvents=1;shape=mxgraph.aws4.sagemaker_shadow_testing;" vertex="1" parent="1">
+          <mxGeometry x="580" y="490" width="60" height="60" as="geometry" />
+        </mxCell>
+        <mxCell id="JDmTQxFqgsX-q80Pc-4e-17" value="" style="shape=step;perimeter=stepPerimeter;whiteSpace=wrap;html=1;fixedSize=1;rotation=-60;size=51.40000000000009;strokeColor=#FFFFFF;" vertex="1" parent="1">
+          <mxGeometry x="320" y="174" width="122.81" height="133.8" as="geometry" />
+        </mxCell>
+        <mxCell id="70666756c84bb93c-2" value="&lt;div&gt;Challenges in&lt;br&gt;&lt;/div&gt;&lt;div&gt;LLMOps&lt;br&gt;&lt;/div&gt;" style="ellipse;whiteSpace=wrap;html=1;rounded=0;shadow=0;comic=0;strokeWidth=1;fontFamily=Verdana;fontSize=22;strokeColor=default;fillColor=#ffffff;fontColor=#000000;fontStyle=1" parent="1" vertex="1">
+          <mxGeometry x="400" y="230" width="260" height="260" as="geometry" />
+        </mxCell>
+        <mxCell id="JDmTQxFqgsX-q80Pc-4e-19" value="" style="ellipse;whiteSpace=wrap;html=1;rounded=0;shadow=0;comic=0;strokeWidth=1;fontFamily=Verdana;fontSize=14;strokeColor=default;fillColor=none;" vertex="1" parent="1">
+          <mxGeometry x="280" y="110" width="500" height="500" as="geometry" />
+        </mxCell>
+        <mxCell id="JDmTQxFqgsX-q80Pc-4e-33" value="&lt;div style=&quot;text-align: left;&quot;&gt;&lt;br&gt;&lt;/div&gt;&lt;div style=&quot;text-align: center;&quot;&gt;&lt;br&gt;&lt;/div&gt;&lt;div style=&quot;text-align: center; font-size: 14px;&quot;&gt;&lt;font style=&quot;font-size: 14px; font-family: Verdana; color: rgb(255, 255, 255);&quot;&gt;&lt;br&gt;&lt;/font&gt;&lt;/div&gt;&lt;div style=&quot;text-align: center; font-size: 14px;&quot;&gt;&lt;font style=&quot;font-size: 14px; font-family: Verdana; color: rgb(255, 255, 255);&quot;&gt;&lt;br&gt;&lt;/font&gt;&lt;/div&gt;&lt;div style=&quot;text-align: center; font-size: 14px;&quot;&gt;&lt;font style=&quot;font-size: 14px; font-family: Verdana; color: rgb(255, 255, 255);&quot;&gt;&lt;br&gt;&lt;/font&gt;&lt;/div&gt;&lt;div style=&quot;text-align: center; font-size: 14px;&quot;&gt;&lt;font style=&quot;font-size: 14px; font-family: Verdana; color: rgb(255, 255, 255);&quot;&gt;&lt;br&gt;&lt;/font&gt;&lt;/div&gt;&lt;div style=&quot;text-align: center; font-size: 14px;&quot;&gt;&lt;font style=&quot;font-size: 14px; font-family: Verdana; color: rgb(255, 255, 255);&quot;&gt;Human&lt;/font&gt;&lt;/div&gt;&lt;div style=&quot;text-align: center; font-size: 14px;&quot;&gt;&lt;font style=&quot;font-size: 14px; font-family: Verdana; color: rgb(255, 255, 255);&quot;&gt;Feedback&lt;br&gt;&lt;/font&gt;&lt;/div&gt;" style="shape=actor;whiteSpace=wrap;html=1;strokeColor=#2AAD7E;align=left;" vertex="1" parent="1">
+          <mxGeometry x="470" y="500" width="47.19" height="60" as="geometry" />
+        </mxCell>
+      </root>
+    </mxGraphModel>
+  </diagram>
+</mxfile>

docs/learn/architecture.mdx

@@ -0,0 +1,34 @@
+---
+title: "System Architecture"
+description: "An overview of the architecture of Agenta"
+---
+
+<Frame caption="Agenta System Architecture">
+<img width="700" src="images/apps_and_configurations_light.png" />
+</Frame>
+
+## Concepts behind agenta
+
+The main concept behind agenta is the separation of the application logic and configuration. The application logic refers to the code that defines the application, whether it's a simple prompt, chain of prompts, RAG, etc. The configuration refers to the parameters that define the application logic, such as the prompt, model, chunk size, etc.
+
+By separating the application logic from the configuration. We simplify the process of iterating on application variants, and enable the collaboration between developers (working from code/cli) and non-developers (working from the UI).
+
+## Architecture Components
+
+### The Application
+
+The application describes the logic written in Python code. An application can be created from a pre-built template in the UI or from code in the CLI. In either case, a new container with the application code is launched. The application can then be accessed via a REST API.
+
+Each application has a default configuration specified in its code. This default configuration can be overridden by the user in the UI or the CLI. Additionally, the user can create new configurations from the UI or the CLI. Each new configuration results in the creation of a new application variant, which is a combination of the application logic and a configuration. A single project can house many variants encompassing multiple application logics and configurations.
+
+## The Backend
+Agenta's backend manages applications and configurations. It is responsible for building images, deploying containers, and managing configurations and prompts for the application.
+
+## The Frontend / UI
+The frontend provides tools to create new applications from a template, create and edit configurations, run evaluations, and deploy applications to different environments (e.g., staging, production, etc.).
+
+## The CLI
+The CLI offers the same capabilities as the frontend. Additionally, it allows for the creation of custom applications not available as templates. When serving a new application from the CLI, Agenta handles container creation and deployment. After creating a new application, users can edit its configuration and evaluate it in the UI.
+
+## The SDK
+The SDK is a Python library used to create new applications from code. It manages the saving of the default configuration, creation of the REST API, and necessary actions to create a playground and integrate the application with the Agenta platform.

docs/mint.json

@@ -56,9 +56,10 @@
   ],
   "navigation": [
     {
-      "group": "Quickstart",
+      "group": "Getting Started",
       "pages": [
         "quickstart/introduction",
+        "quickstart/how-agenta-works",
         "quickstart/getting-started-ui",
         "quickstart/getting-started-code"
       ]
@@ -82,9 +83,8 @@
       "group": "Learn",
       "pages": [
         "learn/the_llmops_workflow",
-        "conceptual/evaluating_llm_apps",
-        "conceptual/concepts",
-        "conceptual/architecture"
+        "learn/evaluating_llm_apps",
+        "learn/architecture"
       ]
     },
     {

docs/quickstart/getting-started-ui.mdx

@@ -1,5 +1,5 @@
 ---
-title: 'Getting Started'
+title: 'Quick Start'
 description: 'Create and deploy your first LLM app in one minute'
 ---
 

docs/quickstart/how-agenta-works.mdx

@@ -0,0 +1,68 @@
+---
+title: 'How does Agenta Work?'
+description: 'Understand the main concepts behind agenta'
+---
+
+## What problems does agenta solve?
+
+Building Large Language Model powered applications (LLM apps) is challenging because these models are hard to control. The most effective way to build LLM apps is to continuously iterate on the prompts, models, parameters, and application logic until finding a robust solution.
+
+This process can be difficult to manage, especially when collaborating in teams, due to several factors:
+1. **Version management**: Keeping track of different versions of prompts, models, and application logic is a difficult task. Especially, since **mixing the application logic with the prompt** creates a disoragnized mix.
+
+2. **Collaboration Challenges**:  Non-developers, who often have insights into creating effective prompts and assessing the outputs, cannot contribute due to the technical barrier.
+
+3. **Organization Issues**: It's challenging to organize different versions, evaluations, and test sets, and human labeling. Especially, when dozens of LLM apps and use cases are in production.
+
+<Frame>
+<img className="dark:hidden" width="500" src="images/problems.light.png" />
+<img className="hidden dark:block" width="500" src="images/problems.dark.png" />
+</Frame>
+
+## How agenta addresses these problems?
+
+Agenta offers an integrated solution for developing LLM applications. It simplifies the development process by:
+
+1. **Version Control**: It makes managing different application variants easier, enhancing the ability to test, compare, and **optimize accuracy** and **reduce latency**.
+2. **Collaboration**: It facilitates collaboration between developers (working on code) and domain experts (working in the UI).
+3. **Evaluation and Deployment Tools**: It supplies all the tools necessary to evaluate and deploy the LLM application to production.
+
+## How does agenta work?
+
+<Frame caption="Agenta decouples the configuration (prompt, model) from the application logic. The configuration is managed by the backend. You can test different configurations both from the UI and the CLI. ">
+<img width="700" src="images/apps_and_configurations_light.png" />
+</Frame>
+<Tip>For more details about our architecture, you can refer to [System Architecture](/learn/architecture).</Tip>
+Agenta's framework is based on three core concepts:
+
+1. **Application Variant**: Building a robust AI application requires iterating on different prompts, models, and architectures. We refer to each of these different iterations as an `application variant`.
+2. **Separation of Logic and Configuration**: Agenta separates the application logic (one prompt, chain-of-prompts, RAG, ..) from the configuration (the prompt, model..). 
+3. **Microservice-based Applications**: Agenta treats each application as a microservice. Creating a new application in Agenta automatically generates a container with a REST API. This is true whether the application is created using a pre-built template from the UI, or whether from CLI using custom application code. Agenta handles the creation of Docker images and container deployment. This means that all interactions with the application (either from the UI, during evaluations, or post-deployment) occur with the container.
+
+
+## How to use agenta?
+
+<Steps>
+  <Step title="Create an application">
+    You can create an application using a pre-built template [directly from the UI](/quickstart/getting-started-code) or by writing [custom code](/quickstart/getting-started-code) and serving it using the CLI.
+  </Step>
+  <Step title="Experiment and add more variants">
+Next, visit the playground to experiment with different configurations, prompts, and models. Directly observe the effects of changes or compare different variants side by side. 
+You can also experiment with different architectures and application logics by adding more custom applications to the same project using the CLI (for instance, if you want to compare using RAG to single prompts).
+  </Step>
+  <Step title="Create a golden set">
+A golden set is a collection of input for your LLM app with expected outputs. Create a golden set in one of the following ways:
+
+1. Upload a CSV or JSON, or manually fill the set using our UI.
+2. Iteratively work in the playground (by interacting with the LLM application and adding edge cases to the golden set).
+3. (In the future) From the logs of your deployed application.
+  </Step> 
+
+  <Step title="Evaluate">
+Run evaluations on your golden set (or for some evaluators with an unlabeled set) from the UI or the CLI. You can also batch run and compare the results for different variants side by side.
+  </Step> 
+  <Step title="Deploy to Production">
+Finally, deploy the best performing LLM applications to different production environments (staging, production) from the UI in the playground.
+  </Step>
+</Steps>
+