+
+
+
+Cognee ist eine auf Entwickler ausgerichtete Plattform, die die Entwicklung von KI-Anwendungen mit skalierbaren, modularen ECL-Pipelines (Extract, Cognify, Load) rationalisiert. Durch die nahtlose Integration mit Milvus ermöglicht Cognee eine effiziente Verbindung und Abfrage von Gesprächen, Dokumenten und Transkriptionen, wodurch Halluzinationen reduziert und Betriebskosten optimiert werden.
+Durch die Unterstützung von Vektorspeichern wie Milvus, Graphdatenbanken und LLMs bietet Cognee ein flexibles und anpassbares Framework für den Aufbau von RAG-Systemen (Retrieval-Augmented Generation). Seine produktionsreife Architektur gewährleistet eine verbesserte Genauigkeit und Effizienz für KI-gestützte Anwendungen.
+In diesem Tutorial zeigen wir Ihnen, wie Sie eine RAG-Pipeline (Retrieval-Augmented Generation) mit Milvus und Cognee erstellen.
+$ pip install pymilvus git+https://github.com/topoteretes/cognee.git
+++Wenn Sie Google Colab verwenden, müssen Sie möglicherweise die Runtime neu starten, um die soeben installierten Abhängigkeiten zu aktivieren (klicken Sie auf das Menü "Runtime" am oberen Bildschirmrand und wählen Sie "Restart session" aus dem Dropdown-Menü).
+
Standardmäßig wird in diesem Beispiel OpenAI als LLM verwendet. Sie sollten den Api-Schlüssel vorbereiten und ihn in der Funktion config set_llm_api_key() einstellen.
Um Milvus als Vektordatenbank zu konfigurieren, setzen Sie VECTOR_DB_PROVIDER auf milvus und geben Sie VECTOR_DB_URL und VECTOR_DB_KEY an. Da wir in dieser Demo Milvus Lite zum Speichern von Daten verwenden, muss nur die VECTOR_DB_URL angegeben werden.
import os
+
+import cognee
+
+cognee.config.set_llm_api_key("YOUR_OPENAI_API_KEY")
+
+
+os.environ["VECTOR_DB_PROVIDER"] = "milvus"
+os.environ["VECTOR_DB_URL"] = "./milvus.db"
+Was die Umgebungsvariablen VECTOR_DB_URL und VECTOR_DB_KEY betrifft:
VECTOR_DB_URL als lokale Datei, z. B../milvus.db, ist die bequemste Methode, da sie automatisch Milvus Lite nutzt, um alle Daten in dieser Datei zu speichern.http://localhost:19530, als VECTOR_DB_URL.VECTOR_DB_URL und VECTOR_DB_KEY an, die dem öffentlichen Endpunkt und dem Api-Schlüssel in Zilliz Cloud entsprechen.Wir verwenden die FAQ-Seiten aus der Milvus-Dokumentation 2.4.x als privates Wissen in unserer RAG, was eine gute Datenquelle für eine einfache RAG-Pipeline ist.
+Laden Sie die Zip-Datei herunter und entpacken Sie die Dokumente in den Ordner milvus_docs.
$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
+Wir laden alle Markdown-Dateien aus dem Ordner milvus_docs/en/faq. Für jedes Dokument verwenden wir einfach "# ", um den Inhalt in der Datei zu trennen, wodurch der Inhalt jedes Hauptteils der Markdown-Datei grob getrennt werden kann.
from glob import glob
+
+text_lines = []
+
+for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
+ with open(file_path, "r") as file:
+ file_text = file.read()
+
+ text_lines += file_text.split("# ")
+await cognee.prune.prune_data()
+await cognee.prune.prune_system(metadata=True)
+Jetzt können wir unseren Datensatz hinzufügen und ihn zu einem Wissensgraphen verarbeiten.
+await cognee.add(data=text_lines, dataset_name="milvus_faq")
+await cognee.cognify()
+
+# [DocumentChunk(id=UUID('6889e7ef-3670-555c-bb16-3eb50d1d30b0'), updated_at=datetime.datetime(2024, 12, 4, 6, 29, 46, 472907, tzinfo=datetime.timezone.utc), text='Does the query perform in memory? What are incremental data and historical data?\n\nYes. When ...
+# ...
+Die Methode add lädt den Datensatz (Milvus FAQs) in Cognee und die Methode cognify verarbeitet die Daten, um Entitäten, Beziehungen und Zusammenfassungen zu extrahieren und einen Wissensgraphen zu erstellen.
Nun, da die Daten verarbeitet wurden, können wir den Wissensgraphen abfragen.
+from cognee.api.v1.search import SearchType
+
+query_text = "How is data stored in milvus?"
+search_results = await cognee.search(SearchType.SUMMARIES, query_text=query_text)
+
+print(search_results[0])
+{'id': 'de5c6713-e079-5d0b-b11d-e9bacd1e0d73', 'text': 'Milvus stores two data types: inserted data and metadata.'}
+Diese Abfrage durchsucht den Wissensgraphen nach einer Zusammenfassung, die mit dem Abfragetext in Beziehung steht, und der am meisten in Beziehung stehende Kandidat wird gedruckt.
+Zusammenfassungen bieten Einblicke auf hoher Ebene, aber für detailliertere Informationen können wir bestimmte Datenpakete direkt aus dem verarbeiteten Datensatz abfragen. Diese Chunks werden aus den ursprünglichen Daten abgeleitet, die während der Erstellung des Wissensgraphen hinzugefügt und analysiert wurden.
+from cognee.api.v1.search import SearchType
+
+query_text = "How is data stored in milvus?"
+search_results = await cognee.search(SearchType.CHUNKS, query_text=query_text)
+Formatieren und zeigen wir sie zur besseren Lesbarkeit an!
+def format_and_print(data):
+ print("ID:", data["id"])
+ print("\nText:\n")
+ paragraphs = data["text"].split("\n\n")
+ for paragraph in paragraphs:
+ print(paragraph.strip())
+ print()
+
+
+format_and_print(search_results[0])
+ID: 4be01c4b-9ee5-541c-9b85-297883934ab3
+
+Text:
+
+Where does Milvus store data?
+
+Milvus deals with two types of data, inserted data and metadata.
+
+Inserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).
+
+Metadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.
+
+###
+In den vorangegangenen Schritten haben wir den Milvus-FAQ-Datensatz sowohl nach Zusammenfassungen als auch nach bestimmten Datenpaketen abgefragt. Dies lieferte zwar detaillierte Einblicke und granulare Informationen, aber der Datensatz war groß, so dass es schwierig war, die Abhängigkeiten innerhalb des Wissensgraphen klar zu visualisieren.
+Um dieses Problem zu lösen, werden wir die Cognee-Umgebung zurücksetzen und mit einem kleineren, gezielteren Datensatz arbeiten. Dadurch können wir die Beziehungen und Abhängigkeiten, die während des Cognify-Prozesses extrahiert wurden, besser darstellen. Durch die Vereinfachung der Daten können wir klar erkennen, wie Cognee die Informationen im Wissensgraphen organisiert und strukturiert.
+await cognee.prune.prune_data()
+await cognee.prune.prune_system(metadata=True)
+Hier wird ein kleinerer Datensatz mit nur einer Textzeile hinzugefügt und verarbeitet, um einen fokussierten und leicht interpretierbaren Wissensgraphen zu erhalten.
+# We only use one line of text as the dataset, which simplifies the output later
+text = """
+ Natural language processing (NLP) is an interdisciplinary
+ subfield of computer science and information retrieval.
+ """
+
+await cognee.add(text)
+await cognee.cognify()
+Durch die Fokussierung auf diesen kleineren Datensatz können wir nun die Beziehungen und Strukturen innerhalb des Wissensgraphen klar analysieren.
+query_text = "Tell me about NLP"
+search_results = await cognee.search(SearchType.INSIGHTS, query_text=query_text)
+
+for result_text in search_results:
+ print(result_text)
+
+# Example output:
+# ({'id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'natural language processing', 'description': 'An interdisciplinary subfield of computer science and information retrieval.'}, {'relationship_name': 'is_a_subfield_of', 'source_node_id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'target_node_id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 15, 473137, tzinfo=datetime.timezone.utc)}, {'id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'computer science', 'description': 'The study of computation and information processing.'})
+# (...)
+#
+# It represents nodes and relationships in the knowledge graph:
+# - The first element is the source node (e.g., 'natural language processing').
+# - The second element is the relationship between nodes (e.g., 'is_a_subfield_of').
+# - The third element is the target node (e.g., 'computer science').
+Diese Ausgabe stellt die Ergebnisse einer Wissensgraphenabfrage dar und zeigt die Entitäten (Knoten) und ihre Beziehungen (Kanten), wie sie aus dem verarbeiteten Datensatz extrahiert wurden. Jedes Tupel enthält eine Quell-Entität, einen Beziehungstyp und eine Ziel-Entität, zusammen mit Metadaten wie eindeutigen IDs, Beschreibungen und Zeitstempeln. Der Graph hebt die wichtigsten Konzepte und ihre semantischen Verbindungen hervor und bietet ein strukturiertes Verständnis des Datensatzes.
+Herzlichen Glückwunsch, Sie haben die grundlegende Verwendung von cognee mit Milvus gelernt. Wenn Sie mehr über die fortgeschrittene Nutzung von cognee erfahren möchten, besuchen Sie bitte die offizielle Seite.
diff --git a/localization/v2.4.x/site/de/integrations/build_RAG_with_milvus_and_gemini.json b/localization/v2.4.x/site/de/integrations/build_RAG_with_milvus_and_gemini.json new file mode 100644 index 000000000..41c010a0f --- /dev/null +++ b/localization/v2.4.x/site/de/integrations/build_RAG_with_milvus_and_gemini.json @@ -0,0 +1 @@ +{"codeList":["$ pip install --upgrade pymilvus google-generativeai requests tqdm\n","import os\n\nos.environ[\"GEMINI_API_KEY\"] = \"***********\"\n","$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip\n$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs\n","from glob import glob\n\ntext_lines = []\n\nfor file_path in glob(\"milvus_docs/en/faq/*.md\", recursive=True):\n with open(file_path, \"r\") as file:\n file_text = file.read()\n\n text_lines += file_text.split(\"# \")\n","import google.generativeai as genai\n\ngenai.configure(api_key=os.environ[\"GEMINI_API_KEY\"])\n\ngemini_model = genai.GenerativeModel(\"gemini-1.5-flash\")\n\nresponse = gemini_model.generate_content(\"who are you\")\nprint(response.text)\n","test_embeddings = genai.embed_content(\n model=\"models/text-embedding-004\", content=[\"This is a test1\", \"This is a test2\"]\n)[\"embedding\"]\n\nembedding_dim = len(test_embeddings[0])\nprint(embedding_dim)\nprint(test_embeddings[0][:10])\n","from pymilvus import MilvusClient\n\nmilvus_client = MilvusClient(uri=\"./milvus_demo.db\")\n\ncollection_name = \"my_rag_collection\"\n","if milvus_client.has_collection(collection_name):\n milvus_client.drop_collection(collection_name)\n","milvus_client.create_collection(\n collection_name=collection_name,\n dimension=embedding_dim,\n metric_type=\"IP\", # Inner product distance\n consistency_level=\"Strong\", # Strong consistency level\n)\n","from tqdm import tqdm\n\ndata = []\n\ndoc_embeddings = genai.embed_content(\n model=\"models/text-embedding-004\", content=text_lines\n)[\"embedding\"]\n\nfor i, line in enumerate(tqdm(text_lines, desc=\"Creating embeddings\")):\n data.append({\"id\": i, \"vector\": doc_embeddings[i], \"text\": line})\n\nmilvus_client.insert(collection_name=collection_name, data=data)\n","question = \"How is data stored in milvus?\"\n","question_embedding = genai.embed_content(\n model=\"models/text-embedding-004\", content=question\n)[\"embedding\"]\n\nsearch_res = milvus_client.search(\n collection_name=collection_name,\n data=[question_embedding],\n limit=3, # Return top 3 results\n search_params={\"metric_type\": \"IP\", \"params\": {}}, # Inner product distance\n output_fields=[\"text\"], # Return the text field\n)\n","import json\n\nretrieved_lines_with_distances = [\n (res[\"entity\"][\"text\"], res[\"distance\"]) for res in search_res[0]\n]\nprint(json.dumps(retrieved_lines_with_distances, indent=4))\n","context = \"\\n\".join(\n [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]\n)\n","SYSTEM_PROMPT = \"\"\"\nHuman: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.\n\"\"\"\nUSER_PROMPT = f\"\"\"\nUse the following pieces of information enclosed inDie Gemini-API und Google AI Studio helfen Ihnen, mit den neuesten Modellen von Google zu arbeiten und Ihre Ideen in skalierbare Anwendungen umzusetzen. Gemini bietet Zugang zu leistungsstarken Sprachmodellen wie Gemini-1.5-Flash, Gemini-1.5-Flash-8B und Gemini-1.5-Pro für Aufgaben wie Texterstellung, Dokumentenverarbeitung, Bildverarbeitung, Audioanalyse und mehr. Die API ermöglicht die Eingabe langer Kontexte mit Millionen von Token, die Feinabstimmung von Modellen für bestimmte Aufgaben, die Erzeugung strukturierter Ausgaben wie JSON und die Nutzung von Funktionen wie semantisches Retrieval und Codeausführung.
In diesem Tutorial zeigen wir Ihnen, wie Sie eine RAG-Pipeline (Retrieval-Augmented Generation) mit Milvus und Gemini aufbauen. Wir werden das Gemini-Modell verwenden, um Text auf der Grundlage einer gegebenen Anfrage zu generieren. Wir werden auch Milvus verwenden, um den generierten Text zu speichern und abzurufen.
+$ pip install --upgrade pymilvus google-generativeai requests tqdm
+Wenn Sie Google Colab verwenden, müssen Sie möglicherweise die Runtime neu starten, um die soeben installierten Abhängigkeiten zu aktivieren (klicken Sie auf das Menü "Runtime" am oberen Rand des Bildschirms und wählen Sie "Restart session" aus dem Dropdown-Menü).
+Sie sollten sich zunächst bei der Google AI Studio-Plattform anmelden und den api-Schlüssel GEMINI_API_KEY als Umgebungsvariable vorbereiten.
import os
+
+os.environ["GEMINI_API_KEY"] = "***********"
+Wir verwenden die FAQ-Seiten aus der Milvus-Dokumentation 2.4.x als privates Wissen in unserem RAG, was eine gute Datenquelle für eine einfache RAG-Pipeline ist.
+Laden Sie die Zip-Datei herunter und entpacken Sie die Dokumente in den Ordner milvus_docs.
$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
+Wir laden alle Markdown-Dateien aus dem Ordner milvus_docs/en/faq. Für jedes Dokument verwenden wir einfach "# ", um den Inhalt in der Datei zu trennen, wodurch der Inhalt jedes Hauptteils der Markdown-Datei grob getrennt werden kann.
from glob import glob
+
+text_lines = []
+
+for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
+ with open(file_path, "r") as file:
+ file_text = file.read()
+
+ text_lines += file_text.split("# ")
+Wir verwenden die gemini-1.5-flash als LLM und die text-embedding-004 als Einbettungsmodell.
Lassen Sie uns versuchen, eine Testantwort aus der LLM zu generieren:
+import google.generativeai as genai
+
+genai.configure(api_key=os.environ["GEMINI_API_KEY"])
+
+gemini_model = genai.GenerativeModel("gemini-1.5-flash")
+
+response = gemini_model.generate_content("who are you")
+print(response.text)
+I am a large language model, trained by Google. I am an AI and don't have a personal identity or consciousness. My purpose is to process information and respond to a wide range of prompts and questions in a helpful and informative way.
+Erzeugen Sie eine Testeinbettung und drucken Sie ihre Dimension und die ersten Elemente aus.
+test_embeddings = genai.embed_content(
+ model="models/text-embedding-004", content=["This is a test1", "This is a test2"]
+)["embedding"]
+
+embedding_dim = len(test_embeddings[0])
+print(embedding_dim)
+print(test_embeddings[0][:10])
+768
+[0.013588584, -0.004361838, -0.08481652, -0.039724775, 0.04723794, -0.0051557426, 0.026071774, 0.045514572, -0.016867816, 0.039378334]
+from pymilvus import MilvusClient
+
+milvus_client = MilvusClient(uri="./milvus_demo.db")
+
+collection_name = "my_rag_collection"
+Wie für das Argument von MilvusClient:
uri als lokale Datei, z. B../milvus.db, ist die bequemste Methode, da sie automatisch Milvus Lite nutzt, um alle Daten in dieser Datei zu speichern.http://localhost:19530, als uri.uri und token an, die dem öffentlichen Endpunkt und dem Api-Schlüssel in Zilliz Cloud entsprechen.Prüfen Sie, ob die Sammlung bereits existiert und löschen Sie sie, wenn dies der Fall ist.
+if milvus_client.has_collection(collection_name):
+ milvus_client.drop_collection(collection_name)
+Erstellen Sie eine neue Sammlung mit den angegebenen Parametern.
+Wenn wir keine Feldinformationen angeben, erstellt Milvus automatisch ein Standardfeld id für den Primärschlüssel und ein Feld vector zum Speichern der Vektordaten. Ein reserviertes JSON-Feld wird verwendet, um nicht schema-definierte Felder und ihre Werte zu speichern.
milvus_client.create_collection(
+ collection_name=collection_name,
+ dimension=embedding_dim,
+ metric_type="IP", # Inner product distance
+ consistency_level="Strong", # Strong consistency level
+)
+Iterieren Sie durch die Textzeilen, erstellen Sie Einbettungen und fügen Sie dann die Daten in Milvus ein.
+Hier ist ein neues Feld text, das ein nicht definiertes Feld im Sammelschema ist. Es wird automatisch dem reservierten dynamischen JSON-Feld hinzugefügt, das auf hoher Ebene wie ein normales Feld behandelt werden kann.
from tqdm import tqdm
+
+data = []
+
+doc_embeddings = genai.embed_content(
+ model="models/text-embedding-004", content=text_lines
+)["embedding"]
+
+for i, line in enumerate(tqdm(text_lines, desc="Creating embeddings")):
+ data.append({"id": i, "vector": doc_embeddings[i], "text": line})
+
+milvus_client.insert(collection_name=collection_name, data=data)
+Creating embeddings: 100%|██████████| 72/72 [00:00<00:00, 468201.38it/s]
+
+
+
+
+
+{'insert_count': 72, 'ids': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71], 'cost': 0}
+Lassen Sie uns eine häufige Frage über Milvus angeben.
+question = "How is data stored in milvus?"
+Suchen Sie nach der Frage in der Sammlung und rufen Sie die semantischen Top-3-Treffer ab.
+question_embedding = genai.embed_content(
+ model="models/text-embedding-004", content=question
+)["embedding"]
+
+search_res = milvus_client.search(
+ collection_name=collection_name,
+ data=[question_embedding],
+ limit=3, # Return top 3 results
+ search_params={"metric_type": "IP", "params": {}}, # Inner product distance
+ output_fields=["text"], # Return the text field
+)
+Werfen wir einen Blick auf die Suchergebnisse der Abfrage
+import json
+
+retrieved_lines_with_distances = [
+ (res["entity"]["text"], res["distance"]) for res in search_res[0]
+]
+print(json.dumps(retrieved_lines_with_distances, indent=4))
+[
+ [
+ " Where does Milvus store data?\n\nMilvus deals with two types of data, inserted data and metadata. \n\nInserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).\n\nMetadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.\n\n###",
+ 0.8048275113105774
+ ],
+ [
+ "Does the query perform in memory? What are incremental data and historical data?\n\nYes. When a query request comes, Milvus searches both incremental data and historical data by loading them into memory. Incremental data are in the growing segments, which are buffered in memory before they reach the threshold to be persisted in storage engine, while historical data are from the sealed segments that are stored in the object storage. Incremental data and historical data together constitute the whole dataset to search.\n\n###",
+ 0.7574886679649353
+ ],
+ [
+ "What is the maximum dataset size Milvus can handle?\n\n \nTheoretically, the maximum dataset size Milvus can handle is determined by the hardware it is run on, specifically system memory and storage:\n\n- Milvus loads all specified collections and partitions into memory before running queries. Therefore, memory size determines the maximum amount of data Milvus can query.\n- When new entities and and collection-related schema (currently only MinIO is supported for data persistence) are added to Milvus, system storage determines the maximum allowable size of inserted data.\n\n###",
+ 0.7453608512878418
+ ]
+]
+Konvertieren Sie die abgerufenen Dokumente in ein String-Format.
+context = "\n".join(
+ [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]
+)
+Definieren Sie System- und Benutzer-Prompts für das Lanage Model. Diese Eingabeaufforderung wird mit den abgerufenen Dokumenten aus Milvus zusammengestellt.
+SYSTEM_PROMPT = """
+Human: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.
+"""
+USER_PROMPT = f"""
+Use the following pieces of information enclosed in <context> tags to provide an answer to the question enclosed in <question> tags.
+<context>
+{context}
+</context>
+<question>
+{question}
+</question>
+"""
+Verwenden Sie den Gemini, um eine Antwort auf der Grundlage der Prompts zu generieren.
+gemini_model = genai.GenerativeModel(
+ "gemini-1.5-flash", system_instruction=SYSTEM_PROMPT
+)
+response = gemini_model.generate_content(USER_PROMPT)
+print(response.text)
+Milvus stores data in two ways: Inserted data (vector data, scalar data, and collection-specific schema) is stored as an incremental log in persistent storage using object storage backends such as MinIO, AWS S3, Google Cloud Storage, Azure Blob Storage, Alibaba Cloud OSS, and Tencent Cloud Object Storage. Metadata, generated by each Milvus module, is stored in etcd.
+Großartig! Wir haben erfolgreich eine RAG-Pipeline mit Milvus und Gemini aufgebaut.
diff --git a/localization/v2.4.x/site/de/integrations/build_RAG_with_milvus_and_ollama.json b/localization/v2.4.x/site/de/integrations/build_RAG_with_milvus_and_ollama.json new file mode 100644 index 000000000..b754f202c --- /dev/null +++ b/localization/v2.4.x/site/de/integrations/build_RAG_with_milvus_and_ollama.json @@ -0,0 +1 @@ +{"codeList":["$ pip install pymilvus ollama\n","$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip\n$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs\n","from glob import glob\n\ntext_lines = []\n\nfor file_path in glob(\"milvus_docs/en/faq/*.md\", recursive=True):\n with open(file_path, \"r\") as file:\n file_text = file.read()\n\n text_lines += file_text.split(\"# \")\n","! ollama pull mxbai-embed-large\n","! ollama pull llama3.2\n","import ollama\n\n\ndef emb_text(text):\n response = ollama.embeddings(model=\"mxbai-embed-large\", prompt=text)\n return response[\"embedding\"]\n","test_embedding = emb_text(\"This is a test\")\nembedding_dim = len(test_embedding)\nprint(embedding_dim)\nprint(test_embedding[:10])\n","from pymilvus import MilvusClient\n\nmilvus_client = MilvusClient(uri=\"./milvus_demo.db\")\n\ncollection_name = \"my_rag_collection\"\n","if milvus_client.has_collection(collection_name):\n milvus_client.drop_collection(collection_name)\n","milvus_client.create_collection(\n collection_name=collection_name,\n dimension=embedding_dim,\n metric_type=\"IP\", # Inner product distance\n consistency_level=\"Strong\", # Strong consistency level\n)\n","from tqdm import tqdm\n\ndata = []\n\nfor i, line in enumerate(tqdm(text_lines, desc=\"Creating embeddings\")):\n data.append({\"id\": i, \"vector\": emb_text(line), \"text\": line})\n\nmilvus_client.insert(collection_name=collection_name, data=data)\n","question = \"How is data stored in milvus?\"\n","search_res = milvus_client.search(\n collection_name=collection_name,\n data=[\n emb_text(question)\n ], # Use the `emb_text` function to convert the question to an embedding vector\n limit=3, # Return top 3 results\n search_params={\"metric_type\": \"IP\", \"params\": {}}, # Inner product distance\n output_fields=[\"text\"], # Return the text field\n)\n","import json\n\nretrieved_lines_with_distances = [\n (res[\"entity\"][\"text\"], res[\"distance\"]) for res in search_res[0]\n]\nprint(json.dumps(retrieved_lines_with_distances, indent=4))\n","context = \"\\n\".join(\n [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]\n)\n","SYSTEM_PROMPT = \"\"\"\nHuman: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.\n\"\"\"\nUSER_PROMPT = f\"\"\"\nUse the following pieces of information enclosed inOllama ist eine Open-Source-Plattform, die die lokale Ausführung und Anpassung großer Sprachmodelle (LLMs) vereinfacht. Sie bietet eine benutzerfreundliche, Cloud-freie Erfahrung, die mühelose Modell-Downloads, Installationen und Interaktionen ermöglicht, ohne dass fortgeschrittene technische Kenntnisse erforderlich sind. Mit einer wachsenden Bibliothek von vortrainierten LLMs - von allgemeinen bis hin zu domänenspezifischen - macht es Ollama einfach, Modelle für verschiedene Anwendungen zu verwalten und anzupassen. Es gewährleistet den Datenschutz und die Flexibilität und ermöglicht es den Nutzern, KI-gesteuerte Lösungen vollständig auf ihren Rechnern zu optimieren und einzusetzen.
+In diesem Leitfaden zeigen wir Ihnen, wie Sie Ollama und Milvus nutzen können, um eine RAG-Pipeline (Retrieval-Augmented Generation) effizient und sicher aufzubauen.
+$ pip install pymilvus ollama
+Wenn Sie Google Colab verwenden, müssen Sie möglicherweise die Runtime neu starten, um die soeben installierten Abhängigkeiten zu aktivieren (klicken Sie auf das Menü "Runtime" am oberen Rand des Bildschirms und wählen Sie "Sitzung neu starten" aus dem Dropdown-Menü).
+Wir verwenden die FAQ-Seiten aus der Milvus-Dokumentation 2.4.x als privates Wissen in unserem RAG, was eine gute Datenquelle für eine einfache RAG-Pipeline ist.
+Laden Sie die Zip-Datei herunter und entpacken Sie die Dokumente in den Ordner milvus_docs.
$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
+--2024-11-26 21:47:19-- https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+Resolving github.com (github.com)... 140.82.112.4
+Connecting to github.com (github.com)|140.82.112.4|:443... connected.
+HTTP request sent, awaiting response... 302 Found
+Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/267273319/c52902a0-e13c-4ca7-92e0-086751098a05?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241127%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241127T024720Z&X-Amz-Expires=300&X-Amz-Signature=7808b77cbdaa7e122196bcd75a73f29f2540333a350c4830bbdf5f286e876304&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dmilvus_docs_2.4.x_en.zip&response-content-type=application%2Foctet-stream [following]
+--2024-11-26 21:47:20-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/267273319/c52902a0-e13c-4ca7-92e0-086751098a05?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241127%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241127T024720Z&X-Amz-Expires=300&X-Amz-Signature=7808b77cbdaa7e122196bcd75a73f29f2540333a350c4830bbdf5f286e876304&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dmilvus_docs_2.4.x_en.zip&response-content-type=application%2Foctet-stream
+Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.109.133, 185.199.111.133, 185.199.108.133, ...
+Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.109.133|:443... connected.
+HTTP request sent, awaiting response... 200 OK
+Length: 613094 (599K) [application/octet-stream]
+Saving to: ‘milvus_docs_2.4.x_en.zip’
+
+milvus_docs_2.4.x_e 100%[===================>] 598.72K 1.20MB/s in 0.5s
+
+2024-11-26 21:47:20 (1.20 MB/s) - ‘milvus_docs_2.4.x_en.zip’ saved [613094/613094]
+Wir laden alle Markdown-Dateien aus dem Ordner milvus_docs/en/faq. Für jedes Dokument verwenden wir einfach "# ", um den Inhalt in der Datei zu trennen, wodurch der Inhalt jedes Hauptteils der Markdown-Datei grob getrennt werden kann.
from glob import glob
+
+text_lines = []
+
+for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
+ with open(file_path, "r") as file:
+ file_text = file.read()
+
+ text_lines += file_text.split("# ")
+Ollama unterstützt mehrere Modelle sowohl für LLM-basierte Aufgaben als auch für die Generierung von Einbettungen, was die Entwicklung von Retrieval-Augmented Generation (RAG)-Anwendungen erleichtert. Für dieses Setup:
+Bevor wir beginnen, stellen wir sicher, dass beide Modelle lokal gezogen werden:
+! ollama pull mxbai-embed-large
+[?25lpulling manifest ⠋ [?25h[?25l[2K[1Gpulling manifest ⠙ [?25h[?25l[2K[1Gpulling manifest ⠹ [?25h[?25l[2K[1Gpulling manifest ⠸ [?25h[?25l[2K[1Gpulling manifest ⠼ [?25h[?25l[2K[1Gpulling manifest ⠴ [?25h[?25l[2K[1Gpulling manifest
+pulling 819c2adf5ce6... 100% ▕████████████████▏ 669 MB
+pulling c71d239df917... 100% ▕████████████████▏ 11 KB
+pulling b837481ff855... 100% ▕████████████████▏ 16 B
+pulling 38badd946f91... 100% ▕████████████████▏ 408 B
+verifying sha256 digest
+writing manifest
+success [?25h
+! ollama pull llama3.2
+[?25lpulling manifest ⠋ [?25h[?25l[2K[1Gpulling manifest ⠙ [?25h[?25l[2K[1Gpulling manifest ⠹ [?25h[?25l[2K[1Gpulling manifest ⠸ [?25h[?25l[2K[1Gpulling manifest ⠼ [?25h[?25l[2K[1Gpulling manifest ⠴ [?25h[?25l[2K[1Gpulling manifest
+pulling dde5aa3fc5ff... 100% ▕████████████████▏ 2.0 GB
+pulling 966de95ca8a6... 100% ▕████████████████▏ 1.4 KB
+pulling fcc5a6bec9da... 100% ▕████████████████▏ 7.7 KB
+pulling a70ff7e570d9... 100% ▕████████████████▏ 6.0 KB
+pulling 56bb8bd477a5... 100% ▕████████████████▏ 96 B
+pulling 34bb5ab01051... 100% ▕████████████████▏ 561 B
+verifying sha256 digest
+writing manifest
+success [?25h
+Wenn diese Modelle bereit sind, können wir mit der Implementierung von LLM-gesteuerten Generierungs- und einbettungsbasierten Retrieval-Workflows fortfahren.
+import ollama
+
+
+def emb_text(text):
+ response = ollama.embeddings(model="mxbai-embed-large", prompt=text)
+ return response["embedding"]
+Erzeugen Sie eine Testeinbettung und geben Sie ihre Dimension und die ersten Elemente aus.
+test_embedding = emb_text("This is a test")
+embedding_dim = len(test_embedding)
+print(embedding_dim)
+print(test_embedding[:10])
+1024
+[0.23276396095752716, 0.4257211685180664, 0.19724100828170776, 0.46120673418045044, -0.46039995551109314, -0.1413791924715042, -0.18261606991291046, -0.07602324336767197, 0.39991313219070435, 0.8337644338607788]
+from pymilvus import MilvusClient
+
+milvus_client = MilvusClient(uri="./milvus_demo.db")
+
+collection_name = "my_rag_collection"
+Wie für das Argument von MilvusClient:
uri als lokale Datei, z. B../milvus.db, ist die bequemste Methode, da Milvus Lite automatisch alle Daten in dieser Datei speichert.http://localhost:19530, als uri.uri und token an, die dem öffentlichen Endpunkt und dem Api-Schlüssel in Zilliz Cloud entsprechen.Prüfen Sie, ob die Sammlung bereits existiert und löschen Sie sie, wenn dies der Fall ist.
+if milvus_client.has_collection(collection_name):
+ milvus_client.drop_collection(collection_name)
+Erstellen Sie eine neue Sammlung mit den angegebenen Parametern.
+Wenn wir keine Feldinformationen angeben, erstellt Milvus automatisch ein Standardfeld id für den Primärschlüssel und ein Feld vector zum Speichern der Vektordaten. Ein reserviertes JSON-Feld wird verwendet, um nicht schema-definierte Felder und ihre Werte zu speichern.
milvus_client.create_collection(
+ collection_name=collection_name,
+ dimension=embedding_dim,
+ metric_type="IP", # Inner product distance
+ consistency_level="Strong", # Strong consistency level
+)
+Iterieren Sie durch die Textzeilen, erstellen Sie Einbettungen und fügen Sie dann die Daten in Milvus ein.
+Hier ist ein neues Feld text, das ein nicht definiertes Feld im Sammelschema ist. Es wird automatisch dem reservierten dynamischen JSON-Feld hinzugefügt, das auf hoher Ebene wie ein normales Feld behandelt werden kann.
from tqdm import tqdm
+
+data = []
+
+for i, line in enumerate(tqdm(text_lines, desc="Creating embeddings")):
+ data.append({"id": i, "vector": emb_text(line), "text": line})
+
+milvus_client.insert(collection_name=collection_name, data=data)
+Creating embeddings: 100%|██████████| 72/72 [00:03<00:00, 22.56it/s]
+
+
+
+
+
+{'insert_count': 72, 'ids': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71], 'cost': 0}
+Lassen Sie uns eine häufige Frage über Milvus angeben.
+question = "How is data stored in milvus?"
+Suchen Sie nach der Frage in der Sammlung und rufen Sie die semantischen Top-3-Treffer ab.
+search_res = milvus_client.search(
+ collection_name=collection_name,
+ data=[
+ emb_text(question)
+ ], # Use the `emb_text` function to convert the question to an embedding vector
+ limit=3, # Return top 3 results
+ search_params={"metric_type": "IP", "params": {}}, # Inner product distance
+ output_fields=["text"], # Return the text field
+)
+Werfen wir einen Blick auf die Suchergebnisse der Abfrage
+import json
+
+retrieved_lines_with_distances = [
+ (res["entity"]["text"], res["distance"]) for res in search_res[0]
+]
+print(json.dumps(retrieved_lines_with_distances, indent=4))
+[
+ [
+ " Where does Milvus store data?\n\nMilvus deals with two types of data, inserted data and metadata. \n\nInserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).\n\nMetadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.\n\n###",
+ 231.9398193359375
+ ],
+ [
+ "How does Milvus flush data?\n\nMilvus returns success when inserted data are loaded to the message queue. However, the data are not yet flushed to the disk. Then Milvus' data node writes the data in the message queue to persistent storage as incremental logs. If `flush()` is called, the data node is forced to write all data in the message queue to persistent storage immediately.\n\n###",
+ 226.48316955566406
+ ],
+ [
+ "What is the maximum dataset size Milvus can handle?\n\n \nTheoretically, the maximum dataset size Milvus can handle is determined by the hardware it is run on, specifically system memory and storage:\n\n- Milvus loads all specified collections and partitions into memory before running queries. Therefore, memory size determines the maximum amount of data Milvus can query.\n- When new entities and and collection-related schema (currently only MinIO is supported for data persistence) are added to Milvus, system storage determines the maximum allowable size of inserted data.\n\n###",
+ 210.60745239257812
+ ]
+]
+Konvertieren Sie die abgerufenen Dokumente in ein String-Format.
+context = "\n".join(
+ [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]
+)
+Definieren Sie System- und Benutzer-Prompts für das Lanage Model. Diese Eingabeaufforderung wird mit den abgerufenen Dokumenten aus Milvus zusammengestellt.
+SYSTEM_PROMPT = """
+Human: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.
+"""
+USER_PROMPT = f"""
+Use the following pieces of information enclosed in <context> tags to provide an answer to the question enclosed in <question> tags.
+<context>
+{context}
+</context>
+<question>
+{question}
+</question>
+"""
+Verwenden Sie das von Ollama bereitgestellte llama3.2 Modell, um eine Antwort auf der Grundlage der Prompts zu generieren.
from ollama import chat
+from ollama import ChatResponse
+
+response: ChatResponse = chat(
+ model="llama3.2",
+ messages=[
+ {"role": "system", "content": SYSTEM_PROMPT},
+ {"role": "user", "content": USER_PROMPT},
+ ],
+)
+print(response["message"]["content"])
+According to the provided context, data in Milvus is stored in two types:
+
+1. **Inserted data**: Storing data in persistent storage as incremental log. It supports multiple object storage backends such as MinIO, AWS S3, Google Cloud Storage (GCS), Azure Blob Storage, Alibaba Cloud OSS, and Tencent Cloud Object Storage.
+
+2. **Metadata**: Generated within Milvus and stored in etcd.
+Großartig! Wir haben erfolgreich eine RAG-Pipeline mit Milvus und Ollama aufgebaut.
diff --git a/localization/v2.4.x/site/de/integrations/milvus_rag_with_dynamiq.json b/localization/v2.4.x/site/de/integrations/milvus_rag_with_dynamiq.json new file mode 100644 index 000000000..3715712c1 --- /dev/null +++ b/localization/v2.4.x/site/de/integrations/milvus_rag_with_dynamiq.json @@ -0,0 +1 @@ +{"codeList":["$ pip install dynamiq pymilvus\n","import os\n\nos.environ[\"OPENAI_API_KEY\"] = \"sk-***********\"\n","# Importing necessary libraries for the workflow\nfrom io import BytesIO\nfrom dynamiq import Workflow\nfrom dynamiq.nodes import InputTransformer\nfrom dynamiq.connections import (\n OpenAI as OpenAIConnection,\n Milvus as MilvusConnection,\n MilvusDeploymentType,\n)\nfrom dynamiq.nodes.converters import PyPDFConverter\nfrom dynamiq.nodes.splitters.document import DocumentSplitter\nfrom dynamiq.nodes.embedders import OpenAIDocumentEmbedder\nfrom dynamiq.nodes.writers import MilvusDocumentWriter\n\n# Initialize the workflow\nrag_wf = Workflow()\n","converter = PyPDFConverter(document_creation_mode=\"one-doc-per-page\")\nconverter_added = rag_wf.flow.add_nodes(\n converter\n) # Add node to the DAG (Directed Acyclic Graph)\n","document_splitter = DocumentSplitter(\n split_by=\"sentence\", # Splits documents into sentences\n split_length=10,\n split_overlap=1,\n input_transformer=InputTransformer(\n selector={\n \"documents\": f\"${[converter.id]}.output.documents\",\n },\n ),\n).depends_on(\n converter\n) # Set dependency on the PDF converter\nsplitter_added = rag_wf.flow.add_nodes(document_splitter) # Add to the DAG\n","embedder = OpenAIDocumentEmbedder(\n connection=OpenAIConnection(api_key=os.environ[\"OPENAI_API_KEY\"]),\n input_transformer=InputTransformer(\n selector={\n \"documents\": f\"${[document_splitter.id]}.output.documents\",\n },\n ),\n).depends_on(\n document_splitter\n) # Set dependency on the splitter\ndocument_embedder_added = rag_wf.flow.add_nodes(embedder) # Add to the DAG\n","vector_store = (\n MilvusDocumentWriter(\n connection=MilvusConnection(\n deployment_type=MilvusDeploymentType.FILE, uri=\"./milvus.db\"\n ),\n index_name=\"my_milvus_collection\",\n dimension=1536,\n create_if_not_exist=True,\n metric_type=\"COSINE\",\n )\n .inputs(documents=embedder.outputs.documents) # Connect to embedder output\n .depends_on(embedder) # Set dependency on the embedder\n)\nmilvus_writer_added = rag_wf.flow.add_nodes(vector_store) # Add to the DAG\n","file_paths = [\"./pdf_files/WhatisMilvus.pdf\"]\ninput_data = {\n \"files\": [BytesIO(open(path, \"rb\").read()) for path in file_paths],\n \"metadata\": [{\"filename\": path} for path in file_paths],\n}\n\n# Run the workflow with the prepared input data\ninserted_data = rag_wf.run(input_data=input_data)\n","from dynamiq import Workflow\nfrom dynamiq.connections import (\n OpenAI as OpenAIConnection,\n Milvus as MilvusConnection,\n MilvusDeploymentType,\n)\nfrom dynamiq.nodes.embedders import OpenAITextEmbedder\nfrom dynamiq.nodes.retrievers import MilvusDocumentRetriever\nfrom dynamiq.nodes.llms import OpenAI\nfrom dynamiq.prompts import Message, Prompt\n\n# Initialize the workflow\nretrieval_wf = Workflow()\n","# Establish OpenAI connection\nopenai_connection = OpenAIConnection(api_key=os.environ[\"OPENAI_API_KEY\"])\n\n# Define the text embedder node\nembedder = OpenAITextEmbedder(\n connection=openai_connection,\n model=\"text-embedding-3-small\",\n)\n\n# Add the embedder node to the workflow\nembedder_added = retrieval_wf.flow.add_nodes(embedder)\n","document_retriever = (\n MilvusDocumentRetriever(\n connection=MilvusConnection(\n deployment_type=MilvusDeploymentType.FILE, uri=\"./milvus.db\"\n ),\n index_name=\"my_milvus_collection\",\n dimension=1536,\n top_k=5,\n )\n .inputs(embedding=embedder.outputs.embedding) # Connect to embedder output\n .depends_on(embedder) # Dependency on the embedder node\n)\n\n# Add the retriever node to the workflow\nmilvus_retriever_added = retrieval_wf.flow.add_nodes(document_retriever)\n","# Define the prompt template for the LLM\nprompt_template = \"\"\"\nPlease answer the question based on the provided context.\n\nQuestion: {{ query }}\n\nContext:\n{% for document in documents %}\n- {{ document.content }}\n{% endfor %}\n\"\"\"\n\n# Create the prompt object\nprompt = Prompt(messages=[Message(content=prompt_template, role=\"user\")])\n","answer_generator = (\n OpenAI(\n connection=openai_connection,\n model=\"gpt-4o\",\n prompt=prompt,\n )\n .inputs(\n documents=document_retriever.outputs.documents,\n query=embedder.outputs.query,\n )\n .depends_on(\n [document_retriever, embedder]\n ) # Dependencies on retriever and embedder\n)\n\n# Add the answer generator node to the workflow\nanswer_generator_added = retrieval_wf.flow.add_nodes(answer_generator)\n","# Run the workflow with a sample query\nsample_query = \"What is the Advanced Search Algorithms in Milvus?\"\n\nresult = retrieval_wf.run(input_data={\"query\": sample_query})\n\nanswer = result.output.get(answer_generator.id).get(\"output\", {}).get(\"content\")\nprint(answer)\n"],"headingContent":"Getting Started with Dynamiq and Milvus","anchorList":[{"label":"Erste Schritte mit Dynamiq und Milvus","href":"Getting-Started-with-Dynamiq-and-Milvus","type":1,"isActive":false},{"label":"Vorbereitung","href":"Preparation","type":2,"isActive":false},{"label":"RAG - Fluss der Dokumentenindizierung","href":"RAG---Document-Indexing-Flow","type":2,"isActive":false},{"label":"Ablauf der RAG-Dokumentenrecherche","href":"RAG-Document-Retrieval-Flow","type":2,"isActive":false}]} \ No newline at end of file diff --git a/localization/v2.4.x/site/de/integrations/milvus_rag_with_dynamiq.md b/localization/v2.4.x/site/de/integrations/milvus_rag_with_dynamiq.md new file mode 100644 index 000000000..d2b6284c5 --- /dev/null +++ b/localization/v2.4.x/site/de/integrations/milvus_rag_with_dynamiq.md @@ -0,0 +1,340 @@ +--- +id: milvus_rag_with_dynamiq.md +summary: >- + In diesem Tutorial erfahren Sie, wie Sie Dynamiq nahtlos mit Milvus, der + leistungsstarken Vektordatenbank, die speziell für RAG-Workflows entwickelt + wurde, verwenden können. Milvus zeichnet sich durch eine effiziente + Speicherung, Indizierung und Abfrage von Vektoreinbettungen aus und ist damit + eine unverzichtbare Komponente für KI-Systeme, die einen schnellen und + präzisen kontextbezogenen Datenzugriff erfordern. +title: Erste Schritte mit Dynamiq und Milvus +--- + +Dynamiq ist ein leistungsstarkes Gen AI-Framework, das die Entwicklung von KI-gestützten Anwendungen vereinfacht. Mit robuster Unterstützung für Retrieval-Augmented Generation (RAG) und Large Language Model (LLM) Agenten ermöglicht Dynamiq Entwicklern die einfache und effiziente Erstellung intelligenter, dynamischer Systeme.
+In diesem Tutorial erfahren Sie, wie Sie Dynamiq nahtlos mit Milvus, der leistungsstarken Vektordatenbank, die speziell für RAG-Workflows entwickelt wurde, einsetzen können. Milvus zeichnet sich durch eine effiziente Speicherung, Indizierung und Abfrage von Vektoreinbettungen aus und ist damit eine unverzichtbare Komponente für KI-Systeme, die einen schnellen und präzisen kontextbezogenen Datenzugriff erfordern.
+Diese Schritt-für-Schritt-Anleitung behandelt zwei zentrale RAG-Workflows:
+Document Indexing Flow: Lernen Sie, wie Sie Eingabedateien (z.B. PDFs) verarbeiten, ihren Inhalt in Vektoreinbettungen umwandeln und sie in Milvus speichern. Die Nutzung der leistungsstarken Indizierungsfunktionen von Milvus stellt sicher, dass Ihre Daten für einen schnellen Abruf bereit sind.
Document Retrieval Flow: Erfahren Sie, wie Sie Milvus nach relevanten Dokumenteneinbettungen abfragen und diese nutzen, um mit den LLM-Agenten von Dynamiq aufschlussreiche, kontextbezogene Antworten zu generieren und so eine nahtlose KI-gestützte Benutzererfahrung zu schaffen.
Am Ende dieses Tutorials werden Sie ein solides Verständnis dafür erlangen, wie Milvus und Dynamiq zusammenarbeiten, um skalierbare, kontextbewusste KI-Systeme zu entwickeln, die auf Ihre Bedürfnisse zugeschnitten sind.
+$ pip install dynamiq pymilvus
+Wenn Sie Google Colab verwenden, müssen Sie möglicherweise die Runtime neu starten, um die soeben installierten Abhängigkeiten zu aktivieren (klicken Sie auf das Menü Runtime" am oberen Rand des Bildschirms und wählen Sie Sitzung neu starten" aus dem Dropdown-Menü).
+Wir werden in diesem Beispiel OpenAI als LLM verwenden. Sie sollten den api-Schlüssel OPENAI_API_KEY als Umgebungsvariable vorbereiten.
import os
+
+os.environ["OPENAI_API_KEY"] = "sk-***********"
+Dieses Tutorial demonstriert einen Retrieval-Augmented Generation (RAG) Workflow zur Indizierung von Dokumenten mit Milvus als Vektordatenbank. Der Workflow nimmt PDF-Eingabedateien, verarbeitet sie in kleinere Teile, erzeugt Vektoreinbettungen mit dem Einbettungsmodell von OpenAI und speichert die Einbettungen in einer Milvus-Sammlung für eine effiziente Suche.
+Am Ende dieses Workflows werden Sie über ein skalierbares und effizientes Dokumentenindizierungssystem verfügen, das zukünftige RAG-Aufgaben wie semantische Suche und Fragenbeantwortung unterstützt.
+# Importing necessary libraries for the workflow
+from io import BytesIO
+from dynamiq import Workflow
+from dynamiq.nodes import InputTransformer
+from dynamiq.connections import (
+ OpenAI as OpenAIConnection,
+ Milvus as MilvusConnection,
+ MilvusDeploymentType,
+)
+from dynamiq.nodes.converters import PyPDFConverter
+from dynamiq.nodes.splitters.document import DocumentSplitter
+from dynamiq.nodes.embedders import OpenAIDocumentEmbedder
+from dynamiq.nodes.writers import MilvusDocumentWriter
+
+# Initialize the workflow
+rag_wf = Workflow()
+converter = PyPDFConverter(document_creation_mode="one-doc-per-page")
+converter_added = rag_wf.flow.add_nodes(
+ converter
+) # Add node to the DAG (Directed Acyclic Graph)
+document_splitter = DocumentSplitter(
+ split_by="sentence", # Splits documents into sentences
+ split_length=10,
+ split_overlap=1,
+ input_transformer=InputTransformer(
+ selector={
+ "documents": f"${[converter.id]}.output.documents",
+ },
+ ),
+).depends_on(
+ converter
+) # Set dependency on the PDF converter
+splitter_added = rag_wf.flow.add_nodes(document_splitter) # Add to the DAG
+embedder = OpenAIDocumentEmbedder(
+ connection=OpenAIConnection(api_key=os.environ["OPENAI_API_KEY"]),
+ input_transformer=InputTransformer(
+ selector={
+ "documents": f"${[document_splitter.id]}.output.documents",
+ },
+ ),
+).depends_on(
+ document_splitter
+) # Set dependency on the splitter
+document_embedder_added = rag_wf.flow.add_nodes(embedder) # Add to the DAG
+vector_store = (
+ MilvusDocumentWriter(
+ connection=MilvusConnection(
+ deployment_type=MilvusDeploymentType.FILE, uri="./milvus.db"
+ ),
+ index_name="my_milvus_collection",
+ dimension=1536,
+ create_if_not_exist=True,
+ metric_type="COSINE",
+ )
+ .inputs(documents=embedder.outputs.documents) # Connect to embedder output
+ .depends_on(embedder) # Set dependency on the embedder
+)
+milvus_writer_added = rag_wf.flow.add_nodes(vector_store) # Add to the DAG
+2024-11-19 22:14:03 - WARNING - Environment variable 'MILVUS_API_TOKEN' not found
+2024-11-19 22:14:03 - INFO - Pass in the local path ./milvus.db, and run it using milvus-lite
+2024-11-19 22:14:04 - DEBUG - Created new connection using: 0bef2849fdb1458a85df8bb9dd27f51d
+2024-11-19 22:14:04 - INFO - Collection my_milvus_collection does not exist. Creating a new collection.
+2024-11-19 22:14:04 - DEBUG - Successfully created collection: my_milvus_collection
+2024-11-19 22:14:05 - DEBUG - Successfully created an index on collection: my_milvus_collection
+2024-11-19 22:14:05 - DEBUG - Successfully created an index on collection: my_milvus_collection
+Milvus bietet zwei Deployment-Typen an, die unterschiedliche Anwendungsfälle abdecken:
+uri auf einen lokalen Dateipfad (z. B. ./milvus.db), um Milvus Lite zu nutzen, das alle Daten automatisch in der angegebenen Datei speichert.Konzipiert für umfangreiche Datenszenarien, z. B. die Verwaltung von über einer Million Vektoren.
+Selbstgehosteter Server
+uri (z. B. http://localhost:19530).<your_username>:<your_password> als token an.token unverändert.Zilliz Cloud (Verwalteter Dienst)
+uri und token entsprechend dem öffentlichen Endpunkt und dem API-Schlüssel ein, die in der Zilliz Cloud-Konsole bereitgestellt werden.file_paths = ["./pdf_files/WhatisMilvus.pdf"]
+input_data = {
+ "files": [BytesIO(open(path, "rb").read()) for path in file_paths],
+ "metadata": [{"filename": path} for path in file_paths],
+}
+
+# Run the workflow with the prepared input data
+inserted_data = rag_wf.run(input_data=input_data)
+/var/folders/09/d0hx80nj35sb5hxb5cpc1q180000gn/T/ipykernel_31319/3145804345.py:4: ResourceWarning: unclosed file <_io.BufferedReader name='./pdf_files/WhatisMilvus.pdf'>
+ BytesIO(open(path, "rb").read()) for path in file_paths
+ResourceWarning: Enable tracemalloc to get the object allocation traceback
+2024-11-19 22:14:09 - INFO - Workflow 87878444-6a3d-43f3-ae32-0127564a959f: execution started.
+2024-11-19 22:14:09 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution started.
+2024-11-19 22:14:09 - INFO - Node PyPDF File Converter - 6eb42b1f-7637-407b-a3ac-4167bcf3b5c4: execution started.
+2024-11-19 22:14:09 - INFO - Node PyPDF File Converter - 6eb42b1f-7637-407b-a3ac-4167bcf3b5c4: execution succeeded in 58ms.
+2024-11-19 22:14:09 - INFO - Node DocumentSplitter - 5baed580-6de0-4dcd-bace-d7d947ab6c7f: execution started.
+/Users/jinhonglin/anaconda3/envs/myenv/lib/python3.11/site-packages/websockets/legacy/__init__.py:6: DeprecationWarning: websockets.legacy is deprecated; see https://websockets.readthedocs.io/en/stable/howto/upgrade.html for upgrade instructions
+ warnings.warn( # deprecated in 14.0 - 2024-11-09
+/Users/jinhonglin/anaconda3/envs/myenv/lib/python3.11/site-packages/pydantic/fields.py:804: PydanticDeprecatedSince20: Using extra keyword arguments on `Field` is deprecated and will be removed. Use `json_schema_extra` instead. (Extra keys: 'is_accessible_to_agent'). Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/
+ warn(
+2024-11-19 22:14:09 - INFO - Node DocumentSplitter - 5baed580-6de0-4dcd-bace-d7d947ab6c7f: execution succeeded in 104ms.
+2024-11-19 22:14:09 - INFO - Node OpenAIDocumentEmbedder - 91928f67-a00f-48f6-a864-f6e21672ec7e: execution started.
+2024-11-19 22:14:09 - INFO - Node OpenAIDocumentEmbedder - d30a4cdc-0fab-4aff-b2e5-6161a62cb6fd: execution started.
+2024-11-19 22:14:10 - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
+2024-11-19 22:14:10 - INFO - Node OpenAIDocumentEmbedder - d30a4cdc-0fab-4aff-b2e5-6161a62cb6fd: execution succeeded in 724ms.
+2024-11-19 22:14:10 - INFO - Node MilvusDocumentWriter - dddab4cc-1dae-4e7e-9101-1ec353f530da: execution started.
+2024-11-19 22:14:10 - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
+2024-11-19 22:14:10 - INFO - Node MilvusDocumentWriter - dddab4cc-1dae-4e7e-9101-1ec353f530da: execution succeeded in 66ms.
+2024-11-19 22:14:10 - INFO - Node OpenAIDocumentEmbedder - 91928f67-a00f-48f6-a864-f6e21672ec7e: execution succeeded in 961ms.
+2024-11-19 22:14:10 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution succeeded in 1.3s.
+2024-11-19 22:14:10 - INFO - Workflow 87878444-6a3d-43f3-ae32-0127564a959f: execution succeeded in 1.3s.
+Mit diesem Workflow haben wir erfolgreich eine Dokumentenindizierungspipeline implementiert, die Milvus als Vektordatenbank und das Einbettungsmodell von OpenAI für die semantische Darstellung verwendet. Dieser Aufbau ermöglicht ein schnelles und genaues vektorbasiertes Retrieval und bildet die Grundlage für RAG-Workflows wie semantische Suche, Dokumentenretrieval und kontextbezogene KI-gesteuerte Interaktionen.
+Mit den skalierbaren Speichermöglichkeiten von Milvus und der Orchestrierung von Dynamiq ist diese Lösung sowohl für das Prototyping als auch für groß angelegte Produktionsimplementierungen geeignet. Sie können diese Pipeline nun erweitern, um zusätzliche Aufgaben wie Retrieval-basierte Fragebeantwortung oder KI-gesteuerte Inhaltsgenerierung einzubeziehen.
+In diesem Tutorial implementieren wir einen RAG-Workflow (Retrieval-Augmented Generation) zum Abrufen von Dokumenten. Dieser Workflow nimmt eine Benutzeranfrage, generiert eine Vektoreinbettung für sie, ruft die relevantesten Dokumente aus einer Milvus-Vektordatenbank ab und verwendet ein großes Sprachmodell (LLM), um eine detaillierte und kontextbezogene Antwort auf der Grundlage der abgerufenen Dokumente zu generieren.
+Wenn Sie diesen Arbeitsablauf befolgen, schaffen Sie eine End-to-End-Lösung für die semantische Suche und die Beantwortung von Fragen, indem Sie die Leistungsfähigkeit der vektorbasierten Dokumentensuche mit den Fähigkeiten der fortschrittlichen LLMs von OpenAI kombinieren. Dieser Ansatz ermöglicht effiziente und intelligente Antworten auf Benutzeranfragen, indem er das gespeicherte Wissen in Ihrer Dokumentendatenbank nutzt.
+from dynamiq import Workflow
+from dynamiq.connections import (
+ OpenAI as OpenAIConnection,
+ Milvus as MilvusConnection,
+ MilvusDeploymentType,
+)
+from dynamiq.nodes.embedders import OpenAITextEmbedder
+from dynamiq.nodes.retrievers import MilvusDocumentRetriever
+from dynamiq.nodes.llms import OpenAI
+from dynamiq.prompts import Message, Prompt
+
+# Initialize the workflow
+retrieval_wf = Workflow()
+# Establish OpenAI connection
+openai_connection = OpenAIConnection(api_key=os.environ["OPENAI_API_KEY"])
+
+# Define the text embedder node
+embedder = OpenAITextEmbedder(
+ connection=openai_connection,
+ model="text-embedding-3-small",
+)
+
+# Add the embedder node to the workflow
+embedder_added = retrieval_wf.flow.add_nodes(embedder)
+document_retriever = (
+ MilvusDocumentRetriever(
+ connection=MilvusConnection(
+ deployment_type=MilvusDeploymentType.FILE, uri="./milvus.db"
+ ),
+ index_name="my_milvus_collection",
+ dimension=1536,
+ top_k=5,
+ )
+ .inputs(embedding=embedder.outputs.embedding) # Connect to embedder output
+ .depends_on(embedder) # Dependency on the embedder node
+)
+
+# Add the retriever node to the workflow
+milvus_retriever_added = retrieval_wf.flow.add_nodes(document_retriever)
+2024-11-19 22:14:19 - WARNING - Environment variable 'MILVUS_API_TOKEN' not found
+2024-11-19 22:14:19 - INFO - Pass in the local path ./milvus.db, and run it using milvus-lite
+2024-11-19 22:14:19 - DEBUG - Created new connection using: 98d1132773af4298a894ad5925845fd2
+2024-11-19 22:14:19 - INFO - Collection my_milvus_collection already exists. Skipping creation.
+# Define the prompt template for the LLM
+prompt_template = """
+Please answer the question based on the provided context.
+
+Question: {{ query }}
+
+Context:
+{% for document in documents %}
+- {{ document.content }}
+{% endfor %}
+"""
+
+# Create the prompt object
+prompt = Prompt(messages=[Message(content=prompt_template, role="user")])
+answer_generator = (
+ OpenAI(
+ connection=openai_connection,
+ model="gpt-4o",
+ prompt=prompt,
+ )
+ .inputs(
+ documents=document_retriever.outputs.documents,
+ query=embedder.outputs.query,
+ )
+ .depends_on(
+ [document_retriever, embedder]
+ ) # Dependencies on retriever and embedder
+)
+
+# Add the answer generator node to the workflow
+answer_generator_added = retrieval_wf.flow.add_nodes(answer_generator)
+# Run the workflow with a sample query
+sample_query = "What is the Advanced Search Algorithms in Milvus?"
+
+result = retrieval_wf.run(input_data={"query": sample_query})
+
+answer = result.output.get(answer_generator.id).get("output", {}).get("content")
+print(answer)
+2024-11-19 22:14:22 - INFO - Workflow f4a073fb-dfb6-499c-8cac-5710a7ad6d47: execution started.
+2024-11-19 22:14:22 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution started.
+2024-11-19 22:14:22 - INFO - Node OpenAITextEmbedder - 47afb0bc-cf96-429d-b58f-11b6c935fec3: execution started.
+2024-11-19 22:14:23 - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
+2024-11-19 22:14:23 - INFO - Node OpenAITextEmbedder - 47afb0bc-cf96-429d-b58f-11b6c935fec3: execution succeeded in 474ms.
+2024-11-19 22:14:23 - INFO - Node MilvusDocumentRetriever - 51c8311b-4837-411f-ba42-21e28239a2ee: execution started.
+2024-11-19 22:14:23 - INFO - Node MilvusDocumentRetriever - 51c8311b-4837-411f-ba42-21e28239a2ee: execution succeeded in 23ms.
+2024-11-19 22:14:23 - INFO - Node LLM - ac722325-bece-453f-a2ed-135b0749ee7a: execution started.
+2024-11-19 22:14:24 - INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
+2024-11-19 22:14:24 - INFO - Node LLM - ac722325-bece-453f-a2ed-135b0749ee7a: execution succeeded in 1.8s.
+2024-11-19 22:14:25 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution succeeded in 2.4s.
+2024-11-19 22:14:25 - INFO - Workflow f4a073fb-dfb6-499c-8cac-5710a7ad6d47: execution succeeded in 2.4s.
+
+
+The advanced search algorithms in Milvus include a variety of in-memory and on-disk indexing/search algorithms such as IVF (Inverted File), HNSW (Hierarchical Navigable Small World), and DiskANN. These algorithms have been deeply optimized to enhance performance, delivering 30%-70% better performance compared to popular implementations like FAISS and HNSWLib. These optimizations are part of Milvus's design to ensure high efficiency and scalability in handling vector data.
+Cognee is a developer-first platform that streamlines AI application development with scalable, modular ECL (Extract, Cognify, Load) pipelines. By integrating seamlessly with Milvus, Cognee enables efficient connection and retrieval of conversations, documents, and transcriptions, reducing hallucinations and optimizing operational costs.
+With strong support for vector stores like Milvus, graph databases, and LLMs, Cognee provides a flexible and customizable framework for building retrieval-augmented generation (RAG) systems. Its production-ready architecture ensures improved accuracy and efficiency for AI-powered applications.
+In this tutorial, we will show you how to build a RAG (Retrieval-Augmented Generation) pipeline with Milvus and Cognee.
+$ pip install pymilvus git+https://github.com/topoteretes/cognee.git
+++If you are using Google Colab, to enable dependencies just installed, you may need to restart the runtime (click on the “Runtime” menu at the top of the screen, and select “Restart session” from the dropdown menu).
+
By default, it use OpenAI as the LLM in this example. You should prepare the api key, and set it in the config set_llm_api_key() function.
To configure Milvus as the vector database, set the VECTOR_DB_PROVIDER to milvus and specify the VECTOR_DB_URL and VECTOR_DB_KEY. Since we are using Milvus Lite to store data in this demo, only the VECTOR_DB_URL needs to be provided.
import os
+
+import cognee
+
+cognee.config.set_llm_api_key("YOUR_OPENAI_API_KEY")
+
+
+os.environ["VECTOR_DB_PROVIDER"] = "milvus"
+os.environ["VECTOR_DB_URL"] = "./milvus.db"
+As for the environment variables VECTOR_DB_URL and VECTOR_DB_KEY:
VECTOR_DB_URL as a local file, e.g../milvus.db, is the most convenient method, as it automatically utilizes Milvus Lite to store all data in this file.http://localhost:19530, as your VECTOR_DB_URL.VECTOR_DB_URL and VECTOR_DB_KEY, which correspond to the Public Endpoint and Api key in Zilliz Cloud.We use the FAQ pages from the Milvus Documentation 2.4.x as the private knowledge in our RAG, which is a good data source for a simple RAG pipeline.
+Download the zip file and extract documents to the folder milvus_docs.
$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
+We load all markdown files from the folder milvus_docs/en/faq. For each document, we just simply use "# " to separate the content in the file, which can roughly separate the content of each main part of the markdown file.
from glob import glob
+
+text_lines = []
+
+for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
+ with open(file_path, "r") as file:
+ file_text = file.read()
+
+ text_lines += file_text.split("# ")
+await cognee.prune.prune_data()
+await cognee.prune.prune_system(metadata=True)
+With a clean slate ready, we can now add our dataset and process it into a knowledge graph.
+await cognee.add(data=text_lines, dataset_name="milvus_faq")
+await cognee.cognify()
+
+# [DocumentChunk(id=UUID('6889e7ef-3670-555c-bb16-3eb50d1d30b0'), updated_at=datetime.datetime(2024, 12, 4, 6, 29, 46, 472907, tzinfo=datetime.timezone.utc), text='Does the query perform in memory? What are incremental data and historical data?\n\nYes. When ...
+# ...
+The add method loads the dataset (Milvus FAQs) into Cognee and the cognify method processes the data to extract entities, relationships, and summaries, constructing a knowledge graph.
Now that the data has been processed, let’s query the knowledge graph.
+from cognee.api.v1.search import SearchType
+
+query_text = "How is data stored in milvus?"
+search_results = await cognee.search(SearchType.SUMMARIES, query_text=query_text)
+
+print(search_results[0])
+{'id': 'de5c6713-e079-5d0b-b11d-e9bacd1e0d73', 'text': 'Milvus stores two data types: inserted data and metadata.'}
+This query searches the knowledge graph for a summary related to the query text, and the most related candidate is printed.
+Summaries offer high-level insights, but for more granular details, we can query specific chunks of data directly from the processed dataset. These chunks are derived from the original data that was added and analyzed during the knowledge graph creation.
+from cognee.api.v1.search import SearchType
+
+query_text = "How is data stored in milvus?"
+search_results = await cognee.search(SearchType.CHUNKS, query_text=query_text)
+Let’s format and display it for better readability!
+def format_and_print(data):
+ print("ID:", data["id"])
+ print("\nText:\n")
+ paragraphs = data["text"].split("\n\n")
+ for paragraph in paragraphs:
+ print(paragraph.strip())
+ print()
+
+
+format_and_print(search_results[0])
+ID: 4be01c4b-9ee5-541c-9b85-297883934ab3
+
+Text:
+
+Where does Milvus store data?
+
+Milvus deals with two types of data, inserted data and metadata.
+
+Inserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).
+
+Metadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.
+
+###
+In our previous steps, we queried the Milvus FAQ dataset for both summaries and specific chunks of data. While this provided detailed insights and granular information, the dataset was large, making it challenging to clearly visualize the dependencies within the knowledge graph.
+To address this, we will reset the Cognee environment and work with a smaller, more focused dataset. This will allow us to better demonstrate the relationships and dependencies extracted during the cognify process. By simplifying the data, we can clearly see how Cognee organizes and structures information in the knowledge graph.
+await cognee.prune.prune_data()
+await cognee.prune.prune_system(metadata=True)
+Here, a smaller dataset with only one line of text is added and processed to ensure a focused and easily interpretable knowledge graph.
+# We only use one line of text as the dataset, which simplifies the output later
+text = """
+ Natural language processing (NLP) is an interdisciplinary
+ subfield of computer science and information retrieval.
+ """
+
+await cognee.add(text)
+await cognee.cognify()
+By focusing on this smaller dataset, we can now clearly analyze the relationships and structure within the knowledge graph.
+query_text = "Tell me about NLP"
+search_results = await cognee.search(SearchType.INSIGHTS, query_text=query_text)
+
+for result_text in search_results:
+ print(result_text)
+
+# Example output:
+# ({'id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'natural language processing', 'description': 'An interdisciplinary subfield of computer science and information retrieval.'}, {'relationship_name': 'is_a_subfield_of', 'source_node_id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'target_node_id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 15, 473137, tzinfo=datetime.timezone.utc)}, {'id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'computer science', 'description': 'The study of computation and information processing.'})
+# (...)
+#
+# It represents nodes and relationships in the knowledge graph:
+# - The first element is the source node (e.g., 'natural language processing').
+# - The second element is the relationship between nodes (e.g., 'is_a_subfield_of').
+# - The third element is the target node (e.g., 'computer science').
+This output represents the results of a knowledge graph query, showcasing entities (nodes) and their relationships (edges) as extracted from the processed dataset. Each tuple includes a source entity, a relationship type, and a target entity, along with metadata like unique IDs, descriptions, and timestamps. The graph highlights key concepts and their semantic connections, providing a structured understanding of the dataset.
+Congratulations, you have learned the basic usage of cognee with Milvus. If you want to know more advanced usage of cognee, please refer to its official page .
diff --git a/localization/v2.4.x/site/en/integrations/build_RAG_with_milvus_and_gemini.json b/localization/v2.4.x/site/en/integrations/build_RAG_with_milvus_and_gemini.json new file mode 100644 index 000000000..e4a5c4973 --- /dev/null +++ b/localization/v2.4.x/site/en/integrations/build_RAG_with_milvus_and_gemini.json @@ -0,0 +1 @@ +{"codeList":["$ pip install --upgrade pymilvus google-generativeai requests tqdm\n","import os\n\nos.environ[\"GEMINI_API_KEY\"] = \"***********\"\n","$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip\n$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs\n","from glob import glob\n\ntext_lines = []\n\nfor file_path in glob(\"milvus_docs/en/faq/*.md\", recursive=True):\n with open(file_path, \"r\") as file:\n file_text = file.read()\n\n text_lines += file_text.split(\"# \")\n","import google.generativeai as genai\n\ngenai.configure(api_key=os.environ[\"GEMINI_API_KEY\"])\n\ngemini_model = genai.GenerativeModel(\"gemini-1.5-flash\")\n\nresponse = gemini_model.generate_content(\"who are you\")\nprint(response.text)\n","test_embeddings = genai.embed_content(\n model=\"models/text-embedding-004\", content=[\"This is a test1\", \"This is a test2\"]\n)[\"embedding\"]\n\nembedding_dim = len(test_embeddings[0])\nprint(embedding_dim)\nprint(test_embeddings[0][:10])\n","from pymilvus import MilvusClient\n\nmilvus_client = MilvusClient(uri=\"./milvus_demo.db\")\n\ncollection_name = \"my_rag_collection\"\n","if milvus_client.has_collection(collection_name):\n milvus_client.drop_collection(collection_name)\n","milvus_client.create_collection(\n collection_name=collection_name,\n dimension=embedding_dim,\n metric_type=\"IP\", # Inner product distance\n consistency_level=\"Strong\", # Strong consistency level\n)\n","from tqdm import tqdm\n\ndata = []\n\ndoc_embeddings = genai.embed_content(\n model=\"models/text-embedding-004\", content=text_lines\n)[\"embedding\"]\n\nfor i, line in enumerate(tqdm(text_lines, desc=\"Creating embeddings\")):\n data.append({\"id\": i, \"vector\": doc_embeddings[i], \"text\": line})\n\nmilvus_client.insert(collection_name=collection_name, data=data)\n","question = \"How is data stored in milvus?\"\n","question_embedding = genai.embed_content(\n model=\"models/text-embedding-004\", content=question\n)[\"embedding\"]\n\nsearch_res = milvus_client.search(\n collection_name=collection_name,\n data=[question_embedding],\n limit=3, # Return top 3 results\n search_params={\"metric_type\": \"IP\", \"params\": {}}, # Inner product distance\n output_fields=[\"text\"], # Return the text field\n)\n","import json\n\nretrieved_lines_with_distances = [\n (res[\"entity\"][\"text\"], res[\"distance\"]) for res in search_res[0]\n]\nprint(json.dumps(retrieved_lines_with_distances, indent=4))\n","context = \"\\n\".join(\n [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]\n)\n","SYSTEM_PROMPT = \"\"\"\nHuman: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.\n\"\"\"\nUSER_PROMPT = f\"\"\"\nUse the following pieces of information enclosed inThe Gemini API and Google AI Studio help you start working with Google’s latest models and turn your ideas into applications that scale. Gemini provides access to powerful language models like Gemini-1.5-Flash, Gemini-1.5-Flash-8B, and Gemini-1.5-Pro for tasks such as text generation, document processing, vision, audio analysis, and more. The API allows you to input long context with millions of tokens, fine-tune models for specific tasks, generate structured outputs like JSON, and leverage capabilities like semantic retrieval and code execution.
In this tutorial, we will show you how to build a RAG (Retrieval-Augmented Generation) pipeline with Milvus and Gemini. We will use the Gemini model to generate text based on a given query. We will also use Milvus to store and retrieve the generated text.
+$ pip install --upgrade pymilvus google-generativeai requests tqdm
+If you are using Google Colab, to enable dependencies just installed, you may need to restart the runtime (click on the “Runtime” menu at the top of the screen, and select “Restart session” from the dropdown menu).
+You should first log in to the Google AI Studio platform and prepare the api key GEMINI_API_KEY as an environment variable.
import os
+
+os.environ["GEMINI_API_KEY"] = "***********"
+We use the FAQ pages from the Milvus Documentation 2.4.x as the private knowledge in our RAG, which is a good data source for a simple RAG pipeline.
+Download the zip file and extract documents to the folder milvus_docs.
$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
+We load all markdown files from the folder milvus_docs/en/faq. For each document, we just simply use "# " to separate the content in the file, which can roughly separate the content of each main part of the markdown file.
from glob import glob
+
+text_lines = []
+
+for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
+ with open(file_path, "r") as file:
+ file_text = file.read()
+
+ text_lines += file_text.split("# ")
+We use the gemini-1.5-flash as LLM, and the text-embedding-004 as embedding model.
Let’s try to generate a test response from the LLM:
+import google.generativeai as genai
+
+genai.configure(api_key=os.environ["GEMINI_API_KEY"])
+
+gemini_model = genai.GenerativeModel("gemini-1.5-flash")
+
+response = gemini_model.generate_content("who are you")
+print(response.text)
+I am a large language model, trained by Google. I am an AI and don't have a personal identity or consciousness. My purpose is to process information and respond to a wide range of prompts and questions in a helpful and informative way.
+Generate a test embedding and print its dimension and first few elements.
+test_embeddings = genai.embed_content(
+ model="models/text-embedding-004", content=["This is a test1", "This is a test2"]
+)["embedding"]
+
+embedding_dim = len(test_embeddings[0])
+print(embedding_dim)
+print(test_embeddings[0][:10])
+768
+[0.013588584, -0.004361838, -0.08481652, -0.039724775, 0.04723794, -0.0051557426, 0.026071774, 0.045514572, -0.016867816, 0.039378334]
+from pymilvus import MilvusClient
+
+milvus_client = MilvusClient(uri="./milvus_demo.db")
+
+collection_name = "my_rag_collection"
+As for the argument of MilvusClient:
uri as a local file, e.g../milvus.db, is the most convenient method, as it automatically utilizes Milvus Lite to store all data in this file.http://localhost:19530, as your uri.uri and token, which correspond to the Public Endpoint and Api key in Zilliz Cloud.Check if the collection already exists and drop it if it does.
+if milvus_client.has_collection(collection_name):
+ milvus_client.drop_collection(collection_name)
+Create a new collection with specified parameters.
+If we don’t specify any field information, Milvus will automatically create a default id field for primary key, and a vector field to store the vector data. A reserved JSON field is used to store non-schema-defined fields and their values.
milvus_client.create_collection(
+ collection_name=collection_name,
+ dimension=embedding_dim,
+ metric_type="IP", # Inner product distance
+ consistency_level="Strong", # Strong consistency level
+)
+Iterate through the text lines, create embeddings, and then insert the data into Milvus.
+Here is a new field text, which is a non-defined field in the collection schema. It will be automatically added to the reserved JSON dynamic field, which can be treated as a normal field at a high level.
from tqdm import tqdm
+
+data = []
+
+doc_embeddings = genai.embed_content(
+ model="models/text-embedding-004", content=text_lines
+)["embedding"]
+
+for i, line in enumerate(tqdm(text_lines, desc="Creating embeddings")):
+ data.append({"id": i, "vector": doc_embeddings[i], "text": line})
+
+milvus_client.insert(collection_name=collection_name, data=data)
+Creating embeddings: 100%|██████████| 72/72 [00:00<00:00, 468201.38it/s]
+
+
+
+
+
+{'insert_count': 72, 'ids': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71], 'cost': 0}
+Let’s specify a frequent question about Milvus.
+question = "How is data stored in milvus?"
+Search for the question in the collection and retrieve the semantic top-3 matches.
+question_embedding = genai.embed_content(
+ model="models/text-embedding-004", content=question
+)["embedding"]
+
+search_res = milvus_client.search(
+ collection_name=collection_name,
+ data=[question_embedding],
+ limit=3, # Return top 3 results
+ search_params={"metric_type": "IP", "params": {}}, # Inner product distance
+ output_fields=["text"], # Return the text field
+)
+Let’s take a look at the search results of the query
+import json
+
+retrieved_lines_with_distances = [
+ (res["entity"]["text"], res["distance"]) for res in search_res[0]
+]
+print(json.dumps(retrieved_lines_with_distances, indent=4))
+[
+ [
+ " Where does Milvus store data?\n\nMilvus deals with two types of data, inserted data and metadata. \n\nInserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).\n\nMetadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.\n\n###",
+ 0.8048275113105774
+ ],
+ [
+ "Does the query perform in memory? What are incremental data and historical data?\n\nYes. When a query request comes, Milvus searches both incremental data and historical data by loading them into memory. Incremental data are in the growing segments, which are buffered in memory before they reach the threshold to be persisted in storage engine, while historical data are from the sealed segments that are stored in the object storage. Incremental data and historical data together constitute the whole dataset to search.\n\n###",
+ 0.7574886679649353
+ ],
+ [
+ "What is the maximum dataset size Milvus can handle?\n\n \nTheoretically, the maximum dataset size Milvus can handle is determined by the hardware it is run on, specifically system memory and storage:\n\n- Milvus loads all specified collections and partitions into memory before running queries. Therefore, memory size determines the maximum amount of data Milvus can query.\n- When new entities and and collection-related schema (currently only MinIO is supported for data persistence) are added to Milvus, system storage determines the maximum allowable size of inserted data.\n\n###",
+ 0.7453608512878418
+ ]
+]
+Convert the retrieved documents into a string format.
+context = "\n".join(
+ [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]
+)
+Define system and user prompts for the Lanage Model. This prompt is assembled with the retrieved documents from Milvus.
+SYSTEM_PROMPT = """
+Human: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.
+"""
+USER_PROMPT = f"""
+Use the following pieces of information enclosed in <context> tags to provide an answer to the question enclosed in <question> tags.
+<context>
+{context}
+</context>
+<question>
+{question}
+</question>
+"""
+Use the Gemini to generate a response based on the prompts.
+gemini_model = genai.GenerativeModel(
+ "gemini-1.5-flash", system_instruction=SYSTEM_PROMPT
+)
+response = gemini_model.generate_content(USER_PROMPT)
+print(response.text)
+Milvus stores data in two ways: Inserted data (vector data, scalar data, and collection-specific schema) is stored as an incremental log in persistent storage using object storage backends such as MinIO, AWS S3, Google Cloud Storage, Azure Blob Storage, Alibaba Cloud OSS, and Tencent Cloud Object Storage. Metadata, generated by each Milvus module, is stored in etcd.
+Great! We have successfully built a RAG pipeline with Milvus and Gemini.
diff --git a/localization/v2.4.x/site/en/integrations/build_RAG_with_milvus_and_ollama.json b/localization/v2.4.x/site/en/integrations/build_RAG_with_milvus_and_ollama.json new file mode 100644 index 000000000..fb0a098e6 --- /dev/null +++ b/localization/v2.4.x/site/en/integrations/build_RAG_with_milvus_and_ollama.json @@ -0,0 +1 @@ +{"codeList":["$ pip install pymilvus ollama\n","$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip\n$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs\n","from glob import glob\n\ntext_lines = []\n\nfor file_path in glob(\"milvus_docs/en/faq/*.md\", recursive=True):\n with open(file_path, \"r\") as file:\n file_text = file.read()\n\n text_lines += file_text.split(\"# \")\n","! ollama pull mxbai-embed-large\n","! ollama pull llama3.2\n","import ollama\n\n\ndef emb_text(text):\n response = ollama.embeddings(model=\"mxbai-embed-large\", prompt=text)\n return response[\"embedding\"]\n","test_embedding = emb_text(\"This is a test\")\nembedding_dim = len(test_embedding)\nprint(embedding_dim)\nprint(test_embedding[:10])\n","from pymilvus import MilvusClient\n\nmilvus_client = MilvusClient(uri=\"./milvus_demo.db\")\n\ncollection_name = \"my_rag_collection\"\n","if milvus_client.has_collection(collection_name):\n milvus_client.drop_collection(collection_name)\n","milvus_client.create_collection(\n collection_name=collection_name,\n dimension=embedding_dim,\n metric_type=\"IP\", # Inner product distance\n consistency_level=\"Strong\", # Strong consistency level\n)\n","from tqdm import tqdm\n\ndata = []\n\nfor i, line in enumerate(tqdm(text_lines, desc=\"Creating embeddings\")):\n data.append({\"id\": i, \"vector\": emb_text(line), \"text\": line})\n\nmilvus_client.insert(collection_name=collection_name, data=data)\n","question = \"How is data stored in milvus?\"\n","search_res = milvus_client.search(\n collection_name=collection_name,\n data=[\n emb_text(question)\n ], # Use the `emb_text` function to convert the question to an embedding vector\n limit=3, # Return top 3 results\n search_params={\"metric_type\": \"IP\", \"params\": {}}, # Inner product distance\n output_fields=[\"text\"], # Return the text field\n)\n","import json\n\nretrieved_lines_with_distances = [\n (res[\"entity\"][\"text\"], res[\"distance\"]) for res in search_res[0]\n]\nprint(json.dumps(retrieved_lines_with_distances, indent=4))\n","context = \"\\n\".join(\n [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]\n)\n","SYSTEM_PROMPT = \"\"\"\nHuman: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.\n\"\"\"\nUSER_PROMPT = f\"\"\"\nUse the following pieces of information enclosed inOllama is an open-source platform that simplifies running and customizing large language models (LLMs) locally. It provides a user-friendly, cloud-free experience, enabling effortless model downloads, installation, and interaction without requiring advanced technical skills. With a growing library of pre-trained LLMs—from general-purpose to domain-specific—Ollama makes it easy to manage and customize models for various applications. It ensures data privacy and flexibility, empowering users to fine-tune, optimize, and deploy AI-driven solutions entirely on their machines.
+In this guide, we’ll show you how to leverage Ollama and Milvus to build a RAG (Retrieval-Augmented Generation) pipeline efficiently and securely.
+$ pip install pymilvus ollama
+If you are using Google Colab, to enable dependencies just installed, you may need to restart the runtime (click on the “Runtime” menu at the top of the screen, and select “Restart session” from the dropdown menu).
+We use the FAQ pages from the Milvus Documentation 2.4.x as the private knowledge in our RAG, which is a good data source for a simple RAG pipeline.
+Download the zip file and extract documents to the folder milvus_docs.
$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
+--2024-11-26 21:47:19-- https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+Resolving github.com (github.com)... 140.82.112.4
+Connecting to github.com (github.com)|140.82.112.4|:443... connected.
+HTTP request sent, awaiting response... 302 Found
+Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/267273319/c52902a0-e13c-4ca7-92e0-086751098a05?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241127%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241127T024720Z&X-Amz-Expires=300&X-Amz-Signature=7808b77cbdaa7e122196bcd75a73f29f2540333a350c4830bbdf5f286e876304&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dmilvus_docs_2.4.x_en.zip&response-content-type=application%2Foctet-stream [following]
+--2024-11-26 21:47:20-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/267273319/c52902a0-e13c-4ca7-92e0-086751098a05?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241127%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241127T024720Z&X-Amz-Expires=300&X-Amz-Signature=7808b77cbdaa7e122196bcd75a73f29f2540333a350c4830bbdf5f286e876304&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dmilvus_docs_2.4.x_en.zip&response-content-type=application%2Foctet-stream
+Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.109.133, 185.199.111.133, 185.199.108.133, ...
+Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.109.133|:443... connected.
+HTTP request sent, awaiting response... 200 OK
+Length: 613094 (599K) [application/octet-stream]
+Saving to: ‘milvus_docs_2.4.x_en.zip’
+
+milvus_docs_2.4.x_e 100%[===================>] 598.72K 1.20MB/s in 0.5s
+
+2024-11-26 21:47:20 (1.20 MB/s) - ‘milvus_docs_2.4.x_en.zip’ saved [613094/613094]
+We load all markdown files from the folder milvus_docs/en/faq. For each document, we just simply use "# " to separate the content in the file, which can roughly separate the content of each main part of the markdown file.
from glob import glob
+
+text_lines = []
+
+for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
+ with open(file_path, "r") as file:
+ file_text = file.read()
+
+ text_lines += file_text.split("# ")
+Ollama supports multiple models for both LLM-based tasks and embedding generation, making it easy to develop retrieval-augmented generation (RAG) applications. For this setup:
+Before starting, ensure both models are pulled locally:
+! ollama pull mxbai-embed-large
+[?25lpulling manifest ⠋ [?25h[?25l[2K[1Gpulling manifest ⠙ [?25h[?25l[2K[1Gpulling manifest ⠹ [?25h[?25l[2K[1Gpulling manifest ⠸ [?25h[?25l[2K[1Gpulling manifest ⠼ [?25h[?25l[2K[1Gpulling manifest ⠴ [?25h[?25l[2K[1Gpulling manifest
+pulling 819c2adf5ce6... 100% ▕████████████████▏ 669 MB
+pulling c71d239df917... 100% ▕████████████████▏ 11 KB
+pulling b837481ff855... 100% ▕████████████████▏ 16 B
+pulling 38badd946f91... 100% ▕████████████████▏ 408 B
+verifying sha256 digest
+writing manifest
+success [?25h
+! ollama pull llama3.2
+[?25lpulling manifest ⠋ [?25h[?25l[2K[1Gpulling manifest ⠙ [?25h[?25l[2K[1Gpulling manifest ⠹ [?25h[?25l[2K[1Gpulling manifest ⠸ [?25h[?25l[2K[1Gpulling manifest ⠼ [?25h[?25l[2K[1Gpulling manifest ⠴ [?25h[?25l[2K[1Gpulling manifest
+pulling dde5aa3fc5ff... 100% ▕████████████████▏ 2.0 GB
+pulling 966de95ca8a6... 100% ▕████████████████▏ 1.4 KB
+pulling fcc5a6bec9da... 100% ▕████████████████▏ 7.7 KB
+pulling a70ff7e570d9... 100% ▕████████████████▏ 6.0 KB
+pulling 56bb8bd477a5... 100% ▕████████████████▏ 96 B
+pulling 34bb5ab01051... 100% ▕████████████████▏ 561 B
+verifying sha256 digest
+writing manifest
+success [?25h
+With these models ready, we can proceed to implement LLM-driven generation and embedding-based retrieval workflows.
+import ollama
+
+
+def emb_text(text):
+ response = ollama.embeddings(model="mxbai-embed-large", prompt=text)
+ return response["embedding"]
+Generate a test embedding and print its dimension and first few elements.
+test_embedding = emb_text("This is a test")
+embedding_dim = len(test_embedding)
+print(embedding_dim)
+print(test_embedding[:10])
+1024
+[0.23276396095752716, 0.4257211685180664, 0.19724100828170776, 0.46120673418045044, -0.46039995551109314, -0.1413791924715042, -0.18261606991291046, -0.07602324336767197, 0.39991313219070435, 0.8337644338607788]
+from pymilvus import MilvusClient
+
+milvus_client = MilvusClient(uri="./milvus_demo.db")
+
+collection_name = "my_rag_collection"
+As for the argument of MilvusClient:
uri as a local file, e.g../milvus.db, is the most convenient method, as it automatically utilizes Milvus Lite to store all data in this file.http://localhost:19530, as your uri.uri and token, which correspond to the Public Endpoint and Api key in Zilliz Cloud.Check if the collection already exists and drop it if it does.
+if milvus_client.has_collection(collection_name):
+ milvus_client.drop_collection(collection_name)
+Create a new collection with specified parameters.
+If we don’t specify any field information, Milvus will automatically create a default id field for primary key, and a vector field to store the vector data. A reserved JSON field is used to store non-schema-defined fields and their values.
milvus_client.create_collection(
+ collection_name=collection_name,
+ dimension=embedding_dim,
+ metric_type="IP", # Inner product distance
+ consistency_level="Strong", # Strong consistency level
+)
+Iterate through the text lines, create embeddings, and then insert the data into Milvus.
+Here is a new field text, which is a non-defined field in the collection schema. It will be automatically added to the reserved JSON dynamic field, which can be treated as a normal field at a high level.
from tqdm import tqdm
+
+data = []
+
+for i, line in enumerate(tqdm(text_lines, desc="Creating embeddings")):
+ data.append({"id": i, "vector": emb_text(line), "text": line})
+
+milvus_client.insert(collection_name=collection_name, data=data)
+Creating embeddings: 100%|██████████| 72/72 [00:03<00:00, 22.56it/s]
+
+
+
+
+
+{'insert_count': 72, 'ids': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71], 'cost': 0}
+Let’s specify a frequent question about Milvus.
+question = "How is data stored in milvus?"
+Search for the question in the collection and retrieve the semantic top-3 matches.
+search_res = milvus_client.search(
+ collection_name=collection_name,
+ data=[
+ emb_text(question)
+ ], # Use the `emb_text` function to convert the question to an embedding vector
+ limit=3, # Return top 3 results
+ search_params={"metric_type": "IP", "params": {}}, # Inner product distance
+ output_fields=["text"], # Return the text field
+)
+Let’s take a look at the search results of the query
+import json
+
+retrieved_lines_with_distances = [
+ (res["entity"]["text"], res["distance"]) for res in search_res[0]
+]
+print(json.dumps(retrieved_lines_with_distances, indent=4))
+[
+ [
+ " Where does Milvus store data?\n\nMilvus deals with two types of data, inserted data and metadata. \n\nInserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).\n\nMetadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.\n\n###",
+ 231.9398193359375
+ ],
+ [
+ "How does Milvus flush data?\n\nMilvus returns success when inserted data are loaded to the message queue. However, the data are not yet flushed to the disk. Then Milvus' data node writes the data in the message queue to persistent storage as incremental logs. If `flush()` is called, the data node is forced to write all data in the message queue to persistent storage immediately.\n\n###",
+ 226.48316955566406
+ ],
+ [
+ "What is the maximum dataset size Milvus can handle?\n\n \nTheoretically, the maximum dataset size Milvus can handle is determined by the hardware it is run on, specifically system memory and storage:\n\n- Milvus loads all specified collections and partitions into memory before running queries. Therefore, memory size determines the maximum amount of data Milvus can query.\n- When new entities and and collection-related schema (currently only MinIO is supported for data persistence) are added to Milvus, system storage determines the maximum allowable size of inserted data.\n\n###",
+ 210.60745239257812
+ ]
+]
+Convert the retrieved documents into a string format.
+context = "\n".join(
+ [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]
+)
+Define system and user prompts for the Lanage Model. This prompt is assembled with the retrieved documents from Milvus.
+SYSTEM_PROMPT = """
+Human: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.
+"""
+USER_PROMPT = f"""
+Use the following pieces of information enclosed in <context> tags to provide an answer to the question enclosed in <question> tags.
+<context>
+{context}
+</context>
+<question>
+{question}
+</question>
+"""
+Use the llama3.2 model provided by Ollama to generate a response based on the prompts.
from ollama import chat
+from ollama import ChatResponse
+
+response: ChatResponse = chat(
+ model="llama3.2",
+ messages=[
+ {"role": "system", "content": SYSTEM_PROMPT},
+ {"role": "user", "content": USER_PROMPT},
+ ],
+)
+print(response["message"]["content"])
+According to the provided context, data in Milvus is stored in two types:
+
+1. **Inserted data**: Storing data in persistent storage as incremental log. It supports multiple object storage backends such as MinIO, AWS S3, Google Cloud Storage (GCS), Azure Blob Storage, Alibaba Cloud OSS, and Tencent Cloud Object Storage.
+
+2. **Metadata**: Generated within Milvus and stored in etcd.
+Great! We have successfully built a RAG pipeline with Milvus and Ollama.
diff --git a/localization/v2.4.x/site/en/integrations/integrations_overview.md b/localization/v2.4.x/site/en/integrations/integrations_overview.md index bb6580886..ad6a73a39 100644 --- a/localization/v2.4.x/site/en/integrations/integrations_overview.md +++ b/localization/v2.4.x/site/en/integrations/integrations_overview.md @@ -71,5 +71,9 @@ title: Integrations OverviewDynamiq is a powerful Gen AI framework that streamlines the development of AI-powered applications. With robust support for retrieval-augmented generation (RAG) and large language model (LLM) agents, Dynamiq empowers developers to create intelligent, dynamic systems with ease and efficiency.
+In this tutorial, we’ll explore how to seamlessly use Dynamiq with Milvus, the high-performance vector database purpose-built for RAG workflows. Milvus excels at efficient storage, indexing, and retrieval of vector embeddings, making it an indispensable component for AI systems that demand fast and precise contextual data access.
+This step-by-step guide will cover two core RAG workflows:
+Document Indexing Flow: Learn how to process input files (e.g., PDFs), transform their content into vector embeddings, and store them in Milvus. Leveraging Milvus’s high-performance indexing capabilities ensures your data is ready for rapid retrieval.
Document Retrieval Flow: Discover how to query Milvus for relevant document embeddings and use them to generate insightful, context-aware responses with Dynamiq’s LLM agents, creating a seamless AI-powered user experience.
By the end of this tutorial, you’ll gain a solid understanding of how Milvus and Dynamiq work together to build scalable, context-aware AI systems tailored to your needs.
+$ pip install dynamiq pymilvus
+If you are using Google Colab, to enable dependencies just installed, you may need to restart the runtime (click on the “Runtime” menu at the top of the screen, and select “Restart session” from the dropdown menu).
+We will use OpenAI as the LLM in this example. You should prepare the api key OPENAI_API_KEY as an environment variable.
import os
+
+os.environ["OPENAI_API_KEY"] = "sk-***********"
+This tutorial demonstrates a Retrieval-Augmented Generation (RAG) workflow for indexing documents with Milvus as the vector database. The workflow takes input PDF files, processes them into smaller chunks, generates vector embeddings using OpenAI’s embedding model, and stores the embeddings in a Milvus collection for efficient retrieval.
+By the end of this workflow, you will have a scalable and efficient document indexing system that supports future RAG tasks like semantic search and question answering.
+# Importing necessary libraries for the workflow
+from io import BytesIO
+from dynamiq import Workflow
+from dynamiq.nodes import InputTransformer
+from dynamiq.connections import (
+ OpenAI as OpenAIConnection,
+ Milvus as MilvusConnection,
+ MilvusDeploymentType,
+)
+from dynamiq.nodes.converters import PyPDFConverter
+from dynamiq.nodes.splitters.document import DocumentSplitter
+from dynamiq.nodes.embedders import OpenAIDocumentEmbedder
+from dynamiq.nodes.writers import MilvusDocumentWriter
+
+# Initialize the workflow
+rag_wf = Workflow()
+converter = PyPDFConverter(document_creation_mode="one-doc-per-page")
+converter_added = rag_wf.flow.add_nodes(
+ converter
+) # Add node to the DAG (Directed Acyclic Graph)
+document_splitter = DocumentSplitter(
+ split_by="sentence", # Splits documents into sentences
+ split_length=10,
+ split_overlap=1,
+ input_transformer=InputTransformer(
+ selector={
+ "documents": f"${[converter.id]}.output.documents",
+ },
+ ),
+).depends_on(
+ converter
+) # Set dependency on the PDF converter
+splitter_added = rag_wf.flow.add_nodes(document_splitter) # Add to the DAG
+embedder = OpenAIDocumentEmbedder(
+ connection=OpenAIConnection(api_key=os.environ["OPENAI_API_KEY"]),
+ input_transformer=InputTransformer(
+ selector={
+ "documents": f"${[document_splitter.id]}.output.documents",
+ },
+ ),
+).depends_on(
+ document_splitter
+) # Set dependency on the splitter
+document_embedder_added = rag_wf.flow.add_nodes(embedder) # Add to the DAG
+vector_store = (
+ MilvusDocumentWriter(
+ connection=MilvusConnection(
+ deployment_type=MilvusDeploymentType.FILE, uri="./milvus.db"
+ ),
+ index_name="my_milvus_collection",
+ dimension=1536,
+ create_if_not_exist=True,
+ metric_type="COSINE",
+ )
+ .inputs(documents=embedder.outputs.documents) # Connect to embedder output
+ .depends_on(embedder) # Set dependency on the embedder
+)
+milvus_writer_added = rag_wf.flow.add_nodes(vector_store) # Add to the DAG
+2024-11-19 22:14:03 - WARNING - Environment variable 'MILVUS_API_TOKEN' not found
+2024-11-19 22:14:03 - INFO - Pass in the local path ./milvus.db, and run it using milvus-lite
+2024-11-19 22:14:04 - DEBUG - Created new connection using: 0bef2849fdb1458a85df8bb9dd27f51d
+2024-11-19 22:14:04 - INFO - Collection my_milvus_collection does not exist. Creating a new collection.
+2024-11-19 22:14:04 - DEBUG - Successfully created collection: my_milvus_collection
+2024-11-19 22:14:05 - DEBUG - Successfully created an index on collection: my_milvus_collection
+2024-11-19 22:14:05 - DEBUG - Successfully created an index on collection: my_milvus_collection
+Milvus offers two deployment types, catering to different use cases:
+uri to a local file path (e.g., ./milvus.db) to leverage Milvus Lite, which automatically stores all data in the specified file.Designed for large-scale data scenarios, such as managing over a million vectors.
+Self-Hosted Server
+uri (e.g., http://localhost:19530).<your_username>:<your_password> as the token.token unset.Zilliz Cloud (Managed Service)
+uri and token according to the Public Endpoint and API key provided in the Zilliz Cloud console.file_paths = ["./pdf_files/WhatisMilvus.pdf"]
+input_data = {
+ "files": [BytesIO(open(path, "rb").read()) for path in file_paths],
+ "metadata": [{"filename": path} for path in file_paths],
+}
+
+# Run the workflow with the prepared input data
+inserted_data = rag_wf.run(input_data=input_data)
+/var/folders/09/d0hx80nj35sb5hxb5cpc1q180000gn/T/ipykernel_31319/3145804345.py:4: ResourceWarning: unclosed file <_io.BufferedReader name='./pdf_files/WhatisMilvus.pdf'>
+ BytesIO(open(path, "rb").read()) for path in file_paths
+ResourceWarning: Enable tracemalloc to get the object allocation traceback
+2024-11-19 22:14:09 - INFO - Workflow 87878444-6a3d-43f3-ae32-0127564a959f: execution started.
+2024-11-19 22:14:09 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution started.
+2024-11-19 22:14:09 - INFO - Node PyPDF File Converter - 6eb42b1f-7637-407b-a3ac-4167bcf3b5c4: execution started.
+2024-11-19 22:14:09 - INFO - Node PyPDF File Converter - 6eb42b1f-7637-407b-a3ac-4167bcf3b5c4: execution succeeded in 58ms.
+2024-11-19 22:14:09 - INFO - Node DocumentSplitter - 5baed580-6de0-4dcd-bace-d7d947ab6c7f: execution started.
+/Users/jinhonglin/anaconda3/envs/myenv/lib/python3.11/site-packages/websockets/legacy/__init__.py:6: DeprecationWarning: websockets.legacy is deprecated; see https://websockets.readthedocs.io/en/stable/howto/upgrade.html for upgrade instructions
+ warnings.warn( # deprecated in 14.0 - 2024-11-09
+/Users/jinhonglin/anaconda3/envs/myenv/lib/python3.11/site-packages/pydantic/fields.py:804: PydanticDeprecatedSince20: Using extra keyword arguments on `Field` is deprecated and will be removed. Use `json_schema_extra` instead. (Extra keys: 'is_accessible_to_agent'). Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/
+ warn(
+2024-11-19 22:14:09 - INFO - Node DocumentSplitter - 5baed580-6de0-4dcd-bace-d7d947ab6c7f: execution succeeded in 104ms.
+2024-11-19 22:14:09 - INFO - Node OpenAIDocumentEmbedder - 91928f67-a00f-48f6-a864-f6e21672ec7e: execution started.
+2024-11-19 22:14:09 - INFO - Node OpenAIDocumentEmbedder - d30a4cdc-0fab-4aff-b2e5-6161a62cb6fd: execution started.
+2024-11-19 22:14:10 - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
+2024-11-19 22:14:10 - INFO - Node OpenAIDocumentEmbedder - d30a4cdc-0fab-4aff-b2e5-6161a62cb6fd: execution succeeded in 724ms.
+2024-11-19 22:14:10 - INFO - Node MilvusDocumentWriter - dddab4cc-1dae-4e7e-9101-1ec353f530da: execution started.
+2024-11-19 22:14:10 - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
+2024-11-19 22:14:10 - INFO - Node MilvusDocumentWriter - dddab4cc-1dae-4e7e-9101-1ec353f530da: execution succeeded in 66ms.
+2024-11-19 22:14:10 - INFO - Node OpenAIDocumentEmbedder - 91928f67-a00f-48f6-a864-f6e21672ec7e: execution succeeded in 961ms.
+2024-11-19 22:14:10 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution succeeded in 1.3s.
+2024-11-19 22:14:10 - INFO - Workflow 87878444-6a3d-43f3-ae32-0127564a959f: execution succeeded in 1.3s.
+Through this workflow, we have successfully implemented a document indexing pipeline using Milvus as the vector database and OpenAI’s embedding model for semantic representation. This setup enables fast and accurate vector-based retrieval, forming the foundation for RAG workflows like semantic search, document retrieval, and contextual AI-driven interactions.
+With Milvus’s scalable storage capabilities and Dynamiq’s orchestration, this solution is ready for both prototyping and large-scale production deployments. You can now extend this pipeline to include additional tasks like retrieval-based question answering or AI-driven content generation.
+In this tutorial, we implement a Retrieval-Augmented Generation (RAG) document retrieval workflow. This workflow takes a user query, generates a vector embedding for it, retrieves the most relevant documents from a Milvus vector database, and uses a large language model (LLM) to generate a detailed and context-aware answer based on the retrieved documents.
+By following this workflow, you will create an end-to-end solution for semantic search and question answering, combining the power of vector-based document retrieval with the capabilities of OpenAI’s advanced LLMs. This approach enables efficient and intelligent responses to user queries by leveraging the stored knowledge in your document database.
+from dynamiq import Workflow
+from dynamiq.connections import (
+ OpenAI as OpenAIConnection,
+ Milvus as MilvusConnection,
+ MilvusDeploymentType,
+)
+from dynamiq.nodes.embedders import OpenAITextEmbedder
+from dynamiq.nodes.retrievers import MilvusDocumentRetriever
+from dynamiq.nodes.llms import OpenAI
+from dynamiq.prompts import Message, Prompt
+
+# Initialize the workflow
+retrieval_wf = Workflow()
+# Establish OpenAI connection
+openai_connection = OpenAIConnection(api_key=os.environ["OPENAI_API_KEY"])
+
+# Define the text embedder node
+embedder = OpenAITextEmbedder(
+ connection=openai_connection,
+ model="text-embedding-3-small",
+)
+
+# Add the embedder node to the workflow
+embedder_added = retrieval_wf.flow.add_nodes(embedder)
+document_retriever = (
+ MilvusDocumentRetriever(
+ connection=MilvusConnection(
+ deployment_type=MilvusDeploymentType.FILE, uri="./milvus.db"
+ ),
+ index_name="my_milvus_collection",
+ dimension=1536,
+ top_k=5,
+ )
+ .inputs(embedding=embedder.outputs.embedding) # Connect to embedder output
+ .depends_on(embedder) # Dependency on the embedder node
+)
+
+# Add the retriever node to the workflow
+milvus_retriever_added = retrieval_wf.flow.add_nodes(document_retriever)
+2024-11-19 22:14:19 - WARNING - Environment variable 'MILVUS_API_TOKEN' not found
+2024-11-19 22:14:19 - INFO - Pass in the local path ./milvus.db, and run it using milvus-lite
+2024-11-19 22:14:19 - DEBUG - Created new connection using: 98d1132773af4298a894ad5925845fd2
+2024-11-19 22:14:19 - INFO - Collection my_milvus_collection already exists. Skipping creation.
+# Define the prompt template for the LLM
+prompt_template = """
+Please answer the question based on the provided context.
+
+Question: {{ query }}
+
+Context:
+{% for document in documents %}
+- {{ document.content }}
+{% endfor %}
+"""
+
+# Create the prompt object
+prompt = Prompt(messages=[Message(content=prompt_template, role="user")])
+answer_generator = (
+ OpenAI(
+ connection=openai_connection,
+ model="gpt-4o",
+ prompt=prompt,
+ )
+ .inputs(
+ documents=document_retriever.outputs.documents,
+ query=embedder.outputs.query,
+ )
+ .depends_on(
+ [document_retriever, embedder]
+ ) # Dependencies on retriever and embedder
+)
+
+# Add the answer generator node to the workflow
+answer_generator_added = retrieval_wf.flow.add_nodes(answer_generator)
+# Run the workflow with a sample query
+sample_query = "What is the Advanced Search Algorithms in Milvus?"
+
+result = retrieval_wf.run(input_data={"query": sample_query})
+
+answer = result.output.get(answer_generator.id).get("output", {}).get("content")
+print(answer)
+2024-11-19 22:14:22 - INFO - Workflow f4a073fb-dfb6-499c-8cac-5710a7ad6d47: execution started.
+2024-11-19 22:14:22 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution started.
+2024-11-19 22:14:22 - INFO - Node OpenAITextEmbedder - 47afb0bc-cf96-429d-b58f-11b6c935fec3: execution started.
+2024-11-19 22:14:23 - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
+2024-11-19 22:14:23 - INFO - Node OpenAITextEmbedder - 47afb0bc-cf96-429d-b58f-11b6c935fec3: execution succeeded in 474ms.
+2024-11-19 22:14:23 - INFO - Node MilvusDocumentRetriever - 51c8311b-4837-411f-ba42-21e28239a2ee: execution started.
+2024-11-19 22:14:23 - INFO - Node MilvusDocumentRetriever - 51c8311b-4837-411f-ba42-21e28239a2ee: execution succeeded in 23ms.
+2024-11-19 22:14:23 - INFO - Node LLM - ac722325-bece-453f-a2ed-135b0749ee7a: execution started.
+2024-11-19 22:14:24 - INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
+2024-11-19 22:14:24 - INFO - Node LLM - ac722325-bece-453f-a2ed-135b0749ee7a: execution succeeded in 1.8s.
+2024-11-19 22:14:25 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution succeeded in 2.4s.
+2024-11-19 22:14:25 - INFO - Workflow f4a073fb-dfb6-499c-8cac-5710a7ad6d47: execution succeeded in 2.4s.
+
+
+The advanced search algorithms in Milvus include a variety of in-memory and on-disk indexing/search algorithms such as IVF (Inverted File), HNSW (Hierarchical Navigable Small World), and DiskANN. These algorithms have been deeply optimized to enhance performance, delivering 30%-70% better performance compared to popular implementations like FAISS and HNSWLib. These optimizations are part of Milvus's design to ensure high efficiency and scalability in handling vector data.
+// 4. Basic search with a JSON field
query_vectors = [[0.6765405125697714, 0.759217474274025, 0.4122471841491111, 0.3346805565394215, 0.09679748345514638]]
diff --git a/localization/v2.4.x/site/es/integrations/build_RAG_with_milvus_and_cognee.json b/localization/v2.4.x/site/es/integrations/build_RAG_with_milvus_and_cognee.json
new file mode 100644
index 000000000..20c8dceb2
--- /dev/null
+++ b/localization/v2.4.x/site/es/integrations/build_RAG_with_milvus_and_cognee.json
@@ -0,0 +1 @@
+{"codeList":["$ pip install pymilvus git+https://github.com/topoteretes/cognee.git\n","import os\n\nimport cognee\n\ncognee.config.set_llm_api_key(\"YOUR_OPENAI_API_KEY\")\n\n\nos.environ[\"VECTOR_DB_PROVIDER\"] = \"milvus\"\nos.environ[\"VECTOR_DB_URL\"] = \"./milvus.db\"\n","$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip\n$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs\n","from glob import glob\n\ntext_lines = []\n\nfor file_path in glob(\"milvus_docs/en/faq/*.md\", recursive=True):\n with open(file_path, \"r\") as file:\n file_text = file.read()\n\n text_lines += file_text.split(\"# \")\n","await cognee.prune.prune_data()\nawait cognee.prune.prune_system(metadata=True)\n","await cognee.add(data=text_lines, dataset_name=\"milvus_faq\")\nawait cognee.cognify()\n\n# [DocumentChunk(id=UUID('6889e7ef-3670-555c-bb16-3eb50d1d30b0'), updated_at=datetime.datetime(2024, 12, 4, 6, 29, 46, 472907, tzinfo=datetime.timezone.utc), text='Does the query perform in memory? What are incremental data and historical data?\\n\\nYes. When ...\n# ...\n","from cognee.api.v1.search import SearchType\n\nquery_text = \"How is data stored in milvus?\"\nsearch_results = await cognee.search(SearchType.SUMMARIES, query_text=query_text)\n\nprint(search_results[0])\n","from cognee.api.v1.search import SearchType\n\nquery_text = \"How is data stored in milvus?\"\nsearch_results = await cognee.search(SearchType.CHUNKS, query_text=query_text)\n","def format_and_print(data):\n print(\"ID:\", data[\"id\"])\n print(\"\\nText:\\n\")\n paragraphs = data[\"text\"].split(\"\\n\\n\")\n for paragraph in paragraphs:\n print(paragraph.strip())\n print()\n\n\nformat_and_print(search_results[0])\n","await cognee.prune.prune_data()\nawait cognee.prune.prune_system(metadata=True)\n","# We only use one line of text as the dataset, which simplifies the output later\ntext = \"\"\"\n Natural language processing (NLP) is an interdisciplinary\n subfield of computer science and information retrieval.\n \"\"\"\n\nawait cognee.add(text)\nawait cognee.cognify()\n","query_text = \"Tell me about NLP\"\nsearch_results = await cognee.search(SearchType.INSIGHTS, query_text=query_text)\n\nfor result_text in search_results:\n print(result_text)\n\n# Example output:\n# ({'id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'natural language processing', 'description': 'An interdisciplinary subfield of computer science and information retrieval.'}, {'relationship_name': 'is_a_subfield_of', 'source_node_id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'target_node_id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 15, 473137, tzinfo=datetime.timezone.utc)}, {'id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'computer science', 'description': 'The study of computation and information processing.'})\n# (...)\n#\n# It represents nodes and relationships in the knowledge graph:\n# - The first element is the source node (e.g., 'natural language processing').\n# - The second element is the relationship between nodes (e.g., 'is_a_subfield_of').\n# - The third element is the target node (e.g., 'computer science').\n"],"headingContent":"","anchorList":[{"label":"Construir RAG","href":"Build-RAG","type":2,"isActive":false}]}
\ No newline at end of file
diff --git a/localization/v2.4.x/site/es/integrations/build_RAG_with_milvus_and_cognee.md b/localization/v2.4.x/site/es/integrations/build_RAG_with_milvus_and_cognee.md
new file mode 100644
index 000000000..3354cc518
--- /dev/null
+++ b/localization/v2.4.x/site/es/integrations/build_RAG_with_milvus_and_cognee.md
@@ -0,0 +1,159 @@
+---
+id: build_RAG_with_milvus_and_cognee.md
+summary: >-
+ En este tutorial, le mostraremos cómo construir un pipeline RAG
+ (Retrieval-Augmented Generation) con Milvus y Cognee.
+title: Construir RAG con Milvus y Cognee
+---
+
+Construir RAG con Milvus y Cognee
Cognee es una plataforma "developer-first" que agiliza el desarrollo de aplicaciones de IA con pipelines ECL (Extract, Cognify, Load) escalables y modulares. Al integrarse perfectamente con Milvus, Cognee permite una conexión y recuperación eficientes de conversaciones, documentos y transcripciones, reduciendo las alucinaciones y optimizando los costes operativos.
+Con un sólido soporte para almacenes vectoriales como Milvus, bases de datos de grafos y LLM, Cognee proporciona un marco flexible y personalizable para construir sistemas de generación aumentada de recuperación (RAG). Su arquitectura lista para la producción garantiza una mayor precisión y eficiencia para las aplicaciones impulsadas por IA.
+En este tutorial, le mostraremos cómo construir una canalización RAG (Retrieval-Augmented Generation) con Milvus y Cognee.
+$ pip install pymilvus git+https://github.com/topoteretes/cognee.git
+
+
+Si estás utilizando Google Colab, para habilitar las dependencias que acabas de instalar, es posible que tengas que reiniciar el tiempo de ejecución (haz clic en el menú "Tiempo de ejecución" en la parte superior de la pantalla, y selecciona "Reiniciar sesión" en el menú desplegable).
+
+Por defecto, utiliza OpenAI como LLM en este ejemplo. Debe preparar la clave api, y establecerla en la función config set_llm_api_key().
+Para configurar Milvus como la base de datos vectorial, establezca el VECTOR_DB_PROVIDER a milvus y especifique el VECTOR_DB_URL y VECTOR_DB_KEY. Dado que en esta demo utilizamos Milvus Lite para almacenar datos, sólo es necesario proporcionar VECTOR_DB_URL.
+import os
+
+import cognee
+
+cognee.config.set_llm_api_key("YOUR_OPENAI_API_KEY")
+
+
+os.environ["VECTOR_DB_PROVIDER"] = "milvus"
+os.environ["VECTOR_DB_URL"] = "./milvus.db"
+
+
+En cuanto a las variables de entorno VECTOR_DB_URL y VECTOR_DB_KEY:
+
+- Establecer
VECTOR_DB_URL como un archivo local, por ejemplo./milvus.db, es el método más conveniente, ya que utiliza automáticamente Milvus Lite para almacenar todos los datos en este archivo.
+- Si tiene una gran escala de datos, puede configurar un servidor Milvus de mayor rendimiento en docker o kubernetes. En esta configuración, por favor utilice la uri del servidor, por ejemplo
http://localhost:19530, como su VECTOR_DB_URL.
+- Si desea utilizar Zilliz Cloud, el servicio en la nube totalmente gestionado para Milvus, ajuste
VECTOR_DB_URL y VECTOR_DB_KEY, que corresponden al punto final público y a la clave Api en Zilliz Cloud.
+
+
+Preparar los datos
Utilizamos las páginas de preguntas frecuentes de la Documentación de Milvus 2.4.x como conocimiento privado en nuestra RAG, que es una buena fuente de datos para una canalización RAG sencilla.
+Descargamos el archivo zip y extraemos los documentos a la carpeta milvus_docs.
+$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
+
+Cargamos todos los archivos markdown de la carpeta milvus_docs/en/faq. Para cada documento, simplemente utilizamos "# " para separar el contenido del archivo, lo que puede separar aproximadamente el contenido de cada parte principal del archivo markdown.
+from glob import glob
+
+text_lines = []
+
+for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
+ with open(file_path, "r") as file:
+ file_text = file.read()
+
+ text_lines += file_text.split("# ")
+
+Construir RAG
Restablecer los datos de Cognee
await cognee.prune.prune_data()
+await cognee.prune.prune_system(metadata=True)
+
+Con una pizarra limpia lista, ahora podemos añadir nuestro conjunto de datos y procesarlo en un gráfico de conocimiento.
+Añadir datos y Cognee
await cognee.add(data=text_lines, dataset_name="milvus_faq")
+await cognee.cognify()
+
+# [DocumentChunk(id=UUID('6889e7ef-3670-555c-bb16-3eb50d1d30b0'), updated_at=datetime.datetime(2024, 12, 4, 6, 29, 46, 472907, tzinfo=datetime.timezone.utc), text='Does the query perform in memory? What are incremental data and historical data?\n\nYes. When ...
+# ...
+
+El método add carga el conjunto de datos (Milvus FAQs) en Cognee y el método cognify procesa los datos para extraer entidades, relaciones y resúmenes, construyendo un grafo de conocimiento.
+Consulta de resúmenes
Una vez procesados los datos, vamos a consultar el grafo de conocimiento.
+from cognee.api.v1.search import SearchType
+
+query_text = "How is data stored in milvus?"
+search_results = await cognee.search(SearchType.SUMMARIES, query_text=query_text)
+
+print(search_results[0])
+
+{'id': 'de5c6713-e079-5d0b-b11d-e9bacd1e0d73', 'text': 'Milvus stores two data types: inserted data and metadata.'}
+
+Esta consulta busca en el grafo de conocimiento un resumen relacionado con el texto de la consulta, y se imprime el candidato más relacionado.
+Consulta de fragmentos
Los resúmenes ofrecen información de alto nivel, pero para obtener detalles más detallados, podemos consultar fragmentos específicos de datos directamente desde el conjunto de datos procesado. Estos fragmentos se derivan de los datos originales que se añadieron y analizaron durante la creación del gráfico de conocimiento.
+from cognee.api.v1.search import SearchType
+
+query_text = "How is data stored in milvus?"
+search_results = await cognee.search(SearchType.CHUNKS, query_text=query_text)
+
+Formateémoslos y visualicémoslos para facilitar su lectura.
+def format_and_print(data):
+ print("ID:", data["id"])
+ print("\nText:\n")
+ paragraphs = data["text"].split("\n\n")
+ for paragraph in paragraphs:
+ print(paragraph.strip())
+ print()
+
+
+format_and_print(search_results[0])
+
+ID: 4be01c4b-9ee5-541c-9b85-297883934ab3
+
+Text:
+
+Where does Milvus store data?
+
+Milvus deals with two types of data, inserted data and metadata.
+
+Inserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).
+
+Metadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.
+
+###
+
+En los pasos anteriores, consultamos el conjunto de datos de preguntas frecuentes de Milvus para obtener resúmenes y fragmentos específicos de datos. Aunque esto proporcionó información detallada y granular, el conjunto de datos era grande, lo que dificultaba la visualización clara de las dependencias dentro del gráfico de conocimiento.
+Para solucionar este problema, reiniciaremos el entorno Cognee y trabajaremos con un conjunto de datos más pequeño y específico. Esto nos permitirá demostrar mejor las relaciones y dependencias extraídas durante el proceso de Cognee. Al simplificar los datos, podemos ver claramente cómo Cognee organiza y estructura la información en el grafo de conocimiento.
+Reiniciar Cognee
await cognee.prune.prune_data()
+await cognee.prune.prune_system(metadata=True)
+
+Añadir el conjunto de datos focalizado
Aquí, un conjunto de datos más pequeño con sólo una línea de texto es añadido y procesado para asegurar un grafo de conocimiento enfocado y fácilmente interpretable.
+# We only use one line of text as the dataset, which simplifies the output later
+text = """
+ Natural language processing (NLP) is an interdisciplinary
+ subfield of computer science and information retrieval.
+ """
+
+await cognee.add(text)
+await cognee.cognify()
+
+Búsqueda de información
Al centrarnos en este conjunto de datos más pequeño, podemos analizar claramente las relaciones y la estructura del gráfico de conocimiento.
+query_text = "Tell me about NLP"
+search_results = await cognee.search(SearchType.INSIGHTS, query_text=query_text)
+
+for result_text in search_results:
+ print(result_text)
+
+# Example output:
+# ({'id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'natural language processing', 'description': 'An interdisciplinary subfield of computer science and information retrieval.'}, {'relationship_name': 'is_a_subfield_of', 'source_node_id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'target_node_id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 15, 473137, tzinfo=datetime.timezone.utc)}, {'id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'computer science', 'description': 'The study of computation and information processing.'})
+# (...)
+#
+# It represents nodes and relationships in the knowledge graph:
+# - The first element is the source node (e.g., 'natural language processing').
+# - The second element is the relationship between nodes (e.g., 'is_a_subfield_of').
+# - The third element is the target node (e.g., 'computer science').
+
+Esta salida representa los resultados de una consulta del grafo de conocimiento, mostrando las entidades (nodos) y sus relaciones (aristas) extraídas del conjunto de datos procesado. Cada tupla incluye una entidad de origen, un tipo de relación y una entidad de destino, junto con metadatos como identificadores únicos, descripciones y marcas de tiempo. El gráfico destaca los conceptos clave y sus conexiones semánticas, proporcionando una comprensión estructurada del conjunto de datos.
+Enhorabuena, ha aprendido el uso básico de cognee con Milvus. Si desea conocer el uso más avanzado de cognee, por favor consulte su página oficial .
diff --git a/localization/v2.4.x/site/es/integrations/build_RAG_with_milvus_and_gemini.json b/localization/v2.4.x/site/es/integrations/build_RAG_with_milvus_and_gemini.json
new file mode 100644
index 000000000..a578ef4d8
--- /dev/null
+++ b/localization/v2.4.x/site/es/integrations/build_RAG_with_milvus_and_gemini.json
@@ -0,0 +1 @@
+{"codeList":["$ pip install --upgrade pymilvus google-generativeai requests tqdm\n","import os\n\nos.environ[\"GEMINI_API_KEY\"] = \"***********\"\n","$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip\n$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs\n","from glob import glob\n\ntext_lines = []\n\nfor file_path in glob(\"milvus_docs/en/faq/*.md\", recursive=True):\n with open(file_path, \"r\") as file:\n file_text = file.read()\n\n text_lines += file_text.split(\"# \")\n","import google.generativeai as genai\n\ngenai.configure(api_key=os.environ[\"GEMINI_API_KEY\"])\n\ngemini_model = genai.GenerativeModel(\"gemini-1.5-flash\")\n\nresponse = gemini_model.generate_content(\"who are you\")\nprint(response.text)\n","test_embeddings = genai.embed_content(\n model=\"models/text-embedding-004\", content=[\"This is a test1\", \"This is a test2\"]\n)[\"embedding\"]\n\nembedding_dim = len(test_embeddings[0])\nprint(embedding_dim)\nprint(test_embeddings[0][:10])\n","from pymilvus import MilvusClient\n\nmilvus_client = MilvusClient(uri=\"./milvus_demo.db\")\n\ncollection_name = \"my_rag_collection\"\n","if milvus_client.has_collection(collection_name):\n milvus_client.drop_collection(collection_name)\n","milvus_client.create_collection(\n collection_name=collection_name,\n dimension=embedding_dim,\n metric_type=\"IP\", # Inner product distance\n consistency_level=\"Strong\", # Strong consistency level\n)\n","from tqdm import tqdm\n\ndata = []\n\ndoc_embeddings = genai.embed_content(\n model=\"models/text-embedding-004\", content=text_lines\n)[\"embedding\"]\n\nfor i, line in enumerate(tqdm(text_lines, desc=\"Creating embeddings\")):\n data.append({\"id\": i, \"vector\": doc_embeddings[i], \"text\": line})\n\nmilvus_client.insert(collection_name=collection_name, data=data)\n","question = \"How is data stored in milvus?\"\n","question_embedding = genai.embed_content(\n model=\"models/text-embedding-004\", content=question\n)[\"embedding\"]\n\nsearch_res = milvus_client.search(\n collection_name=collection_name,\n data=[question_embedding],\n limit=3, # Return top 3 results\n search_params={\"metric_type\": \"IP\", \"params\": {}}, # Inner product distance\n output_fields=[\"text\"], # Return the text field\n)\n","import json\n\nretrieved_lines_with_distances = [\n (res[\"entity\"][\"text\"], res[\"distance\"]) for res in search_res[0]\n]\nprint(json.dumps(retrieved_lines_with_distances, indent=4))\n","context = \"\\n\".join(\n [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]\n)\n","SYSTEM_PROMPT = \"\"\"\nHuman: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.\n\"\"\"\nUSER_PROMPT = f\"\"\"\nUse the following pieces of information enclosed in tags to provide an answer to the question enclosed in tags.\n\n{context}\n \n\n{question}\n \n\"\"\"\n","gemini_model = genai.GenerativeModel(\n \"gemini-1.5-flash\", system_instruction=SYSTEM_PROMPT\n)\nresponse = gemini_model.generate_content(USER_PROMPT)\nprint(response.text)\n"],"headingContent":"Build RAG with Milvus and Gemini","anchorList":[{"label":"Construir RAG con Milvus y Gemini","href":"Build-RAG-with-Milvus-and-Gemini","type":1,"isActive":false},{"label":"Preparación","href":"Preparation","type":2,"isActive":false},{"label":"Cargar datos en Milvus","href":"Load-data-into-Milvus","type":2,"isActive":false},{"label":"Construir RAG","href":"Build-RAG","type":2,"isActive":false}]}
\ No newline at end of file
diff --git a/localization/v2.4.x/site/es/integrations/build_RAG_with_milvus_and_gemini.md b/localization/v2.4.x/site/es/integrations/build_RAG_with_milvus_and_gemini.md
new file mode 100644
index 000000000..dbfba0f0b
--- /dev/null
+++ b/localization/v2.4.x/site/es/integrations/build_RAG_with_milvus_and_gemini.md
@@ -0,0 +1,246 @@
+---
+id: build_RAG_with_milvus_and_gemini.md
+summary: >-
+ En este tutorial, le mostraremos cómo construir una canalización RAG
+ (Retrieval-Augmented Generation) con Milvus y Gemini. Utilizaremos el modelo
+ Gemini para generar texto basado en una consulta dada. También utilizaremos
+ Milvus para almacenar y recuperar el texto generado.
+title: Construir RAG con Milvus y Gemini
+---
+
+Construir RAG con Milvus y Gemini
La API Gemini y Google AI Studio te ayudan a empezar a trabajar con los modelos más recientes de Google y a convertir tus ideas en aplicaciones escalables. Gemini proporciona acceso a potentes modelos lingüísticos como Gemini-1.5-Flash, Gemini-1.5-Flash-8B, y Gemini-1.5-Pro para tareas como la generación de textos, el procesamiento de documentos, la visión, el análisis de audio y mucho más. La API permite introducir contextos largos con millones de tokens, ajustar modelos para tareas específicas, generar resultados estructurados como JSON y aprovechar funciones como la recuperación semántica y la ejecución de código.
+En este tutorial, le mostraremos cómo construir una canalización RAG (Retrieval-Augmented Generation) con Milvus y Gemini. Utilizaremos el modelo Gemini para generar texto basado en una consulta dada. También utilizaremos Milvus para almacenar y recuperar el texto generado.
+Preparación
Dependencias y entorno
$ pip install --upgrade pymilvus google-generativeai requests tqdm
+
+
+Si utilizas Google Colab, para habilitar las dependencias que acabas de instalar, es posible que tengas que reiniciar el tiempo de ejecución (haz clic en el menú "Tiempo de ejecución" en la parte superior de la pantalla y selecciona "Reiniciar sesión" en el menú desplegable).
+
+Primero debes iniciar sesión en la plataforma Google AI Studio y preparar la clave api GEMINI_API_KEY como variable de entorno.
+import os
+
+os.environ["GEMINI_API_KEY"] = "***********"
+
+Preparar los datos
Utilizamos las páginas de preguntas frecuentes de la Documentación de Milvus 2.4.x como conocimiento privado en nuestra RAG, que es una buena fuente de datos para una canalización RAG sencilla.
+Descargamos el archivo zip y extraemos los documentos a la carpeta milvus_docs.
+$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
+
+Cargamos todos los archivos markdown de la carpeta milvus_docs/en/faq. Para cada documento, simplemente utilizamos "# " para separar el contenido en el archivo, lo que puede separar aproximadamente el contenido de cada parte principal del archivo markdown.
+from glob import glob
+
+text_lines = []
+
+for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
+ with open(file_path, "r") as file:
+ file_text = file.read()
+
+ text_lines += file_text.split("# ")
+
+Preparar el LLM y el modelo de incrustación
Utilizamos gemini-1.5-flash como LLM y text-embedding-004 como modelo de incrustación.
+Intentemos generar una respuesta de prueba a partir del LLM:
+import google.generativeai as genai
+
+genai.configure(api_key=os.environ["GEMINI_API_KEY"])
+
+gemini_model = genai.GenerativeModel("gemini-1.5-flash")
+
+response = gemini_model.generate_content("who are you")
+print(response.text)
+
+I am a large language model, trained by Google. I am an AI and don't have a personal identity or consciousness. My purpose is to process information and respond to a wide range of prompts and questions in a helpful and informative way.
+
+Generar un embedding de prueba e imprimir su dimensión y sus primeros elementos.
+test_embeddings = genai.embed_content(
+ model="models/text-embedding-004", content=["This is a test1", "This is a test2"]
+)["embedding"]
+
+embedding_dim = len(test_embeddings[0])
+print(embedding_dim)
+print(test_embeddings[0][:10])
+
+768
+[0.013588584, -0.004361838, -0.08481652, -0.039724775, 0.04723794, -0.0051557426, 0.026071774, 0.045514572, -0.016867816, 0.039378334]
+
+Cargar datos en Milvus
Crear la colección
from pymilvus import MilvusClient
+
+milvus_client = MilvusClient(uri="./milvus_demo.db")
+
+collection_name = "my_rag_collection"
+
+
+En cuanto al argumento de MilvusClient:
+
+- Establecer el
uri como un archivo local, por ejemplo./milvus.db, es el método más conveniente, ya que utiliza automáticamente Milvus Lite para almacenar todos los datos en este archivo.
+- Si tiene una gran escala de datos, puede configurar un servidor Milvus más eficiente en docker o kubernetes. En esta configuración, por favor utilice la uri del servidor, por ejemplo
http://localhost:19530, como su uri.
+- Si desea utilizar Zilliz Cloud, el servicio en la nube totalmente gestionado para Milvus, ajuste
uri y token, que corresponden al punto final público y a la clave Api en Zilliz Cloud.
+
+
+Compruebe si la colección ya existe y elimínela en caso afirmativo.
+if milvus_client.has_collection(collection_name):
+ milvus_client.drop_collection(collection_name)
+
+Crear una nueva colección con los parámetros especificados.
+Si no especificamos ninguna información de campo, Milvus creará automáticamente un campo id por defecto para la clave primaria, y un campo vector para almacenar los datos vectoriales. Se utiliza un campo JSON reservado para almacenar campos no definidos por el esquema y sus valores.
+milvus_client.create_collection(
+ collection_name=collection_name,
+ dimension=embedding_dim,
+ metric_type="IP", # Inner product distance
+ consistency_level="Strong", # Strong consistency level
+)
+
+Insertar datos
Iterar a través de las líneas de texto, crear incrustaciones, y luego insertar los datos en Milvus.
+Aquí hay un nuevo campo text, que es un campo no definido en el esquema de la colección. Se añadirá automáticamente al campo dinámico JSON reservado, que puede tratarse como un campo normal a alto nivel.
+from tqdm import tqdm
+
+data = []
+
+doc_embeddings = genai.embed_content(
+ model="models/text-embedding-004", content=text_lines
+)["embedding"]
+
+for i, line in enumerate(tqdm(text_lines, desc="Creating embeddings")):
+ data.append({"id": i, "vector": doc_embeddings[i], "text": line})
+
+milvus_client.insert(collection_name=collection_name, data=data)
+
+Creating embeddings: 100%|██████████| 72/72 [00:00<00:00, 468201.38it/s]
+
+
+
+
+
+{'insert_count': 72, 'ids': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71], 'cost': 0}
+
+Construir RAG
Recuperar datos para una consulta
Especifiquemos una pregunta frecuente sobre Milvus.
+question = "How is data stored in milvus?"
+
+Busquemos la pregunta en la colección y recuperemos las 3 primeras coincidencias semánticas.
+question_embedding = genai.embed_content(
+ model="models/text-embedding-004", content=question
+)["embedding"]
+
+search_res = milvus_client.search(
+ collection_name=collection_name,
+ data=[question_embedding],
+ limit=3, # Return top 3 results
+ search_params={"metric_type": "IP", "params": {}}, # Inner product distance
+ output_fields=["text"], # Return the text field
+)
+
+Veamos los resultados de la consulta
+import json
+
+retrieved_lines_with_distances = [
+ (res["entity"]["text"], res["distance"]) for res in search_res[0]
+]
+print(json.dumps(retrieved_lines_with_distances, indent=4))
+
+[
+ [
+ " Where does Milvus store data?\n\nMilvus deals with two types of data, inserted data and metadata. \n\nInserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).\n\nMetadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.\n\n###",
+ 0.8048275113105774
+ ],
+ [
+ "Does the query perform in memory? What are incremental data and historical data?\n\nYes. When a query request comes, Milvus searches both incremental data and historical data by loading them into memory. Incremental data are in the growing segments, which are buffered in memory before they reach the threshold to be persisted in storage engine, while historical data are from the sealed segments that are stored in the object storage. Incremental data and historical data together constitute the whole dataset to search.\n\n###",
+ 0.7574886679649353
+ ],
+ [
+ "What is the maximum dataset size Milvus can handle?\n\n \nTheoretically, the maximum dataset size Milvus can handle is determined by the hardware it is run on, specifically system memory and storage:\n\n- Milvus loads all specified collections and partitions into memory before running queries. Therefore, memory size determines the maximum amount of data Milvus can query.\n- When new entities and and collection-related schema (currently only MinIO is supported for data persistence) are added to Milvus, system storage determines the maximum allowable size of inserted data.\n\n###",
+ 0.7453608512878418
+ ]
+]
+
+Utilizar LLM para obtener una respuesta RAG
Convertir los documentos recuperados en un formato de cadena.
+context = "\n".join(
+ [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]
+)
+
+Definir avisos de sistema y de usuario para el modelo de lenguaje. Este prompt se ensambla con los documentos recuperados de Milvus.
+SYSTEM_PROMPT = """
+Human: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.
+"""
+USER_PROMPT = f"""
+Use the following pieces of information enclosed in <context> tags to provide an answer to the question enclosed in <question> tags.
+<context>
+{context}
+</context>
+<question>
+{question}
+</question>
+"""
+
+Utilice Gemini para generar una respuesta basada en las instrucciones.
+gemini_model = genai.GenerativeModel(
+ "gemini-1.5-flash", system_instruction=SYSTEM_PROMPT
+)
+response = gemini_model.generate_content(USER_PROMPT)
+print(response.text)
+
+Milvus stores data in two ways: Inserted data (vector data, scalar data, and collection-specific schema) is stored as an incremental log in persistent storage using object storage backends such as MinIO, AWS S3, Google Cloud Storage, Azure Blob Storage, Alibaba Cloud OSS, and Tencent Cloud Object Storage. Metadata, generated by each Milvus module, is stored in etcd.
+
+Muy bien. Hemos construido con éxito una canalización RAG con Milvus y Gemini.
diff --git a/localization/v2.4.x/site/es/integrations/build_RAG_with_milvus_and_ollama.json b/localization/v2.4.x/site/es/integrations/build_RAG_with_milvus_and_ollama.json
new file mode 100644
index 000000000..10f9e33e3
--- /dev/null
+++ b/localization/v2.4.x/site/es/integrations/build_RAG_with_milvus_and_ollama.json
@@ -0,0 +1 @@
+{"codeList":["$ pip install pymilvus ollama\n","$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip\n$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs\n","from glob import glob\n\ntext_lines = []\n\nfor file_path in glob(\"milvus_docs/en/faq/*.md\", recursive=True):\n with open(file_path, \"r\") as file:\n file_text = file.read()\n\n text_lines += file_text.split(\"# \")\n","! ollama pull mxbai-embed-large\n","! ollama pull llama3.2\n","import ollama\n\n\ndef emb_text(text):\n response = ollama.embeddings(model=\"mxbai-embed-large\", prompt=text)\n return response[\"embedding\"]\n","test_embedding = emb_text(\"This is a test\")\nembedding_dim = len(test_embedding)\nprint(embedding_dim)\nprint(test_embedding[:10])\n","from pymilvus import MilvusClient\n\nmilvus_client = MilvusClient(uri=\"./milvus_demo.db\")\n\ncollection_name = \"my_rag_collection\"\n","if milvus_client.has_collection(collection_name):\n milvus_client.drop_collection(collection_name)\n","milvus_client.create_collection(\n collection_name=collection_name,\n dimension=embedding_dim,\n metric_type=\"IP\", # Inner product distance\n consistency_level=\"Strong\", # Strong consistency level\n)\n","from tqdm import tqdm\n\ndata = []\n\nfor i, line in enumerate(tqdm(text_lines, desc=\"Creating embeddings\")):\n data.append({\"id\": i, \"vector\": emb_text(line), \"text\": line})\n\nmilvus_client.insert(collection_name=collection_name, data=data)\n","question = \"How is data stored in milvus?\"\n","search_res = milvus_client.search(\n collection_name=collection_name,\n data=[\n emb_text(question)\n ], # Use the `emb_text` function to convert the question to an embedding vector\n limit=3, # Return top 3 results\n search_params={\"metric_type\": \"IP\", \"params\": {}}, # Inner product distance\n output_fields=[\"text\"], # Return the text field\n)\n","import json\n\nretrieved_lines_with_distances = [\n (res[\"entity\"][\"text\"], res[\"distance\"]) for res in search_res[0]\n]\nprint(json.dumps(retrieved_lines_with_distances, indent=4))\n","context = \"\\n\".join(\n [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]\n)\n","SYSTEM_PROMPT = \"\"\"\nHuman: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.\n\"\"\"\nUSER_PROMPT = f\"\"\"\nUse the following pieces of information enclosed in tags to provide an answer to the question enclosed in tags.\n\n{context}\n \n\n{question}\n \n\"\"\"\n","from ollama import chat\nfrom ollama import ChatResponse\n\nresponse: ChatResponse = chat(\n model=\"llama3.2\",\n messages=[\n {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n {\"role\": \"user\", \"content\": USER_PROMPT},\n ],\n)\nprint(response[\"message\"][\"content\"])\n"],"headingContent":"Build RAG with Milvus and Ollama","anchorList":[{"label":"Construir RAG con Milvus y Ollama","href":"Build-RAG-with-Milvus-and-Ollama","type":1,"isActive":false},{"label":"Preparación","href":"Preparation","type":2,"isActive":false},{"label":"Cargar datos en Milvus","href":"Load-data-into-Milvus","type":2,"isActive":false},{"label":"Construir RAG","href":"Build-RAG","type":2,"isActive":false}]}
\ No newline at end of file
diff --git a/localization/v2.4.x/site/es/integrations/build_RAG_with_milvus_and_ollama.md b/localization/v2.4.x/site/es/integrations/build_RAG_with_milvus_and_ollama.md
new file mode 100644
index 000000000..3366667ca
--- /dev/null
+++ b/localization/v2.4.x/site/es/integrations/build_RAG_with_milvus_and_ollama.md
@@ -0,0 +1,282 @@
+---
+id: build_RAG_with_milvus_and_ollama.md
+summary: >-
+ En esta guía, le mostraremos cómo aprovechar Ollama y Milvus para construir
+ una canalización RAG (Retrieval-Augmented Generation) de forma eficiente y
+ segura.
+title: Construir RAG con Milvus y Ollama
+---
+
+Construir RAG con Milvus y Ollama
Ollama es una plataforma de código abierto que simplifica la ejecución y personalización local de grandes modelos lingüísticos (LLM). Proporciona una experiencia fácil de usar y sin nubes, que permite descargar, instalar e interactuar con los modelos sin esfuerzo y sin necesidad de conocimientos técnicos avanzados. Con una creciente biblioteca de LLM preentrenados -desde los de propósito general hasta los específicos de un dominio-, Ollama facilita la gestión y personalización de modelos para diversas aplicaciones. Garantiza la privacidad de los datos y la flexibilidad, permitiendo a los usuarios ajustar, optimizar y desplegar soluciones basadas en IA completamente en sus máquinas.
+En esta guía, le mostraremos cómo aprovechar Ollama y Milvus para construir una canalización RAG (Retrieval-Augmented Generation) de forma eficiente y segura.
+Preparación
Dependencias y entorno
$ pip install pymilvus ollama
+
+
+Si utilizas Google Colab, para habilitar las dependencias que acabas de instalar, es posible que tengas que reiniciar el tiempo de ejecución (haz clic en el menú "Tiempo de ejecución" en la parte superior de la pantalla y selecciona "Reiniciar sesión" en el menú desplegable).
+
+Preparar los datos
Utilizamos las páginas de preguntas frecuentes de la Documentación de Milvus 2.4.x como conocimiento privado en nuestra RAG, que es una buena fuente de datos para una canalización RAG sencilla.
+Descargamos el archivo zip y extraemos los documentos a la carpeta milvus_docs.
+$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
+
+--2024-11-26 21:47:19-- https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+Resolving github.com (github.com)... 140.82.112.4
+Connecting to github.com (github.com)|140.82.112.4|:443... connected.
+HTTP request sent, awaiting response... 302 Found
+Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/267273319/c52902a0-e13c-4ca7-92e0-086751098a05?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241127%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241127T024720Z&X-Amz-Expires=300&X-Amz-Signature=7808b77cbdaa7e122196bcd75a73f29f2540333a350c4830bbdf5f286e876304&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dmilvus_docs_2.4.x_en.zip&response-content-type=application%2Foctet-stream [following]
+--2024-11-26 21:47:20-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/267273319/c52902a0-e13c-4ca7-92e0-086751098a05?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241127%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241127T024720Z&X-Amz-Expires=300&X-Amz-Signature=7808b77cbdaa7e122196bcd75a73f29f2540333a350c4830bbdf5f286e876304&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dmilvus_docs_2.4.x_en.zip&response-content-type=application%2Foctet-stream
+Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.109.133, 185.199.111.133, 185.199.108.133, ...
+Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.109.133|:443... connected.
+HTTP request sent, awaiting response... 200 OK
+Length: 613094 (599K) [application/octet-stream]
+Saving to: ‘milvus_docs_2.4.x_en.zip’
+
+milvus_docs_2.4.x_e 100%[===================>] 598.72K 1.20MB/s in 0.5s
+
+2024-11-26 21:47:20 (1.20 MB/s) - ‘milvus_docs_2.4.x_en.zip’ saved [613094/613094]
+
+Cargamos todos los archivos markdown de la carpeta milvus_docs/en/faq. Para cada documento, simplemente utilizamos "# " para separar el contenido en el archivo, lo que puede separar aproximadamente el contenido de cada parte principal del archivo markdown.
+from glob import glob
+
+text_lines = []
+
+for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
+ with open(file_path, "r") as file:
+ file_text = file.read()
+
+ text_lines += file_text.split("# ")
+
+Preparar el LLM y el modelo de incrustación
Ollama soporta múltiples modelos tanto para tareas basadas en LLM como para la generación de incrustaciones, lo que facilita el desarrollo de aplicaciones de generación aumentada por recuperación (RAG). Para esta configuración:
+
+- Utilizaremos Llama 3.2 (3B) como LLM para tareas de generación de texto.
+- Para la generación de incrustaciones, utilizaremos mxbai-embed-large, un modelo de 334M parámetros optimizado para la similitud semántica.
+
+Antes de empezar, asegúrese de que ambos modelos se extraen localmente:
+! ollama pull mxbai-embed-large
+
+[?25lpulling manifest ⠋ [?25h[?25l[2K[1Gpulling manifest ⠙ [?25h[?25l[2K[1Gpulling manifest ⠹ [?25h[?25l[2K[1Gpulling manifest ⠸ [?25h[?25l[2K[1Gpulling manifest ⠼ [?25h[?25l[2K[1Gpulling manifest ⠴ [?25h[?25l[2K[1Gpulling manifest
+pulling 819c2adf5ce6... 100% ▕████████████████▏ 669 MB
+pulling c71d239df917... 100% ▕████████████████▏ 11 KB
+pulling b837481ff855... 100% ▕████████████████▏ 16 B
+pulling 38badd946f91... 100% ▕████████████████▏ 408 B
+verifying sha256 digest
+writing manifest
+success [?25h
+
+! ollama pull llama3.2
+
+[?25lpulling manifest ⠋ [?25h[?25l[2K[1Gpulling manifest ⠙ [?25h[?25l[2K[1Gpulling manifest ⠹ [?25h[?25l[2K[1Gpulling manifest ⠸ [?25h[?25l[2K[1Gpulling manifest ⠼ [?25h[?25l[2K[1Gpulling manifest ⠴ [?25h[?25l[2K[1Gpulling manifest
+pulling dde5aa3fc5ff... 100% ▕████████████████▏ 2.0 GB
+pulling 966de95ca8a6... 100% ▕████████████████▏ 1.4 KB
+pulling fcc5a6bec9da... 100% ▕████████████████▏ 7.7 KB
+pulling a70ff7e570d9... 100% ▕████████████████▏ 6.0 KB
+pulling 56bb8bd477a5... 100% ▕████████████████▏ 96 B
+pulling 34bb5ab01051... 100% ▕████████████████▏ 561 B
+verifying sha256 digest
+writing manifest
+success [?25h
+
+Con estos modelos listos, podemos proceder a implementar la generación basada en LLM y los flujos de trabajo de recuperación basados en incrustaciones.
+import ollama
+
+
+def emb_text(text):
+ response = ollama.embeddings(model="mxbai-embed-large", prompt=text)
+ return response["embedding"]
+
+Genere una incrustación de prueba e imprima su dimensión y sus primeros elementos.
+test_embedding = emb_text("This is a test")
+embedding_dim = len(test_embedding)
+print(embedding_dim)
+print(test_embedding[:10])
+
+1024
+[0.23276396095752716, 0.4257211685180664, 0.19724100828170776, 0.46120673418045044, -0.46039995551109314, -0.1413791924715042, -0.18261606991291046, -0.07602324336767197, 0.39991313219070435, 0.8337644338607788]
+
+Cargar datos en Milvus
Crear la colección
from pymilvus import MilvusClient
+
+milvus_client = MilvusClient(uri="./milvus_demo.db")
+
+collection_name = "my_rag_collection"
+
+
+Como para el argumento de MilvusClient:
+
+- Establecer el
uri como un archivo local, por ejemplo./milvus.db, es el método más conveniente, ya que utiliza automáticamente Milvus Lite para almacenar todos los datos en este archivo.
+- Si tiene una gran escala de datos, puede configurar un servidor Milvus más eficiente en docker o kubernetes. En esta configuración, por favor utilice la uri del servidor, por ejemplo
http://localhost:19530, como su uri.
+- Si desea utilizar Zilliz Cloud, el servicio en la nube totalmente gestionado para Milvus, ajuste
uri y token, que corresponden al punto final público y a la clave Api en Zilliz Cloud.
+
+
+Compruebe si la colección ya existe y elimínela en caso afirmativo.
+if milvus_client.has_collection(collection_name):
+ milvus_client.drop_collection(collection_name)
+
+Crear una nueva colección con los parámetros especificados.
+Si no especificamos ninguna información de campo, Milvus creará automáticamente un campo id por defecto para la clave primaria, y un campo vector para almacenar los datos vectoriales. Se utiliza un campo JSON reservado para almacenar campos no definidos por el esquema y sus valores.
+milvus_client.create_collection(
+ collection_name=collection_name,
+ dimension=embedding_dim,
+ metric_type="IP", # Inner product distance
+ consistency_level="Strong", # Strong consistency level
+)
+
+Insertar datos
Iterar a través de las líneas de texto, crear incrustaciones, y luego insertar los datos en Milvus.
+Aquí hay un nuevo campo text, que es un campo no definido en el esquema de la colección. Se añadirá automáticamente al campo dinámico JSON reservado, que puede tratarse como un campo normal a alto nivel.
+from tqdm import tqdm
+
+data = []
+
+for i, line in enumerate(tqdm(text_lines, desc="Creating embeddings")):
+ data.append({"id": i, "vector": emb_text(line), "text": line})
+
+milvus_client.insert(collection_name=collection_name, data=data)
+
+Creating embeddings: 100%|██████████| 72/72 [00:03<00:00, 22.56it/s]
+
+
+
+
+
+{'insert_count': 72, 'ids': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71], 'cost': 0}
+
+Construir RAG
Recuperar datos para una consulta
Especifiquemos una pregunta frecuente sobre Milvus.
+question = "How is data stored in milvus?"
+
+Busquemos la pregunta en la colección y recuperemos las 3 primeras coincidencias semánticas.
+search_res = milvus_client.search(
+ collection_name=collection_name,
+ data=[
+ emb_text(question)
+ ], # Use the `emb_text` function to convert the question to an embedding vector
+ limit=3, # Return top 3 results
+ search_params={"metric_type": "IP", "params": {}}, # Inner product distance
+ output_fields=["text"], # Return the text field
+)
+
+Veamos los resultados de la consulta
+import json
+
+retrieved_lines_with_distances = [
+ (res["entity"]["text"], res["distance"]) for res in search_res[0]
+]
+print(json.dumps(retrieved_lines_with_distances, indent=4))
+
+[
+ [
+ " Where does Milvus store data?\n\nMilvus deals with two types of data, inserted data and metadata. \n\nInserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).\n\nMetadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.\n\n###",
+ 231.9398193359375
+ ],
+ [
+ "How does Milvus flush data?\n\nMilvus returns success when inserted data are loaded to the message queue. However, the data are not yet flushed to the disk. Then Milvus' data node writes the data in the message queue to persistent storage as incremental logs. If `flush()` is called, the data node is forced to write all data in the message queue to persistent storage immediately.\n\n###",
+ 226.48316955566406
+ ],
+ [
+ "What is the maximum dataset size Milvus can handle?\n\n \nTheoretically, the maximum dataset size Milvus can handle is determined by the hardware it is run on, specifically system memory and storage:\n\n- Milvus loads all specified collections and partitions into memory before running queries. Therefore, memory size determines the maximum amount of data Milvus can query.\n- When new entities and and collection-related schema (currently only MinIO is supported for data persistence) are added to Milvus, system storage determines the maximum allowable size of inserted data.\n\n###",
+ 210.60745239257812
+ ]
+]
+
+Utilizar LLM para obtener una respuesta RAG
Convertir los documentos recuperados en un formato de cadena.
+context = "\n".join(
+ [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]
+)
+
+Definir avisos de sistema y de usuario para el modelo de lenguaje. Este prompt se ensambla con los documentos recuperados de Milvus.
+SYSTEM_PROMPT = """
+Human: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.
+"""
+USER_PROMPT = f"""
+Use the following pieces of information enclosed in <context> tags to provide an answer to the question enclosed in <question> tags.
+<context>
+{context}
+</context>
+<question>
+{question}
+</question>
+"""
+
+Utilice el modelo llama3.2 proporcionado por Ollama para generar una respuesta basada en las instrucciones.
+from ollama import chat
+from ollama import ChatResponse
+
+response: ChatResponse = chat(
+ model="llama3.2",
+ messages=[
+ {"role": "system", "content": SYSTEM_PROMPT},
+ {"role": "user", "content": USER_PROMPT},
+ ],
+)
+print(response["message"]["content"])
+
+According to the provided context, data in Milvus is stored in two types:
+
+1. **Inserted data**: Storing data in persistent storage as incremental log. It supports multiple object storage backends such as MinIO, AWS S3, Google Cloud Storage (GCS), Azure Blob Storage, Alibaba Cloud OSS, and Tencent Cloud Object Storage.
+
+2. **Metadata**: Generated within Milvus and stored in etcd.
+
+Muy bien. Hemos construido con éxito una tubería RAG con Milvus y Ollama.
diff --git a/localization/v2.4.x/site/es/integrations/milvus_rag_with_dynamiq.json b/localization/v2.4.x/site/es/integrations/milvus_rag_with_dynamiq.json
new file mode 100644
index 000000000..3530ec323
--- /dev/null
+++ b/localization/v2.4.x/site/es/integrations/milvus_rag_with_dynamiq.json
@@ -0,0 +1 @@
+{"codeList":["$ pip install dynamiq pymilvus\n","import os\n\nos.environ[\"OPENAI_API_KEY\"] = \"sk-***********\"\n","# Importing necessary libraries for the workflow\nfrom io import BytesIO\nfrom dynamiq import Workflow\nfrom dynamiq.nodes import InputTransformer\nfrom dynamiq.connections import (\n OpenAI as OpenAIConnection,\n Milvus as MilvusConnection,\n MilvusDeploymentType,\n)\nfrom dynamiq.nodes.converters import PyPDFConverter\nfrom dynamiq.nodes.splitters.document import DocumentSplitter\nfrom dynamiq.nodes.embedders import OpenAIDocumentEmbedder\nfrom dynamiq.nodes.writers import MilvusDocumentWriter\n\n# Initialize the workflow\nrag_wf = Workflow()\n","converter = PyPDFConverter(document_creation_mode=\"one-doc-per-page\")\nconverter_added = rag_wf.flow.add_nodes(\n converter\n) # Add node to the DAG (Directed Acyclic Graph)\n","document_splitter = DocumentSplitter(\n split_by=\"sentence\", # Splits documents into sentences\n split_length=10,\n split_overlap=1,\n input_transformer=InputTransformer(\n selector={\n \"documents\": f\"${[converter.id]}.output.documents\",\n },\n ),\n).depends_on(\n converter\n) # Set dependency on the PDF converter\nsplitter_added = rag_wf.flow.add_nodes(document_splitter) # Add to the DAG\n","embedder = OpenAIDocumentEmbedder(\n connection=OpenAIConnection(api_key=os.environ[\"OPENAI_API_KEY\"]),\n input_transformer=InputTransformer(\n selector={\n \"documents\": f\"${[document_splitter.id]}.output.documents\",\n },\n ),\n).depends_on(\n document_splitter\n) # Set dependency on the splitter\ndocument_embedder_added = rag_wf.flow.add_nodes(embedder) # Add to the DAG\n","vector_store = (\n MilvusDocumentWriter(\n connection=MilvusConnection(\n deployment_type=MilvusDeploymentType.FILE, uri=\"./milvus.db\"\n ),\n index_name=\"my_milvus_collection\",\n dimension=1536,\n create_if_not_exist=True,\n metric_type=\"COSINE\",\n )\n .inputs(documents=embedder.outputs.documents) # Connect to embedder output\n .depends_on(embedder) # Set dependency on the embedder\n)\nmilvus_writer_added = rag_wf.flow.add_nodes(vector_store) # Add to the DAG\n","file_paths = [\"./pdf_files/WhatisMilvus.pdf\"]\ninput_data = {\n \"files\": [BytesIO(open(path, \"rb\").read()) for path in file_paths],\n \"metadata\": [{\"filename\": path} for path in file_paths],\n}\n\n# Run the workflow with the prepared input data\ninserted_data = rag_wf.run(input_data=input_data)\n","from dynamiq import Workflow\nfrom dynamiq.connections import (\n OpenAI as OpenAIConnection,\n Milvus as MilvusConnection,\n MilvusDeploymentType,\n)\nfrom dynamiq.nodes.embedders import OpenAITextEmbedder\nfrom dynamiq.nodes.retrievers import MilvusDocumentRetriever\nfrom dynamiq.nodes.llms import OpenAI\nfrom dynamiq.prompts import Message, Prompt\n\n# Initialize the workflow\nretrieval_wf = Workflow()\n","# Establish OpenAI connection\nopenai_connection = OpenAIConnection(api_key=os.environ[\"OPENAI_API_KEY\"])\n\n# Define the text embedder node\nembedder = OpenAITextEmbedder(\n connection=openai_connection,\n model=\"text-embedding-3-small\",\n)\n\n# Add the embedder node to the workflow\nembedder_added = retrieval_wf.flow.add_nodes(embedder)\n","document_retriever = (\n MilvusDocumentRetriever(\n connection=MilvusConnection(\n deployment_type=MilvusDeploymentType.FILE, uri=\"./milvus.db\"\n ),\n index_name=\"my_milvus_collection\",\n dimension=1536,\n top_k=5,\n )\n .inputs(embedding=embedder.outputs.embedding) # Connect to embedder output\n .depends_on(embedder) # Dependency on the embedder node\n)\n\n# Add the retriever node to the workflow\nmilvus_retriever_added = retrieval_wf.flow.add_nodes(document_retriever)\n","# Define the prompt template for the LLM\nprompt_template = \"\"\"\nPlease answer the question based on the provided context.\n\nQuestion: {{ query }}\n\nContext:\n{% for document in documents %}\n- {{ document.content }}\n{% endfor %}\n\"\"\"\n\n# Create the prompt object\nprompt = Prompt(messages=[Message(content=prompt_template, role=\"user\")])\n","answer_generator = (\n OpenAI(\n connection=openai_connection,\n model=\"gpt-4o\",\n prompt=prompt,\n )\n .inputs(\n documents=document_retriever.outputs.documents,\n query=embedder.outputs.query,\n )\n .depends_on(\n [document_retriever, embedder]\n ) # Dependencies on retriever and embedder\n)\n\n# Add the answer generator node to the workflow\nanswer_generator_added = retrieval_wf.flow.add_nodes(answer_generator)\n","# Run the workflow with a sample query\nsample_query = \"What is the Advanced Search Algorithms in Milvus?\"\n\nresult = retrieval_wf.run(input_data={\"query\": sample_query})\n\nanswer = result.output.get(answer_generator.id).get(\"output\", {}).get(\"content\")\nprint(answer)\n"],"headingContent":"Getting Started with Dynamiq and Milvus","anchorList":[{"label":"Primeros pasos con Dynamiq y Milvus","href":"Getting-Started-with-Dynamiq-and-Milvus","type":1,"isActive":false},{"label":"Preparación","href":"Preparation","type":2,"isActive":false},{"label":"RAG - Flujo de indexación de documentos","href":"RAG---Document-Indexing-Flow","type":2,"isActive":false},{"label":"Flujo de recuperación de documentos RAG","href":"RAG-Document-Retrieval-Flow","type":2,"isActive":false}]}
\ No newline at end of file
diff --git a/localization/v2.4.x/site/es/integrations/milvus_rag_with_dynamiq.md b/localization/v2.4.x/site/es/integrations/milvus_rag_with_dynamiq.md
new file mode 100644
index 000000000..3c6079964
--- /dev/null
+++ b/localization/v2.4.x/site/es/integrations/milvus_rag_with_dynamiq.md
@@ -0,0 +1,340 @@
+---
+id: milvus_rag_with_dynamiq.md
+summary: >-
+ En este tutorial, exploraremos cómo utilizar Dynamiq con Milvus, la base de
+ datos vectorial de alto rendimiento diseñada especialmente para flujos de
+ trabajo RAG. Milvus sobresale en el almacenamiento eficiente, indexación y
+ recuperación de incrustaciones vectoriales, por lo que es un componente
+ indispensable para los sistemas de IA que exigen un acceso rápido y preciso a
+ los datos contextuales.
+title: Primeros pasos con Dynamiq y Milvus
+---
+
+Primeros pasos con Dynamiq y Milvus
Dynamiq es un potente marco de Gen AI que agiliza el desarrollo de aplicaciones basadas en IA. Gracias a su sólida compatibilidad con la generación aumentada por recuperación (RAG) y los grandes agentes de modelos de lenguaje (LLM), Dynamiq permite a los desarrolladores crear sistemas inteligentes y dinámicos con facilidad y eficacia.
+En este tutorial, exploraremos cómo utilizar Dynamiq sin problemas con Milvus, la base de datos vectorial de alto rendimiento creada especialmente para flujos de trabajo RAG. Milvus destaca en el almacenamiento, indexación y recuperación eficientes de incrustaciones vectoriales, lo que lo convierte en un componente indispensable para los sistemas de IA que exigen un acceso a datos contextuales rápido y preciso.
+Esta guía paso a paso cubrirá dos flujos de trabajo principales de RAG:
+
+Flujo de indexación de documentos: Aprenda a procesar archivos de entrada (por ejemplo, PDF), transformar su contenido en incrustaciones vectoriales y almacenarlos en Milvus. Aprovechar las capacidades de indexación de alto rendimiento de Milvus garantiza que sus datos estén listos para una rápida recuperación.
Flujo de recuperación de documentos: Descubra cómo consultar Milvus para incrustaciones de documentos relevantes y utilizarlos para generar respuestas perspicaces y conscientes del contexto con los agentes LLM de Dynamiq, creando una experiencia de usuario sin fisuras impulsada por la IA.
+Al final de este tutorial, obtendrá una sólida comprensión de cómo Milvus y Dynamiq trabajan juntos para construir sistemas de IA escalables y conscientes del contexto adaptados a sus necesidades.
+Preparación
Descarga de las bibliotecas necesarias
$ pip install dynamiq pymilvus
+
+
+Si utilizas Google Colab, para habilitar las dependencias que acabas de instalar, es posible que tengas que reiniciar el tiempo de ejecución (haz clic en el menú "Tiempo de ejecución" en la parte superior de la pantalla y selecciona "Reiniciar sesión" en el menú desplegable).
+
+Configurar el agente LLM
En este ejemplo utilizaremos OpenAI como LLM. Deberás preparar la clave api OPENAI_API_KEY como variable de entorno.
+import os
+
+os.environ["OPENAI_API_KEY"] = "sk-***********"
+
+RAG - Flujo de indexación de documentos
Este tutorial muestra un flujo de trabajo RAG (Retrieval-Augmented Generation) para indexar documentos con Milvus como base de datos vectorial. El flujo de trabajo toma archivos PDF de entrada, los procesa en trozos más pequeños, genera incrustaciones vectoriales utilizando el modelo de incrustación de OpenAI y almacena las incrustaciones en una colección Milvus para una recuperación eficiente.
+Al final de este flujo de trabajo, tendrá un sistema de indexación de documentos escalable y eficiente que soporta futuras tareas RAG como la búsqueda semántica y la respuesta a preguntas.
+Importe las bibliotecas necesarias e inicie el flujo de trabajo
# Importing necessary libraries for the workflow
+from io import BytesIO
+from dynamiq import Workflow
+from dynamiq.nodes import InputTransformer
+from dynamiq.connections import (
+ OpenAI as OpenAIConnection,
+ Milvus as MilvusConnection,
+ MilvusDeploymentType,
+)
+from dynamiq.nodes.converters import PyPDFConverter
+from dynamiq.nodes.splitters.document import DocumentSplitter
+from dynamiq.nodes.embedders import OpenAIDocumentEmbedder
+from dynamiq.nodes.writers import MilvusDocumentWriter
+
+# Initialize the workflow
+rag_wf = Workflow()
+
+Definir nodo conversor de PDF
converter = PyPDFConverter(document_creation_mode="one-doc-per-page")
+converter_added = rag_wf.flow.add_nodes(
+ converter
+) # Add node to the DAG (Directed Acyclic Graph)
+
+Definir nodo divisor de documentos
document_splitter = DocumentSplitter(
+ split_by="sentence", # Splits documents into sentences
+ split_length=10,
+ split_overlap=1,
+ input_transformer=InputTransformer(
+ selector={
+ "documents": f"${[converter.id]}.output.documents",
+ },
+ ),
+).depends_on(
+ converter
+) # Set dependency on the PDF converter
+splitter_added = rag_wf.flow.add_nodes(document_splitter) # Add to the DAG
+
+Definir nodo de incrustación
embedder = OpenAIDocumentEmbedder(
+ connection=OpenAIConnection(api_key=os.environ["OPENAI_API_KEY"]),
+ input_transformer=InputTransformer(
+ selector={
+ "documents": f"${[document_splitter.id]}.output.documents",
+ },
+ ),
+).depends_on(
+ document_splitter
+) # Set dependency on the splitter
+document_embedder_added = rag_wf.flow.add_nodes(embedder) # Add to the DAG
+
+Definir nodo Milvus Vector Store
vector_store = (
+ MilvusDocumentWriter(
+ connection=MilvusConnection(
+ deployment_type=MilvusDeploymentType.FILE, uri="./milvus.db"
+ ),
+ index_name="my_milvus_collection",
+ dimension=1536,
+ create_if_not_exist=True,
+ metric_type="COSINE",
+ )
+ .inputs(documents=embedder.outputs.documents) # Connect to embedder output
+ .depends_on(embedder) # Set dependency on the embedder
+)
+milvus_writer_added = rag_wf.flow.add_nodes(vector_store) # Add to the DAG
+
+2024-11-19 22:14:03 - WARNING - Environment variable 'MILVUS_API_TOKEN' not found
+2024-11-19 22:14:03 - INFO - Pass in the local path ./milvus.db, and run it using milvus-lite
+2024-11-19 22:14:04 - DEBUG - Created new connection using: 0bef2849fdb1458a85df8bb9dd27f51d
+2024-11-19 22:14:04 - INFO - Collection my_milvus_collection does not exist. Creating a new collection.
+2024-11-19 22:14:04 - DEBUG - Successfully created collection: my_milvus_collection
+2024-11-19 22:14:05 - DEBUG - Successfully created an index on collection: my_milvus_collection
+2024-11-19 22:14:05 - DEBUG - Successfully created an index on collection: my_milvus_collection
+
+
+Milvus ofrece dos tipos de despliegue, para diferentes casos de uso:
+
+- MilvusDeploymentType.FILE
+
+
+- Ideal para prototipos locales o almacenamiento de datos a pequeña escala.
+- Establezca
uri en una ruta de archivo local (por ejemplo, ./milvus.db) para aprovechar Milvus Lite, que almacena automáticamente todos los datos en el archivo especificado.
+- Esta es una opción conveniente para una rápida configuración y experimentación.
+
+
+- MilvusTipoDespliegue.HOST
+
+
+Diseñado para escenarios de datos a gran escala, como la gestión de más de un millón de vectores.
+Servidor autoalojado
+
+- Despliegue un servidor Milvus de alto rendimiento utilizando Docker o Kubernetes.
+- Configure la dirección y el puerto del servidor como
uri (por ejemplo, http://localhost:19530).
+- Si la autenticación está habilitada:
+- Proporcione
<your_username>:<your_password> como el token.
+- Si la autenticación está deshabilitada:
+- Deje
token sin configurar.
+
+Zilliz Cloud (servicio gestionado)
+
+- Para una experiencia Milvus totalmente gestionada y basada en la nube, utilice Zilliz Cloud.
+- Configure
uri y token de acuerdo con el punto final público y la clave de API proporcionados en la consola de Zilliz Cloud.
+
+
+Defina los datos de entrada y ejecute el flujo de trabajo
file_paths = ["./pdf_files/WhatisMilvus.pdf"]
+input_data = {
+ "files": [BytesIO(open(path, "rb").read()) for path in file_paths],
+ "metadata": [{"filename": path} for path in file_paths],
+}
+
+# Run the workflow with the prepared input data
+inserted_data = rag_wf.run(input_data=input_data)
+
+/var/folders/09/d0hx80nj35sb5hxb5cpc1q180000gn/T/ipykernel_31319/3145804345.py:4: ResourceWarning: unclosed file <_io.BufferedReader name='./pdf_files/WhatisMilvus.pdf'>
+ BytesIO(open(path, "rb").read()) for path in file_paths
+ResourceWarning: Enable tracemalloc to get the object allocation traceback
+2024-11-19 22:14:09 - INFO - Workflow 87878444-6a3d-43f3-ae32-0127564a959f: execution started.
+2024-11-19 22:14:09 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution started.
+2024-11-19 22:14:09 - INFO - Node PyPDF File Converter - 6eb42b1f-7637-407b-a3ac-4167bcf3b5c4: execution started.
+2024-11-19 22:14:09 - INFO - Node PyPDF File Converter - 6eb42b1f-7637-407b-a3ac-4167bcf3b5c4: execution succeeded in 58ms.
+2024-11-19 22:14:09 - INFO - Node DocumentSplitter - 5baed580-6de0-4dcd-bace-d7d947ab6c7f: execution started.
+/Users/jinhonglin/anaconda3/envs/myenv/lib/python3.11/site-packages/websockets/legacy/__init__.py:6: DeprecationWarning: websockets.legacy is deprecated; see https://websockets.readthedocs.io/en/stable/howto/upgrade.html for upgrade instructions
+ warnings.warn( # deprecated in 14.0 - 2024-11-09
+/Users/jinhonglin/anaconda3/envs/myenv/lib/python3.11/site-packages/pydantic/fields.py:804: PydanticDeprecatedSince20: Using extra keyword arguments on `Field` is deprecated and will be removed. Use `json_schema_extra` instead. (Extra keys: 'is_accessible_to_agent'). Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/
+ warn(
+2024-11-19 22:14:09 - INFO - Node DocumentSplitter - 5baed580-6de0-4dcd-bace-d7d947ab6c7f: execution succeeded in 104ms.
+2024-11-19 22:14:09 - INFO - Node OpenAIDocumentEmbedder - 91928f67-a00f-48f6-a864-f6e21672ec7e: execution started.
+2024-11-19 22:14:09 - INFO - Node OpenAIDocumentEmbedder - d30a4cdc-0fab-4aff-b2e5-6161a62cb6fd: execution started.
+2024-11-19 22:14:10 - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
+2024-11-19 22:14:10 - INFO - Node OpenAIDocumentEmbedder - d30a4cdc-0fab-4aff-b2e5-6161a62cb6fd: execution succeeded in 724ms.
+2024-11-19 22:14:10 - INFO - Node MilvusDocumentWriter - dddab4cc-1dae-4e7e-9101-1ec353f530da: execution started.
+2024-11-19 22:14:10 - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
+2024-11-19 22:14:10 - INFO - Node MilvusDocumentWriter - dddab4cc-1dae-4e7e-9101-1ec353f530da: execution succeeded in 66ms.
+2024-11-19 22:14:10 - INFO - Node OpenAIDocumentEmbedder - 91928f67-a00f-48f6-a864-f6e21672ec7e: execution succeeded in 961ms.
+2024-11-19 22:14:10 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution succeeded in 1.3s.
+2024-11-19 22:14:10 - INFO - Workflow 87878444-6a3d-43f3-ae32-0127564a959f: execution succeeded in 1.3s.
+
+A través de este flujo de trabajo, hemos implementado con éxito una canalización de indexación de documentos utilizando Milvus como base de datos vectorial y el modelo de incrustación de OpenAI para la representación semántica. Esta configuración permite una recuperación rápida y precisa basada en vectores, formando la base para flujos de trabajo RAG como la búsqueda semántica, la recuperación de documentos y las interacciones contextuales impulsadas por IA.
+Con las capacidades de almacenamiento escalable de Milvus y la orquestación de Dynamiq, esta solución está lista tanto para prototipos como para despliegues de producción a gran escala. Ahora puede ampliar esta canalización para incluir tareas adicionales, como la respuesta a preguntas basada en la recuperación o la generación de contenidos impulsada por la IA.
+Flujo de recuperación de documentos RAG
En este tutorial, implementamos un flujo de trabajo de recuperación de documentos con generación mejorada de recuperación (RAG). Este flujo de trabajo toma una consulta de usuario, genera una incrustación vectorial para ella, recupera los documentos más relevantes de una base de datos vectorial Milvus y utiliza un modelo de lenguaje amplio (LLM) para generar una respuesta detallada y consciente del contexto basada en los documentos recuperados.
+Siguiendo este flujo de trabajo, creará una solución integral para la búsqueda semántica y la respuesta a preguntas, combinando la potencia de la recuperación de documentos basada en vectores con las capacidades de los LLM avanzados de OpenAI. Este enfoque permite respuestas eficientes e inteligentes a las consultas de los usuarios aprovechando el conocimiento almacenado en su base de datos de documentos.
+Importe las bibliotecas necesarias e inicialice el flujo de trabajo
from dynamiq import Workflow
+from dynamiq.connections import (
+ OpenAI as OpenAIConnection,
+ Milvus as MilvusConnection,
+ MilvusDeploymentType,
+)
+from dynamiq.nodes.embedders import OpenAITextEmbedder
+from dynamiq.nodes.retrievers import MilvusDocumentRetriever
+from dynamiq.nodes.llms import OpenAI
+from dynamiq.prompts import Message, Prompt
+
+# Initialize the workflow
+retrieval_wf = Workflow()
+
+Definir la conexión OpenAI y el incrustador de texto
# Establish OpenAI connection
+openai_connection = OpenAIConnection(api_key=os.environ["OPENAI_API_KEY"])
+
+# Define the text embedder node
+embedder = OpenAITextEmbedder(
+ connection=openai_connection,
+ model="text-embedding-3-small",
+)
+
+# Add the embedder node to the workflow
+embedder_added = retrieval_wf.flow.add_nodes(embedder)
+
+Definir el recuperador de documentos Milvus
document_retriever = (
+ MilvusDocumentRetriever(
+ connection=MilvusConnection(
+ deployment_type=MilvusDeploymentType.FILE, uri="./milvus.db"
+ ),
+ index_name="my_milvus_collection",
+ dimension=1536,
+ top_k=5,
+ )
+ .inputs(embedding=embedder.outputs.embedding) # Connect to embedder output
+ .depends_on(embedder) # Dependency on the embedder node
+)
+
+# Add the retriever node to the workflow
+milvus_retriever_added = retrieval_wf.flow.add_nodes(document_retriever)
+
+2024-11-19 22:14:19 - WARNING - Environment variable 'MILVUS_API_TOKEN' not found
+2024-11-19 22:14:19 - INFO - Pass in the local path ./milvus.db, and run it using milvus-lite
+2024-11-19 22:14:19 - DEBUG - Created new connection using: 98d1132773af4298a894ad5925845fd2
+2024-11-19 22:14:19 - INFO - Collection my_milvus_collection already exists. Skipping creation.
+
+Definir la plantilla de solicitud
# Define the prompt template for the LLM
+prompt_template = """
+Please answer the question based on the provided context.
+
+Question: {{ query }}
+
+Context:
+{% for document in documents %}
+- {{ document.content }}
+{% endfor %}
+"""
+
+# Create the prompt object
+prompt = Prompt(messages=[Message(content=prompt_template, role="user")])
+
+Definir el generador de respuestas
answer_generator = (
+ OpenAI(
+ connection=openai_connection,
+ model="gpt-4o",
+ prompt=prompt,
+ )
+ .inputs(
+ documents=document_retriever.outputs.documents,
+ query=embedder.outputs.query,
+ )
+ .depends_on(
+ [document_retriever, embedder]
+ ) # Dependencies on retriever and embedder
+)
+
+# Add the answer generator node to the workflow
+answer_generator_added = retrieval_wf.flow.add_nodes(answer_generator)
+
+Ejecutar el flujo de trabajo
# Run the workflow with a sample query
+sample_query = "What is the Advanced Search Algorithms in Milvus?"
+
+result = retrieval_wf.run(input_data={"query": sample_query})
+
+answer = result.output.get(answer_generator.id).get("output", {}).get("content")
+print(answer)
+
+2024-11-19 22:14:22 - INFO - Workflow f4a073fb-dfb6-499c-8cac-5710a7ad6d47: execution started.
+2024-11-19 22:14:22 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution started.
+2024-11-19 22:14:22 - INFO - Node OpenAITextEmbedder - 47afb0bc-cf96-429d-b58f-11b6c935fec3: execution started.
+2024-11-19 22:14:23 - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
+2024-11-19 22:14:23 - INFO - Node OpenAITextEmbedder - 47afb0bc-cf96-429d-b58f-11b6c935fec3: execution succeeded in 474ms.
+2024-11-19 22:14:23 - INFO - Node MilvusDocumentRetriever - 51c8311b-4837-411f-ba42-21e28239a2ee: execution started.
+2024-11-19 22:14:23 - INFO - Node MilvusDocumentRetriever - 51c8311b-4837-411f-ba42-21e28239a2ee: execution succeeded in 23ms.
+2024-11-19 22:14:23 - INFO - Node LLM - ac722325-bece-453f-a2ed-135b0749ee7a: execution started.
+2024-11-19 22:14:24 - INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
+2024-11-19 22:14:24 - INFO - Node LLM - ac722325-bece-453f-a2ed-135b0749ee7a: execution succeeded in 1.8s.
+2024-11-19 22:14:25 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution succeeded in 2.4s.
+2024-11-19 22:14:25 - INFO - Workflow f4a073fb-dfb6-499c-8cac-5710a7ad6d47: execution succeeded in 2.4s.
+
+
+The advanced search algorithms in Milvus include a variety of in-memory and on-disk indexing/search algorithms such as IVF (Inverted File), HNSW (Hierarchical Navigable Small World), and DiskANN. These algorithms have been deeply optimized to enhance performance, delivering 30%-70% better performance compared to popular implementations like FAISS and HNSWLib. These optimizations are part of Milvus's design to ensure high efficiency and scalability in handling vector data.
+
diff --git a/localization/v2.4.x/site/es/menuStructure/es.json b/localization/v2.4.x/site/es/menuStructure/es.json
index e793ecad2..5111e4bfd 100644
--- a/localization/v2.4.x/site/es/menuStructure/es.json
+++ b/localization/v2.4.x/site/es/menuStructure/es.json
@@ -839,7 +839,7 @@
"children": []
},
{
- "label": "Configurar Chunk Cache",
+ "label": "Configurar la caché de trozos",
"id": "chunk_cache.md",
"order": 3,
"children": []
@@ -1036,7 +1036,7 @@
"isMenu": true,
"children": [
{
- "label": "Seguimiento de Jaeger",
+ "label": "Rastreo de Jaeger",
"id": "config_jaeger_tracing.md",
"order": 0,
"children": []
@@ -1344,6 +1344,24 @@
"id": "build_RAG_with_milvus_and_siliconflow.md",
"order": 4,
"children": []
+ },
+ {
+ "label": "SambaNova",
+ "id": "use_milvus_with_sambanova.md",
+ "order": 5,
+ "children": []
+ },
+ {
+ "label": "Géminis",
+ "id": "build_RAG_with_milvus_and_gemini.md",
+ "order": 6,
+ "children": []
+ },
+ {
+ "label": "Ollama",
+ "id": "build_RAG_with_milvus_and_ollama.md",
+ "order": 7,
+ "children": []
}
]
},
@@ -1433,15 +1451,15 @@
"children": []
},
{
- "label": "SambaNova",
- "id": "use_milvus_with_sambanova.md",
+ "label": "PrivadoGPT",
+ "id": "use_milvus_in_private_gpt.md",
"order": 10,
"children": []
},
{
- "label": "PrivadoGPT",
- "id": "use_milvus_in_private_gpt.md",
- "order": 8,
+ "label": "Dynamiq",
+ "id": "milvus_rag_with_dynamiq.md",
+ "order": 11,
"children": []
}
]
@@ -1493,6 +1511,12 @@
"id": "knowledge_table_with_milvus.md",
"order": 2,
"children": []
+ },
+ {
+ "label": "Cognee",
+ "id": "build_RAG_with_milvus_and_cognee.md",
+ "order": 3,
+ "children": []
}
]
},
diff --git a/localization/v2.4.x/site/fr/integrations/build_RAG_with_milvus_and_cognee.json b/localization/v2.4.x/site/fr/integrations/build_RAG_with_milvus_and_cognee.json
new file mode 100644
index 000000000..1aef19ebf
--- /dev/null
+++ b/localization/v2.4.x/site/fr/integrations/build_RAG_with_milvus_and_cognee.json
@@ -0,0 +1 @@
+{"codeList":["$ pip install pymilvus git+https://github.com/topoteretes/cognee.git\n","import os\n\nimport cognee\n\ncognee.config.set_llm_api_key(\"YOUR_OPENAI_API_KEY\")\n\n\nos.environ[\"VECTOR_DB_PROVIDER\"] = \"milvus\"\nos.environ[\"VECTOR_DB_URL\"] = \"./milvus.db\"\n","$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip\n$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs\n","from glob import glob\n\ntext_lines = []\n\nfor file_path in glob(\"milvus_docs/en/faq/*.md\", recursive=True):\n with open(file_path, \"r\") as file:\n file_text = file.read()\n\n text_lines += file_text.split(\"# \")\n","await cognee.prune.prune_data()\nawait cognee.prune.prune_system(metadata=True)\n","await cognee.add(data=text_lines, dataset_name=\"milvus_faq\")\nawait cognee.cognify()\n\n# [DocumentChunk(id=UUID('6889e7ef-3670-555c-bb16-3eb50d1d30b0'), updated_at=datetime.datetime(2024, 12, 4, 6, 29, 46, 472907, tzinfo=datetime.timezone.utc), text='Does the query perform in memory? What are incremental data and historical data?\\n\\nYes. When ...\n# ...\n","from cognee.api.v1.search import SearchType\n\nquery_text = \"How is data stored in milvus?\"\nsearch_results = await cognee.search(SearchType.SUMMARIES, query_text=query_text)\n\nprint(search_results[0])\n","from cognee.api.v1.search import SearchType\n\nquery_text = \"How is data stored in milvus?\"\nsearch_results = await cognee.search(SearchType.CHUNKS, query_text=query_text)\n","def format_and_print(data):\n print(\"ID:\", data[\"id\"])\n print(\"\\nText:\\n\")\n paragraphs = data[\"text\"].split(\"\\n\\n\")\n for paragraph in paragraphs:\n print(paragraph.strip())\n print()\n\n\nformat_and_print(search_results[0])\n","await cognee.prune.prune_data()\nawait cognee.prune.prune_system(metadata=True)\n","# We only use one line of text as the dataset, which simplifies the output later\ntext = \"\"\"\n Natural language processing (NLP) is an interdisciplinary\n subfield of computer science and information retrieval.\n \"\"\"\n\nawait cognee.add(text)\nawait cognee.cognify()\n","query_text = \"Tell me about NLP\"\nsearch_results = await cognee.search(SearchType.INSIGHTS, query_text=query_text)\n\nfor result_text in search_results:\n print(result_text)\n\n# Example output:\n# ({'id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'natural language processing', 'description': 'An interdisciplinary subfield of computer science and information retrieval.'}, {'relationship_name': 'is_a_subfield_of', 'source_node_id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'target_node_id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 15, 473137, tzinfo=datetime.timezone.utc)}, {'id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'computer science', 'description': 'The study of computation and information processing.'})\n# (...)\n#\n# It represents nodes and relationships in the knowledge graph:\n# - The first element is the source node (e.g., 'natural language processing').\n# - The second element is the relationship between nodes (e.g., 'is_a_subfield_of').\n# - The third element is the target node (e.g., 'computer science').\n"],"headingContent":"","anchorList":[{"label":"Construire RAG","href":"Build-RAG","type":2,"isActive":false}]}
\ No newline at end of file
diff --git a/localization/v2.4.x/site/fr/integrations/build_RAG_with_milvus_and_cognee.md b/localization/v2.4.x/site/fr/integrations/build_RAG_with_milvus_and_cognee.md
new file mode 100644
index 000000000..c84057d54
--- /dev/null
+++ b/localization/v2.4.x/site/fr/integrations/build_RAG_with_milvus_and_cognee.md
@@ -0,0 +1,159 @@
+---
+id: build_RAG_with_milvus_and_cognee.md
+summary: >-
+ Dans ce tutoriel, nous allons vous montrer comment construire un pipeline RAG
+ (Retrieval-Augmented Generation) avec Milvus et Cognee.
+title: Construire un RAG avec Milvus et Cognee
+---
+
+Construire RAG avec Milvus et Cognee
Cognee est une plate-forme axée sur le développement qui rationalise le développement d'applications d'IA grâce à des pipelines ECL (Extract, Cognify, Load) modulaires et évolutifs. En s'intégrant de manière transparente à Milvus, Cognee permet une connexion et une récupération efficaces des conversations, des documents et des transcriptions, réduisant ainsi les hallucinations et optimisant les coûts opérationnels.
+Grâce à une prise en charge solide des magasins vectoriels tels que Milvus, des bases de données de graphes et des LLM, Cognee offre un cadre flexible et personnalisable pour la création de systèmes de génération augmentée par récupération (RAG). Son architecture prête à la production garantit une précision et une efficacité accrues pour les applications alimentées par l'IA.
+Dans ce tutoriel, nous allons vous montrer comment construire un pipeline RAG (Retrieval-Augmented Generation) avec Milvus et Cognee.
+$ pip install pymilvus git+https://github.com/topoteretes/cognee.git
+
+
+Si vous utilisez Google Colab, pour activer les dépendances qui viennent d'être installées, vous devrez peut-être redémarrer le runtime (cliquez sur le menu "Runtime" en haut de l'écran, et sélectionnez "Restart session" dans le menu déroulant).
+
+Par défaut, il utilise OpenAI comme LLM dans cet exemple. Vous devez préparer la clé api et la définir dans la fonction config set_llm_api_key().
+Pour configurer Milvus en tant que base de données vectorielle, définissez VECTOR_DB_PROVIDER sur milvus et spécifiez VECTOR_DB_URL et VECTOR_DB_KEY. Comme nous utilisons Milvus Lite pour stocker les données dans cette démo, seul le VECTOR_DB_URL doit être fourni.
+import os
+
+import cognee
+
+cognee.config.set_llm_api_key("YOUR_OPENAI_API_KEY")
+
+
+os.environ["VECTOR_DB_PROVIDER"] = "milvus"
+os.environ["VECTOR_DB_URL"] = "./milvus.db"
+
+
+Quant aux variables d'environnement VECTOR_DB_URL et VECTOR_DB_KEY:
+
+- Définir
VECTOR_DB_URL comme un fichier local, par exemple./milvus.db, est la méthode la plus pratique, car elle utilise automatiquement Milvus Lite pour stocker toutes les données dans ce fichier.
+- Si vous avez des données à grande échelle, vous pouvez configurer un serveur Milvus plus performant sur docker ou kubernetes. Dans cette configuration, veuillez utiliser l'uri du serveur, par exemple
http://localhost:19530, comme votre VECTOR_DB_URL.
+- Si vous souhaitez utiliser Zilliz Cloud, le service cloud entièrement géré pour Milvus, ajustez les adresses
VECTOR_DB_URL et VECTOR_DB_KEY, qui correspondent au point final public et à la clé Api dans Zilliz Cloud.
+
+
+Préparer les données
Nous utilisons les pages FAQ de la documentation Milvus 2.4.x comme connaissance privée dans notre RAG, ce qui constitue une bonne source de données pour un pipeline RAG simple.
+Téléchargez le fichier zip et extrayez les documents dans le dossier milvus_docs.
+$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
+
+Nous chargeons tous les fichiers markdown à partir du dossier milvus_docs/en/faq. Pour chaque document, nous utilisons simplement "# " pour séparer le contenu du fichier, ce qui permet de séparer grossièrement le contenu de chaque partie principale du fichier markdown.
+from glob import glob
+
+text_lines = []
+
+for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
+ with open(file_path, "r") as file:
+ file_text = file.read()
+
+ text_lines += file_text.split("# ")
+
+Construire RAG
Réinitialisation des données de Cognee
await cognee.prune.prune_data()
+await cognee.prune.prune_system(metadata=True)
+
+Une fois que nous avons fait table rase du passé, nous pouvons maintenant ajouter notre jeu de données et le traiter pour en faire un graphe de connaissances.
+Ajout de données et cognification
await cognee.add(data=text_lines, dataset_name="milvus_faq")
+await cognee.cognify()
+
+# [DocumentChunk(id=UUID('6889e7ef-3670-555c-bb16-3eb50d1d30b0'), updated_at=datetime.datetime(2024, 12, 4, 6, 29, 46, 472907, tzinfo=datetime.timezone.utc), text='Does the query perform in memory? What are incremental data and historical data?\n\nYes. When ...
+# ...
+
+La méthode add charge l'ensemble de données (Milvus FAQs) dans Cognee et la méthode cognify traite les données pour extraire les entités, les relations et les résumés, construisant ainsi un graphe de connaissances.
+Recherche de résumés
Maintenant que les données ont été traitées, interrogeons le graphe de connaissances.
+from cognee.api.v1.search import SearchType
+
+query_text = "How is data stored in milvus?"
+search_results = await cognee.search(SearchType.SUMMARIES, query_text=query_text)
+
+print(search_results[0])
+
+{'id': 'de5c6713-e079-5d0b-b11d-e9bacd1e0d73', 'text': 'Milvus stores two data types: inserted data and metadata.'}
+
+Cette requête recherche dans le graphe de connaissances un résumé lié au texte de la requête, et le candidat le plus lié est imprimé.
+Recherche de morceaux
Les résumés offrent des informations de haut niveau, mais pour obtenir des détails plus précis, nous pouvons interroger des morceaux de données spécifiques directement à partir de l'ensemble de données traitées. Ces morceaux sont dérivés des données originales qui ont été ajoutées et analysées lors de la création du graphe de connaissances.
+from cognee.api.v1.search import SearchType
+
+query_text = "How is data stored in milvus?"
+search_results = await cognee.search(SearchType.CHUNKS, query_text=query_text)
+
+Formatons et affichons-les pour une meilleure lisibilité !
+def format_and_print(data):
+ print("ID:", data["id"])
+ print("\nText:\n")
+ paragraphs = data["text"].split("\n\n")
+ for paragraph in paragraphs:
+ print(paragraph.strip())
+ print()
+
+
+format_and_print(search_results[0])
+
+ID: 4be01c4b-9ee5-541c-9b85-297883934ab3
+
+Text:
+
+Where does Milvus store data?
+
+Milvus deals with two types of data, inserted data and metadata.
+
+Inserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).
+
+Metadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.
+
+###
+
+Dans les étapes précédentes, nous avons interrogé l'ensemble de données Milvus FAQ pour obtenir à la fois des résumés et des morceaux de données spécifiques. Bien que cela ait permis d'obtenir des informations détaillées et granulaires, l'ensemble de données était volumineux, ce qui rendait difficile la visualisation claire des dépendances au sein du graphe de connaissances.
+Pour résoudre ce problème, nous réinitialiserons l'environnement Cognee et travaillerons avec un ensemble de données plus petit et plus ciblé. Cela nous permettra de mieux mettre en évidence les relations et les dépendances extraites au cours du processus de cognification. En simplifiant les données, nous pouvons clairement voir comment Cognee organise et structure les informations dans le graphe de connaissances.
+Réinitialiser Cognee
await cognee.prune.prune_data()
+await cognee.prune.prune_system(metadata=True)
+
+Ajout de l'ensemble de données ciblé
Ici, un ensemble de données plus petit ne comportant qu'une seule ligne de texte est ajouté et traité pour garantir un graphe de connaissances ciblé et facilement interprétable.
+# We only use one line of text as the dataset, which simplifies the output later
+text = """
+ Natural language processing (NLP) is an interdisciplinary
+ subfield of computer science and information retrieval.
+ """
+
+await cognee.add(text)
+await cognee.cognify()
+
+Recherche d'informations
En nous concentrant sur cet ensemble de données plus petit, nous pouvons maintenant analyser clairement les relations et la structure du graphe de connaissances.
+query_text = "Tell me about NLP"
+search_results = await cognee.search(SearchType.INSIGHTS, query_text=query_text)
+
+for result_text in search_results:
+ print(result_text)
+
+# Example output:
+# ({'id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'natural language processing', 'description': 'An interdisciplinary subfield of computer science and information retrieval.'}, {'relationship_name': 'is_a_subfield_of', 'source_node_id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'target_node_id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 15, 473137, tzinfo=datetime.timezone.utc)}, {'id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'computer science', 'description': 'The study of computation and information processing.'})
+# (...)
+#
+# It represents nodes and relationships in the knowledge graph:
+# - The first element is the source node (e.g., 'natural language processing').
+# - The second element is the relationship between nodes (e.g., 'is_a_subfield_of').
+# - The third element is the target node (e.g., 'computer science').
+
+Cette sortie représente les résultats d'une requête de graphe de connaissances, mettant en évidence les entités (nœuds) et leurs relations (arêtes) telles qu'elles ont été extraites de l'ensemble de données traité. Chaque tuple comprend une entité source, un type de relation et une entité cible, ainsi que des métadonnées telles que des identifiants uniques, des descriptions et des horodatages. Le graphique met en évidence les concepts clés et leurs connexions sémantiques, ce qui permet une compréhension structurée de l'ensemble de données.
+Félicitations, vous avez appris l'utilisation de base de Cognee avec Milvus. Si vous souhaitez connaître l'utilisation plus avancée de Cognee, veuillez vous référer à sa page officielle.
diff --git a/localization/v2.4.x/site/fr/integrations/build_RAG_with_milvus_and_gemini.json b/localization/v2.4.x/site/fr/integrations/build_RAG_with_milvus_and_gemini.json
new file mode 100644
index 000000000..0d9313b94
--- /dev/null
+++ b/localization/v2.4.x/site/fr/integrations/build_RAG_with_milvus_and_gemini.json
@@ -0,0 +1 @@
+{"codeList":["$ pip install --upgrade pymilvus google-generativeai requests tqdm\n","import os\n\nos.environ[\"GEMINI_API_KEY\"] = \"***********\"\n","$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip\n$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs\n","from glob import glob\n\ntext_lines = []\n\nfor file_path in glob(\"milvus_docs/en/faq/*.md\", recursive=True):\n with open(file_path, \"r\") as file:\n file_text = file.read()\n\n text_lines += file_text.split(\"# \")\n","import google.generativeai as genai\n\ngenai.configure(api_key=os.environ[\"GEMINI_API_KEY\"])\n\ngemini_model = genai.GenerativeModel(\"gemini-1.5-flash\")\n\nresponse = gemini_model.generate_content(\"who are you\")\nprint(response.text)\n","test_embeddings = genai.embed_content(\n model=\"models/text-embedding-004\", content=[\"This is a test1\", \"This is a test2\"]\n)[\"embedding\"]\n\nembedding_dim = len(test_embeddings[0])\nprint(embedding_dim)\nprint(test_embeddings[0][:10])\n","from pymilvus import MilvusClient\n\nmilvus_client = MilvusClient(uri=\"./milvus_demo.db\")\n\ncollection_name = \"my_rag_collection\"\n","if milvus_client.has_collection(collection_name):\n milvus_client.drop_collection(collection_name)\n","milvus_client.create_collection(\n collection_name=collection_name,\n dimension=embedding_dim,\n metric_type=\"IP\", # Inner product distance\n consistency_level=\"Strong\", # Strong consistency level\n)\n","from tqdm import tqdm\n\ndata = []\n\ndoc_embeddings = genai.embed_content(\n model=\"models/text-embedding-004\", content=text_lines\n)[\"embedding\"]\n\nfor i, line in enumerate(tqdm(text_lines, desc=\"Creating embeddings\")):\n data.append({\"id\": i, \"vector\": doc_embeddings[i], \"text\": line})\n\nmilvus_client.insert(collection_name=collection_name, data=data)\n","question = \"How is data stored in milvus?\"\n","question_embedding = genai.embed_content(\n model=\"models/text-embedding-004\", content=question\n)[\"embedding\"]\n\nsearch_res = milvus_client.search(\n collection_name=collection_name,\n data=[question_embedding],\n limit=3, # Return top 3 results\n search_params={\"metric_type\": \"IP\", \"params\": {}}, # Inner product distance\n output_fields=[\"text\"], # Return the text field\n)\n","import json\n\nretrieved_lines_with_distances = [\n (res[\"entity\"][\"text\"], res[\"distance\"]) for res in search_res[0]\n]\nprint(json.dumps(retrieved_lines_with_distances, indent=4))\n","context = \"\\n\".join(\n [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]\n)\n","SYSTEM_PROMPT = \"\"\"\nHuman: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.\n\"\"\"\nUSER_PROMPT = f\"\"\"\nUse the following pieces of information enclosed in tags to provide an answer to the question enclosed in tags.\n\n{context}\n \n\n{question}\n \n\"\"\"\n","gemini_model = genai.GenerativeModel(\n \"gemini-1.5-flash\", system_instruction=SYSTEM_PROMPT\n)\nresponse = gemini_model.generate_content(USER_PROMPT)\nprint(response.text)\n"],"headingContent":"Build RAG with Milvus and Gemini","anchorList":[{"label":"Construire RAG avec Milvus et Gemini","href":"Build-RAG-with-Milvus-and-Gemini","type":1,"isActive":false},{"label":"Préparation","href":"Preparation","type":2,"isActive":false},{"label":"Chargement des données dans Milvus","href":"Load-data-into-Milvus","type":2,"isActive":false},{"label":"Construire un RAG","href":"Build-RAG","type":2,"isActive":false}]}
\ No newline at end of file
diff --git a/localization/v2.4.x/site/fr/integrations/build_RAG_with_milvus_and_gemini.md b/localization/v2.4.x/site/fr/integrations/build_RAG_with_milvus_and_gemini.md
new file mode 100644
index 000000000..9cbfcd0bd
--- /dev/null
+++ b/localization/v2.4.x/site/fr/integrations/build_RAG_with_milvus_and_gemini.md
@@ -0,0 +1,246 @@
+---
+id: build_RAG_with_milvus_and_gemini.md
+summary: >-
+ Dans ce tutoriel, nous allons vous montrer comment construire un pipeline RAG
+ (Retrieval-Augmented Generation) avec Milvus et Gemini. Nous utiliserons le
+ modèle Gemini pour générer du texte sur la base d'une requête donnée. Nous
+ utiliserons également Milvus pour stocker et récupérer le texte généré.
+title: Construire RAG avec Milvus et Gemini
+---
+
+Construire RAG avec Milvus et Gemini
L'API Gemini et Google AI Studio vous permettent de commencer à travailler avec les derniers modèles de Google et de transformer vos idées en applications évolutives. Gemini permet d'accéder à de puissants modèles de langage tels que Gemini-1.5-Flash, Gemini-1.5-Flash-8B et Gemini-1.5-Pro pour des tâches telles que la génération de texte, le traitement de documents, la vision, l'analyse audio, etc. L'API vous permet de saisir des contextes longs avec des millions de tokens, d'affiner les modèles pour des tâches spécifiques, de générer des sorties structurées comme JSON, et d'exploiter des capacités comme la récupération sémantique et l'exécution de code.
+Dans ce tutoriel, nous allons vous montrer comment construire un pipeline RAG (Retrieval-Augmented Generation) avec Milvus et Gemini. Nous utiliserons le modèle Gemini pour générer du texte sur la base d'une requête donnée. Nous utiliserons également Milvus pour stocker et récupérer le texte généré.
+Préparation
Dépendances et environnement
$ pip install --upgrade pymilvus google-generativeai requests tqdm
+
+
+Si vous utilisez Google Colab, pour activer les dépendances qui viennent d'être installées, vous devrez peut-être redémarrer le runtime (cliquez sur le menu "Runtime" en haut de l'écran, et sélectionnez "Restart session" (Redémarrer la session) dans le menu déroulant).
+
+Vous devez d'abord vous connecter à la plateforme Google AI Studio et préparer la clé api GEMINI_API_KEY en tant que variable d'environnement.
+import os
+
+os.environ["GEMINI_API_KEY"] = "***********"
+
+Préparer les données
Nous utilisons les pages FAQ de la documentation Milvus 2.4.x comme connaissance privée dans notre RAG, ce qui constitue une bonne source de données pour un pipeline RAG simple.
+Téléchargez le fichier zip et extrayez les documents dans le dossier milvus_docs.
+$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
+
+Nous chargeons tous les fichiers markdown à partir du dossier milvus_docs/en/faq. Pour chaque document, nous utilisons simplement "# " pour séparer le contenu du fichier, ce qui permet de séparer grossièrement le contenu de chaque partie principale du fichier markdown.
+from glob import glob
+
+text_lines = []
+
+for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
+ with open(file_path, "r") as file:
+ file_text = file.read()
+
+ text_lines += file_text.split("# ")
+
+Préparation du LLM et du modèle d'intégration
Nous utilisons gemini-1.5-flash comme LLM et text-embedding-004 comme modèle d'intégration.
+Essayons de générer une réponse de test à partir du LLM :
+import google.generativeai as genai
+
+genai.configure(api_key=os.environ["GEMINI_API_KEY"])
+
+gemini_model = genai.GenerativeModel("gemini-1.5-flash")
+
+response = gemini_model.generate_content("who are you")
+print(response.text)
+
+I am a large language model, trained by Google. I am an AI and don't have a personal identity or consciousness. My purpose is to process information and respond to a wide range of prompts and questions in a helpful and informative way.
+
+Générer un embedding de test et imprimer sa dimension et ses premiers éléments.
+test_embeddings = genai.embed_content(
+ model="models/text-embedding-004", content=["This is a test1", "This is a test2"]
+)["embedding"]
+
+embedding_dim = len(test_embeddings[0])
+print(embedding_dim)
+print(test_embeddings[0][:10])
+
+768
+[0.013588584, -0.004361838, -0.08481652, -0.039724775, 0.04723794, -0.0051557426, 0.026071774, 0.045514572, -0.016867816, 0.039378334]
+
+Chargement des données dans Milvus
Créer la collection
from pymilvus import MilvusClient
+
+milvus_client = MilvusClient(uri="./milvus_demo.db")
+
+collection_name = "my_rag_collection"
+
+
+Comme pour l'argument de MilvusClient:
+
+- Définir
uri comme fichier local, par exemple./milvus.db, est la méthode la plus pratique, car elle utilise automatiquement Milvus Lite pour stocker toutes les données dans ce fichier.
+- Si vous avez des données à grande échelle, vous pouvez configurer un serveur Milvus plus performant sur docker ou kubernetes. Dans cette configuration, veuillez utiliser l'uri du serveur, par exemple
http://localhost:19530, comme votre uri.
+- Si vous souhaitez utiliser Zilliz Cloud, le service cloud entièrement géré pour Milvus, ajustez les adresses
uri et token, qui correspondent au point de terminaison public et à la clé Api dans Zilliz Cloud.
+
+
+Vérifier si la collection existe déjà et la supprimer si c'est le cas.
+if milvus_client.has_collection(collection_name):
+ milvus_client.drop_collection(collection_name)
+
+Créer une nouvelle collection avec les paramètres spécifiés.
+Si nous ne spécifions aucune information de champ, Milvus créera automatiquement un champ id par défaut pour la clé primaire et un champ vector pour stocker les données vectorielles. Un champ JSON réservé est utilisé pour stocker les champs non définis par le schéma et leurs valeurs.
+milvus_client.create_collection(
+ collection_name=collection_name,
+ dimension=embedding_dim,
+ metric_type="IP", # Inner product distance
+ consistency_level="Strong", # Strong consistency level
+)
+
+Insérer des données
Parcourez les lignes de texte, créez des enchâssements, puis insérez les données dans Milvus.
+Voici un nouveau champ text, qui est un champ non défini dans le schéma de la collection. Il sera automatiquement ajouté au champ dynamique JSON réservé, qui peut être traité comme un champ normal à un niveau élevé.
+from tqdm import tqdm
+
+data = []
+
+doc_embeddings = genai.embed_content(
+ model="models/text-embedding-004", content=text_lines
+)["embedding"]
+
+for i, line in enumerate(tqdm(text_lines, desc="Creating embeddings")):
+ data.append({"id": i, "vector": doc_embeddings[i], "text": line})
+
+milvus_client.insert(collection_name=collection_name, data=data)
+
+Creating embeddings: 100%|██████████| 72/72 [00:00<00:00, 468201.38it/s]
+
+
+
+
+
+{'insert_count': 72, 'ids': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71], 'cost': 0}
+
+Construire un RAG
Récupérer des données pour une requête
Spécifions une question fréquente sur Milvus.
+question = "How is data stored in milvus?"
+
+Cherchons la question dans la collection et récupérons les 3 meilleures réponses sémantiques.
+question_embedding = genai.embed_content(
+ model="models/text-embedding-004", content=question
+)["embedding"]
+
+search_res = milvus_client.search(
+ collection_name=collection_name,
+ data=[question_embedding],
+ limit=3, # Return top 3 results
+ search_params={"metric_type": "IP", "params": {}}, # Inner product distance
+ output_fields=["text"], # Return the text field
+)
+
+Jetons un coup d'œil aux résultats de la recherche de la question.
+import json
+
+retrieved_lines_with_distances = [
+ (res["entity"]["text"], res["distance"]) for res in search_res[0]
+]
+print(json.dumps(retrieved_lines_with_distances, indent=4))
+
+[
+ [
+ " Where does Milvus store data?\n\nMilvus deals with two types of data, inserted data and metadata. \n\nInserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).\n\nMetadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.\n\n###",
+ 0.8048275113105774
+ ],
+ [
+ "Does the query perform in memory? What are incremental data and historical data?\n\nYes. When a query request comes, Milvus searches both incremental data and historical data by loading them into memory. Incremental data are in the growing segments, which are buffered in memory before they reach the threshold to be persisted in storage engine, while historical data are from the sealed segments that are stored in the object storage. Incremental data and historical data together constitute the whole dataset to search.\n\n###",
+ 0.7574886679649353
+ ],
+ [
+ "What is the maximum dataset size Milvus can handle?\n\n \nTheoretically, the maximum dataset size Milvus can handle is determined by the hardware it is run on, specifically system memory and storage:\n\n- Milvus loads all specified collections and partitions into memory before running queries. Therefore, memory size determines the maximum amount of data Milvus can query.\n- When new entities and and collection-related schema (currently only MinIO is supported for data persistence) are added to Milvus, system storage determines the maximum allowable size of inserted data.\n\n###",
+ 0.7453608512878418
+ ]
+]
+
+Utiliser LLM pour obtenir une réponse RAG
Convertir les documents récupérés dans un format de chaîne.
+context = "\n".join(
+ [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]
+)
+
+Définir les messages-guides du système et de l'utilisateur pour le modèle de langue. Cette invite est assemblée avec les documents extraits de Milvus.
+SYSTEM_PROMPT = """
+Human: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.
+"""
+USER_PROMPT = f"""
+Use the following pieces of information enclosed in <context> tags to provide an answer to the question enclosed in <question> tags.
+<context>
+{context}
+</context>
+<question>
+{question}
+</question>
+"""
+
+Utiliser Gemini pour générer une réponse basée sur les invites.
+gemini_model = genai.GenerativeModel(
+ "gemini-1.5-flash", system_instruction=SYSTEM_PROMPT
+)
+response = gemini_model.generate_content(USER_PROMPT)
+print(response.text)
+
+Milvus stores data in two ways: Inserted data (vector data, scalar data, and collection-specific schema) is stored as an incremental log in persistent storage using object storage backends such as MinIO, AWS S3, Google Cloud Storage, Azure Blob Storage, Alibaba Cloud OSS, and Tencent Cloud Object Storage. Metadata, generated by each Milvus module, is stored in etcd.
+
+C'est très bien ! Nous avons réussi à construire un pipeline RAG avec Milvus et Gemini.
diff --git a/localization/v2.4.x/site/fr/integrations/build_RAG_with_milvus_and_ollama.json b/localization/v2.4.x/site/fr/integrations/build_RAG_with_milvus_and_ollama.json
new file mode 100644
index 000000000..d89c1cedd
--- /dev/null
+++ b/localization/v2.4.x/site/fr/integrations/build_RAG_with_milvus_and_ollama.json
@@ -0,0 +1 @@
+{"codeList":["$ pip install pymilvus ollama\n","$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip\n$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs\n","from glob import glob\n\ntext_lines = []\n\nfor file_path in glob(\"milvus_docs/en/faq/*.md\", recursive=True):\n with open(file_path, \"r\") as file:\n file_text = file.read()\n\n text_lines += file_text.split(\"# \")\n","! ollama pull mxbai-embed-large\n","! ollama pull llama3.2\n","import ollama\n\n\ndef emb_text(text):\n response = ollama.embeddings(model=\"mxbai-embed-large\", prompt=text)\n return response[\"embedding\"]\n","test_embedding = emb_text(\"This is a test\")\nembedding_dim = len(test_embedding)\nprint(embedding_dim)\nprint(test_embedding[:10])\n","from pymilvus import MilvusClient\n\nmilvus_client = MilvusClient(uri=\"./milvus_demo.db\")\n\ncollection_name = \"my_rag_collection\"\n","if milvus_client.has_collection(collection_name):\n milvus_client.drop_collection(collection_name)\n","milvus_client.create_collection(\n collection_name=collection_name,\n dimension=embedding_dim,\n metric_type=\"IP\", # Inner product distance\n consistency_level=\"Strong\", # Strong consistency level\n)\n","from tqdm import tqdm\n\ndata = []\n\nfor i, line in enumerate(tqdm(text_lines, desc=\"Creating embeddings\")):\n data.append({\"id\": i, \"vector\": emb_text(line), \"text\": line})\n\nmilvus_client.insert(collection_name=collection_name, data=data)\n","question = \"How is data stored in milvus?\"\n","search_res = milvus_client.search(\n collection_name=collection_name,\n data=[\n emb_text(question)\n ], # Use the `emb_text` function to convert the question to an embedding vector\n limit=3, # Return top 3 results\n search_params={\"metric_type\": \"IP\", \"params\": {}}, # Inner product distance\n output_fields=[\"text\"], # Return the text field\n)\n","import json\n\nretrieved_lines_with_distances = [\n (res[\"entity\"][\"text\"], res[\"distance\"]) for res in search_res[0]\n]\nprint(json.dumps(retrieved_lines_with_distances, indent=4))\n","context = \"\\n\".join(\n [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]\n)\n","SYSTEM_PROMPT = \"\"\"\nHuman: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.\n\"\"\"\nUSER_PROMPT = f\"\"\"\nUse the following pieces of information enclosed in tags to provide an answer to the question enclosed in tags.\n\n{context}\n \n\n{question}\n \n\"\"\"\n","from ollama import chat\nfrom ollama import ChatResponse\n\nresponse: ChatResponse = chat(\n model=\"llama3.2\",\n messages=[\n {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n {\"role\": \"user\", \"content\": USER_PROMPT},\n ],\n)\nprint(response[\"message\"][\"content\"])\n"],"headingContent":"Build RAG with Milvus and Ollama","anchorList":[{"label":"Construire RAG avec Milvus et Ollama","href":"Build-RAG-with-Milvus-and-Ollama","type":1,"isActive":false},{"label":"Préparation","href":"Preparation","type":2,"isActive":false},{"label":"Charger les données dans Milvus","href":"Load-data-into-Milvus","type":2,"isActive":false},{"label":"Construire un RAG","href":"Build-RAG","type":2,"isActive":false}]}
\ No newline at end of file
diff --git a/localization/v2.4.x/site/fr/integrations/build_RAG_with_milvus_and_ollama.md b/localization/v2.4.x/site/fr/integrations/build_RAG_with_milvus_and_ollama.md
new file mode 100644
index 000000000..6b4d9987b
--- /dev/null
+++ b/localization/v2.4.x/site/fr/integrations/build_RAG_with_milvus_and_ollama.md
@@ -0,0 +1,282 @@
+---
+id: build_RAG_with_milvus_and_ollama.md
+summary: >-
+ Dans ce guide, nous vous montrons comment tirer parti d'Ollama et de Milvus
+ pour construire un pipeline RAG (Retrieval-Augmented Generation) de manière
+ efficace et sécurisée.
+title: Construire RAG avec Milvus et Ollama
+---
+
+Construire RAG avec Milvus et Ollama
Ollama est une plateforme open-source qui simplifie l'exécution et la personnalisation de grands modèles de langage (LLM) au niveau local. Elle offre une expérience conviviale, sans nuage, qui permet de télécharger, d'installer et d'interagir sans effort avec les modèles, sans nécessiter de compétences techniques avancées. Grâce à une bibliothèque croissante de LLMs pré-entraînés, allant des modèles à usage général aux modèles spécifiques à un domaine, Ellama facilite la gestion et la personnalisation des modèles pour diverses applications. Il garantit la confidentialité des données et la flexibilité, permettant aux utilisateurs d'affiner, d'optimiser et de déployer des solutions basées sur l'IA entièrement sur leurs machines.
+Dans ce guide, nous allons vous montrer comment tirer parti d'Ollama et de Milvus pour construire un pipeline RAG (Retrieval-Augmented Generation) de manière efficace et sécurisée.
+Préparation
Dépendances et environnement
$ pip install pymilvus ollama
+
+
+Si vous utilisez Google Colab, pour activer les dépendances qui viennent d'être installées, vous devrez peut-être redémarrer le runtime (cliquez sur le menu "Runtime" en haut de l'écran, et sélectionnez "Restart session" dans le menu déroulant).
+
+Préparer les données
Nous utilisons les pages FAQ de la documentation Milvus 2.4.x comme connaissance privée dans notre RAG, qui est une bonne source de données pour un pipeline RAG simple.
+Téléchargez le fichier zip et extrayez les documents dans le dossier milvus_docs.
+$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
+
+--2024-11-26 21:47:19-- https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+Resolving github.com (github.com)... 140.82.112.4
+Connecting to github.com (github.com)|140.82.112.4|:443... connected.
+HTTP request sent, awaiting response... 302 Found
+Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/267273319/c52902a0-e13c-4ca7-92e0-086751098a05?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241127%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241127T024720Z&X-Amz-Expires=300&X-Amz-Signature=7808b77cbdaa7e122196bcd75a73f29f2540333a350c4830bbdf5f286e876304&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dmilvus_docs_2.4.x_en.zip&response-content-type=application%2Foctet-stream [following]
+--2024-11-26 21:47:20-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/267273319/c52902a0-e13c-4ca7-92e0-086751098a05?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241127%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241127T024720Z&X-Amz-Expires=300&X-Amz-Signature=7808b77cbdaa7e122196bcd75a73f29f2540333a350c4830bbdf5f286e876304&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dmilvus_docs_2.4.x_en.zip&response-content-type=application%2Foctet-stream
+Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.109.133, 185.199.111.133, 185.199.108.133, ...
+Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.109.133|:443... connected.
+HTTP request sent, awaiting response... 200 OK
+Length: 613094 (599K) [application/octet-stream]
+Saving to: ‘milvus_docs_2.4.x_en.zip’
+
+milvus_docs_2.4.x_e 100%[===================>] 598.72K 1.20MB/s in 0.5s
+
+2024-11-26 21:47:20 (1.20 MB/s) - ‘milvus_docs_2.4.x_en.zip’ saved [613094/613094]
+
+Nous chargeons tous les fichiers markdown à partir du dossier milvus_docs/en/faq. Pour chaque document, nous utilisons simplement "# " pour séparer le contenu du fichier, ce qui permet de séparer grossièrement le contenu de chaque partie principale du fichier markdown.
+from glob import glob
+
+text_lines = []
+
+for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
+ with open(file_path, "r") as file:
+ file_text = file.read()
+
+ text_lines += file_text.split("# ")
+
+Préparer le LLM et le modèle d'intégration
Ollama supporte plusieurs modèles pour les tâches basées sur le LLM et la génération d'embedding, ce qui facilite le développement d'applications de génération augmentée par la recherche (RAG). Pour cette configuration :
+
+- Nous utiliserons Llama 3.2 (3B) comme LLM pour les tâches de génération de texte.
+- Pour la génération d'embedding, nous utiliserons mxbai-embed-large, un modèle à 334M paramètres optimisé pour la similarité sémantique.
+
+Avant de commencer, assurez-vous que les deux modèles sont tirés localement :
+! ollama pull mxbai-embed-large
+
+[?25lpulling manifest ⠋ [?25h[?25l[2K[1Gpulling manifest ⠙ [?25h[?25l[2K[1Gpulling manifest ⠹ [?25h[?25l[2K[1Gpulling manifest ⠸ [?25h[?25l[2K[1Gpulling manifest ⠼ [?25h[?25l[2K[1Gpulling manifest ⠴ [?25h[?25l[2K[1Gpulling manifest
+pulling 819c2adf5ce6... 100% ▕████████████████▏ 669 MB
+pulling c71d239df917... 100% ▕████████████████▏ 11 KB
+pulling b837481ff855... 100% ▕████████████████▏ 16 B
+pulling 38badd946f91... 100% ▕████████████████▏ 408 B
+verifying sha256 digest
+writing manifest
+success [?25h
+
+! ollama pull llama3.2
+
+[?25lpulling manifest ⠋ [?25h[?25l[2K[1Gpulling manifest ⠙ [?25h[?25l[2K[1Gpulling manifest ⠹ [?25h[?25l[2K[1Gpulling manifest ⠸ [?25h[?25l[2K[1Gpulling manifest ⠼ [?25h[?25l[2K[1Gpulling manifest ⠴ [?25h[?25l[2K[1Gpulling manifest
+pulling dde5aa3fc5ff... 100% ▕████████████████▏ 2.0 GB
+pulling 966de95ca8a6... 100% ▕████████████████▏ 1.4 KB
+pulling fcc5a6bec9da... 100% ▕████████████████▏ 7.7 KB
+pulling a70ff7e570d9... 100% ▕████████████████▏ 6.0 KB
+pulling 56bb8bd477a5... 100% ▕████████████████▏ 96 B
+pulling 34bb5ab01051... 100% ▕████████████████▏ 561 B
+verifying sha256 digest
+writing manifest
+success [?25h
+
+Avec ces modèles prêts, nous pouvons procéder à l'implémentation des flux de travail de génération pilotée par LLM et de récupération basée sur l'incorporation.
+import ollama
+
+
+def emb_text(text):
+ response = ollama.embeddings(model="mxbai-embed-large", prompt=text)
+ return response["embedding"]
+
+Générer un embedding de test et imprimer sa dimension et ses premiers éléments.
+test_embedding = emb_text("This is a test")
+embedding_dim = len(test_embedding)
+print(embedding_dim)
+print(test_embedding[:10])
+
+1024
+[0.23276396095752716, 0.4257211685180664, 0.19724100828170776, 0.46120673418045044, -0.46039995551109314, -0.1413791924715042, -0.18261606991291046, -0.07602324336767197, 0.39991313219070435, 0.8337644338607788]
+
+Charger les données dans Milvus
Créer la collection
from pymilvus import MilvusClient
+
+milvus_client = MilvusClient(uri="./milvus_demo.db")
+
+collection_name = "my_rag_collection"
+
+
+Comme pour l'argument de MilvusClient:
+
+- Définir
uri comme un fichier local, par exemple./milvus.db, est la méthode la plus pratique, car elle utilise automatiquement Milvus Lite pour stocker toutes les données dans ce fichier.
+- Si vous avez des données à grande échelle, vous pouvez configurer un serveur Milvus plus performant sur docker ou kubernetes. Dans cette configuration, veuillez utiliser l'uri du serveur, par exemple
http://localhost:19530, comme votre uri.
+- Si vous souhaitez utiliser Zilliz Cloud, le service cloud entièrement géré pour Milvus, ajustez les adresses
uri et token, qui correspondent au point de terminaison public et à la clé Api dans Zilliz Cloud.
+
+
+Vérifier si la collection existe déjà et la supprimer si c'est le cas.
+if milvus_client.has_collection(collection_name):
+ milvus_client.drop_collection(collection_name)
+
+Créer une nouvelle collection avec les paramètres spécifiés.
+Si nous ne spécifions aucune information de champ, Milvus créera automatiquement un champ id par défaut pour la clé primaire et un champ vector pour stocker les données vectorielles. Un champ JSON réservé est utilisé pour stocker les champs non définis par le schéma et leurs valeurs.
+milvus_client.create_collection(
+ collection_name=collection_name,
+ dimension=embedding_dim,
+ metric_type="IP", # Inner product distance
+ consistency_level="Strong", # Strong consistency level
+)
+
+Insérer des données
Parcourez les lignes de texte, créez des enchâssements, puis insérez les données dans Milvus.
+Voici un nouveau champ text, qui est un champ non défini dans le schéma de la collection. Il sera automatiquement ajouté au champ dynamique JSON réservé, qui peut être traité comme un champ normal à un niveau élevé.
+from tqdm import tqdm
+
+data = []
+
+for i, line in enumerate(tqdm(text_lines, desc="Creating embeddings")):
+ data.append({"id": i, "vector": emb_text(line), "text": line})
+
+milvus_client.insert(collection_name=collection_name, data=data)
+
+Creating embeddings: 100%|██████████| 72/72 [00:03<00:00, 22.56it/s]
+
+
+
+
+
+{'insert_count': 72, 'ids': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71], 'cost': 0}
+
+Construire un RAG
Récupérer des données pour une requête
Spécifions une question fréquente sur Milvus.
+question = "How is data stored in milvus?"
+
+Cherchons la question dans la collection et récupérons les 3 meilleures réponses sémantiques.
+search_res = milvus_client.search(
+ collection_name=collection_name,
+ data=[
+ emb_text(question)
+ ], # Use the `emb_text` function to convert the question to an embedding vector
+ limit=3, # Return top 3 results
+ search_params={"metric_type": "IP", "params": {}}, # Inner product distance
+ output_fields=["text"], # Return the text field
+)
+
+Jetons un coup d'œil aux résultats de la recherche de la question.
+import json
+
+retrieved_lines_with_distances = [
+ (res["entity"]["text"], res["distance"]) for res in search_res[0]
+]
+print(json.dumps(retrieved_lines_with_distances, indent=4))
+
+[
+ [
+ " Where does Milvus store data?\n\nMilvus deals with two types of data, inserted data and metadata. \n\nInserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).\n\nMetadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.\n\n###",
+ 231.9398193359375
+ ],
+ [
+ "How does Milvus flush data?\n\nMilvus returns success when inserted data are loaded to the message queue. However, the data are not yet flushed to the disk. Then Milvus' data node writes the data in the message queue to persistent storage as incremental logs. If `flush()` is called, the data node is forced to write all data in the message queue to persistent storage immediately.\n\n###",
+ 226.48316955566406
+ ],
+ [
+ "What is the maximum dataset size Milvus can handle?\n\n \nTheoretically, the maximum dataset size Milvus can handle is determined by the hardware it is run on, specifically system memory and storage:\n\n- Milvus loads all specified collections and partitions into memory before running queries. Therefore, memory size determines the maximum amount of data Milvus can query.\n- When new entities and and collection-related schema (currently only MinIO is supported for data persistence) are added to Milvus, system storage determines the maximum allowable size of inserted data.\n\n###",
+ 210.60745239257812
+ ]
+]
+
+Utiliser LLM pour obtenir une réponse RAG
Convertir les documents récupérés dans un format de chaîne.
+context = "\n".join(
+ [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]
+)
+
+Définir les messages-guides du système et de l'utilisateur pour le modèle de langue. Cette invite est assemblée avec les documents récupérés de Milvus.
+SYSTEM_PROMPT = """
+Human: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.
+"""
+USER_PROMPT = f"""
+Use the following pieces of information enclosed in <context> tags to provide an answer to the question enclosed in <question> tags.
+<context>
+{context}
+</context>
+<question>
+{question}
+</question>
+"""
+
+Utiliser le modèle llama3.2 fourni par Ollama pour générer une réponse basée sur les invites.
+from ollama import chat
+from ollama import ChatResponse
+
+response: ChatResponse = chat(
+ model="llama3.2",
+ messages=[
+ {"role": "system", "content": SYSTEM_PROMPT},
+ {"role": "user", "content": USER_PROMPT},
+ ],
+)
+print(response["message"]["content"])
+
+According to the provided context, data in Milvus is stored in two types:
+
+1. **Inserted data**: Storing data in persistent storage as incremental log. It supports multiple object storage backends such as MinIO, AWS S3, Google Cloud Storage (GCS), Azure Blob Storage, Alibaba Cloud OSS, and Tencent Cloud Object Storage.
+
+2. **Metadata**: Generated within Milvus and stored in etcd.
+
+Génial ! Nous avons réussi à construire un pipeline RAG avec Milvus et Ollama.
diff --git a/localization/v2.4.x/site/fr/integrations/milvus_rag_with_dynamiq.json b/localization/v2.4.x/site/fr/integrations/milvus_rag_with_dynamiq.json
new file mode 100644
index 000000000..e079338b1
--- /dev/null
+++ b/localization/v2.4.x/site/fr/integrations/milvus_rag_with_dynamiq.json
@@ -0,0 +1 @@
+{"codeList":["$ pip install dynamiq pymilvus\n","import os\n\nos.environ[\"OPENAI_API_KEY\"] = \"sk-***********\"\n","# Importing necessary libraries for the workflow\nfrom io import BytesIO\nfrom dynamiq import Workflow\nfrom dynamiq.nodes import InputTransformer\nfrom dynamiq.connections import (\n OpenAI as OpenAIConnection,\n Milvus as MilvusConnection,\n MilvusDeploymentType,\n)\nfrom dynamiq.nodes.converters import PyPDFConverter\nfrom dynamiq.nodes.splitters.document import DocumentSplitter\nfrom dynamiq.nodes.embedders import OpenAIDocumentEmbedder\nfrom dynamiq.nodes.writers import MilvusDocumentWriter\n\n# Initialize the workflow\nrag_wf = Workflow()\n","converter = PyPDFConverter(document_creation_mode=\"one-doc-per-page\")\nconverter_added = rag_wf.flow.add_nodes(\n converter\n) # Add node to the DAG (Directed Acyclic Graph)\n","document_splitter = DocumentSplitter(\n split_by=\"sentence\", # Splits documents into sentences\n split_length=10,\n split_overlap=1,\n input_transformer=InputTransformer(\n selector={\n \"documents\": f\"${[converter.id]}.output.documents\",\n },\n ),\n).depends_on(\n converter\n) # Set dependency on the PDF converter\nsplitter_added = rag_wf.flow.add_nodes(document_splitter) # Add to the DAG\n","embedder = OpenAIDocumentEmbedder(\n connection=OpenAIConnection(api_key=os.environ[\"OPENAI_API_KEY\"]),\n input_transformer=InputTransformer(\n selector={\n \"documents\": f\"${[document_splitter.id]}.output.documents\",\n },\n ),\n).depends_on(\n document_splitter\n) # Set dependency on the splitter\ndocument_embedder_added = rag_wf.flow.add_nodes(embedder) # Add to the DAG\n","vector_store = (\n MilvusDocumentWriter(\n connection=MilvusConnection(\n deployment_type=MilvusDeploymentType.FILE, uri=\"./milvus.db\"\n ),\n index_name=\"my_milvus_collection\",\n dimension=1536,\n create_if_not_exist=True,\n metric_type=\"COSINE\",\n )\n .inputs(documents=embedder.outputs.documents) # Connect to embedder output\n .depends_on(embedder) # Set dependency on the embedder\n)\nmilvus_writer_added = rag_wf.flow.add_nodes(vector_store) # Add to the DAG\n","file_paths = [\"./pdf_files/WhatisMilvus.pdf\"]\ninput_data = {\n \"files\": [BytesIO(open(path, \"rb\").read()) for path in file_paths],\n \"metadata\": [{\"filename\": path} for path in file_paths],\n}\n\n# Run the workflow with the prepared input data\ninserted_data = rag_wf.run(input_data=input_data)\n","from dynamiq import Workflow\nfrom dynamiq.connections import (\n OpenAI as OpenAIConnection,\n Milvus as MilvusConnection,\n MilvusDeploymentType,\n)\nfrom dynamiq.nodes.embedders import OpenAITextEmbedder\nfrom dynamiq.nodes.retrievers import MilvusDocumentRetriever\nfrom dynamiq.nodes.llms import OpenAI\nfrom dynamiq.prompts import Message, Prompt\n\n# Initialize the workflow\nretrieval_wf = Workflow()\n","# Establish OpenAI connection\nopenai_connection = OpenAIConnection(api_key=os.environ[\"OPENAI_API_KEY\"])\n\n# Define the text embedder node\nembedder = OpenAITextEmbedder(\n connection=openai_connection,\n model=\"text-embedding-3-small\",\n)\n\n# Add the embedder node to the workflow\nembedder_added = retrieval_wf.flow.add_nodes(embedder)\n","document_retriever = (\n MilvusDocumentRetriever(\n connection=MilvusConnection(\n deployment_type=MilvusDeploymentType.FILE, uri=\"./milvus.db\"\n ),\n index_name=\"my_milvus_collection\",\n dimension=1536,\n top_k=5,\n )\n .inputs(embedding=embedder.outputs.embedding) # Connect to embedder output\n .depends_on(embedder) # Dependency on the embedder node\n)\n\n# Add the retriever node to the workflow\nmilvus_retriever_added = retrieval_wf.flow.add_nodes(document_retriever)\n","# Define the prompt template for the LLM\nprompt_template = \"\"\"\nPlease answer the question based on the provided context.\n\nQuestion: {{ query }}\n\nContext:\n{% for document in documents %}\n- {{ document.content }}\n{% endfor %}\n\"\"\"\n\n# Create the prompt object\nprompt = Prompt(messages=[Message(content=prompt_template, role=\"user\")])\n","answer_generator = (\n OpenAI(\n connection=openai_connection,\n model=\"gpt-4o\",\n prompt=prompt,\n )\n .inputs(\n documents=document_retriever.outputs.documents,\n query=embedder.outputs.query,\n )\n .depends_on(\n [document_retriever, embedder]\n ) # Dependencies on retriever and embedder\n)\n\n# Add the answer generator node to the workflow\nanswer_generator_added = retrieval_wf.flow.add_nodes(answer_generator)\n","# Run the workflow with a sample query\nsample_query = \"What is the Advanced Search Algorithms in Milvus?\"\n\nresult = retrieval_wf.run(input_data={\"query\": sample_query})\n\nanswer = result.output.get(answer_generator.id).get(\"output\", {}).get(\"content\")\nprint(answer)\n"],"headingContent":"Getting Started with Dynamiq and Milvus","anchorList":[{"label":"Démarrer avec Dynamiq et Milvus","href":"Getting-Started-with-Dynamiq-and-Milvus","type":1,"isActive":false},{"label":"Préparation","href":"Preparation","type":2,"isActive":false},{"label":"RAG - Flux d'indexation de documents","href":"RAG---Document-Indexing-Flow","type":2,"isActive":false},{"label":"Flux de recherche de documents RAG","href":"RAG-Document-Retrieval-Flow","type":2,"isActive":false}]}
\ No newline at end of file
diff --git a/localization/v2.4.x/site/fr/integrations/milvus_rag_with_dynamiq.md b/localization/v2.4.x/site/fr/integrations/milvus_rag_with_dynamiq.md
new file mode 100644
index 000000000..de07e47c9
--- /dev/null
+++ b/localization/v2.4.x/site/fr/integrations/milvus_rag_with_dynamiq.md
@@ -0,0 +1,340 @@
+---
+id: milvus_rag_with_dynamiq.md
+summary: >-
+ Dans ce tutoriel, nous allons explorer comment utiliser Dynamiq avec Milvus,
+ la base de données vectorielle haute performance conçue pour les flux de
+ travail RAG. Milvus excelle dans le stockage, l'indexation et la récupération
+ efficaces des embeddings vectoriels, ce qui en fait un composant indispensable
+ pour les systèmes d'IA qui exigent un accès rapide et précis aux données
+ contextuelles.
+title: Démarrer avec Dynamiq et Milvus
+---
+
+Démarrer avec Dynamiq et Milvus
Dynamiq est un puissant framework Gen AI qui rationalise le développement d'applications basées sur l'IA. Avec un support robuste pour les agents RAG (retrieval-augmented generation) et LLM (large language model), Dynamiq permet aux développeurs de créer des systèmes intelligents et dynamiques avec facilité et efficacité.
+Dans ce tutoriel, nous allons explorer comment utiliser Dynamiq avec Milvus, la base de données vectorielle de haute performance conçue pour les flux de travail RAG. Milvus excelle dans le stockage, l'indexation et la récupération efficaces des embeddings vectoriels, ce qui en fait un composant indispensable pour les systèmes d'IA qui exigent un accès rapide et précis aux données contextuelles.
+Ce guide étape par étape couvrira deux flux de travail RAG fondamentaux :
+
+Flux d'indexation de documents: apprenez à traiter les fichiers d'entrée (par exemple, les PDF), à transformer leur contenu en incorporations vectorielles et à les stocker dans Milvus. L'exploitation des capacités d'indexation hautes performances de Milvus garantit que vos données sont prêtes à être récupérées rapidement.
Flux d'extraction de documents: Découvrez comment interroger Milvus pour des encastrements de documents pertinents et les utiliser pour générer des réponses perspicaces et contextuelles avec les agents LLM de Dynamiq, créant ainsi une expérience utilisateur transparente alimentée par l'IA.
+À la fin de ce tutoriel, vous aurez acquis une solide compréhension de la façon dont Milvus et Dynamiq travaillent ensemble pour construire des systèmes d'IA évolutifs et contextuels adaptés à vos besoins.
+Préparation
Télécharger les bibliothèques nécessaires
$ pip install dynamiq pymilvus
+
+
+Si vous utilisez Google Colab, pour activer les dépendances qui viennent d'être installées, vous devrez peut-être redémarrer le runtime (cliquez sur le menu "Runtime" en haut de l'écran, et sélectionnez "Restart session" dans le menu déroulant).
+
+Configurer l'agent LLM
Nous utiliserons OpenAI comme LLM dans cet exemple. Vous devez préparer la clé api OPENAI_API_KEY en tant que variable d'environnement.
+import os
+
+os.environ["OPENAI_API_KEY"] = "sk-***********"
+
+RAG - Flux d'indexation de documents
Ce tutoriel présente un flux de travail RAG (Retrieval-Augmented Generation) pour l'indexation de documents avec Milvus comme base de données vectorielle. Le flux de travail prend des fichiers PDF en entrée, les traite en plus petits morceaux, génère des embeddings vectoriels en utilisant le modèle d'embedding d'OpenAI, et stocke les embeddings dans une collection Milvus pour une récupération efficace.
+À la fin de ce flux de travail, vous disposerez d'un système d'indexation de documents évolutif et efficace qui prendra en charge les futures tâches RAG telles que la recherche sémantique et la réponse aux questions.
+Importer les bibliothèques requises et initialiser le flux de travail
# Importing necessary libraries for the workflow
+from io import BytesIO
+from dynamiq import Workflow
+from dynamiq.nodes import InputTransformer
+from dynamiq.connections import (
+ OpenAI as OpenAIConnection,
+ Milvus as MilvusConnection,
+ MilvusDeploymentType,
+)
+from dynamiq.nodes.converters import PyPDFConverter
+from dynamiq.nodes.splitters.document import DocumentSplitter
+from dynamiq.nodes.embedders import OpenAIDocumentEmbedder
+from dynamiq.nodes.writers import MilvusDocumentWriter
+
+# Initialize the workflow
+rag_wf = Workflow()
+
+Définir le nœud de conversion PDF
converter = PyPDFConverter(document_creation_mode="one-doc-per-page")
+converter_added = rag_wf.flow.add_nodes(
+ converter
+) # Add node to the DAG (Directed Acyclic Graph)
+
+Définir le nœud de séparation de documents
document_splitter = DocumentSplitter(
+ split_by="sentence", # Splits documents into sentences
+ split_length=10,
+ split_overlap=1,
+ input_transformer=InputTransformer(
+ selector={
+ "documents": f"${[converter.id]}.output.documents",
+ },
+ ),
+).depends_on(
+ converter
+) # Set dependency on the PDF converter
+splitter_added = rag_wf.flow.add_nodes(document_splitter) # Add to the DAG
+
+Définir le nœud d'intégration
embedder = OpenAIDocumentEmbedder(
+ connection=OpenAIConnection(api_key=os.environ["OPENAI_API_KEY"]),
+ input_transformer=InputTransformer(
+ selector={
+ "documents": f"${[document_splitter.id]}.output.documents",
+ },
+ ),
+).depends_on(
+ document_splitter
+) # Set dependency on the splitter
+document_embedder_added = rag_wf.flow.add_nodes(embedder) # Add to the DAG
+
+Définir le nœud Milvus Vector Store
vector_store = (
+ MilvusDocumentWriter(
+ connection=MilvusConnection(
+ deployment_type=MilvusDeploymentType.FILE, uri="./milvus.db"
+ ),
+ index_name="my_milvus_collection",
+ dimension=1536,
+ create_if_not_exist=True,
+ metric_type="COSINE",
+ )
+ .inputs(documents=embedder.outputs.documents) # Connect to embedder output
+ .depends_on(embedder) # Set dependency on the embedder
+)
+milvus_writer_added = rag_wf.flow.add_nodes(vector_store) # Add to the DAG
+
+2024-11-19 22:14:03 - WARNING - Environment variable 'MILVUS_API_TOKEN' not found
+2024-11-19 22:14:03 - INFO - Pass in the local path ./milvus.db, and run it using milvus-lite
+2024-11-19 22:14:04 - DEBUG - Created new connection using: 0bef2849fdb1458a85df8bb9dd27f51d
+2024-11-19 22:14:04 - INFO - Collection my_milvus_collection does not exist. Creating a new collection.
+2024-11-19 22:14:04 - DEBUG - Successfully created collection: my_milvus_collection
+2024-11-19 22:14:05 - DEBUG - Successfully created an index on collection: my_milvus_collection
+2024-11-19 22:14:05 - DEBUG - Successfully created an index on collection: my_milvus_collection
+
+
+Milvus propose deux types de déploiement, répondant à des cas d'utilisation différents :
+
+- MilvusDeploymentType.FILE
+
+
+- Idéal pour le prototypage local ou le stockage de données à petite échelle.
+- Définissez
uri sur un chemin de fichier local (par exemple, ./milvus.db) pour tirer parti de Milvus Lite, qui stocke automatiquement toutes les données dans le fichier spécifié.
+- Il s'agit d'une option pratique pour une installation et une expérimentation rapides.
+
+
+- MilvusDeploymentType.HOST
+
+
+Conçu pour les scénarios de données à grande échelle, tels que la gestion de plus d'un million de vecteurs.
+Serveur auto-hébergé
+
+- Déployez un serveur Milvus haute performance à l'aide de Docker ou de Kubernetes.
+- Configurez l'adresse et le port du serveur en tant que
uri (par exemple, http://localhost:19530).
+- Si l'authentification est activée :
+- Fournir
<your_username>:<your_password> comme token.
+- Si l'authentification est désactivée :
+- Ne pas configurer
token.
+
+Zilliz Cloud (service géré)
+
+- Pour une expérience Milvus entièrement gérée et basée sur le cloud, utilisez Zilliz Cloud.
+- Définissez les paramètres
uri et token en fonction du point de terminaison public et de la clé API fournis dans la console Zilliz Cloud.
+
+
+Définir les données d'entrée et exécuter le flux de travail
file_paths = ["./pdf_files/WhatisMilvus.pdf"]
+input_data = {
+ "files": [BytesIO(open(path, "rb").read()) for path in file_paths],
+ "metadata": [{"filename": path} for path in file_paths],
+}
+
+# Run the workflow with the prepared input data
+inserted_data = rag_wf.run(input_data=input_data)
+
+/var/folders/09/d0hx80nj35sb5hxb5cpc1q180000gn/T/ipykernel_31319/3145804345.py:4: ResourceWarning: unclosed file <_io.BufferedReader name='./pdf_files/WhatisMilvus.pdf'>
+ BytesIO(open(path, "rb").read()) for path in file_paths
+ResourceWarning: Enable tracemalloc to get the object allocation traceback
+2024-11-19 22:14:09 - INFO - Workflow 87878444-6a3d-43f3-ae32-0127564a959f: execution started.
+2024-11-19 22:14:09 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution started.
+2024-11-19 22:14:09 - INFO - Node PyPDF File Converter - 6eb42b1f-7637-407b-a3ac-4167bcf3b5c4: execution started.
+2024-11-19 22:14:09 - INFO - Node PyPDF File Converter - 6eb42b1f-7637-407b-a3ac-4167bcf3b5c4: execution succeeded in 58ms.
+2024-11-19 22:14:09 - INFO - Node DocumentSplitter - 5baed580-6de0-4dcd-bace-d7d947ab6c7f: execution started.
+/Users/jinhonglin/anaconda3/envs/myenv/lib/python3.11/site-packages/websockets/legacy/__init__.py:6: DeprecationWarning: websockets.legacy is deprecated; see https://websockets.readthedocs.io/en/stable/howto/upgrade.html for upgrade instructions
+ warnings.warn( # deprecated in 14.0 - 2024-11-09
+/Users/jinhonglin/anaconda3/envs/myenv/lib/python3.11/site-packages/pydantic/fields.py:804: PydanticDeprecatedSince20: Using extra keyword arguments on `Field` is deprecated and will be removed. Use `json_schema_extra` instead. (Extra keys: 'is_accessible_to_agent'). Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/
+ warn(
+2024-11-19 22:14:09 - INFO - Node DocumentSplitter - 5baed580-6de0-4dcd-bace-d7d947ab6c7f: execution succeeded in 104ms.
+2024-11-19 22:14:09 - INFO - Node OpenAIDocumentEmbedder - 91928f67-a00f-48f6-a864-f6e21672ec7e: execution started.
+2024-11-19 22:14:09 - INFO - Node OpenAIDocumentEmbedder - d30a4cdc-0fab-4aff-b2e5-6161a62cb6fd: execution started.
+2024-11-19 22:14:10 - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
+2024-11-19 22:14:10 - INFO - Node OpenAIDocumentEmbedder - d30a4cdc-0fab-4aff-b2e5-6161a62cb6fd: execution succeeded in 724ms.
+2024-11-19 22:14:10 - INFO - Node MilvusDocumentWriter - dddab4cc-1dae-4e7e-9101-1ec353f530da: execution started.
+2024-11-19 22:14:10 - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
+2024-11-19 22:14:10 - INFO - Node MilvusDocumentWriter - dddab4cc-1dae-4e7e-9101-1ec353f530da: execution succeeded in 66ms.
+2024-11-19 22:14:10 - INFO - Node OpenAIDocumentEmbedder - 91928f67-a00f-48f6-a864-f6e21672ec7e: execution succeeded in 961ms.
+2024-11-19 22:14:10 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution succeeded in 1.3s.
+2024-11-19 22:14:10 - INFO - Workflow 87878444-6a3d-43f3-ae32-0127564a959f: execution succeeded in 1.3s.
+
+Grâce à ce flux de travail, nous avons réussi à mettre en œuvre un pipeline d'indexation de documents en utilisant Milvus comme base de données vectorielle et le modèle d'intégration d'OpenAI pour la représentation sémantique. Cette configuration permet une recherche vectorielle rapide et précise, constituant la base des flux de travail RAG tels que la recherche sémantique, la recherche de documents et les interactions contextuelles pilotées par l'IA.
+Grâce aux capacités de stockage évolutives de Milvus et à l'orchestration de Dynamiq, cette solution est prête à la fois pour le prototypage et les déploiements de production à grande échelle. Vous pouvez maintenant étendre ce pipeline pour inclure des tâches supplémentaires telles que la réponse à des questions basées sur l'extraction ou la génération de contenu pilotée par l'IA.
+Flux de recherche de documents RAG
Dans ce tutoriel, nous mettons en œuvre un flux de recherche de documents RAG (Retrieval-Augmented Generation). Ce flux de travail prend une requête utilisateur, génère une intégration vectorielle pour celle-ci, récupère les documents les plus pertinents à partir d'une base de données vectorielle Milvus, et utilise un grand modèle de langage (LLM) pour générer une réponse détaillée et contextuelle basée sur les documents récupérés.
+En suivant ce flux de travail, vous créerez une solution de bout en bout pour la recherche sémantique et la réponse aux questions, en combinant la puissance de la recherche de documents basée sur les vecteurs avec les capacités des LLM avancés d'OpenAI. Cette approche permet des réponses efficaces et intelligentes aux requêtes des utilisateurs en exploitant les connaissances stockées dans votre base de données de documents.
+Importer les bibliothèques requises et initialiser le flux de travail
from dynamiq import Workflow
+from dynamiq.connections import (
+ OpenAI as OpenAIConnection,
+ Milvus as MilvusConnection,
+ MilvusDeploymentType,
+)
+from dynamiq.nodes.embedders import OpenAITextEmbedder
+from dynamiq.nodes.retrievers import MilvusDocumentRetriever
+from dynamiq.nodes.llms import OpenAI
+from dynamiq.prompts import Message, Prompt
+
+# Initialize the workflow
+retrieval_wf = Workflow()
+
+Définir la connexion OpenAI et l'intégrateur de texte
# Establish OpenAI connection
+openai_connection = OpenAIConnection(api_key=os.environ["OPENAI_API_KEY"])
+
+# Define the text embedder node
+embedder = OpenAITextEmbedder(
+ connection=openai_connection,
+ model="text-embedding-3-small",
+)
+
+# Add the embedder node to the workflow
+embedder_added = retrieval_wf.flow.add_nodes(embedder)
+
+Définir le récupérateur de documents Milvus
document_retriever = (
+ MilvusDocumentRetriever(
+ connection=MilvusConnection(
+ deployment_type=MilvusDeploymentType.FILE, uri="./milvus.db"
+ ),
+ index_name="my_milvus_collection",
+ dimension=1536,
+ top_k=5,
+ )
+ .inputs(embedding=embedder.outputs.embedding) # Connect to embedder output
+ .depends_on(embedder) # Dependency on the embedder node
+)
+
+# Add the retriever node to the workflow
+milvus_retriever_added = retrieval_wf.flow.add_nodes(document_retriever)
+
+2024-11-19 22:14:19 - WARNING - Environment variable 'MILVUS_API_TOKEN' not found
+2024-11-19 22:14:19 - INFO - Pass in the local path ./milvus.db, and run it using milvus-lite
+2024-11-19 22:14:19 - DEBUG - Created new connection using: 98d1132773af4298a894ad5925845fd2
+2024-11-19 22:14:19 - INFO - Collection my_milvus_collection already exists. Skipping creation.
+
+Définir le modèle d'invite
# Define the prompt template for the LLM
+prompt_template = """
+Please answer the question based on the provided context.
+
+Question: {{ query }}
+
+Context:
+{% for document in documents %}
+- {{ document.content }}
+{% endfor %}
+"""
+
+# Create the prompt object
+prompt = Prompt(messages=[Message(content=prompt_template, role="user")])
+
+Définir le générateur de réponses
answer_generator = (
+ OpenAI(
+ connection=openai_connection,
+ model="gpt-4o",
+ prompt=prompt,
+ )
+ .inputs(
+ documents=document_retriever.outputs.documents,
+ query=embedder.outputs.query,
+ )
+ .depends_on(
+ [document_retriever, embedder]
+ ) # Dependencies on retriever and embedder
+)
+
+# Add the answer generator node to the workflow
+answer_generator_added = retrieval_wf.flow.add_nodes(answer_generator)
+
+Exécuter le flux de travail
# Run the workflow with a sample query
+sample_query = "What is the Advanced Search Algorithms in Milvus?"
+
+result = retrieval_wf.run(input_data={"query": sample_query})
+
+answer = result.output.get(answer_generator.id).get("output", {}).get("content")
+print(answer)
+
+2024-11-19 22:14:22 - INFO - Workflow f4a073fb-dfb6-499c-8cac-5710a7ad6d47: execution started.
+2024-11-19 22:14:22 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution started.
+2024-11-19 22:14:22 - INFO - Node OpenAITextEmbedder - 47afb0bc-cf96-429d-b58f-11b6c935fec3: execution started.
+2024-11-19 22:14:23 - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
+2024-11-19 22:14:23 - INFO - Node OpenAITextEmbedder - 47afb0bc-cf96-429d-b58f-11b6c935fec3: execution succeeded in 474ms.
+2024-11-19 22:14:23 - INFO - Node MilvusDocumentRetriever - 51c8311b-4837-411f-ba42-21e28239a2ee: execution started.
+2024-11-19 22:14:23 - INFO - Node MilvusDocumentRetriever - 51c8311b-4837-411f-ba42-21e28239a2ee: execution succeeded in 23ms.
+2024-11-19 22:14:23 - INFO - Node LLM - ac722325-bece-453f-a2ed-135b0749ee7a: execution started.
+2024-11-19 22:14:24 - INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
+2024-11-19 22:14:24 - INFO - Node LLM - ac722325-bece-453f-a2ed-135b0749ee7a: execution succeeded in 1.8s.
+2024-11-19 22:14:25 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution succeeded in 2.4s.
+2024-11-19 22:14:25 - INFO - Workflow f4a073fb-dfb6-499c-8cac-5710a7ad6d47: execution succeeded in 2.4s.
+
+
+The advanced search algorithms in Milvus include a variety of in-memory and on-disk indexing/search algorithms such as IVF (Inverted File), HNSW (Hierarchical Navigable Small World), and DiskANN. These algorithms have been deeply optimized to enhance performance, delivering 30%-70% better performance compared to popular implementations like FAISS and HNSWLib. These optimizations are part of Milvus's design to ensure high efficiency and scalability in handling vector data.
+
diff --git a/localization/v2.4.x/site/fr/menuStructure/fr.json b/localization/v2.4.x/site/fr/menuStructure/fr.json
index 5b4a4a11d..a66bde12d 100644
--- a/localization/v2.4.x/site/fr/menuStructure/fr.json
+++ b/localization/v2.4.x/site/fr/menuStructure/fr.json
@@ -74,7 +74,7 @@
"children": []
},
{
- "label": "Run Milvus Standalone",
+ "label": "Exécuter Milvus en mode autonome",
"id": "run_milvus_docker",
"order": 2,
"isMenu": true,
@@ -1344,6 +1344,24 @@
"id": "build_RAG_with_milvus_and_siliconflow.md",
"order": 4,
"children": []
+ },
+ {
+ "label": "SambaNova",
+ "id": "use_milvus_with_sambanova.md",
+ "order": 5,
+ "children": []
+ },
+ {
+ "label": "Gémeaux",
+ "id": "build_RAG_with_milvus_and_gemini.md",
+ "order": 6,
+ "children": []
+ },
+ {
+ "label": "Ollama",
+ "id": "build_RAG_with_milvus_and_ollama.md",
+ "order": 7,
+ "children": []
}
]
},
@@ -1433,15 +1451,15 @@
"children": []
},
{
- "label": "SambaNova",
- "id": "use_milvus_with_sambanova.md",
+ "label": "PrivateGPT",
+ "id": "use_milvus_in_private_gpt.md",
"order": 10,
"children": []
},
{
- "label": "PrivéGPT",
- "id": "use_milvus_in_private_gpt.md",
- "order": 8,
+ "label": "Dynamiq",
+ "id": "milvus_rag_with_dynamiq.md",
+ "order": 11,
"children": []
}
]
@@ -1493,6 +1511,12 @@
"id": "knowledge_table_with_milvus.md",
"order": 2,
"children": []
+ },
+ {
+ "label": "Cognee",
+ "id": "build_RAG_with_milvus_and_cognee.md",
+ "order": 3,
+ "children": []
}
]
},
diff --git a/localization/v2.4.x/site/it/integrations/build_RAG_with_milvus_and_cognee.json b/localization/v2.4.x/site/it/integrations/build_RAG_with_milvus_and_cognee.json
new file mode 100644
index 000000000..615408ae7
--- /dev/null
+++ b/localization/v2.4.x/site/it/integrations/build_RAG_with_milvus_and_cognee.json
@@ -0,0 +1 @@
+{"codeList":["$ pip install pymilvus git+https://github.com/topoteretes/cognee.git\n","import os\n\nimport cognee\n\ncognee.config.set_llm_api_key(\"YOUR_OPENAI_API_KEY\")\n\n\nos.environ[\"VECTOR_DB_PROVIDER\"] = \"milvus\"\nos.environ[\"VECTOR_DB_URL\"] = \"./milvus.db\"\n","$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip\n$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs\n","from glob import glob\n\ntext_lines = []\n\nfor file_path in glob(\"milvus_docs/en/faq/*.md\", recursive=True):\n with open(file_path, \"r\") as file:\n file_text = file.read()\n\n text_lines += file_text.split(\"# \")\n","await cognee.prune.prune_data()\nawait cognee.prune.prune_system(metadata=True)\n","await cognee.add(data=text_lines, dataset_name=\"milvus_faq\")\nawait cognee.cognify()\n\n# [DocumentChunk(id=UUID('6889e7ef-3670-555c-bb16-3eb50d1d30b0'), updated_at=datetime.datetime(2024, 12, 4, 6, 29, 46, 472907, tzinfo=datetime.timezone.utc), text='Does the query perform in memory? What are incremental data and historical data?\\n\\nYes. When ...\n# ...\n","from cognee.api.v1.search import SearchType\n\nquery_text = \"How is data stored in milvus?\"\nsearch_results = await cognee.search(SearchType.SUMMARIES, query_text=query_text)\n\nprint(search_results[0])\n","from cognee.api.v1.search import SearchType\n\nquery_text = \"How is data stored in milvus?\"\nsearch_results = await cognee.search(SearchType.CHUNKS, query_text=query_text)\n","def format_and_print(data):\n print(\"ID:\", data[\"id\"])\n print(\"\\nText:\\n\")\n paragraphs = data[\"text\"].split(\"\\n\\n\")\n for paragraph in paragraphs:\n print(paragraph.strip())\n print()\n\n\nformat_and_print(search_results[0])\n","await cognee.prune.prune_data()\nawait cognee.prune.prune_system(metadata=True)\n","# We only use one line of text as the dataset, which simplifies the output later\ntext = \"\"\"\n Natural language processing (NLP) is an interdisciplinary\n subfield of computer science and information retrieval.\n \"\"\"\n\nawait cognee.add(text)\nawait cognee.cognify()\n","query_text = \"Tell me about NLP\"\nsearch_results = await cognee.search(SearchType.INSIGHTS, query_text=query_text)\n\nfor result_text in search_results:\n print(result_text)\n\n# Example output:\n# ({'id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'natural language processing', 'description': 'An interdisciplinary subfield of computer science and information retrieval.'}, {'relationship_name': 'is_a_subfield_of', 'source_node_id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'target_node_id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 15, 473137, tzinfo=datetime.timezone.utc)}, {'id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'computer science', 'description': 'The study of computation and information processing.'})\n# (...)\n#\n# It represents nodes and relationships in the knowledge graph:\n# - The first element is the source node (e.g., 'natural language processing').\n# - The second element is the relationship between nodes (e.g., 'is_a_subfield_of').\n# - The third element is the target node (e.g., 'computer science').\n"],"headingContent":"","anchorList":[{"label":"Creare RAG","href":"Build-RAG","type":2,"isActive":false}]}
\ No newline at end of file
diff --git a/localization/v2.4.x/site/it/integrations/build_RAG_with_milvus_and_cognee.md b/localization/v2.4.x/site/it/integrations/build_RAG_with_milvus_and_cognee.md
new file mode 100644
index 000000000..9d4debd07
--- /dev/null
+++ b/localization/v2.4.x/site/it/integrations/build_RAG_with_milvus_and_cognee.md
@@ -0,0 +1,159 @@
+---
+id: build_RAG_with_milvus_and_cognee.md
+summary: >-
+ In questo tutorial vi mostreremo come costruire una pipeline RAG
+ (Retrieval-Augmented Generation) con Milvus e Cognee.
+title: Costruire RAG con Milvus e Cognee
+---
+
+Costruire RAG con Milvus e Cognee
Cognee è una piattaforma per sviluppatori che semplifica lo sviluppo di applicazioni AI con pipeline ECL (Extract, Cognify, Load) scalabili e modulari. Integrandosi perfettamente con Milvus, Cognee consente di collegare e recuperare in modo efficiente conversazioni, documenti e trascrizioni, riducendo le allucinazioni e ottimizzando i costi operativi.
+Grazie al forte supporto per archivi vettoriali come Milvus, database a grafo e LLM, Cognee offre un framework flessibile e personalizzabile per la creazione di sistemi di generazione aumentata del reperimento (RAG). La sua architettura pronta per la produzione garantisce una maggiore precisione ed efficienza per le applicazioni basate sull'intelligenza artificiale.
+In questo tutorial vi mostreremo come costruire una pipeline RAG (Retrieval-Augmented Generation) con Milvus e Cognee.
+$ pip install pymilvus git+https://github.com/topoteretes/cognee.git
+
+
+Se si utilizza Google Colab, per abilitare le dipendenze appena installate, potrebbe essere necessario riavviare il runtime (fare clic sul menu "Runtime" nella parte superiore dello schermo e selezionare "Riavvia sessione" dal menu a discesa).
+
+Per impostazione predefinita, in questo esempio viene utilizzato OpenAI come LLM. È necessario preparare la chiave api e impostarla nella funzione config set_llm_api_key().
+Per configurare Milvus come database vettoriale, impostare VECTOR_DB_PROVIDER su milvus e specificare VECTOR_DB_URL e VECTOR_DB_KEY. Poiché in questa demo utilizziamo Milvus Lite per memorizzare i dati, è necessario fornire solo VECTOR_DB_URL.
+import os
+
+import cognee
+
+cognee.config.set_llm_api_key("YOUR_OPENAI_API_KEY")
+
+
+os.environ["VECTOR_DB_PROVIDER"] = "milvus"
+os.environ["VECTOR_DB_URL"] = "./milvus.db"
+
+
+Per quanto riguarda le variabili d'ambiente VECTOR_DB_URL e VECTOR_DB_KEY:
+
+- L'impostazione di
VECTOR_DB_URL come file locale, ad esempio./milvus.db, è il metodo più conveniente, poiché utilizza automaticamente Milvus Lite per memorizzare tutti i dati in questo file.
+- Se si dispone di una grande quantità di dati, è possibile configurare un server Milvus più performante su docker o kubernetes. In questa configurazione, utilizzare l'uri del server, ad esempio
http://localhost:19530, come VECTOR_DB_URL.
+- Se si desidera utilizzare Zilliz Cloud, il servizio cloud completamente gestito da Milvus, è necessario impostare
VECTOR_DB_URL e VECTOR_DB_KEY, che corrispondono all'endpoint pubblico e alla chiave Api di Zilliz Cloud.
+
+
+Preparare i dati
Come conoscenza privata nel nostro RAG utilizziamo le pagine FAQ della Documentazione di Milvus 2.4.x, una buona fonte di dati per una semplice pipeline RAG.
+Scaricare il file zip ed estrarre i documenti nella cartella milvus_docs.
+$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
+
+Carichiamo tutti i file markdown dalla cartella milvus_docs/en/faq. Per ogni documento, usiamo semplicemente "# " per separare il contenuto del file, che può separare approssimativamente il contenuto di ogni parte principale del file markdown.
+from glob import glob
+
+text_lines = []
+
+for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
+ with open(file_path, "r") as file:
+ file_text = file.read()
+
+ text_lines += file_text.split("# ")
+
+Creare RAG
Ripristino dei dati di Cognee
await cognee.prune.prune_data()
+await cognee.prune.prune_system(metadata=True)
+
+Dopo aver fatto tabula rasa, possiamo aggiungere il nostro set di dati ed elaborarlo in un grafo della conoscenza.
+Aggiungere i dati e cognificare
await cognee.add(data=text_lines, dataset_name="milvus_faq")
+await cognee.cognify()
+
+# [DocumentChunk(id=UUID('6889e7ef-3670-555c-bb16-3eb50d1d30b0'), updated_at=datetime.datetime(2024, 12, 4, 6, 29, 46, 472907, tzinfo=datetime.timezone.utc), text='Does the query perform in memory? What are incremental data and historical data?\n\nYes. When ...
+# ...
+
+Il metodo add carica il set di dati (Milvus FAQs) in Cognee e il metodo cognify elabora i dati per estrarre entità, relazioni e sommari, costruendo un grafo della conoscenza.
+Interrogazione dei sommari
Ora che i dati sono stati elaborati, interroghiamo il grafo della conoscenza.
+from cognee.api.v1.search import SearchType
+
+query_text = "How is data stored in milvus?"
+search_results = await cognee.search(SearchType.SUMMARIES, query_text=query_text)
+
+print(search_results[0])
+
+{'id': 'de5c6713-e079-5d0b-b11d-e9bacd1e0d73', 'text': 'Milvus stores two data types: inserted data and metadata.'}
+
+Questa interrogazione cerca nel grafo della conoscenza un sommario correlato al testo dell'interrogazione e il candidato più correlato viene stampato.
+Interrogazione per pezzi
I riepiloghi offrono approfondimenti di alto livello, ma per ottenere dettagli più dettagliati è possibile interrogare pezzi specifici di dati direttamente dal set di dati elaborati. Questi pezzi sono derivati dai dati originali che sono stati aggiunti e analizzati durante la creazione del grafo della conoscenza.
+from cognee.api.v1.search import SearchType
+
+query_text = "How is data stored in milvus?"
+search_results = await cognee.search(SearchType.CHUNKS, query_text=query_text)
+
+Formattiamoli e visualizziamoli per una migliore leggibilità!
+def format_and_print(data):
+ print("ID:", data["id"])
+ print("\nText:\n")
+ paragraphs = data["text"].split("\n\n")
+ for paragraph in paragraphs:
+ print(paragraph.strip())
+ print()
+
+
+format_and_print(search_results[0])
+
+ID: 4be01c4b-9ee5-541c-9b85-297883934ab3
+
+Text:
+
+Where does Milvus store data?
+
+Milvus deals with two types of data, inserted data and metadata.
+
+Inserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).
+
+Metadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.
+
+###
+
+Nelle fasi precedenti, abbiamo interrogato il dataset Milvus FAQ per ottenere sia riepiloghi che pezzi specifici di dati. Se da un lato questo ha fornito approfondimenti dettagliati e informazioni granulari, dall'altro il set di dati era di grandi dimensioni, il che ha reso difficile visualizzare chiaramente le dipendenze all'interno del grafo della conoscenza.
+Per risolvere questo problema, ripristineremo l'ambiente Cognee e lavoreremo con un set di dati più piccolo e mirato. Questo ci permetterà di mostrare meglio le relazioni e le dipendenze estratte durante il processo di cognificazione. Semplificando i dati, possiamo vedere chiaramente come Cognee organizza e struttura le informazioni nel grafo della conoscenza.
+Resettare Cognee
await cognee.prune.prune_data()
+await cognee.prune.prune_system(metadata=True)
+
+Aggiunta del set di dati focalizzato
In questo caso, viene aggiunto ed elaborato un set di dati più piccolo con una sola riga di testo, per garantire un grafico della conoscenza mirato e facilmente interpretabile.
+# We only use one line of text as the dataset, which simplifies the output later
+text = """
+ Natural language processing (NLP) is an interdisciplinary
+ subfield of computer science and information retrieval.
+ """
+
+await cognee.add(text)
+await cognee.cognify()
+
+Interrogazione per gli approfondimenti
Concentrandoci su questo set di dati più piccolo, possiamo ora analizzare chiaramente le relazioni e la struttura all'interno del grafico della conoscenza.
+query_text = "Tell me about NLP"
+search_results = await cognee.search(SearchType.INSIGHTS, query_text=query_text)
+
+for result_text in search_results:
+ print(result_text)
+
+# Example output:
+# ({'id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'natural language processing', 'description': 'An interdisciplinary subfield of computer science and information retrieval.'}, {'relationship_name': 'is_a_subfield_of', 'source_node_id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'target_node_id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 15, 473137, tzinfo=datetime.timezone.utc)}, {'id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'computer science', 'description': 'The study of computation and information processing.'})
+# (...)
+#
+# It represents nodes and relationships in the knowledge graph:
+# - The first element is the source node (e.g., 'natural language processing').
+# - The second element is the relationship between nodes (e.g., 'is_a_subfield_of').
+# - The third element is the target node (e.g., 'computer science').
+
+Questo output rappresenta i risultati di una query sul grafo della conoscenza, che mostra le entità (nodi) e le loro relazioni (spigoli) estratte dal dataset elaborato. Ogni tupla include un'entità di origine, un tipo di relazione e un'entità di destinazione, insieme a metadati come ID univoci, descrizioni e timestamp. Il grafico evidenzia i concetti chiave e le loro connessioni semantiche, fornendo una comprensione strutturata del set di dati.
+Complimenti, avete imparato l'uso di base di cognee con Milvus. Se volete conoscere un uso più avanzato di cognee, consultate la sua pagina ufficiale.
diff --git a/localization/v2.4.x/site/it/integrations/build_RAG_with_milvus_and_gemini.json b/localization/v2.4.x/site/it/integrations/build_RAG_with_milvus_and_gemini.json
new file mode 100644
index 000000000..f1233f3e2
--- /dev/null
+++ b/localization/v2.4.x/site/it/integrations/build_RAG_with_milvus_and_gemini.json
@@ -0,0 +1 @@
+{"codeList":["$ pip install --upgrade pymilvus google-generativeai requests tqdm\n","import os\n\nos.environ[\"GEMINI_API_KEY\"] = \"***********\"\n","$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip\n$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs\n","from glob import glob\n\ntext_lines = []\n\nfor file_path in glob(\"milvus_docs/en/faq/*.md\", recursive=True):\n with open(file_path, \"r\") as file:\n file_text = file.read()\n\n text_lines += file_text.split(\"# \")\n","import google.generativeai as genai\n\ngenai.configure(api_key=os.environ[\"GEMINI_API_KEY\"])\n\ngemini_model = genai.GenerativeModel(\"gemini-1.5-flash\")\n\nresponse = gemini_model.generate_content(\"who are you\")\nprint(response.text)\n","test_embeddings = genai.embed_content(\n model=\"models/text-embedding-004\", content=[\"This is a test1\", \"This is a test2\"]\n)[\"embedding\"]\n\nembedding_dim = len(test_embeddings[0])\nprint(embedding_dim)\nprint(test_embeddings[0][:10])\n","from pymilvus import MilvusClient\n\nmilvus_client = MilvusClient(uri=\"./milvus_demo.db\")\n\ncollection_name = \"my_rag_collection\"\n","if milvus_client.has_collection(collection_name):\n milvus_client.drop_collection(collection_name)\n","milvus_client.create_collection(\n collection_name=collection_name,\n dimension=embedding_dim,\n metric_type=\"IP\", # Inner product distance\n consistency_level=\"Strong\", # Strong consistency level\n)\n","from tqdm import tqdm\n\ndata = []\n\ndoc_embeddings = genai.embed_content(\n model=\"models/text-embedding-004\", content=text_lines\n)[\"embedding\"]\n\nfor i, line in enumerate(tqdm(text_lines, desc=\"Creating embeddings\")):\n data.append({\"id\": i, \"vector\": doc_embeddings[i], \"text\": line})\n\nmilvus_client.insert(collection_name=collection_name, data=data)\n","question = \"How is data stored in milvus?\"\n","question_embedding = genai.embed_content(\n model=\"models/text-embedding-004\", content=question\n)[\"embedding\"]\n\nsearch_res = milvus_client.search(\n collection_name=collection_name,\n data=[question_embedding],\n limit=3, # Return top 3 results\n search_params={\"metric_type\": \"IP\", \"params\": {}}, # Inner product distance\n output_fields=[\"text\"], # Return the text field\n)\n","import json\n\nretrieved_lines_with_distances = [\n (res[\"entity\"][\"text\"], res[\"distance\"]) for res in search_res[0]\n]\nprint(json.dumps(retrieved_lines_with_distances, indent=4))\n","context = \"\\n\".join(\n [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]\n)\n","SYSTEM_PROMPT = \"\"\"\nHuman: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.\n\"\"\"\nUSER_PROMPT = f\"\"\"\nUse the following pieces of information enclosed in tags to provide an answer to the question enclosed in tags.\n\n{context}\n \n\n{question}\n \n\"\"\"\n","gemini_model = genai.GenerativeModel(\n \"gemini-1.5-flash\", system_instruction=SYSTEM_PROMPT\n)\nresponse = gemini_model.generate_content(USER_PROMPT)\nprint(response.text)\n"],"headingContent":"Build RAG with Milvus and Gemini","anchorList":[{"label":"Costruire RAG con Milvus e Gemini","href":"Build-RAG-with-Milvus-and-Gemini","type":1,"isActive":false},{"label":"Preparazione","href":"Preparation","type":2,"isActive":false},{"label":"Caricare i dati in Milvus","href":"Load-data-into-Milvus","type":2,"isActive":false},{"label":"Costruire la RAG","href":"Build-RAG","type":2,"isActive":false}]}
\ No newline at end of file
diff --git a/localization/v2.4.x/site/it/integrations/build_RAG_with_milvus_and_gemini.md b/localization/v2.4.x/site/it/integrations/build_RAG_with_milvus_and_gemini.md
new file mode 100644
index 000000000..b4b4e4b29
--- /dev/null
+++ b/localization/v2.4.x/site/it/integrations/build_RAG_with_milvus_and_gemini.md
@@ -0,0 +1,246 @@
+---
+id: build_RAG_with_milvus_and_gemini.md
+summary: >-
+ In questo tutorial vi mostreremo come costruire una pipeline RAG
+ (Retrieval-Augmented Generation) con Milvus e Gemini. Utilizzeremo il modello
+ Gemini per generare testo in base a una determinata query. Utilizzeremo anche
+ Milvus per memorizzare e recuperare il testo generato.
+title: Costruire RAG con Milvus e Gemini
+---
+
+Costruire RAG con Milvus e Gemini
L'API Gemini e Google AI Studio consentono di iniziare a lavorare con i modelli più recenti di Google e di trasformare le proprie idee in applicazioni scalabili. Gemini fornisce l'accesso a potenti modelli linguistici come Gemini-1.5-Flash, Gemini-1.5-Flash-8B e Gemini-1.5-Pro per attività come la generazione di testi, l'elaborazione di documenti, la visione, l'analisi audio e altro ancora. L'API consente di inserire contesti lunghi con milioni di token, di mettere a punto i modelli per compiti specifici, di generare output strutturati come JSON e di sfruttare funzionalità come il recupero semantico e l'esecuzione di codice.
+In questo tutorial vi mostreremo come costruire una pipeline RAG (Retrieval-Augmented Generation) con Milvus e Gemini. Utilizzeremo il modello Gemini per generare testo in base a una determinata query. Utilizzeremo anche Milvus per memorizzare e recuperare il testo generato.
+Preparazione
Dipendenze e ambiente
$ pip install --upgrade pymilvus google-generativeai requests tqdm
+
+
+Se si utilizza Google Colab, per abilitare le dipendenze appena installate potrebbe essere necessario riavviare il runtime (fare clic sul menu "Runtime" nella parte superiore dello schermo e selezionare "Riavvia sessione" dal menu a discesa).
+
+Per prima cosa è necessario accedere alla piattaforma Google AI Studio e preparare la chiave api GEMINI_API_KEY come variabile d'ambiente.
+import os
+
+os.environ["GEMINI_API_KEY"] = "***********"
+
+Preparare i dati
Nel nostro RAG utilizziamo le pagine FAQ della Documentazione Milvus 2.4.x come conoscenza privata, che è una buona fonte di dati per una semplice pipeline RAG.
+Scaricare il file zip ed estrarre i documenti nella cartella milvus_docs.
+$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
+
+Carichiamo tutti i file markdown dalla cartella milvus_docs/en/faq. Per ogni documento, usiamo semplicemente "# " per separare il contenuto del file, che può separare approssimativamente il contenuto di ogni parte principale del file markdown.
+from glob import glob
+
+text_lines = []
+
+for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
+ with open(file_path, "r") as file:
+ file_text = file.read()
+
+ text_lines += file_text.split("# ")
+
+Preparare l'LLM e il modello di incorporamento
Utilizziamo gemini-1.5-flash come LLM e text-embedding-004 come modello di incorporamento.
+Proviamo a generare una risposta di prova dall'LLM:
+import google.generativeai as genai
+
+genai.configure(api_key=os.environ["GEMINI_API_KEY"])
+
+gemini_model = genai.GenerativeModel("gemini-1.5-flash")
+
+response = gemini_model.generate_content("who are you")
+print(response.text)
+
+I am a large language model, trained by Google. I am an AI and don't have a personal identity or consciousness. My purpose is to process information and respond to a wide range of prompts and questions in a helpful and informative way.
+
+Generare un embedding di prova e stamparne la dimensione e i primi elementi.
+test_embeddings = genai.embed_content(
+ model="models/text-embedding-004", content=["This is a test1", "This is a test2"]
+)["embedding"]
+
+embedding_dim = len(test_embeddings[0])
+print(embedding_dim)
+print(test_embeddings[0][:10])
+
+768
+[0.013588584, -0.004361838, -0.08481652, -0.039724775, 0.04723794, -0.0051557426, 0.026071774, 0.045514572, -0.016867816, 0.039378334]
+
+Caricare i dati in Milvus
Creare la collezione
from pymilvus import MilvusClient
+
+milvus_client = MilvusClient(uri="./milvus_demo.db")
+
+collection_name = "my_rag_collection"
+
+
+Come per l'argomento di MilvusClient:
+
+- L'impostazione di
uri come file locale, ad esempio./milvus.db, è il metodo più conveniente, poiché utilizza automaticamente Milvus Lite per memorizzare tutti i dati in questo file.
+- Se si dispone di una grande quantità di dati, è possibile configurare un server Milvus più performante su docker o kubernetes. In questa configurazione, utilizzare l'uri del server, ad esempio
http://localhost:19530, come uri.
+- Se si desidera utilizzare Zilliz Cloud, il servizio cloud completamente gestito per Milvus, regolare
uri e token, che corrispondono all'endpoint pubblico e alla chiave Api di Zilliz Cloud.
+
+
+Verificare se la raccolta esiste già e, in caso affermativo, eliminarla.
+if milvus_client.has_collection(collection_name):
+ milvus_client.drop_collection(collection_name)
+
+Creare una nuova raccolta con i parametri specificati.
+Se non si specifica alcun campo, Milvus creerà automaticamente un campo predefinito id per la chiave primaria e un campo vector per memorizzare i dati vettoriali. Un campo JSON riservato viene utilizzato per memorizzare campi non definiti da schemi e i loro valori.
+milvus_client.create_collection(
+ collection_name=collection_name,
+ dimension=embedding_dim,
+ metric_type="IP", # Inner product distance
+ consistency_level="Strong", # Strong consistency level
+)
+
+Inserire i dati
Si intersecano le righe di testo, si creano le incorporazioni e si inseriscono i dati in Milvus.
+Ecco un nuovo campo text, che è un campo non definito nello schema della collezione. Verrà aggiunto automaticamente al campo dinamico JSON riservato, che può essere trattato come un campo normale ad alto livello.
+from tqdm import tqdm
+
+data = []
+
+doc_embeddings = genai.embed_content(
+ model="models/text-embedding-004", content=text_lines
+)["embedding"]
+
+for i, line in enumerate(tqdm(text_lines, desc="Creating embeddings")):
+ data.append({"id": i, "vector": doc_embeddings[i], "text": line})
+
+milvus_client.insert(collection_name=collection_name, data=data)
+
+Creating embeddings: 100%|██████████| 72/72 [00:00<00:00, 468201.38it/s]
+
+
+
+
+
+{'insert_count': 72, 'ids': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71], 'cost': 0}
+
+Costruire la RAG
Recuperare i dati per una query
Specifichiamo una domanda frequente su Milvus.
+question = "How is data stored in milvus?"
+
+Cerchiamo la domanda nella raccolta e recuperiamo le prime 3 corrispondenze semantiche.
+question_embedding = genai.embed_content(
+ model="models/text-embedding-004", content=question
+)["embedding"]
+
+search_res = milvus_client.search(
+ collection_name=collection_name,
+ data=[question_embedding],
+ limit=3, # Return top 3 results
+ search_params={"metric_type": "IP", "params": {}}, # Inner product distance
+ output_fields=["text"], # Return the text field
+)
+
+Diamo un'occhiata ai risultati della ricerca della domanda
+import json
+
+retrieved_lines_with_distances = [
+ (res["entity"]["text"], res["distance"]) for res in search_res[0]
+]
+print(json.dumps(retrieved_lines_with_distances, indent=4))
+
+[
+ [
+ " Where does Milvus store data?\n\nMilvus deals with two types of data, inserted data and metadata. \n\nInserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).\n\nMetadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.\n\n###",
+ 0.8048275113105774
+ ],
+ [
+ "Does the query perform in memory? What are incremental data and historical data?\n\nYes. When a query request comes, Milvus searches both incremental data and historical data by loading them into memory. Incremental data are in the growing segments, which are buffered in memory before they reach the threshold to be persisted in storage engine, while historical data are from the sealed segments that are stored in the object storage. Incremental data and historical data together constitute the whole dataset to search.\n\n###",
+ 0.7574886679649353
+ ],
+ [
+ "What is the maximum dataset size Milvus can handle?\n\n \nTheoretically, the maximum dataset size Milvus can handle is determined by the hardware it is run on, specifically system memory and storage:\n\n- Milvus loads all specified collections and partitions into memory before running queries. Therefore, memory size determines the maximum amount of data Milvus can query.\n- When new entities and and collection-related schema (currently only MinIO is supported for data persistence) are added to Milvus, system storage determines the maximum allowable size of inserted data.\n\n###",
+ 0.7453608512878418
+ ]
+]
+
+Utilizzare LLM per ottenere una risposta RAG
Convertire i documenti recuperati in un formato stringa.
+context = "\n".join(
+ [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]
+)
+
+Definire i prompt del sistema e dell'utente per il Lanage Model. Questo prompt viene assemblato con i documenti recuperati da Milvus.
+SYSTEM_PROMPT = """
+Human: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.
+"""
+USER_PROMPT = f"""
+Use the following pieces of information enclosed in <context> tags to provide an answer to the question enclosed in <question> tags.
+<context>
+{context}
+</context>
+<question>
+{question}
+</question>
+"""
+
+Utilizzare Gemini per generare una risposta basata sui prompt.
+gemini_model = genai.GenerativeModel(
+ "gemini-1.5-flash", system_instruction=SYSTEM_PROMPT
+)
+response = gemini_model.generate_content(USER_PROMPT)
+print(response.text)
+
+Milvus stores data in two ways: Inserted data (vector data, scalar data, and collection-specific schema) is stored as an incremental log in persistent storage using object storage backends such as MinIO, AWS S3, Google Cloud Storage, Azure Blob Storage, Alibaba Cloud OSS, and Tencent Cloud Object Storage. Metadata, generated by each Milvus module, is stored in etcd.
+
+Ottimo! Abbiamo costruito con successo una pipeline RAG con Milvus e Gemini.
diff --git a/localization/v2.4.x/site/it/integrations/build_RAG_with_milvus_and_ollama.json b/localization/v2.4.x/site/it/integrations/build_RAG_with_milvus_and_ollama.json
new file mode 100644
index 000000000..06a7c911a
--- /dev/null
+++ b/localization/v2.4.x/site/it/integrations/build_RAG_with_milvus_and_ollama.json
@@ -0,0 +1 @@
+{"codeList":["$ pip install pymilvus ollama\n","$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip\n$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs\n","from glob import glob\n\ntext_lines = []\n\nfor file_path in glob(\"milvus_docs/en/faq/*.md\", recursive=True):\n with open(file_path, \"r\") as file:\n file_text = file.read()\n\n text_lines += file_text.split(\"# \")\n","! ollama pull mxbai-embed-large\n","! ollama pull llama3.2\n","import ollama\n\n\ndef emb_text(text):\n response = ollama.embeddings(model=\"mxbai-embed-large\", prompt=text)\n return response[\"embedding\"]\n","test_embedding = emb_text(\"This is a test\")\nembedding_dim = len(test_embedding)\nprint(embedding_dim)\nprint(test_embedding[:10])\n","from pymilvus import MilvusClient\n\nmilvus_client = MilvusClient(uri=\"./milvus_demo.db\")\n\ncollection_name = \"my_rag_collection\"\n","if milvus_client.has_collection(collection_name):\n milvus_client.drop_collection(collection_name)\n","milvus_client.create_collection(\n collection_name=collection_name,\n dimension=embedding_dim,\n metric_type=\"IP\", # Inner product distance\n consistency_level=\"Strong\", # Strong consistency level\n)\n","from tqdm import tqdm\n\ndata = []\n\nfor i, line in enumerate(tqdm(text_lines, desc=\"Creating embeddings\")):\n data.append({\"id\": i, \"vector\": emb_text(line), \"text\": line})\n\nmilvus_client.insert(collection_name=collection_name, data=data)\n","question = \"How is data stored in milvus?\"\n","search_res = milvus_client.search(\n collection_name=collection_name,\n data=[\n emb_text(question)\n ], # Use the `emb_text` function to convert the question to an embedding vector\n limit=3, # Return top 3 results\n search_params={\"metric_type\": \"IP\", \"params\": {}}, # Inner product distance\n output_fields=[\"text\"], # Return the text field\n)\n","import json\n\nretrieved_lines_with_distances = [\n (res[\"entity\"][\"text\"], res[\"distance\"]) for res in search_res[0]\n]\nprint(json.dumps(retrieved_lines_with_distances, indent=4))\n","context = \"\\n\".join(\n [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]\n)\n","SYSTEM_PROMPT = \"\"\"\nHuman: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.\n\"\"\"\nUSER_PROMPT = f\"\"\"\nUse the following pieces of information enclosed in tags to provide an answer to the question enclosed in tags.\n\n{context}\n \n\n{question}\n \n\"\"\"\n","from ollama import chat\nfrom ollama import ChatResponse\n\nresponse: ChatResponse = chat(\n model=\"llama3.2\",\n messages=[\n {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n {\"role\": \"user\", \"content\": USER_PROMPT},\n ],\n)\nprint(response[\"message\"][\"content\"])\n"],"headingContent":"Build RAG with Milvus and Ollama","anchorList":[{"label":"Costruire RAG con Milvus e Ollama","href":"Build-RAG-with-Milvus-and-Ollama","type":1,"isActive":false},{"label":"Preparazione","href":"Preparation","type":2,"isActive":false},{"label":"Caricare i dati in Milvus","href":"Load-data-into-Milvus","type":2,"isActive":false},{"label":"Costruire la RAG","href":"Build-RAG","type":2,"isActive":false}]}
\ No newline at end of file
diff --git a/localization/v2.4.x/site/it/integrations/build_RAG_with_milvus_and_ollama.md b/localization/v2.4.x/site/it/integrations/build_RAG_with_milvus_and_ollama.md
new file mode 100644
index 000000000..df286dfc2
--- /dev/null
+++ b/localization/v2.4.x/site/it/integrations/build_RAG_with_milvus_and_ollama.md
@@ -0,0 +1,281 @@
+---
+id: build_RAG_with_milvus_and_ollama.md
+summary: >-
+ In questa guida vi mostreremo come sfruttare Ollama e Milvus per costruire una
+ pipeline RAG (Retrieval-Augmented Generation) in modo efficiente e sicuro.
+title: Costruire RAG con Milvus e Ollama
+---
+
+Costruire RAG con Milvus e Ollama
Ollama è una piattaforma open-source che semplifica l'esecuzione e la personalizzazione di grandi modelli linguistici (LLM) a livello locale. Offre un'esperienza facile da usare e senza cloud, consentendo di scaricare, installare e interagire con i modelli senza richiedere competenze tecniche avanzate. Grazie a una libreria crescente di LLM pre-addestrati, da quelli generici a quelli specifici per il dominio, Ellama semplifica la gestione e la personalizzazione dei modelli per varie applicazioni. Garantisce la privacy dei dati e la flessibilità, consentendo agli utenti di mettere a punto, ottimizzare e distribuire soluzioni basate sull'intelligenza artificiale interamente sulle proprie macchine.
+In questa guida vi mostreremo come sfruttare Ollama e Milvus per costruire una pipeline RAG (Retrieval-Augmented Generation) in modo efficiente e sicuro.
+Preparazione
Dipendenze e ambiente
$ pip install pymilvus ollama
+
+
+Se si utilizza Google Colab, per abilitare le dipendenze appena installate potrebbe essere necessario riavviare il runtime (fare clic sul menu "Runtime" nella parte superiore dello schermo e selezionare "Riavvia sessione" dal menu a discesa).
+
+Preparare i dati
Come conoscenza privata nel nostro RAG utilizziamo le pagine FAQ della Documentazione Milvus 2.4.x, che è una buona fonte di dati per una semplice pipeline RAG.
+Scaricare il file zip ed estrarre i documenti nella cartella milvus_docs.
+$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
+
+--2024-11-26 21:47:19-- https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+Resolving github.com (github.com)... 140.82.112.4
+Connecting to github.com (github.com)|140.82.112.4|:443... connected.
+HTTP request sent, awaiting response... 302 Found
+Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/267273319/c52902a0-e13c-4ca7-92e0-086751098a05?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241127%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241127T024720Z&X-Amz-Expires=300&X-Amz-Signature=7808b77cbdaa7e122196bcd75a73f29f2540333a350c4830bbdf5f286e876304&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dmilvus_docs_2.4.x_en.zip&response-content-type=application%2Foctet-stream [following]
+--2024-11-26 21:47:20-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/267273319/c52902a0-e13c-4ca7-92e0-086751098a05?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241127%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241127T024720Z&X-Amz-Expires=300&X-Amz-Signature=7808b77cbdaa7e122196bcd75a73f29f2540333a350c4830bbdf5f286e876304&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dmilvus_docs_2.4.x_en.zip&response-content-type=application%2Foctet-stream
+Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.109.133, 185.199.111.133, 185.199.108.133, ...
+Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.109.133|:443... connected.
+HTTP request sent, awaiting response... 200 OK
+Length: 613094 (599K) [application/octet-stream]
+Saving to: ‘milvus_docs_2.4.x_en.zip’
+
+milvus_docs_2.4.x_e 100%[===================>] 598.72K 1.20MB/s in 0.5s
+
+2024-11-26 21:47:20 (1.20 MB/s) - ‘milvus_docs_2.4.x_en.zip’ saved [613094/613094]
+
+Carichiamo tutti i file markdown dalla cartella milvus_docs/en/faq. Per ogni documento, usiamo semplicemente "# " per separare il contenuto del file, che può separare approssimativamente il contenuto di ogni parte principale del file markdown.
+from glob import glob
+
+text_lines = []
+
+for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
+ with open(file_path, "r") as file:
+ file_text = file.read()
+
+ text_lines += file_text.split("# ")
+
+Preparare il modello LLM e il modello di incorporamento
Ollama supporta diversi modelli sia per le attività basate su LLM sia per la generazione di embedding, rendendo più semplice lo sviluppo di applicazioni di retrieval-augmented generation (RAG). Per questa configurazione:
+
+- Utilizzeremo Llama 3.2 (3B) come LLM per la generazione di testo.
+- Per la generazione di embedding, utilizzeremo mxbai-embed-large, un modello a 334M parametri ottimizzato per la similarità semantica.
+
+Prima di iniziare, assicurarsi che entrambi i modelli siano estratti localmente:
+! ollama pull mxbai-embed-large
+
+[?25lpulling manifest ⠋ [?25h[?25l[2K[1Gpulling manifest ⠙ [?25h[?25l[2K[1Gpulling manifest ⠹ [?25h[?25l[2K[1Gpulling manifest ⠸ [?25h[?25l[2K[1Gpulling manifest ⠼ [?25h[?25l[2K[1Gpulling manifest ⠴ [?25h[?25l[2K[1Gpulling manifest
+pulling 819c2adf5ce6... 100% ▕████████████████▏ 669 MB
+pulling c71d239df917... 100% ▕████████████████▏ 11 KB
+pulling b837481ff855... 100% ▕████████████████▏ 16 B
+pulling 38badd946f91... 100% ▕████████████████▏ 408 B
+verifying sha256 digest
+writing manifest
+success [?25h
+
+! ollama pull llama3.2
+
+[?25lpulling manifest ⠋ [?25h[?25l[2K[1Gpulling manifest ⠙ [?25h[?25l[2K[1Gpulling manifest ⠹ [?25h[?25l[2K[1Gpulling manifest ⠸ [?25h[?25l[2K[1Gpulling manifest ⠼ [?25h[?25l[2K[1Gpulling manifest ⠴ [?25h[?25l[2K[1Gpulling manifest
+pulling dde5aa3fc5ff... 100% ▕████████████████▏ 2.0 GB
+pulling 966de95ca8a6... 100% ▕████████████████▏ 1.4 KB
+pulling fcc5a6bec9da... 100% ▕████████████████▏ 7.7 KB
+pulling a70ff7e570d9... 100% ▕████████████████▏ 6.0 KB
+pulling 56bb8bd477a5... 100% ▕████████████████▏ 96 B
+pulling 34bb5ab01051... 100% ▕████████████████▏ 561 B
+verifying sha256 digest
+writing manifest
+success [?25h
+
+Con questi modelli pronti, possiamo procedere all'implementazione dei flussi di lavoro di generazione guidata da LLM e di recupero basato su embedding.
+import ollama
+
+
+def emb_text(text):
+ response = ollama.embeddings(model="mxbai-embed-large", prompt=text)
+ return response["embedding"]
+
+Generare un embedding di prova e stamparne la dimensione e i primi elementi.
+test_embedding = emb_text("This is a test")
+embedding_dim = len(test_embedding)
+print(embedding_dim)
+print(test_embedding[:10])
+
+1024
+[0.23276396095752716, 0.4257211685180664, 0.19724100828170776, 0.46120673418045044, -0.46039995551109314, -0.1413791924715042, -0.18261606991291046, -0.07602324336767197, 0.39991313219070435, 0.8337644338607788]
+
+Caricare i dati in Milvus
Creare la collezione
from pymilvus import MilvusClient
+
+milvus_client = MilvusClient(uri="./milvus_demo.db")
+
+collection_name = "my_rag_collection"
+
+
+Come per l'argomento di MilvusClient:
+
+- L'impostazione di
uri come file locale, ad esempio./milvus.db, è il metodo più conveniente, poiché utilizza automaticamente Milvus Lite per memorizzare tutti i dati in questo file.
+- Se si dispone di una grande quantità di dati, è possibile configurare un server Milvus più performante su docker o kubernetes. In questa configurazione, utilizzare l'uri del server, ad esempio
http://localhost:19530, come uri.
+- Se si desidera utilizzare Zilliz Cloud, il servizio cloud completamente gestito per Milvus, regolare
uri e token, che corrispondono all'endpoint pubblico e alla chiave Api di Zilliz Cloud.
+
+
+Verificare se la raccolta esiste già e, in caso affermativo, eliminarla.
+if milvus_client.has_collection(collection_name):
+ milvus_client.drop_collection(collection_name)
+
+Creare una nuova raccolta con i parametri specificati.
+Se non si specifica alcun campo, Milvus creerà automaticamente un campo predefinito id per la chiave primaria e un campo vector per memorizzare i dati vettoriali. Un campo JSON riservato viene utilizzato per memorizzare campi non definiti dalla mappa e i loro valori.
+milvus_client.create_collection(
+ collection_name=collection_name,
+ dimension=embedding_dim,
+ metric_type="IP", # Inner product distance
+ consistency_level="Strong", # Strong consistency level
+)
+
+Inserire i dati
Si intersecano le righe di testo, si creano le incorporazioni e si inseriscono i dati in Milvus.
+Ecco un nuovo campo text, che è un campo non definito nello schema della collezione. Verrà aggiunto automaticamente al campo dinamico JSON riservato, che può essere trattato come un campo normale ad alto livello.
+from tqdm import tqdm
+
+data = []
+
+for i, line in enumerate(tqdm(text_lines, desc="Creating embeddings")):
+ data.append({"id": i, "vector": emb_text(line), "text": line})
+
+milvus_client.insert(collection_name=collection_name, data=data)
+
+Creating embeddings: 100%|██████████| 72/72 [00:03<00:00, 22.56it/s]
+
+
+
+
+
+{'insert_count': 72, 'ids': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71], 'cost': 0}
+
+Costruire la RAG
Recuperare i dati per una query
Specifichiamo una domanda frequente su Milvus.
+question = "How is data stored in milvus?"
+
+Cerchiamo la domanda nella raccolta e recuperiamo le prime 3 corrispondenze semantiche.
+search_res = milvus_client.search(
+ collection_name=collection_name,
+ data=[
+ emb_text(question)
+ ], # Use the `emb_text` function to convert the question to an embedding vector
+ limit=3, # Return top 3 results
+ search_params={"metric_type": "IP", "params": {}}, # Inner product distance
+ output_fields=["text"], # Return the text field
+)
+
+Diamo un'occhiata ai risultati della ricerca della domanda
+import json
+
+retrieved_lines_with_distances = [
+ (res["entity"]["text"], res["distance"]) for res in search_res[0]
+]
+print(json.dumps(retrieved_lines_with_distances, indent=4))
+
+[
+ [
+ " Where does Milvus store data?\n\nMilvus deals with two types of data, inserted data and metadata. \n\nInserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).\n\nMetadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.\n\n###",
+ 231.9398193359375
+ ],
+ [
+ "How does Milvus flush data?\n\nMilvus returns success when inserted data are loaded to the message queue. However, the data are not yet flushed to the disk. Then Milvus' data node writes the data in the message queue to persistent storage as incremental logs. If `flush()` is called, the data node is forced to write all data in the message queue to persistent storage immediately.\n\n###",
+ 226.48316955566406
+ ],
+ [
+ "What is the maximum dataset size Milvus can handle?\n\n \nTheoretically, the maximum dataset size Milvus can handle is determined by the hardware it is run on, specifically system memory and storage:\n\n- Milvus loads all specified collections and partitions into memory before running queries. Therefore, memory size determines the maximum amount of data Milvus can query.\n- When new entities and and collection-related schema (currently only MinIO is supported for data persistence) are added to Milvus, system storage determines the maximum allowable size of inserted data.\n\n###",
+ 210.60745239257812
+ ]
+]
+
+Utilizzare LLM per ottenere una risposta RAG
Convertire i documenti recuperati in un formato stringa.
+context = "\n".join(
+ [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]
+)
+
+Definire i prompt del sistema e dell'utente per il Lanage Model. Questo prompt viene assemblato con i documenti recuperati da Milvus.
+SYSTEM_PROMPT = """
+Human: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.
+"""
+USER_PROMPT = f"""
+Use the following pieces of information enclosed in <context> tags to provide an answer to the question enclosed in <question> tags.
+<context>
+{context}
+</context>
+<question>
+{question}
+</question>
+"""
+
+Utilizzare il modello llama3.2 fornito da Ollama per generare una risposta basata sui prompt.
+from ollama import chat
+from ollama import ChatResponse
+
+response: ChatResponse = chat(
+ model="llama3.2",
+ messages=[
+ {"role": "system", "content": SYSTEM_PROMPT},
+ {"role": "user", "content": USER_PROMPT},
+ ],
+)
+print(response["message"]["content"])
+
+According to the provided context, data in Milvus is stored in two types:
+
+1. **Inserted data**: Storing data in persistent storage as incremental log. It supports multiple object storage backends such as MinIO, AWS S3, Google Cloud Storage (GCS), Azure Blob Storage, Alibaba Cloud OSS, and Tencent Cloud Object Storage.
+
+2. **Metadata**: Generated within Milvus and stored in etcd.
+
+Ottimo! Abbiamo costruito con successo una pipeline RAG con Milvus e Ollama.
diff --git a/localization/v2.4.x/site/it/integrations/milvus_rag_with_dynamiq.json b/localization/v2.4.x/site/it/integrations/milvus_rag_with_dynamiq.json
new file mode 100644
index 000000000..149d96547
--- /dev/null
+++ b/localization/v2.4.x/site/it/integrations/milvus_rag_with_dynamiq.json
@@ -0,0 +1 @@
+{"codeList":["$ pip install dynamiq pymilvus\n","import os\n\nos.environ[\"OPENAI_API_KEY\"] = \"sk-***********\"\n","# Importing necessary libraries for the workflow\nfrom io import BytesIO\nfrom dynamiq import Workflow\nfrom dynamiq.nodes import InputTransformer\nfrom dynamiq.connections import (\n OpenAI as OpenAIConnection,\n Milvus as MilvusConnection,\n MilvusDeploymentType,\n)\nfrom dynamiq.nodes.converters import PyPDFConverter\nfrom dynamiq.nodes.splitters.document import DocumentSplitter\nfrom dynamiq.nodes.embedders import OpenAIDocumentEmbedder\nfrom dynamiq.nodes.writers import MilvusDocumentWriter\n\n# Initialize the workflow\nrag_wf = Workflow()\n","converter = PyPDFConverter(document_creation_mode=\"one-doc-per-page\")\nconverter_added = rag_wf.flow.add_nodes(\n converter\n) # Add node to the DAG (Directed Acyclic Graph)\n","document_splitter = DocumentSplitter(\n split_by=\"sentence\", # Splits documents into sentences\n split_length=10,\n split_overlap=1,\n input_transformer=InputTransformer(\n selector={\n \"documents\": f\"${[converter.id]}.output.documents\",\n },\n ),\n).depends_on(\n converter\n) # Set dependency on the PDF converter\nsplitter_added = rag_wf.flow.add_nodes(document_splitter) # Add to the DAG\n","embedder = OpenAIDocumentEmbedder(\n connection=OpenAIConnection(api_key=os.environ[\"OPENAI_API_KEY\"]),\n input_transformer=InputTransformer(\n selector={\n \"documents\": f\"${[document_splitter.id]}.output.documents\",\n },\n ),\n).depends_on(\n document_splitter\n) # Set dependency on the splitter\ndocument_embedder_added = rag_wf.flow.add_nodes(embedder) # Add to the DAG\n","vector_store = (\n MilvusDocumentWriter(\n connection=MilvusConnection(\n deployment_type=MilvusDeploymentType.FILE, uri=\"./milvus.db\"\n ),\n index_name=\"my_milvus_collection\",\n dimension=1536,\n create_if_not_exist=True,\n metric_type=\"COSINE\",\n )\n .inputs(documents=embedder.outputs.documents) # Connect to embedder output\n .depends_on(embedder) # Set dependency on the embedder\n)\nmilvus_writer_added = rag_wf.flow.add_nodes(vector_store) # Add to the DAG\n","file_paths = [\"./pdf_files/WhatisMilvus.pdf\"]\ninput_data = {\n \"files\": [BytesIO(open(path, \"rb\").read()) for path in file_paths],\n \"metadata\": [{\"filename\": path} for path in file_paths],\n}\n\n# Run the workflow with the prepared input data\ninserted_data = rag_wf.run(input_data=input_data)\n","from dynamiq import Workflow\nfrom dynamiq.connections import (\n OpenAI as OpenAIConnection,\n Milvus as MilvusConnection,\n MilvusDeploymentType,\n)\nfrom dynamiq.nodes.embedders import OpenAITextEmbedder\nfrom dynamiq.nodes.retrievers import MilvusDocumentRetriever\nfrom dynamiq.nodes.llms import OpenAI\nfrom dynamiq.prompts import Message, Prompt\n\n# Initialize the workflow\nretrieval_wf = Workflow()\n","# Establish OpenAI connection\nopenai_connection = OpenAIConnection(api_key=os.environ[\"OPENAI_API_KEY\"])\n\n# Define the text embedder node\nembedder = OpenAITextEmbedder(\n connection=openai_connection,\n model=\"text-embedding-3-small\",\n)\n\n# Add the embedder node to the workflow\nembedder_added = retrieval_wf.flow.add_nodes(embedder)\n","document_retriever = (\n MilvusDocumentRetriever(\n connection=MilvusConnection(\n deployment_type=MilvusDeploymentType.FILE, uri=\"./milvus.db\"\n ),\n index_name=\"my_milvus_collection\",\n dimension=1536,\n top_k=5,\n )\n .inputs(embedding=embedder.outputs.embedding) # Connect to embedder output\n .depends_on(embedder) # Dependency on the embedder node\n)\n\n# Add the retriever node to the workflow\nmilvus_retriever_added = retrieval_wf.flow.add_nodes(document_retriever)\n","# Define the prompt template for the LLM\nprompt_template = \"\"\"\nPlease answer the question based on the provided context.\n\nQuestion: {{ query }}\n\nContext:\n{% for document in documents %}\n- {{ document.content }}\n{% endfor %}\n\"\"\"\n\n# Create the prompt object\nprompt = Prompt(messages=[Message(content=prompt_template, role=\"user\")])\n","answer_generator = (\n OpenAI(\n connection=openai_connection,\n model=\"gpt-4o\",\n prompt=prompt,\n )\n .inputs(\n documents=document_retriever.outputs.documents,\n query=embedder.outputs.query,\n )\n .depends_on(\n [document_retriever, embedder]\n ) # Dependencies on retriever and embedder\n)\n\n# Add the answer generator node to the workflow\nanswer_generator_added = retrieval_wf.flow.add_nodes(answer_generator)\n","# Run the workflow with a sample query\nsample_query = \"What is the Advanced Search Algorithms in Milvus?\"\n\nresult = retrieval_wf.run(input_data={\"query\": sample_query})\n\nanswer = result.output.get(answer_generator.id).get(\"output\", {}).get(\"content\")\nprint(answer)\n"],"headingContent":"Getting Started with Dynamiq and Milvus","anchorList":[{"label":"Come iniziare con Dynamiq e Milvus","href":"Getting-Started-with-Dynamiq-and-Milvus","type":1,"isActive":false},{"label":"Preparazione","href":"Preparation","type":2,"isActive":false},{"label":"RAG - Flusso di indicizzazione dei documenti","href":"RAG---Document-Indexing-Flow","type":2,"isActive":false},{"label":"Flusso di recupero dei documenti RAG","href":"RAG-Document-Retrieval-Flow","type":2,"isActive":false}]}
\ No newline at end of file
diff --git a/localization/v2.4.x/site/it/integrations/milvus_rag_with_dynamiq.md b/localization/v2.4.x/site/it/integrations/milvus_rag_with_dynamiq.md
new file mode 100644
index 000000000..4e60dca51
--- /dev/null
+++ b/localization/v2.4.x/site/it/integrations/milvus_rag_with_dynamiq.md
@@ -0,0 +1,340 @@
+---
+id: milvus_rag_with_dynamiq.md
+summary: >-
+ In questo tutorial, esploreremo come utilizzare senza problemi Dynamiq con
+ Milvus, il database vettoriale ad alte prestazioni costruito appositamente per
+ i flussi di lavoro RAG. Milvus eccelle per l'archiviazione, l'indicizzazione e
+ il recupero efficiente delle incorporazioni vettoriali, rendendolo un
+ componente indispensabile per i sistemi di intelligenza artificiale che
+ richiedono un accesso rapido e preciso ai dati contestuali.
+title: Come iniziare con Dynamiq e Milvus
+---
+
+Come iniziare con Dynamiq e Milvus
Dynamiq è un potente framework Gen AI che semplifica lo sviluppo di applicazioni basate sull'AI. Grazie al solido supporto per gli agenti RAG (retrieval-augmented generation) e LLM (large language model), Dynamiq consente agli sviluppatori di creare sistemi intelligenti e dinamici con facilità ed efficienza.
+In questo tutorial, vedremo come utilizzare Dynamiq con Milvus, il database vettoriale ad alte prestazioni creato appositamente per i flussi di lavoro RAG. Milvus eccelle per l'archiviazione, l'indicizzazione e il recupero efficiente delle incorporazioni vettoriali, rendendolo un componente indispensabile per i sistemi di intelligenza artificiale che richiedono un accesso rapido e preciso ai dati contestuali.
+Questa guida passo passo coprirà due flussi di lavoro RAG fondamentali:
+
+Flusso di indicizzazione dei documenti: imparare a elaborare i file di input (ad esempio, i PDF), trasformare il loro contenuto in embeddings vettoriali e memorizzarli in Milvus. Sfruttando le capacità di indicizzazione ad alte prestazioni di Milvus, i dati sono pronti per essere recuperati rapidamente.
Flusso di recupero dei documenti: scoprite come interrogare Milvus per trovare embeddings di documenti rilevanti e usarli per generare risposte perspicaci e consapevoli del contesto con gli agenti LLM di Dynamiq, creando un'esperienza utente AI senza soluzione di continuità.
+Alla fine di questo tutorial, avrete una solida comprensione di come Milvus e Dynamiq lavorano insieme per costruire sistemi di intelligenza artificiale scalabili e consapevoli del contesto, adatti alle vostre esigenze.
+Preparazione
Scaricare le librerie necessarie
$ pip install dynamiq pymilvus
+
+
+Se si utilizza Google Colab, per abilitare le dipendenze appena installate, potrebbe essere necessario riavviare il runtime (fare clic sul menu "Runtime" nella parte superiore dello schermo e selezionare "Riavvia sessione" dal menu a discesa).
+
+Configurare l'agente LLM
In questo esempio utilizzeremo OpenAI come LLM. È necessario preparare la chiave api OPENAI_API_KEY come variabile d'ambiente.
+import os
+
+os.environ["OPENAI_API_KEY"] = "sk-***********"
+
+RAG - Flusso di indicizzazione dei documenti
Questa esercitazione mostra un flusso di lavoro RAG (Retrieval-Augmented Generation) per l'indicizzazione di documenti con Milvus come database vettoriale. Il flusso di lavoro prende in input i file PDF, li elabora in pezzi più piccoli, genera incorporazioni vettoriali utilizzando il modello di incorporazione di OpenAI e memorizza le incorporazioni in una raccolta Milvus per un recupero efficiente.
+Alla fine di questo flusso di lavoro, avrete un sistema di indicizzazione dei documenti scalabile ed efficiente, in grado di supportare le future attività di RAG, come la ricerca semantica e la risposta alle domande.
+Importare le librerie necessarie e inizializzare il flusso di lavoro
# Importing necessary libraries for the workflow
+from io import BytesIO
+from dynamiq import Workflow
+from dynamiq.nodes import InputTransformer
+from dynamiq.connections import (
+ OpenAI as OpenAIConnection,
+ Milvus as MilvusConnection,
+ MilvusDeploymentType,
+)
+from dynamiq.nodes.converters import PyPDFConverter
+from dynamiq.nodes.splitters.document import DocumentSplitter
+from dynamiq.nodes.embedders import OpenAIDocumentEmbedder
+from dynamiq.nodes.writers import MilvusDocumentWriter
+
+# Initialize the workflow
+rag_wf = Workflow()
+
+Definire il nodo convertitore PDF
converter = PyPDFConverter(document_creation_mode="one-doc-per-page")
+converter_added = rag_wf.flow.add_nodes(
+ converter
+) # Add node to the DAG (Directed Acyclic Graph)
+
+Definire il nodo di divisione dei documenti
document_splitter = DocumentSplitter(
+ split_by="sentence", # Splits documents into sentences
+ split_length=10,
+ split_overlap=1,
+ input_transformer=InputTransformer(
+ selector={
+ "documents": f"${[converter.id]}.output.documents",
+ },
+ ),
+).depends_on(
+ converter
+) # Set dependency on the PDF converter
+splitter_added = rag_wf.flow.add_nodes(document_splitter) # Add to the DAG
+
+Definire il nodo di incorporamento
embedder = OpenAIDocumentEmbedder(
+ connection=OpenAIConnection(api_key=os.environ["OPENAI_API_KEY"]),
+ input_transformer=InputTransformer(
+ selector={
+ "documents": f"${[document_splitter.id]}.output.documents",
+ },
+ ),
+).depends_on(
+ document_splitter
+) # Set dependency on the splitter
+document_embedder_added = rag_wf.flow.add_nodes(embedder) # Add to the DAG
+
+Definire il nodo Milvus Vector Store
vector_store = (
+ MilvusDocumentWriter(
+ connection=MilvusConnection(
+ deployment_type=MilvusDeploymentType.FILE, uri="./milvus.db"
+ ),
+ index_name="my_milvus_collection",
+ dimension=1536,
+ create_if_not_exist=True,
+ metric_type="COSINE",
+ )
+ .inputs(documents=embedder.outputs.documents) # Connect to embedder output
+ .depends_on(embedder) # Set dependency on the embedder
+)
+milvus_writer_added = rag_wf.flow.add_nodes(vector_store) # Add to the DAG
+
+2024-11-19 22:14:03 - WARNING - Environment variable 'MILVUS_API_TOKEN' not found
+2024-11-19 22:14:03 - INFO - Pass in the local path ./milvus.db, and run it using milvus-lite
+2024-11-19 22:14:04 - DEBUG - Created new connection using: 0bef2849fdb1458a85df8bb9dd27f51d
+2024-11-19 22:14:04 - INFO - Collection my_milvus_collection does not exist. Creating a new collection.
+2024-11-19 22:14:04 - DEBUG - Successfully created collection: my_milvus_collection
+2024-11-19 22:14:05 - DEBUG - Successfully created an index on collection: my_milvus_collection
+2024-11-19 22:14:05 - DEBUG - Successfully created an index on collection: my_milvus_collection
+
+
+Milvus offre due tipi di distribuzione, adatti a diversi casi d'uso:
+
+- MilvusDeploymentType.FILE
+
+
+- Ideale per la prototipazione locale o l'archiviazione di dati su piccola scala.
+- Impostare
uri su un percorso di file locale (ad esempio, ./milvus.db) per sfruttare Milvus Lite, che memorizza automaticamente tutti i dati nel file specificato.
+- Si tratta di un'opzione comoda per una rapida configurazione e sperimentazione.
+
+
+- MilvusDeploymentType.HOST
+
+
+Progettato per scenari di dati su larga scala, come la gestione di oltre un milione di vettori.
+Server auto-ospitato
+
+- Distribuisce un server Milvus ad alte prestazioni utilizzando Docker o Kubernetes.
+- Configurare l'indirizzo e la porta del server come
uri (ad esempio, http://localhost:19530).
+- Se l'autenticazione è abilitata:
+- Fornire
<your_username>:<your_password> come token.
+- Se l'autenticazione è disabilitata:
+- Lasciare
token non impostato.
+
+Zilliz Cloud (Servizio gestito)
+
+- Per un'esperienza Milvus completamente gestita e basata sul cloud, utilizzate Zilliz Cloud.
+- Impostare
uri e token in base all'endpoint pubblico e alla chiave API forniti nella console di Zilliz Cloud.
+
+
+Definizione dei dati di input ed esecuzione del flusso di lavoro
file_paths = ["./pdf_files/WhatisMilvus.pdf"]
+input_data = {
+ "files": [BytesIO(open(path, "rb").read()) for path in file_paths],
+ "metadata": [{"filename": path} for path in file_paths],
+}
+
+# Run the workflow with the prepared input data
+inserted_data = rag_wf.run(input_data=input_data)
+
+/var/folders/09/d0hx80nj35sb5hxb5cpc1q180000gn/T/ipykernel_31319/3145804345.py:4: ResourceWarning: unclosed file <_io.BufferedReader name='./pdf_files/WhatisMilvus.pdf'>
+ BytesIO(open(path, "rb").read()) for path in file_paths
+ResourceWarning: Enable tracemalloc to get the object allocation traceback
+2024-11-19 22:14:09 - INFO - Workflow 87878444-6a3d-43f3-ae32-0127564a959f: execution started.
+2024-11-19 22:14:09 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution started.
+2024-11-19 22:14:09 - INFO - Node PyPDF File Converter - 6eb42b1f-7637-407b-a3ac-4167bcf3b5c4: execution started.
+2024-11-19 22:14:09 - INFO - Node PyPDF File Converter - 6eb42b1f-7637-407b-a3ac-4167bcf3b5c4: execution succeeded in 58ms.
+2024-11-19 22:14:09 - INFO - Node DocumentSplitter - 5baed580-6de0-4dcd-bace-d7d947ab6c7f: execution started.
+/Users/jinhonglin/anaconda3/envs/myenv/lib/python3.11/site-packages/websockets/legacy/__init__.py:6: DeprecationWarning: websockets.legacy is deprecated; see https://websockets.readthedocs.io/en/stable/howto/upgrade.html for upgrade instructions
+ warnings.warn( # deprecated in 14.0 - 2024-11-09
+/Users/jinhonglin/anaconda3/envs/myenv/lib/python3.11/site-packages/pydantic/fields.py:804: PydanticDeprecatedSince20: Using extra keyword arguments on `Field` is deprecated and will be removed. Use `json_schema_extra` instead. (Extra keys: 'is_accessible_to_agent'). Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/
+ warn(
+2024-11-19 22:14:09 - INFO - Node DocumentSplitter - 5baed580-6de0-4dcd-bace-d7d947ab6c7f: execution succeeded in 104ms.
+2024-11-19 22:14:09 - INFO - Node OpenAIDocumentEmbedder - 91928f67-a00f-48f6-a864-f6e21672ec7e: execution started.
+2024-11-19 22:14:09 - INFO - Node OpenAIDocumentEmbedder - d30a4cdc-0fab-4aff-b2e5-6161a62cb6fd: execution started.
+2024-11-19 22:14:10 - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
+2024-11-19 22:14:10 - INFO - Node OpenAIDocumentEmbedder - d30a4cdc-0fab-4aff-b2e5-6161a62cb6fd: execution succeeded in 724ms.
+2024-11-19 22:14:10 - INFO - Node MilvusDocumentWriter - dddab4cc-1dae-4e7e-9101-1ec353f530da: execution started.
+2024-11-19 22:14:10 - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
+2024-11-19 22:14:10 - INFO - Node MilvusDocumentWriter - dddab4cc-1dae-4e7e-9101-1ec353f530da: execution succeeded in 66ms.
+2024-11-19 22:14:10 - INFO - Node OpenAIDocumentEmbedder - 91928f67-a00f-48f6-a864-f6e21672ec7e: execution succeeded in 961ms.
+2024-11-19 22:14:10 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution succeeded in 1.3s.
+2024-11-19 22:14:10 - INFO - Workflow 87878444-6a3d-43f3-ae32-0127564a959f: execution succeeded in 1.3s.
+
+Attraverso questo flusso di lavoro, abbiamo implementato con successo una pipeline di indicizzazione di documenti utilizzando Milvus come database vettoriale e il modello di embedding di OpenAI per la rappresentazione semantica. Questa configurazione consente un recupero rapido e accurato basato su vettori, costituendo la base per i flussi di lavoro RAG come la ricerca semantica, il recupero di documenti e le interazioni contestuali guidate dall'intelligenza artificiale.
+Grazie alle capacità di archiviazione scalabili di Milvus e all'orchestrazione di Dynamiq, questa soluzione è pronta sia per la prototipazione che per le implementazioni di produzione su larga scala. È ora possibile estendere questa pipeline per includere attività aggiuntive come la risposta a domande basate sul reperimento o la generazione di contenuti guidati dall'intelligenza artificiale.
+Flusso di recupero dei documenti RAG
In questa esercitazione, implementiamo un flusso di recupero di documenti RAG (Retrieval-Augmented Generation). Questo flusso di lavoro prende una query dell'utente, genera un embedding vettoriale per essa, recupera i documenti più rilevanti da un database vettoriale Milvus e utilizza un modello linguistico di grandi dimensioni (LLM) per generare una risposta dettagliata e consapevole del contesto in base ai documenti recuperati.
+Seguendo questo flusso di lavoro, si crea una soluzione end-to-end per la ricerca semantica e la risposta alle domande, combinando la potenza del recupero di documenti basato su vettori con le capacità dei modelli linguistici avanzati di OpenAI. Questo approccio consente di rispondere in modo efficiente e intelligente alle domande degli utenti, sfruttando le conoscenze memorizzate nel database dei documenti.
+Importazione delle librerie necessarie e inizializzazione del flusso di lavoro
from dynamiq import Workflow
+from dynamiq.connections import (
+ OpenAI as OpenAIConnection,
+ Milvus as MilvusConnection,
+ MilvusDeploymentType,
+)
+from dynamiq.nodes.embedders import OpenAITextEmbedder
+from dynamiq.nodes.retrievers import MilvusDocumentRetriever
+from dynamiq.nodes.llms import OpenAI
+from dynamiq.prompts import Message, Prompt
+
+# Initialize the workflow
+retrieval_wf = Workflow()
+
+Definire la connessione e l'incorporatore di testo OpenAI
# Establish OpenAI connection
+openai_connection = OpenAIConnection(api_key=os.environ["OPENAI_API_KEY"])
+
+# Define the text embedder node
+embedder = OpenAITextEmbedder(
+ connection=openai_connection,
+ model="text-embedding-3-small",
+)
+
+# Add the embedder node to the workflow
+embedder_added = retrieval_wf.flow.add_nodes(embedder)
+
+Definire il recuperatore di documenti Milvus
document_retriever = (
+ MilvusDocumentRetriever(
+ connection=MilvusConnection(
+ deployment_type=MilvusDeploymentType.FILE, uri="./milvus.db"
+ ),
+ index_name="my_milvus_collection",
+ dimension=1536,
+ top_k=5,
+ )
+ .inputs(embedding=embedder.outputs.embedding) # Connect to embedder output
+ .depends_on(embedder) # Dependency on the embedder node
+)
+
+# Add the retriever node to the workflow
+milvus_retriever_added = retrieval_wf.flow.add_nodes(document_retriever)
+
+2024-11-19 22:14:19 - WARNING - Environment variable 'MILVUS_API_TOKEN' not found
+2024-11-19 22:14:19 - INFO - Pass in the local path ./milvus.db, and run it using milvus-lite
+2024-11-19 22:14:19 - DEBUG - Created new connection using: 98d1132773af4298a894ad5925845fd2
+2024-11-19 22:14:19 - INFO - Collection my_milvus_collection already exists. Skipping creation.
+
+Definire il modello di prompt
# Define the prompt template for the LLM
+prompt_template = """
+Please answer the question based on the provided context.
+
+Question: {{ query }}
+
+Context:
+{% for document in documents %}
+- {{ document.content }}
+{% endfor %}
+"""
+
+# Create the prompt object
+prompt = Prompt(messages=[Message(content=prompt_template, role="user")])
+
+Definire il generatore di risposte
answer_generator = (
+ OpenAI(
+ connection=openai_connection,
+ model="gpt-4o",
+ prompt=prompt,
+ )
+ .inputs(
+ documents=document_retriever.outputs.documents,
+ query=embedder.outputs.query,
+ )
+ .depends_on(
+ [document_retriever, embedder]
+ ) # Dependencies on retriever and embedder
+)
+
+# Add the answer generator node to the workflow
+answer_generator_added = retrieval_wf.flow.add_nodes(answer_generator)
+
+Eseguire il flusso di lavoro
# Run the workflow with a sample query
+sample_query = "What is the Advanced Search Algorithms in Milvus?"
+
+result = retrieval_wf.run(input_data={"query": sample_query})
+
+answer = result.output.get(answer_generator.id).get("output", {}).get("content")
+print(answer)
+
+2024-11-19 22:14:22 - INFO - Workflow f4a073fb-dfb6-499c-8cac-5710a7ad6d47: execution started.
+2024-11-19 22:14:22 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution started.
+2024-11-19 22:14:22 - INFO - Node OpenAITextEmbedder - 47afb0bc-cf96-429d-b58f-11b6c935fec3: execution started.
+2024-11-19 22:14:23 - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
+2024-11-19 22:14:23 - INFO - Node OpenAITextEmbedder - 47afb0bc-cf96-429d-b58f-11b6c935fec3: execution succeeded in 474ms.
+2024-11-19 22:14:23 - INFO - Node MilvusDocumentRetriever - 51c8311b-4837-411f-ba42-21e28239a2ee: execution started.
+2024-11-19 22:14:23 - INFO - Node MilvusDocumentRetriever - 51c8311b-4837-411f-ba42-21e28239a2ee: execution succeeded in 23ms.
+2024-11-19 22:14:23 - INFO - Node LLM - ac722325-bece-453f-a2ed-135b0749ee7a: execution started.
+2024-11-19 22:14:24 - INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
+2024-11-19 22:14:24 - INFO - Node LLM - ac722325-bece-453f-a2ed-135b0749ee7a: execution succeeded in 1.8s.
+2024-11-19 22:14:25 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution succeeded in 2.4s.
+2024-11-19 22:14:25 - INFO - Workflow f4a073fb-dfb6-499c-8cac-5710a7ad6d47: execution succeeded in 2.4s.
+
+
+The advanced search algorithms in Milvus include a variety of in-memory and on-disk indexing/search algorithms such as IVF (Inverted File), HNSW (Hierarchical Navigable Small World), and DiskANN. These algorithms have been deeply optimized to enhance performance, delivering 30%-70% better performance compared to popular implementations like FAISS and HNSWLib. These optimizations are part of Milvus's design to ensure high efficiency and scalability in handling vector data.
+
diff --git a/localization/v2.4.x/site/it/menuStructure/it.json b/localization/v2.4.x/site/it/menuStructure/it.json
index 08cc06008..100fb3156 100644
--- a/localization/v2.4.x/site/it/menuStructure/it.json
+++ b/localization/v2.4.x/site/it/menuStructure/it.json
@@ -1344,6 +1344,24 @@
"id": "build_RAG_with_milvus_and_siliconflow.md",
"order": 4,
"children": []
+ },
+ {
+ "label": "SambaNova",
+ "id": "use_milvus_with_sambanova.md",
+ "order": 5,
+ "children": []
+ },
+ {
+ "label": "Gemelli",
+ "id": "build_RAG_with_milvus_and_gemini.md",
+ "order": 6,
+ "children": []
+ },
+ {
+ "label": "Ollama",
+ "id": "build_RAG_with_milvus_and_ollama.md",
+ "order": 7,
+ "children": []
}
]
},
@@ -1433,15 +1451,15 @@
"children": []
},
{
- "label": "SambaNova",
- "id": "use_milvus_with_sambanova.md",
+ "label": "PrivatoGPT",
+ "id": "use_milvus_in_private_gpt.md",
"order": 10,
"children": []
},
{
- "label": "PrivatoGPT",
- "id": "use_milvus_in_private_gpt.md",
- "order": 8,
+ "label": "Dynamiq",
+ "id": "milvus_rag_with_dynamiq.md",
+ "order": 11,
"children": []
}
]
@@ -1493,6 +1511,12 @@
"id": "knowledge_table_with_milvus.md",
"order": 2,
"children": []
+ },
+ {
+ "label": "Cognee",
+ "id": "build_RAG_with_milvus_and_cognee.md",
+ "order": 3,
+ "children": []
}
]
},
@@ -1664,7 +1688,7 @@
"children": []
},
{
- "label": "Ricerca a imbuto con le matrici Matryoshka",
+ "label": "Ricerca a imbuto con matrioske embeddings",
"id": "funnel_search_with_matryoshka.md",
"order": 10,
"children": []
@@ -1693,7 +1717,7 @@
"children": []
},
{
- "label": "Ricerca di somiglianza audio",
+ "label": "Ricerca della somiglianza audio",
"id": "audio_similarity_search.md",
"order": 8,
"children": []
diff --git a/localization/v2.4.x/site/ja/integrations/build_RAG_with_milvus_and_cognee.json b/localization/v2.4.x/site/ja/integrations/build_RAG_with_milvus_and_cognee.json
new file mode 100644
index 000000000..cc3a539a4
--- /dev/null
+++ b/localization/v2.4.x/site/ja/integrations/build_RAG_with_milvus_and_cognee.json
@@ -0,0 +1 @@
+{"codeList":["$ pip install pymilvus git+https://github.com/topoteretes/cognee.git\n","import os\n\nimport cognee\n\ncognee.config.set_llm_api_key(\"YOUR_OPENAI_API_KEY\")\n\n\nos.environ[\"VECTOR_DB_PROVIDER\"] = \"milvus\"\nos.environ[\"VECTOR_DB_URL\"] = \"./milvus.db\"\n","$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip\n$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs\n","from glob import glob\n\ntext_lines = []\n\nfor file_path in glob(\"milvus_docs/en/faq/*.md\", recursive=True):\n with open(file_path, \"r\") as file:\n file_text = file.read()\n\n text_lines += file_text.split(\"# \")\n","await cognee.prune.prune_data()\nawait cognee.prune.prune_system(metadata=True)\n","await cognee.add(data=text_lines, dataset_name=\"milvus_faq\")\nawait cognee.cognify()\n\n# [DocumentChunk(id=UUID('6889e7ef-3670-555c-bb16-3eb50d1d30b0'), updated_at=datetime.datetime(2024, 12, 4, 6, 29, 46, 472907, tzinfo=datetime.timezone.utc), text='Does the query perform in memory? What are incremental data and historical data?\\n\\nYes. When ...\n# ...\n","from cognee.api.v1.search import SearchType\n\nquery_text = \"How is data stored in milvus?\"\nsearch_results = await cognee.search(SearchType.SUMMARIES, query_text=query_text)\n\nprint(search_results[0])\n","from cognee.api.v1.search import SearchType\n\nquery_text = \"How is data stored in milvus?\"\nsearch_results = await cognee.search(SearchType.CHUNKS, query_text=query_text)\n","def format_and_print(data):\n print(\"ID:\", data[\"id\"])\n print(\"\\nText:\\n\")\n paragraphs = data[\"text\"].split(\"\\n\\n\")\n for paragraph in paragraphs:\n print(paragraph.strip())\n print()\n\n\nformat_and_print(search_results[0])\n","await cognee.prune.prune_data()\nawait cognee.prune.prune_system(metadata=True)\n","# We only use one line of text as the dataset, which simplifies the output later\ntext = \"\"\"\n Natural language processing (NLP) is an interdisciplinary\n subfield of computer science and information retrieval.\n \"\"\"\n\nawait cognee.add(text)\nawait cognee.cognify()\n","query_text = \"Tell me about NLP\"\nsearch_results = await cognee.search(SearchType.INSIGHTS, query_text=query_text)\n\nfor result_text in search_results:\n print(result_text)\n\n# Example output:\n# ({'id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'natural language processing', 'description': 'An interdisciplinary subfield of computer science and information retrieval.'}, {'relationship_name': 'is_a_subfield_of', 'source_node_id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'target_node_id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 15, 473137, tzinfo=datetime.timezone.utc)}, {'id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'computer science', 'description': 'The study of computation and information processing.'})\n# (...)\n#\n# It represents nodes and relationships in the knowledge graph:\n# - The first element is the source node (e.g., 'natural language processing').\n# - The second element is the relationship between nodes (e.g., 'is_a_subfield_of').\n# - The third element is the target node (e.g., 'computer science').\n"],"headingContent":"","anchorList":[{"label":"RAGの構築","href":"Build-RAG","type":2,"isActive":false}]}
\ No newline at end of file
diff --git a/localization/v2.4.x/site/ja/integrations/build_RAG_with_milvus_and_cognee.md b/localization/v2.4.x/site/ja/integrations/build_RAG_with_milvus_and_cognee.md
new file mode 100644
index 000000000..38e8bbcfe
--- /dev/null
+++ b/localization/v2.4.x/site/ja/integrations/build_RAG_with_milvus_and_cognee.md
@@ -0,0 +1,159 @@
+---
+id: build_RAG_with_milvus_and_cognee.md
+summary: >-
+ このチュートリアルでは、MilvusとCogneeを使ってRAG（Retrieval-Augmented
+ Generation）パイプラインを構築する方法を紹介します。
+title: MilvusとCogneeでRAGを構築する
+---
+
+MilvusとCogneeでRAGを構築する
Cogneeは、スケーラブルでモジュール化されたECL（Extract、Cognify、Load）パイプラインによりAIアプリケーション開発を合理化する開発者ファーストのプラットフォームです。CogneeはMilvusとシームレスに統合することで、会話、ドキュメント、トランスクリプションの効率的な接続と検索を可能にし、幻覚を減らし、運用コストを最適化します。
+Milvusのようなベクトルストア、グラフデータベース、LLMを強力にサポートするCogneeは、検索支援世代（RAG）システムを構築するための柔軟でカスタマイズ可能なフレームワークを提供します。そのプロダクション・レディなアーキテクチャは、AIを活用したアプリケーションの精度と効率の向上を保証します。
+このチュートリアルでは、MilvusとCogneeを使ってRAG（Retrieval-Augmented Generation）パイプラインを構築する方法を紹介します。
+$ pip install pymilvus git+https://github.com/topoteretes/cognee.git
+
+
+Google Colabを使用している場合、インストールしたばかりの依存関係を有効にするために、ランタイムを再起動する必要があるかもしれません（画面上部の "Runtime "メニューをクリックし、ドロップダウンメニューから "Restart session "を選択してください）。
+
+この例では、デフォルトでOpenAIをLLMとして使用しています。apiキーを用意し、configset_llm_api_key() 。
+Milvusをベクターデータベースとして設定するには、VECTOR_DB_PROVIDER をmilvus に設定し、VECTOR_DB_URL とVECTOR_DB_KEY を指定します。このデモではMilvus Liteを使ってデータを保存しているので、VECTOR_DB_URL のみを指定すればよい。
+import os
+
+import cognee
+
+cognee.config.set_llm_api_key("YOUR_OPENAI_API_KEY")
+
+
+os.environ["VECTOR_DB_PROVIDER"] = "milvus"
+os.environ["VECTOR_DB_URL"] = "./milvus.db"
+
+
+環境変数としてはVECTOR_DB_URL とVECTOR_DB_KEY を指定する：
+
+VECTOR_DB_URL をローカルファイル、例えば./milvus.db に設定するのが最も便利な方法である。- データ規模が大きい場合は、dockerやkubernetes上に、よりパフォーマンスの高いMilvusサーバを構築することができます。このセットアップでは、サーバの uri、例えば
http://localhost:19530 をVECTOR_DB_URL として使用してください。
+- MilvusのフルマネージドクラウドサービスであるZilliz Cloudを利用する場合は、Zilliz CloudのPublic EndpointとApi keyに対応する
VECTOR_DB_URL とVECTOR_DB_KEY を調整してください。
+
+
+データの準備
Milvusドキュメント2.4.xのFAQページをRAGのプライベートナレッジとして使用する。
+zipファイルをダウンロードし、milvus_docs フォルダにドキュメントを展開する。
+$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
+
+フォルダmilvus_docs/en/faq からすべてのマークダウン・ファイルをロードする。各ドキュメントについて、単に "# "を使ってファイル内のコンテンツを区切るだけで、マークダウン・ファイルの各主要部分のコンテンツを大まかに区切ることができる。
+from glob import glob
+
+text_lines = []
+
+for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
+ with open(file_path, "r") as file:
+ file_text = file.read()
+
+ text_lines += file_text.split("# ")
+
+RAGの構築
Cogneeデータのリセット
await cognee.prune.prune_data()
+await cognee.prune.prune_system(metadata=True)
+
+データセットを追加し、ナレッジグラフに加工することができる。
+データの追加とコグニファイ
await cognee.add(data=text_lines, dataset_name="milvus_faq")
+await cognee.cognify()
+
+# [DocumentChunk(id=UUID('6889e7ef-3670-555c-bb16-3eb50d1d30b0'), updated_at=datetime.datetime(2024, 12, 4, 6, 29, 46, 472907, tzinfo=datetime.timezone.utc), text='Does the query perform in memory? What are incremental data and historical data?\n\nYes. When ...
+# ...
+
+add メソッドはデータセット（Milvus FAQs）をCogneeにロードし、cognify メソッドはデータを処理してエンティティ、リレーションシップ、サマリーを抽出し、ナレッジグラフを構築します。
+要約のクエリ
データが処理されたので、ナレッジグラフをクエリしてみましょう。
+from cognee.api.v1.search import SearchType
+
+query_text = "How is data stored in milvus?"
+search_results = await cognee.search(SearchType.SUMMARIES, query_text=query_text)
+
+print(search_results[0])
+
+{'id': 'de5c6713-e079-5d0b-b11d-e9bacd1e0d73', 'text': 'Milvus stores two data types: inserted data and metadata.'}
+
+このクエリは、クエリテキストに関連する要約をナレッジグラフから検索し、最も関連する候補を表示します。
+チャンクのクエリ
要約はハイレベルな洞察を提供しますが、より詳細な詳細については、処理されたデータセットから直接データの特定のチャンクをクエリできます。これらのチャンクは、ナレッジグラフ作成時に追加・分析された元のデータから派生したものです。
+from cognee.api.v1.search import SearchType
+
+query_text = "How is data stored in milvus?"
+search_results = await cognee.search(SearchType.CHUNKS, query_text=query_text)
+
+読みやすいように整形して表示しましょう！
+def format_and_print(data):
+ print("ID:", data["id"])
+ print("\nText:\n")
+ paragraphs = data["text"].split("\n\n")
+ for paragraph in paragraphs:
+ print(paragraph.strip())
+ print()
+
+
+format_and_print(search_results[0])
+
+ID: 4be01c4b-9ee5-541c-9b85-297883934ab3
+
+Text:
+
+Where does Milvus store data?
+
+Milvus deals with two types of data, inserted data and metadata.
+
+Inserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).
+
+Metadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.
+
+###
+
+これまでのステップでは、MilvusのFAQデータセットに対して、要約と特定のデータチャンクの両方を照会しました。これは詳細な洞察と粒度の細かい情報を提供しましたが、データセットが大きかったため、ナレッジグラフ内の依存関係を明確に視覚化することが困難でした。
+この問題に対処するため、Cognee環境をリセットし、より小さく、より焦点を絞ったデータセットで作業する。これにより、コグニファイ処理中に抽出された関係や依存関係をより明確に示すことができるようになります。データを単純化することで、Cogneeがどのようにナレッジグラフ内の情報を整理し、構造化するかを明確に見ることができます。
+Cogneeのリセット
await cognee.prune.prune_data()
+await cognee.prune.prune_system(metadata=True)
+
+フォーカスされたデータセットの追加
ここでは、1行のテキストのみの小さなデータセットが追加され、ナレッジグラフがフォーカスされ解釈しやすくなるように処理されます。
+# We only use one line of text as the dataset, which simplifies the output later
+text = """
+ Natural language processing (NLP) is an interdisciplinary
+ subfield of computer science and information retrieval.
+ """
+
+await cognee.add(text)
+await cognee.cognify()
+
+インサイトのクエリ
この小さなデータセットに焦点を当てることで、ナレッジグラフ内の関係と構造を明確に分析できるようになりました。
+query_text = "Tell me about NLP"
+search_results = await cognee.search(SearchType.INSIGHTS, query_text=query_text)
+
+for result_text in search_results:
+ print(result_text)
+
+# Example output:
+# ({'id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'natural language processing', 'description': 'An interdisciplinary subfield of computer science and information retrieval.'}, {'relationship_name': 'is_a_subfield_of', 'source_node_id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'target_node_id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 15, 473137, tzinfo=datetime.timezone.utc)}, {'id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'computer science', 'description': 'The study of computation and information processing.'})
+# (...)
+#
+# It represents nodes and relationships in the knowledge graph:
+# - The first element is the source node (e.g., 'natural language processing').
+# - The second element is the relationship between nodes (e.g., 'is_a_subfield_of').
+# - The third element is the target node (e.g., 'computer science').
+
+この出力はナレッジグラフのクエリの結果を表しており、処理されたデータセットから抽出されたエンティティ（ノード）とその関係（エッジ）を示しています。各タプルには、一意のID、説明、タイムスタンプなどのメタデータとともに、ソース・エンティティ、関係タイプ、ターゲット・エンティティが含まれます。グラフは、主要な概念とその意味的なつながりを強調し、データセットの構造的な理解を提供します。
+Milvusを使ったcogneeの基本的な使い方はご理解いただけたと思います。より高度な使い方を知りたい方はcogneeの公式ページをご覧ください。
diff --git a/localization/v2.4.x/site/ja/integrations/build_RAG_with_milvus_and_gemini.json b/localization/v2.4.x/site/ja/integrations/build_RAG_with_milvus_and_gemini.json
new file mode 100644
index 000000000..6f15f8310
--- /dev/null
+++ b/localization/v2.4.x/site/ja/integrations/build_RAG_with_milvus_and_gemini.json
@@ -0,0 +1 @@
+{"codeList":["$ pip install --upgrade pymilvus google-generativeai requests tqdm\n","import os\n\nos.environ[\"GEMINI_API_KEY\"] = \"***********\"\n","$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip\n$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs\n","from glob import glob\n\ntext_lines = []\n\nfor file_path in glob(\"milvus_docs/en/faq/*.md\", recursive=True):\n with open(file_path, \"r\") as file:\n file_text = file.read()\n\n text_lines += file_text.split(\"# \")\n","import google.generativeai as genai\n\ngenai.configure(api_key=os.environ[\"GEMINI_API_KEY\"])\n\ngemini_model = genai.GenerativeModel(\"gemini-1.5-flash\")\n\nresponse = gemini_model.generate_content(\"who are you\")\nprint(response.text)\n","test_embeddings = genai.embed_content(\n model=\"models/text-embedding-004\", content=[\"This is a test1\", \"This is a test2\"]\n)[\"embedding\"]\n\nembedding_dim = len(test_embeddings[0])\nprint(embedding_dim)\nprint(test_embeddings[0][:10])\n","from pymilvus import MilvusClient\n\nmilvus_client = MilvusClient(uri=\"./milvus_demo.db\")\n\ncollection_name = \"my_rag_collection\"\n","if milvus_client.has_collection(collection_name):\n milvus_client.drop_collection(collection_name)\n","milvus_client.create_collection(\n collection_name=collection_name,\n dimension=embedding_dim,\n metric_type=\"IP\", # Inner product distance\n consistency_level=\"Strong\", # Strong consistency level\n)\n","from tqdm import tqdm\n\ndata = []\n\ndoc_embeddings = genai.embed_content(\n model=\"models/text-embedding-004\", content=text_lines\n)[\"embedding\"]\n\nfor i, line in enumerate(tqdm(text_lines, desc=\"Creating embeddings\")):\n data.append({\"id\": i, \"vector\": doc_embeddings[i], \"text\": line})\n\nmilvus_client.insert(collection_name=collection_name, data=data)\n","question = \"How is data stored in milvus?\"\n","question_embedding = genai.embed_content(\n model=\"models/text-embedding-004\", content=question\n)[\"embedding\"]\n\nsearch_res = milvus_client.search(\n collection_name=collection_name,\n data=[question_embedding],\n limit=3, # Return top 3 results\n search_params={\"metric_type\": \"IP\", \"params\": {}}, # Inner product distance\n output_fields=[\"text\"], # Return the text field\n)\n","import json\n\nretrieved_lines_with_distances = [\n (res[\"entity\"][\"text\"], res[\"distance\"]) for res in search_res[0]\n]\nprint(json.dumps(retrieved_lines_with_distances, indent=4))\n","context = \"\\n\".join(\n [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]\n)\n","SYSTEM_PROMPT = \"\"\"\nHuman: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.\n\"\"\"\nUSER_PROMPT = f\"\"\"\nUse the following pieces of information enclosed in tags to provide an answer to the question enclosed in tags.\n\n{context}\n \n\n{question}\n \n\"\"\"\n","gemini_model = genai.GenerativeModel(\n \"gemini-1.5-flash\", system_instruction=SYSTEM_PROMPT\n)\nresponse = gemini_model.generate_content(USER_PROMPT)\nprint(response.text)\n"],"headingContent":"Build RAG with Milvus and Gemini","anchorList":[{"label":"MilvusとGeminiでRAGを構築する","href":"Build-RAG-with-Milvus-and-Gemini","type":1,"isActive":false},{"label":"準備","href":"Preparation","type":2,"isActive":false},{"label":"Milvusにデータをロードする。","href":"Load-data-into-Milvus","type":2,"isActive":false},{"label":"RAGの構築","href":"Build-RAG","type":2,"isActive":false}]}
\ No newline at end of file
diff --git a/localization/v2.4.x/site/ja/integrations/build_RAG_with_milvus_and_gemini.md b/localization/v2.4.x/site/ja/integrations/build_RAG_with_milvus_and_gemini.md
new file mode 100644
index 000000000..3df3ddc2d
--- /dev/null
+++ b/localization/v2.4.x/site/ja/integrations/build_RAG_with_milvus_and_gemini.md
@@ -0,0 +1,244 @@
+---
+id: build_RAG_with_milvus_and_gemini.md
+summary: >-
+ このチュートリアルでは、MilvusとGeminiを使ってRAG（Retrieval-Augmented
+ Generation）パイプラインを構築する方法を紹介する。与えられたクエリに基づいてテキストを生成するためにGeminiモデルを使用します。また、Milvusを使用して、生成されたテキストの保存と取得を行う。
+title: MilvusとGeminiでRAGを構築する
+---
+
+MilvusとGeminiでRAGを構築する
Gemini APIと Google AI Studioを使えば、Googleの最新モデルを使って作業を開始し、あなたのアイデアをスケールの大きなアプリケーションにすることができます。Geminiは、テキスト生成、文書処理、視覚、音声分析などのタスクのために、Gemini-1.5-Flash 、Gemini-1.5-Flash-8B 、Gemini-1.5-Pro のような強力な言語モデルへのアクセスを提供します。APIを使用することで、何百万ものトークンを含む長いコンテキストを入力したり、特定のタスクのためにモデルを微調整したり、JSONのような構造化された出力を生成したり、セマンティック検索やコード実行のような機能を活用したりすることができる。
+このチュートリアルでは、MilvusとGeminiを使ってRAG（Retrieval-Augmented Generation）パイプラインを構築する方法を紹介する。与えられたクエリに基づいてテキストを生成するためにGeminiモデルを使用します。また、Milvusを使用して、生成されたテキストの保存と検索を行う。
+準備
依存関係と環境
$ pip install --upgrade pymilvus google-generativeai requests tqdm
+
+
+Google Colabを使用している場合、インストールしたばかりの依存関係を有効にするために、ランタイムを再起動する必要があるかもしれません（画面上部の "Runtime "メニューをクリックし、ドロップダウンメニューから "Restart session "を選択してください）。
+
+まずGoogle AI Studioプラットフォームにログインし、api key GEMINI_API_KEY を環境変数として用意します。
+import os
+
+os.environ["GEMINI_API_KEY"] = "***********"
+
+データの準備
Milvusドキュメント2.4.xのFAQページをRAGのプライベートナレッジとして使用する。
+zipファイルをダウンロードし、milvus_docs フォルダにドキュメントを展開する。
+$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
+
+フォルダmilvus_docs/en/faq からすべてのマークダウン・ファイルをロードする。各ドキュメントについて、単に "# "を使ってファイル内のコンテンツを区切るだけで、マークダウン・ファイルの各主要部分のコンテンツを大まかに区切ることができる。
+from glob import glob
+
+text_lines = []
+
+for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
+ with open(file_path, "r") as file:
+ file_text = file.read()
+
+ text_lines += file_text.split("# ")
+
+LLMと埋め込みモデルの準備
LLMとしてgemini-1.5-flash 、埋め込みモデルとしてtext-embedding-004 。
+LLMからテスト応答を生成してみましょう：
+import google.generativeai as genai
+
+genai.configure(api_key=os.environ["GEMINI_API_KEY"])
+
+gemini_model = genai.GenerativeModel("gemini-1.5-flash")
+
+response = gemini_model.generate_content("who are you")
+print(response.text)
+
+I am a large language model, trained by Google. I am an AI and don't have a personal identity or consciousness. My purpose is to process information and respond to a wide range of prompts and questions in a helpful and informative way.
+
+テスト埋め込みを生成し、その次元と最初の数要素を表示します。
+test_embeddings = genai.embed_content(
+ model="models/text-embedding-004", content=["This is a test1", "This is a test2"]
+)["embedding"]
+
+embedding_dim = len(test_embeddings[0])
+print(embedding_dim)
+print(test_embeddings[0][:10])
+
+768
+[0.013588584, -0.004361838, -0.08481652, -0.039724775, 0.04723794, -0.0051557426, 0.026071774, 0.045514572, -0.016867816, 0.039378334]
+
+Milvusにデータをロードする。
コレクションの作成
from pymilvus import MilvusClient
+
+milvus_client = MilvusClient(uri="./milvus_demo.db")
+
+collection_name = "my_rag_collection"
+
+
+MilvusClient ：
+
+uri をローカルファイル、例えば./milvus.db とするのが最も便利な方法です。- データ規模が大きい場合は、dockerやkubernetes上に、よりパフォーマンスの高いMilvusサーバを構築することができます。このセットアップでは、サーバの uri、例えば
http://localhost:19530 をuri として使用してください。
+- MilvusのフルマネージドクラウドサービスであるZilliz Cloudを利用する場合は、Zilliz CloudのPublic EndpointとApi keyに対応する
uri とtoken を調整してください。
+
+
+コレクションが既に存在するか確認し、存在する場合は削除します。
+if milvus_client.has_collection(collection_name):
+ milvus_client.drop_collection(collection_name)
+
+指定したパラメータで新しいコレクションを作成します。
+フィールド情報を指定しない場合、Milvusは自動的にプライマリキー用のデフォルトid フィールドと、ベクトルデータを格納するためのvector フィールドを作成します。予約されたJSONフィールドは、スキーマで定義されていないフィールドとその値を格納するために使用されます。
+milvus_client.create_collection(
+ collection_name=collection_name,
+ dimension=embedding_dim,
+ metric_type="IP", # Inner product distance
+ consistency_level="Strong", # Strong consistency level
+)
+
+データの挿入
テキスト行を繰り返し、エンベッディングを作成し、milvusにデータを挿入します。
+ここに新しいフィールドtext 、コレクションスキーマで定義されていないフィールドです。これは予約されたJSONダイナミックフィールドに自動的に追加され、高レベルでは通常のフィールドとして扱うことができる。
+from tqdm import tqdm
+
+data = []
+
+doc_embeddings = genai.embed_content(
+ model="models/text-embedding-004", content=text_lines
+)["embedding"]
+
+for i, line in enumerate(tqdm(text_lines, desc="Creating embeddings")):
+ data.append({"id": i, "vector": doc_embeddings[i], "text": line})
+
+milvus_client.insert(collection_name=collection_name, data=data)
+
+Creating embeddings: 100%|██████████| 72/72 [00:00<00:00, 468201.38it/s]
+
+
+
+
+
+{'insert_count': 72, 'ids': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71], 'cost': 0}
+
+RAGの構築
クエリのデータを取得する
Milvusに関するよくある質問を指定してみましょう。
+question = "How is data stored in milvus?"
+
+コレクションで質問を検索し、セマンティックトップ3マッチを取得します。
+question_embedding = genai.embed_content(
+ model="models/text-embedding-004", content=question
+)["embedding"]
+
+search_res = milvus_client.search(
+ collection_name=collection_name,
+ data=[question_embedding],
+ limit=3, # Return top 3 results
+ search_params={"metric_type": "IP", "params": {}}, # Inner product distance
+ output_fields=["text"], # Return the text field
+)
+
+クエリの検索結果を見てみましょう。
+import json
+
+retrieved_lines_with_distances = [
+ (res["entity"]["text"], res["distance"]) for res in search_res[0]
+]
+print(json.dumps(retrieved_lines_with_distances, indent=4))
+
+[
+ [
+ " Where does Milvus store data?\n\nMilvus deals with two types of data, inserted data and metadata. \n\nInserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).\n\nMetadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.\n\n###",
+ 0.8048275113105774
+ ],
+ [
+ "Does the query perform in memory? What are incremental data and historical data?\n\nYes. When a query request comes, Milvus searches both incremental data and historical data by loading them into memory. Incremental data are in the growing segments, which are buffered in memory before they reach the threshold to be persisted in storage engine, while historical data are from the sealed segments that are stored in the object storage. Incremental data and historical data together constitute the whole dataset to search.\n\n###",
+ 0.7574886679649353
+ ],
+ [
+ "What is the maximum dataset size Milvus can handle?\n\n \nTheoretically, the maximum dataset size Milvus can handle is determined by the hardware it is run on, specifically system memory and storage:\n\n- Milvus loads all specified collections and partitions into memory before running queries. Therefore, memory size determines the maximum amount of data Milvus can query.\n- When new entities and and collection-related schema (currently only MinIO is supported for data persistence) are added to Milvus, system storage determines the maximum allowable size of inserted data.\n\n###",
+ 0.7453608512878418
+ ]
+]
+
+LLMを使ってRAGレスポンスを取得する
検索されたドキュメントを文字列フォーマットに変換する。
+context = "\n".join(
+ [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]
+)
+
+ラネージ・モデルのシステム・プロンプトとユーザー・プロンプトを定義する。このプロンプトをmilvusから検索された文書で組み立てる。
+SYSTEM_PROMPT = """
+Human: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.
+"""
+USER_PROMPT = f"""
+Use the following pieces of information enclosed in <context> tags to provide an answer to the question enclosed in <question> tags.
+<context>
+{context}
+</context>
+<question>
+{question}
+</question>
+"""
+
+Geminiを使用してプロンプトに基づいた応答を生成する。
+gemini_model = genai.GenerativeModel(
+ "gemini-1.5-flash", system_instruction=SYSTEM_PROMPT
+)
+response = gemini_model.generate_content(USER_PROMPT)
+print(response.text)
+
+Milvus stores data in two ways: Inserted data (vector data, scalar data, and collection-specific schema) is stored as an incremental log in persistent storage using object storage backends such as MinIO, AWS S3, Google Cloud Storage, Azure Blob Storage, Alibaba Cloud OSS, and Tencent Cloud Object Storage. Metadata, generated by each Milvus module, is stored in etcd.
+
+素晴らしい！MilvusとGeminiを使ってRAGパイプラインを構築することに成功した。
diff --git a/localization/v2.4.x/site/ja/integrations/build_RAG_with_milvus_and_ollama.json b/localization/v2.4.x/site/ja/integrations/build_RAG_with_milvus_and_ollama.json
new file mode 100644
index 000000000..3b0ff69b6
--- /dev/null
+++ b/localization/v2.4.x/site/ja/integrations/build_RAG_with_milvus_and_ollama.json
@@ -0,0 +1 @@
+{"codeList":["$ pip install pymilvus ollama\n","$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip\n$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs\n","from glob import glob\n\ntext_lines = []\n\nfor file_path in glob(\"milvus_docs/en/faq/*.md\", recursive=True):\n with open(file_path, \"r\") as file:\n file_text = file.read()\n\n text_lines += file_text.split(\"# \")\n","! ollama pull mxbai-embed-large\n","! ollama pull llama3.2\n","import ollama\n\n\ndef emb_text(text):\n response = ollama.embeddings(model=\"mxbai-embed-large\", prompt=text)\n return response[\"embedding\"]\n","test_embedding = emb_text(\"This is a test\")\nembedding_dim = len(test_embedding)\nprint(embedding_dim)\nprint(test_embedding[:10])\n","from pymilvus import MilvusClient\n\nmilvus_client = MilvusClient(uri=\"./milvus_demo.db\")\n\ncollection_name = \"my_rag_collection\"\n","if milvus_client.has_collection(collection_name):\n milvus_client.drop_collection(collection_name)\n","milvus_client.create_collection(\n collection_name=collection_name,\n dimension=embedding_dim,\n metric_type=\"IP\", # Inner product distance\n consistency_level=\"Strong\", # Strong consistency level\n)\n","from tqdm import tqdm\n\ndata = []\n\nfor i, line in enumerate(tqdm(text_lines, desc=\"Creating embeddings\")):\n data.append({\"id\": i, \"vector\": emb_text(line), \"text\": line})\n\nmilvus_client.insert(collection_name=collection_name, data=data)\n","question = \"How is data stored in milvus?\"\n","search_res = milvus_client.search(\n collection_name=collection_name,\n data=[\n emb_text(question)\n ], # Use the `emb_text` function to convert the question to an embedding vector\n limit=3, # Return top 3 results\n search_params={\"metric_type\": \"IP\", \"params\": {}}, # Inner product distance\n output_fields=[\"text\"], # Return the text field\n)\n","import json\n\nretrieved_lines_with_distances = [\n (res[\"entity\"][\"text\"], res[\"distance\"]) for res in search_res[0]\n]\nprint(json.dumps(retrieved_lines_with_distances, indent=4))\n","context = \"\\n\".join(\n [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]\n)\n","SYSTEM_PROMPT = \"\"\"\nHuman: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.\n\"\"\"\nUSER_PROMPT = f\"\"\"\nUse the following pieces of information enclosed in tags to provide an answer to the question enclosed in tags.\n\n{context}\n \n\n{question}\n \n\"\"\"\n","from ollama import chat\nfrom ollama import ChatResponse\n\nresponse: ChatResponse = chat(\n model=\"llama3.2\",\n messages=[\n {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n {\"role\": \"user\", \"content\": USER_PROMPT},\n ],\n)\nprint(response[\"message\"][\"content\"])\n"],"headingContent":"Build RAG with Milvus and Ollama","anchorList":[{"label":"MilvusとOllamaでRAGを構築する","href":"Build-RAG-with-Milvus-and-Ollama","type":1,"isActive":false},{"label":"準備","href":"Preparation","type":2,"isActive":false},{"label":"Milvusにデータをロードする。","href":"Load-data-into-Milvus","type":2,"isActive":false},{"label":"RAGの構築","href":"Build-RAG","type":2,"isActive":false}]}
\ No newline at end of file
diff --git a/localization/v2.4.x/site/ja/integrations/build_RAG_with_milvus_and_ollama.md b/localization/v2.4.x/site/ja/integrations/build_RAG_with_milvus_and_ollama.md
new file mode 100644
index 000000000..cc443a8c4
--- /dev/null
+++ b/localization/v2.4.x/site/ja/integrations/build_RAG_with_milvus_and_ollama.md
@@ -0,0 +1,281 @@
+---
+id: build_RAG_with_milvus_and_ollama.md
+summary: >-
+ このガイドでは、OllamaとMilvusを活用し、RAG（Retrieval-Augmented
+ Generation）パイプラインを効率的かつ安全に構築する方法をご紹介します。
+title: MilvusとOllamaでRAGを構築する
+---
+
+MilvusとOllamaでRAGを構築する
Ollamaは、大規模言語モデル（LLM）のローカルでの実行とカスタマイズを簡素化するオープンソースのプラットフォームです。高度な技術スキルを必要とすることなく、モデルのダウンロード、インストール、インタラクションを簡単に行うことができます。汎用的なものからドメインに特化したものまで、トレーニング済みのLLMライブラリが充実しているため、Ollamaは様々なアプリケーション向けにモデルの管理やカスタマイズを簡単に行うことができます。データのプライバシーと柔軟性が確保され、ユーザーは自分のマシン上でAI主導のソリューションを微調整、最適化、展開できるようになります。
+このガイドでは、OllamaとMilvusを活用してRAG（Retrieval-Augmented Generation）パイプラインを効率的かつ安全に構築する方法を紹介します。
+準備
依存関係と環境
$ pip install pymilvus ollama
+
+
+Google Colabを使用している場合、インストールしたばかりの依存関係を有効にするために、ランタイムを再起動する必要があるかもしれない（画面上部の "Runtime "メニューをクリックし、ドロップダウンメニューから "Restart session "を選択する）。
+
+データの準備
Milvusドキュメント2.4.xのFAQページをRAGのプライベートナレッジとして使用する。
+zipファイルをダウンロードし、milvus_docs フォルダにドキュメントを展開する。
+$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
+
+--2024-11-26 21:47:19-- https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+Resolving github.com (github.com)... 140.82.112.4
+Connecting to github.com (github.com)|140.82.112.4|:443... connected.
+HTTP request sent, awaiting response... 302 Found
+Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/267273319/c52902a0-e13c-4ca7-92e0-086751098a05?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241127%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241127T024720Z&X-Amz-Expires=300&X-Amz-Signature=7808b77cbdaa7e122196bcd75a73f29f2540333a350c4830bbdf5f286e876304&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dmilvus_docs_2.4.x_en.zip&response-content-type=application%2Foctet-stream [following]
+--2024-11-26 21:47:20-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/267273319/c52902a0-e13c-4ca7-92e0-086751098a05?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241127%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241127T024720Z&X-Amz-Expires=300&X-Amz-Signature=7808b77cbdaa7e122196bcd75a73f29f2540333a350c4830bbdf5f286e876304&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dmilvus_docs_2.4.x_en.zip&response-content-type=application%2Foctet-stream
+Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.109.133, 185.199.111.133, 185.199.108.133, ...
+Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.109.133|:443... connected.
+HTTP request sent, awaiting response... 200 OK
+Length: 613094 (599K) [application/octet-stream]
+Saving to: ‘milvus_docs_2.4.x_en.zip’
+
+milvus_docs_2.4.x_e 100%[===================>] 598.72K 1.20MB/s in 0.5s
+
+2024-11-26 21:47:20 (1.20 MB/s) - ‘milvus_docs_2.4.x_en.zip’ saved [613094/613094]
+
+フォルダmilvus_docs/en/faq からすべてのマークダウン・ファイルをロードする。各ドキュメントについて、単に "# "を使ってファイル内のコンテンツを区切るだけで、マークダウン・ファイルの各主要部分のコンテンツを大まかに区切ることができる。
+from glob import glob
+
+text_lines = []
+
+for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
+ with open(file_path, "r") as file:
+ file_text = file.read()
+
+ text_lines += file_text.split("# ")
+
+LLMと埋め込みモデルの準備
OllamaはLLMベースのタスクと埋め込み生成の両方に複数のモデルをサポートしており、検索支援生成（RAG）アプリケーションを簡単に開発することができる。このセットアップでは
+
+- テキスト生成タスクでは、LLMとしてLlama 3.2（3B）を使用します。
+- 埋め込み生成には、意味的類似性に最適化された334Mのパラメータを持つモデル、mxbai-embed-largeを使う。
+
+開始する前に、両方のモデルがローカルに引き込まれていることを確認する：
+! ollama pull mxbai-embed-large
+
+[?25lpulling manifest ⠋ [?25h[?25l[2K[1Gpulling manifest ⠙ [?25h[?25l[2K[1Gpulling manifest ⠹ [?25h[?25l[2K[1Gpulling manifest ⠸ [?25h[?25l[2K[1Gpulling manifest ⠼ [?25h[?25l[2K[1Gpulling manifest ⠴ [?25h[?25l[2K[1Gpulling manifest
+pulling 819c2adf5ce6... 100% ▕████████████████▏ 669 MB
+pulling c71d239df917... 100% ▕████████████████▏ 11 KB
+pulling b837481ff855... 100% ▕████████████████▏ 16 B
+pulling 38badd946f91... 100% ▕████████████████▏ 408 B
+verifying sha256 digest
+writing manifest
+success [?25h
+
+! ollama pull llama3.2
+
+[?25lpulling manifest ⠋ [?25h[?25l[2K[1Gpulling manifest ⠙ [?25h[?25l[2K[1Gpulling manifest ⠹ [?25h[?25l[2K[1Gpulling manifest ⠸ [?25h[?25l[2K[1Gpulling manifest ⠼ [?25h[?25l[2K[1Gpulling manifest ⠴ [?25h[?25l[2K[1Gpulling manifest
+pulling dde5aa3fc5ff... 100% ▕████████████████▏ 2.0 GB
+pulling 966de95ca8a6... 100% ▕████████████████▏ 1.4 KB
+pulling fcc5a6bec9da... 100% ▕████████████████▏ 7.7 KB
+pulling a70ff7e570d9... 100% ▕████████████████▏ 6.0 KB
+pulling 56bb8bd477a5... 100% ▕████████████████▏ 96 B
+pulling 34bb5ab01051... 100% ▕████████████████▏ 561 B
+verifying sha256 digest
+writing manifest
+success [?25h
+
+これらのモデルの準備ができたので、LLM駆動生成と埋め込みベースの検索ワークフローの実装に進むことができます。
+import ollama
+
+
+def emb_text(text):
+ response = ollama.embeddings(model="mxbai-embed-large", prompt=text)
+ return response["embedding"]
+
+テスト埋め込みを生成し、その次元と最初のいくつかの要素を表示します。
+test_embedding = emb_text("This is a test")
+embedding_dim = len(test_embedding)
+print(embedding_dim)
+print(test_embedding[:10])
+
+1024
+[0.23276396095752716, 0.4257211685180664, 0.19724100828170776, 0.46120673418045044, -0.46039995551109314, -0.1413791924715042, -0.18261606991291046, -0.07602324336767197, 0.39991313219070435, 0.8337644338607788]
+
+Milvusにデータをロードする。
コレクションの作成
from pymilvus import MilvusClient
+
+milvus_client = MilvusClient(uri="./milvus_demo.db")
+
+collection_name = "my_rag_collection"
+
+
+MilvusClient の引数については、以下の通りです：
+
+uri をローカルファイル、例えば./milvus.db とするのが最も便利な方法です。- データ規模が大きい場合は、dockerやkubernetes上に、よりパフォーマンスの高いMilvusサーバを構築することができます。このセットアップでは、サーバの uri、例えば
http://localhost:19530 をuri として使用してください。
+- MilvusのフルマネージドクラウドサービスであるZilliz Cloudを利用する場合は、Zilliz CloudのPublic EndpointとApi keyに対応する
uri とtoken を調整してください。
+
+
+コレクションが既に存在するか確認し、存在する場合は削除します。
+if milvus_client.has_collection(collection_name):
+ milvus_client.drop_collection(collection_name)
+
+指定したパラメータで新しいコレクションを作成します。
+フィールド情報を指定しない場合、Milvusは自動的にプライマリキー用のデフォルトid フィールドと、ベクトルデータを格納するためのvector フィールドを作成します。予約されたJSONフィールドは、スキーマで定義されていないフィールドとその値を格納するために使用されます。
+milvus_client.create_collection(
+ collection_name=collection_name,
+ dimension=embedding_dim,
+ metric_type="IP", # Inner product distance
+ consistency_level="Strong", # Strong consistency level
+)
+
+データの挿入
テキスト行を繰り返し、エンベッディングを作成し、milvusにデータを挿入します。
+ここに新しいフィールドtext 、コレクションスキーマで定義されていないフィールドがあります。これは予約されたJSONダイナミックフィールドに自動的に追加され、高レベルでは通常のフィールドとして扱うことができる。
+from tqdm import tqdm
+
+data = []
+
+for i, line in enumerate(tqdm(text_lines, desc="Creating embeddings")):
+ data.append({"id": i, "vector": emb_text(line), "text": line})
+
+milvus_client.insert(collection_name=collection_name, data=data)
+
+Creating embeddings: 100%|██████████| 72/72 [00:03<00:00, 22.56it/s]
+
+
+
+
+
+{'insert_count': 72, 'ids': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71], 'cost': 0}
+
+RAGの構築
クエリのデータを取得する
Milvusに関するよくある質問を指定してみましょう。
+question = "How is data stored in milvus?"
+
+コレクションで質問を検索し、セマンティックトップ3マッチを取得します。
+search_res = milvus_client.search(
+ collection_name=collection_name,
+ data=[
+ emb_text(question)
+ ], # Use the `emb_text` function to convert the question to an embedding vector
+ limit=3, # Return top 3 results
+ search_params={"metric_type": "IP", "params": {}}, # Inner product distance
+ output_fields=["text"], # Return the text field
+)
+
+クエリの検索結果を見てみましょう。
+import json
+
+retrieved_lines_with_distances = [
+ (res["entity"]["text"], res["distance"]) for res in search_res[0]
+]
+print(json.dumps(retrieved_lines_with_distances, indent=4))
+
+[
+ [
+ " Where does Milvus store data?\n\nMilvus deals with two types of data, inserted data and metadata. \n\nInserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).\n\nMetadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.\n\n###",
+ 231.9398193359375
+ ],
+ [
+ "How does Milvus flush data?\n\nMilvus returns success when inserted data are loaded to the message queue. However, the data are not yet flushed to the disk. Then Milvus' data node writes the data in the message queue to persistent storage as incremental logs. If `flush()` is called, the data node is forced to write all data in the message queue to persistent storage immediately.\n\n###",
+ 226.48316955566406
+ ],
+ [
+ "What is the maximum dataset size Milvus can handle?\n\n \nTheoretically, the maximum dataset size Milvus can handle is determined by the hardware it is run on, specifically system memory and storage:\n\n- Milvus loads all specified collections and partitions into memory before running queries. Therefore, memory size determines the maximum amount of data Milvus can query.\n- When new entities and and collection-related schema (currently only MinIO is supported for data persistence) are added to Milvus, system storage determines the maximum allowable size of inserted data.\n\n###",
+ 210.60745239257812
+ ]
+]
+
+LLMを使ってRAGレスポンスを取得する
検索されたドキュメントを文字列形式に変換する。
+context = "\n".join(
+ [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]
+)
+
+ラネージ・モデルのシステム・プロンプトとユーザー・プロンプトを定義する。このプロンプトはmilvusから検索された文書で組み立てられる。
+SYSTEM_PROMPT = """
+Human: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.
+"""
+USER_PROMPT = f"""
+Use the following pieces of information enclosed in <context> tags to provide an answer to the question enclosed in <question> tags.
+<context>
+{context}
+</context>
+<question>
+{question}
+</question>
+"""
+
+Ollamaが提供するllama3.2 モデルを使って、プロンプトに基づいたレスポンスを生成する。
+from ollama import chat
+from ollama import ChatResponse
+
+response: ChatResponse = chat(
+ model="llama3.2",
+ messages=[
+ {"role": "system", "content": SYSTEM_PROMPT},
+ {"role": "user", "content": USER_PROMPT},
+ ],
+)
+print(response["message"]["content"])
+
+According to the provided context, data in Milvus is stored in two types:
+
+1. **Inserted data**: Storing data in persistent storage as incremental log. It supports multiple object storage backends such as MinIO, AWS S3, Google Cloud Storage (GCS), Azure Blob Storage, Alibaba Cloud OSS, and Tencent Cloud Object Storage.
+
+2. **Metadata**: Generated within Milvus and stored in etcd.
+
+素晴らしい！MilvusとOllamaでRAGパイプラインの構築に成功した。
diff --git a/localization/v2.4.x/site/ja/integrations/milvus_rag_with_dynamiq.json b/localization/v2.4.x/site/ja/integrations/milvus_rag_with_dynamiq.json
new file mode 100644
index 000000000..dcfdf2d33
--- /dev/null
+++ b/localization/v2.4.x/site/ja/integrations/milvus_rag_with_dynamiq.json
@@ -0,0 +1 @@
+{"codeList":["$ pip install dynamiq pymilvus\n","import os\n\nos.environ[\"OPENAI_API_KEY\"] = \"sk-***********\"\n","# Importing necessary libraries for the workflow\nfrom io import BytesIO\nfrom dynamiq import Workflow\nfrom dynamiq.nodes import InputTransformer\nfrom dynamiq.connections import (\n OpenAI as OpenAIConnection,\n Milvus as MilvusConnection,\n MilvusDeploymentType,\n)\nfrom dynamiq.nodes.converters import PyPDFConverter\nfrom dynamiq.nodes.splitters.document import DocumentSplitter\nfrom dynamiq.nodes.embedders import OpenAIDocumentEmbedder\nfrom dynamiq.nodes.writers import MilvusDocumentWriter\n\n# Initialize the workflow\nrag_wf = Workflow()\n","converter = PyPDFConverter(document_creation_mode=\"one-doc-per-page\")\nconverter_added = rag_wf.flow.add_nodes(\n converter\n) # Add node to the DAG (Directed Acyclic Graph)\n","document_splitter = DocumentSplitter(\n split_by=\"sentence\", # Splits documents into sentences\n split_length=10,\n split_overlap=1,\n input_transformer=InputTransformer(\n selector={\n \"documents\": f\"${[converter.id]}.output.documents\",\n },\n ),\n).depends_on(\n converter\n) # Set dependency on the PDF converter\nsplitter_added = rag_wf.flow.add_nodes(document_splitter) # Add to the DAG\n","embedder = OpenAIDocumentEmbedder(\n connection=OpenAIConnection(api_key=os.environ[\"OPENAI_API_KEY\"]),\n input_transformer=InputTransformer(\n selector={\n \"documents\": f\"${[document_splitter.id]}.output.documents\",\n },\n ),\n).depends_on(\n document_splitter\n) # Set dependency on the splitter\ndocument_embedder_added = rag_wf.flow.add_nodes(embedder) # Add to the DAG\n","vector_store = (\n MilvusDocumentWriter(\n connection=MilvusConnection(\n deployment_type=MilvusDeploymentType.FILE, uri=\"./milvus.db\"\n ),\n index_name=\"my_milvus_collection\",\n dimension=1536,\n create_if_not_exist=True,\n metric_type=\"COSINE\",\n )\n .inputs(documents=embedder.outputs.documents) # Connect to embedder output\n .depends_on(embedder) # Set dependency on the embedder\n)\nmilvus_writer_added = rag_wf.flow.add_nodes(vector_store) # Add to the DAG\n","file_paths = [\"./pdf_files/WhatisMilvus.pdf\"]\ninput_data = {\n \"files\": [BytesIO(open(path, \"rb\").read()) for path in file_paths],\n \"metadata\": [{\"filename\": path} for path in file_paths],\n}\n\n# Run the workflow with the prepared input data\ninserted_data = rag_wf.run(input_data=input_data)\n","from dynamiq import Workflow\nfrom dynamiq.connections import (\n OpenAI as OpenAIConnection,\n Milvus as MilvusConnection,\n MilvusDeploymentType,\n)\nfrom dynamiq.nodes.embedders import OpenAITextEmbedder\nfrom dynamiq.nodes.retrievers import MilvusDocumentRetriever\nfrom dynamiq.nodes.llms import OpenAI\nfrom dynamiq.prompts import Message, Prompt\n\n# Initialize the workflow\nretrieval_wf = Workflow()\n","# Establish OpenAI connection\nopenai_connection = OpenAIConnection(api_key=os.environ[\"OPENAI_API_KEY\"])\n\n# Define the text embedder node\nembedder = OpenAITextEmbedder(\n connection=openai_connection,\n model=\"text-embedding-3-small\",\n)\n\n# Add the embedder node to the workflow\nembedder_added = retrieval_wf.flow.add_nodes(embedder)\n","document_retriever = (\n MilvusDocumentRetriever(\n connection=MilvusConnection(\n deployment_type=MilvusDeploymentType.FILE, uri=\"./milvus.db\"\n ),\n index_name=\"my_milvus_collection\",\n dimension=1536,\n top_k=5,\n )\n .inputs(embedding=embedder.outputs.embedding) # Connect to embedder output\n .depends_on(embedder) # Dependency on the embedder node\n)\n\n# Add the retriever node to the workflow\nmilvus_retriever_added = retrieval_wf.flow.add_nodes(document_retriever)\n","# Define the prompt template for the LLM\nprompt_template = \"\"\"\nPlease answer the question based on the provided context.\n\nQuestion: {{ query }}\n\nContext:\n{% for document in documents %}\n- {{ document.content }}\n{% endfor %}\n\"\"\"\n\n# Create the prompt object\nprompt = Prompt(messages=[Message(content=prompt_template, role=\"user\")])\n","answer_generator = (\n OpenAI(\n connection=openai_connection,\n model=\"gpt-4o\",\n prompt=prompt,\n )\n .inputs(\n documents=document_retriever.outputs.documents,\n query=embedder.outputs.query,\n )\n .depends_on(\n [document_retriever, embedder]\n ) # Dependencies on retriever and embedder\n)\n\n# Add the answer generator node to the workflow\nanswer_generator_added = retrieval_wf.flow.add_nodes(answer_generator)\n","# Run the workflow with a sample query\nsample_query = \"What is the Advanced Search Algorithms in Milvus?\"\n\nresult = retrieval_wf.run(input_data={\"query\": sample_query})\n\nanswer = result.output.get(answer_generator.id).get(\"output\", {}).get(\"content\")\nprint(answer)\n"],"headingContent":"Getting Started with Dynamiq and Milvus","anchorList":[{"label":"DynamiqとMilvusをはじめよう","href":"Getting-Started-with-Dynamiq-and-Milvus","type":1,"isActive":false},{"label":"準備","href":"Preparation","type":2,"isActive":false},{"label":"RAG - ドキュメントのインデックス作成フロー","href":"RAG---Document-Indexing-Flow","type":2,"isActive":false},{"label":"RAGドキュメント検索フロー","href":"RAG-Document-Retrieval-Flow","type":2,"isActive":false}]}
\ No newline at end of file
diff --git a/localization/v2.4.x/site/ja/integrations/milvus_rag_with_dynamiq.md b/localization/v2.4.x/site/ja/integrations/milvus_rag_with_dynamiq.md
new file mode 100644
index 000000000..fb27d06bf
--- /dev/null
+++ b/localization/v2.4.x/site/ja/integrations/milvus_rag_with_dynamiq.md
@@ -0,0 +1,335 @@
+---
+id: milvus_rag_with_dynamiq.md
+summary: >-
+ このチュートリアルでは、RAGワークフローのために作られた高性能ベクトルデータベースMilvusとDynamiqをシームレスに使用する方法を探ります。Milvusは、ベクトル埋め込みデータの効率的な保存、インデックス付け、検索に優れており、高速かつ正確なコンテキストデータへのアクセスを必要とするAIシステムに不可欠なコンポーネントです。
+title: DynamiqとMilvusをはじめよう
+---
+
+DynamiqとMilvusをはじめよう
Dynamiqは、AIを搭載したアプリケーションの開発を効率化する強力なGen AIフレームワークです。検索拡張世代（RAG）と大規模言語モデル（LLM）エージェントを強力にサポートするDynamiqは、開発者が簡単かつ効率的にインテリジェントで動的なシステムを作成できるようにします。
+このチュートリアルでは、RAGワークフローのために作られた高性能ベクトルデータベースであるMilvusとDynamiqをシームレスに使用する方法を探ります。Milvusは、ベクトル埋め込みデータの効率的な保存、インデックス付け、検索に優れており、高速かつ正確なコンテキストデータへのアクセスを必要とするAIシステムにとって不可欠なコンポーネントとなっています。
+このステップバイステップガイドでは、2つのコアなRAGワークフローをカバーします：
+
+ドキュメントインデキシングフロー：入力ファイル（PDFなど）を処理し、その内容をベクトル埋め込みデータに変換し、Milvusに格納する方法を学びます。Milvusの高性能なインデックス作成機能を活用することで、データを迅速に検索できる状態にします。
ドキュメント検索フロー： Milvusに関連するドキュメントの埋め込みをクエリし、DynamiqのLLMエージェントでコンテキストを考慮した洞察に満ちたレスポンスを生成するためにそれらを使用し、AIを活用したシームレスなユーザーエクスペリエンスを実現する方法をご覧ください。
+このチュートリアルが終わる頃には、MilvusとDynamiqがどのように連携し、ニーズに合わせたスケーラブルでコンテキストを考慮したAIシステムを構築するのかについて、しっかりと理解できるようになるでしょう。
+準備
必要なライブラリのダウンロード
$ pip install dynamiq pymilvus
+
+
+Google Colabを使用している場合、インストールした依存関係を有効にするために、ランタイムを再起動する必要があります（画面上部の "Runtime "メニューをクリックし、ドロップダウンメニューから "Restart session "を選択してください）。
+
+LLMエージェントの設定
この例では、LLMとしてOpenAIを使います。api key OPENAI_API_KEY を環境変数として用意してください。
+import os
+
+os.environ["OPENAI_API_KEY"] = "sk-***********"
+
+RAG - ドキュメントのインデックス作成フロー
このチュートリアルでは、Milvusをベクターデータベースとして文書をインデックス化するRAG（Retrieval-Augmented Generation）ワークフローを示します。このワークフローは、入力されたPDFファイルを受け取り、より小さな塊に処理し、OpenAIの埋め込みモデルを使用してベクトル埋め込みを生成し、効率的な検索のために埋め込みをMilvusコレクションに保存します。
+このワークフローが終了する頃には、セマンティック検索や質問応答のような将来のRAGタスクをサポートする、スケーラブルで効率的なドキュメントインデキシングシステムが完成していることでしょう。
+必要なライブラリのインポートとワークフローの初期化
# Importing necessary libraries for the workflow
+from io import BytesIO
+from dynamiq import Workflow
+from dynamiq.nodes import InputTransformer
+from dynamiq.connections import (
+ OpenAI as OpenAIConnection,
+ Milvus as MilvusConnection,
+ MilvusDeploymentType,
+)
+from dynamiq.nodes.converters import PyPDFConverter
+from dynamiq.nodes.splitters.document import DocumentSplitter
+from dynamiq.nodes.embedders import OpenAIDocumentEmbedder
+from dynamiq.nodes.writers import MilvusDocumentWriter
+
+# Initialize the workflow
+rag_wf = Workflow()
+
+PDFコンバータノードの定義
converter = PyPDFConverter(document_creation_mode="one-doc-per-page")
+converter_added = rag_wf.flow.add_nodes(
+ converter
+) # Add node to the DAG (Directed Acyclic Graph)
+
+ドキュメント分割ノードの定義
document_splitter = DocumentSplitter(
+ split_by="sentence", # Splits documents into sentences
+ split_length=10,
+ split_overlap=1,
+ input_transformer=InputTransformer(
+ selector={
+ "documents": f"${[converter.id]}.output.documents",
+ },
+ ),
+).depends_on(
+ converter
+) # Set dependency on the PDF converter
+splitter_added = rag_wf.flow.add_nodes(document_splitter) # Add to the DAG
+
+埋め込みノードの定義
embedder = OpenAIDocumentEmbedder(
+ connection=OpenAIConnection(api_key=os.environ["OPENAI_API_KEY"]),
+ input_transformer=InputTransformer(
+ selector={
+ "documents": f"${[document_splitter.id]}.output.documents",
+ },
+ ),
+).depends_on(
+ document_splitter
+) # Set dependency on the splitter
+document_embedder_added = rag_wf.flow.add_nodes(embedder) # Add to the DAG
+
+Milvusベクターストアノードの定義
vector_store = (
+ MilvusDocumentWriter(
+ connection=MilvusConnection(
+ deployment_type=MilvusDeploymentType.FILE, uri="./milvus.db"
+ ),
+ index_name="my_milvus_collection",
+ dimension=1536,
+ create_if_not_exist=True,
+ metric_type="COSINE",
+ )
+ .inputs(documents=embedder.outputs.documents) # Connect to embedder output
+ .depends_on(embedder) # Set dependency on the embedder
+)
+milvus_writer_added = rag_wf.flow.add_nodes(vector_store) # Add to the DAG
+
+2024-11-19 22:14:03 - WARNING - Environment variable 'MILVUS_API_TOKEN' not found
+2024-11-19 22:14:03 - INFO - Pass in the local path ./milvus.db, and run it using milvus-lite
+2024-11-19 22:14:04 - DEBUG - Created new connection using: 0bef2849fdb1458a85df8bb9dd27f51d
+2024-11-19 22:14:04 - INFO - Collection my_milvus_collection does not exist. Creating a new collection.
+2024-11-19 22:14:04 - DEBUG - Successfully created collection: my_milvus_collection
+2024-11-19 22:14:05 - DEBUG - Successfully created an index on collection: my_milvus_collection
+2024-11-19 22:14:05 - DEBUG - Successfully created an index on collection: my_milvus_collection
+
+
+Milvusは2つのデプロイメントタイプを提供し、異なるユースケースに対応します：
+
+- MilvusDeploymentType.FILE
+
+
+- ローカルのプロトタイピングや 小規模なデータ保存に最適です。
+uri をローカルファイルパス(例:./milvus.db)に設定すると、Milvus Liteが自動的にすべてのデータを指定されたファイルに保存します。- これは迅速なセットアップと 実験に便利なオプションです。
+
+
+- MilvusDeploymentType.HOST
+
+
+100万以上のベクターを管理するような大規模データシナリオ用に設計されています。
+セルフホストサーバ
+
+- DockerまたはKubernetesを使用して高性能なMilvusサーバーをデプロイします。
+- サーバのアドレスとポートを
uri (例:http://localhost:19530) に設定します。
+- 認証が有効な場合：
+token に<your_username>:<your_password> を指定する。- 認証が無効の場合：
+token は未設定のままにします。
+Zillizクラウド（マネージドサービス）
+
+- フルマネージドでクラウドベースのMilvusをご利用いただくには、Zilliz Cloudをご利用ください。
+- Zilliz Cloud コンソールで提供されるPublic Endpoint と API キーに従って
uri とtoken を設定します。
+
+
+入力データの定義とワークフローの実行
file_paths = ["./pdf_files/WhatisMilvus.pdf"]
+input_data = {
+ "files": [BytesIO(open(path, "rb").read()) for path in file_paths],
+ "metadata": [{"filename": path} for path in file_paths],
+}
+
+# Run the workflow with the prepared input data
+inserted_data = rag_wf.run(input_data=input_data)
+
+/var/folders/09/d0hx80nj35sb5hxb5cpc1q180000gn/T/ipykernel_31319/3145804345.py:4: ResourceWarning: unclosed file <_io.BufferedReader name='./pdf_files/WhatisMilvus.pdf'>
+ BytesIO(open(path, "rb").read()) for path in file_paths
+ResourceWarning: Enable tracemalloc to get the object allocation traceback
+2024-11-19 22:14:09 - INFO - Workflow 87878444-6a3d-43f3-ae32-0127564a959f: execution started.
+2024-11-19 22:14:09 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution started.
+2024-11-19 22:14:09 - INFO - Node PyPDF File Converter - 6eb42b1f-7637-407b-a3ac-4167bcf3b5c4: execution started.
+2024-11-19 22:14:09 - INFO - Node PyPDF File Converter - 6eb42b1f-7637-407b-a3ac-4167bcf3b5c4: execution succeeded in 58ms.
+2024-11-19 22:14:09 - INFO - Node DocumentSplitter - 5baed580-6de0-4dcd-bace-d7d947ab6c7f: execution started.
+/Users/jinhonglin/anaconda3/envs/myenv/lib/python3.11/site-packages/websockets/legacy/__init__.py:6: DeprecationWarning: websockets.legacy is deprecated; see https://websockets.readthedocs.io/en/stable/howto/upgrade.html for upgrade instructions
+ warnings.warn( # deprecated in 14.0 - 2024-11-09
+/Users/jinhonglin/anaconda3/envs/myenv/lib/python3.11/site-packages/pydantic/fields.py:804: PydanticDeprecatedSince20: Using extra keyword arguments on `Field` is deprecated and will be removed. Use `json_schema_extra` instead. (Extra keys: 'is_accessible_to_agent'). Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/
+ warn(
+2024-11-19 22:14:09 - INFO - Node DocumentSplitter - 5baed580-6de0-4dcd-bace-d7d947ab6c7f: execution succeeded in 104ms.
+2024-11-19 22:14:09 - INFO - Node OpenAIDocumentEmbedder - 91928f67-a00f-48f6-a864-f6e21672ec7e: execution started.
+2024-11-19 22:14:09 - INFO - Node OpenAIDocumentEmbedder - d30a4cdc-0fab-4aff-b2e5-6161a62cb6fd: execution started.
+2024-11-19 22:14:10 - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
+2024-11-19 22:14:10 - INFO - Node OpenAIDocumentEmbedder - d30a4cdc-0fab-4aff-b2e5-6161a62cb6fd: execution succeeded in 724ms.
+2024-11-19 22:14:10 - INFO - Node MilvusDocumentWriter - dddab4cc-1dae-4e7e-9101-1ec353f530da: execution started.
+2024-11-19 22:14:10 - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
+2024-11-19 22:14:10 - INFO - Node MilvusDocumentWriter - dddab4cc-1dae-4e7e-9101-1ec353f530da: execution succeeded in 66ms.
+2024-11-19 22:14:10 - INFO - Node OpenAIDocumentEmbedder - 91928f67-a00f-48f6-a864-f6e21672ec7e: execution succeeded in 961ms.
+2024-11-19 22:14:10 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution succeeded in 1.3s.
+2024-11-19 22:14:10 - INFO - Workflow 87878444-6a3d-43f3-ae32-0127564a959f: execution succeeded in 1.3s.
+
+このワークフローを通して、Milvusをベクトルデータベースとして使用し、OpenAIの埋め込みモデルをセマンティック表現に使用したドキュメントインデキシングパイプラインの実装に成功した。このセットアップにより、高速かつ正確なベクトルベースの検索が可能になり、意味検索、文書検索、コンテキストAI主導型インタラクションのようなRAGワークフローの基盤が形成される。
+Milvusのスケーラブルなストレージ機能とDynamiqのオーケストレーションにより、このソリューションはプロトタイピングと大規模な本番導入の両方に対応しています。このパイプラインを拡張して、検索ベースの質問応答やAI主導のコンテンツ生成などのタスクを追加することができます。
+RAGドキュメント検索フロー
このチュートリアルでは、RAG（Retrieval-Augmented Generation）文書検索ワークフローを実装します。このワークフローでは、ユーザークエリを受け取り、それに対するベクトル埋め込みを生成し、Milvusベクトルデータベースから最も関連性の高い文書を検索し、大規模言語モデル（LLM）を使用して、検索された文書に基づいて詳細かつコンテキストを考慮した回答を生成します。
+このワークフローに従うことで、ベクトルベースの文書検索のパワーとOpenAIの高度なLLMの機能を組み合わせた、意味検索と質問応答のためのエンドツーエンドのソリューションを作成することができます。このアプローチは、ドキュメントデータベースに保存された知識を活用することで、ユーザーからの問い合わせに対して効率的でインテリジェントな応答を可能にします。
+必要なライブラリのインポートとワークフローの初期化
from dynamiq import Workflow
+from dynamiq.connections import (
+ OpenAI as OpenAIConnection,
+ Milvus as MilvusConnection,
+ MilvusDeploymentType,
+)
+from dynamiq.nodes.embedders import OpenAITextEmbedder
+from dynamiq.nodes.retrievers import MilvusDocumentRetriever
+from dynamiq.nodes.llms import OpenAI
+from dynamiq.prompts import Message, Prompt
+
+# Initialize the workflow
+retrieval_wf = Workflow()
+
+OpenAI接続とテキストエンベッダーの定義
# Establish OpenAI connection
+openai_connection = OpenAIConnection(api_key=os.environ["OPENAI_API_KEY"])
+
+# Define the text embedder node
+embedder = OpenAITextEmbedder(
+ connection=openai_connection,
+ model="text-embedding-3-small",
+)
+
+# Add the embedder node to the workflow
+embedder_added = retrieval_wf.flow.add_nodes(embedder)
+
+Milvusドキュメントレトリバーの定義
document_retriever = (
+ MilvusDocumentRetriever(
+ connection=MilvusConnection(
+ deployment_type=MilvusDeploymentType.FILE, uri="./milvus.db"
+ ),
+ index_name="my_milvus_collection",
+ dimension=1536,
+ top_k=5,
+ )
+ .inputs(embedding=embedder.outputs.embedding) # Connect to embedder output
+ .depends_on(embedder) # Dependency on the embedder node
+)
+
+# Add the retriever node to the workflow
+milvus_retriever_added = retrieval_wf.flow.add_nodes(document_retriever)
+
+2024-11-19 22:14:19 - WARNING - Environment variable 'MILVUS_API_TOKEN' not found
+2024-11-19 22:14:19 - INFO - Pass in the local path ./milvus.db, and run it using milvus-lite
+2024-11-19 22:14:19 - DEBUG - Created new connection using: 98d1132773af4298a894ad5925845fd2
+2024-11-19 22:14:19 - INFO - Collection my_milvus_collection already exists. Skipping creation.
+
+プロンプトテンプレートの定義
# Define the prompt template for the LLM
+prompt_template = """
+Please answer the question based on the provided context.
+
+Question: {{ query }}
+
+Context:
+{% for document in documents %}
+- {{ document.content }}
+{% endfor %}
+"""
+
+# Create the prompt object
+prompt = Prompt(messages=[Message(content=prompt_template, role="user")])
+
+回答ジェネレータの定義
answer_generator = (
+ OpenAI(
+ connection=openai_connection,
+ model="gpt-4o",
+ prompt=prompt,
+ )
+ .inputs(
+ documents=document_retriever.outputs.documents,
+ query=embedder.outputs.query,
+ )
+ .depends_on(
+ [document_retriever, embedder]
+ ) # Dependencies on retriever and embedder
+)
+
+# Add the answer generator node to the workflow
+answer_generator_added = retrieval_wf.flow.add_nodes(answer_generator)
+
+ワークフローの実行
# Run the workflow with a sample query
+sample_query = "What is the Advanced Search Algorithms in Milvus?"
+
+result = retrieval_wf.run(input_data={"query": sample_query})
+
+answer = result.output.get(answer_generator.id).get("output", {}).get("content")
+print(answer)
+
+2024-11-19 22:14:22 - INFO - Workflow f4a073fb-dfb6-499c-8cac-5710a7ad6d47: execution started.
+2024-11-19 22:14:22 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution started.
+2024-11-19 22:14:22 - INFO - Node OpenAITextEmbedder - 47afb0bc-cf96-429d-b58f-11b6c935fec3: execution started.
+2024-11-19 22:14:23 - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
+2024-11-19 22:14:23 - INFO - Node OpenAITextEmbedder - 47afb0bc-cf96-429d-b58f-11b6c935fec3: execution succeeded in 474ms.
+2024-11-19 22:14:23 - INFO - Node MilvusDocumentRetriever - 51c8311b-4837-411f-ba42-21e28239a2ee: execution started.
+2024-11-19 22:14:23 - INFO - Node MilvusDocumentRetriever - 51c8311b-4837-411f-ba42-21e28239a2ee: execution succeeded in 23ms.
+2024-11-19 22:14:23 - INFO - Node LLM - ac722325-bece-453f-a2ed-135b0749ee7a: execution started.
+2024-11-19 22:14:24 - INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
+2024-11-19 22:14:24 - INFO - Node LLM - ac722325-bece-453f-a2ed-135b0749ee7a: execution succeeded in 1.8s.
+2024-11-19 22:14:25 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution succeeded in 2.4s.
+2024-11-19 22:14:25 - INFO - Workflow f4a073fb-dfb6-499c-8cac-5710a7ad6d47: execution succeeded in 2.4s.
+
+
+The advanced search algorithms in Milvus include a variety of in-memory and on-disk indexing/search algorithms such as IVF (Inverted File), HNSW (Hierarchical Navigable Small World), and DiskANN. These algorithms have been deeply optimized to enhance performance, delivering 30%-70% better performance compared to popular implementations like FAISS and HNSWLib. These optimizations are part of Milvus's design to ensure high efficiency and scalability in handling vector data.
+
diff --git a/localization/v2.4.x/site/ja/menuStructure/ja.json b/localization/v2.4.x/site/ja/menuStructure/ja.json
index 49962fc26..e6116c7a4 100644
--- a/localization/v2.4.x/site/ja/menuStructure/ja.json
+++ b/localization/v2.4.x/site/ja/menuStructure/ja.json
@@ -641,7 +641,7 @@
]
},
{
- "label": "Milvusマイグレーション",
+ "label": "Milvus移住",
"id": "milvus_migration",
"isMenu": true,
"order": 5,
@@ -1344,6 +1344,24 @@
"id": "build_RAG_with_milvus_and_siliconflow.md",
"order": 4,
"children": []
+ },
+ {
+ "label": "サンバ・ノヴァ",
+ "id": "use_milvus_with_sambanova.md",
+ "order": 5,
+ "children": []
+ },
+ {
+ "label": "ジェミニ",
+ "id": "build_RAG_with_milvus_and_gemini.md",
+ "order": 6,
+ "children": []
+ },
+ {
+ "label": "オーラマ",
+ "id": "build_RAG_with_milvus_and_ollama.md",
+ "order": 7,
+ "children": []
}
]
},
@@ -1433,15 +1451,15 @@
"children": []
},
{
- "label": "サンバ・ノヴァ",
- "id": "use_milvus_with_sambanova.md",
+ "label": "プライベートGPT",
+ "id": "use_milvus_in_private_gpt.md",
"order": 10,
"children": []
},
{
- "label": "プライベートGPT",
- "id": "use_milvus_in_private_gpt.md",
- "order": 8,
+ "label": "ダイナミック",
+ "id": "milvus_rag_with_dynamiq.md",
+ "order": 11,
"children": []
}
]
@@ -1493,6 +1511,12 @@
"id": "knowledge_table_with_milvus.md",
"order": 2,
"children": []
+ },
+ {
+ "label": "コグニー",
+ "id": "build_RAG_with_milvus_and_cognee.md",
+ "order": 3,
+ "children": []
}
]
},
diff --git a/localization/v2.4.x/site/ko/integrations/build_RAG_with_milvus_and_cognee.json b/localization/v2.4.x/site/ko/integrations/build_RAG_with_milvus_and_cognee.json
new file mode 100644
index 000000000..65a11fc1e
--- /dev/null
+++ b/localization/v2.4.x/site/ko/integrations/build_RAG_with_milvus_and_cognee.json
@@ -0,0 +1 @@
+{"codeList":["$ pip install pymilvus git+https://github.com/topoteretes/cognee.git\n","import os\n\nimport cognee\n\ncognee.config.set_llm_api_key(\"YOUR_OPENAI_API_KEY\")\n\n\nos.environ[\"VECTOR_DB_PROVIDER\"] = \"milvus\"\nos.environ[\"VECTOR_DB_URL\"] = \"./milvus.db\"\n","$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip\n$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs\n","from glob import glob\n\ntext_lines = []\n\nfor file_path in glob(\"milvus_docs/en/faq/*.md\", recursive=True):\n with open(file_path, \"r\") as file:\n file_text = file.read()\n\n text_lines += file_text.split(\"# \")\n","await cognee.prune.prune_data()\nawait cognee.prune.prune_system(metadata=True)\n","await cognee.add(data=text_lines, dataset_name=\"milvus_faq\")\nawait cognee.cognify()\n\n# [DocumentChunk(id=UUID('6889e7ef-3670-555c-bb16-3eb50d1d30b0'), updated_at=datetime.datetime(2024, 12, 4, 6, 29, 46, 472907, tzinfo=datetime.timezone.utc), text='Does the query perform in memory? What are incremental data and historical data?\\n\\nYes. When ...\n# ...\n","from cognee.api.v1.search import SearchType\n\nquery_text = \"How is data stored in milvus?\"\nsearch_results = await cognee.search(SearchType.SUMMARIES, query_text=query_text)\n\nprint(search_results[0])\n","from cognee.api.v1.search import SearchType\n\nquery_text = \"How is data stored in milvus?\"\nsearch_results = await cognee.search(SearchType.CHUNKS, query_text=query_text)\n","def format_and_print(data):\n print(\"ID:\", data[\"id\"])\n print(\"\\nText:\\n\")\n paragraphs = data[\"text\"].split(\"\\n\\n\")\n for paragraph in paragraphs:\n print(paragraph.strip())\n print()\n\n\nformat_and_print(search_results[0])\n","await cognee.prune.prune_data()\nawait cognee.prune.prune_system(metadata=True)\n","# We only use one line of text as the dataset, which simplifies the output later\ntext = \"\"\"\n Natural language processing (NLP) is an interdisciplinary\n subfield of computer science and information retrieval.\n \"\"\"\n\nawait cognee.add(text)\nawait cognee.cognify()\n","query_text = \"Tell me about NLP\"\nsearch_results = await cognee.search(SearchType.INSIGHTS, query_text=query_text)\n\nfor result_text in search_results:\n print(result_text)\n\n# Example output:\n# ({'id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'natural language processing', 'description': 'An interdisciplinary subfield of computer science and information retrieval.'}, {'relationship_name': 'is_a_subfield_of', 'source_node_id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'target_node_id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 15, 473137, tzinfo=datetime.timezone.utc)}, {'id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'computer science', 'description': 'The study of computation and information processing.'})\n# (...)\n#\n# It represents nodes and relationships in the knowledge graph:\n# - The first element is the source node (e.g., 'natural language processing').\n# - The second element is the relationship between nodes (e.g., 'is_a_subfield_of').\n# - The third element is the target node (e.g., 'computer science').\n"],"headingContent":"","anchorList":[{"label":"RAG 빌드","href":"Build-RAG","type":2,"isActive":false}]}
\ No newline at end of file
diff --git a/localization/v2.4.x/site/ko/integrations/build_RAG_with_milvus_and_cognee.md b/localization/v2.4.x/site/ko/integrations/build_RAG_with_milvus_and_cognee.md
new file mode 100644
index 000000000..e29d90220
--- /dev/null
+++ b/localization/v2.4.x/site/ko/integrations/build_RAG_with_milvus_and_cognee.md
@@ -0,0 +1,157 @@
+---
+id: build_RAG_with_milvus_and_cognee.md
+summary: 이 튜토리얼에서는 Milvus와 Cognee를 사용하여 RAG(검색 증강 생성) 파이프라인을 구축하는 방법을 보여드리겠습니다.
+title: 밀버스 및 코그니로 RAG 구축하기
+---
+
+Milvus와 Cognee로 RAG 구축하기
Cognee는 확장 가능한 모듈식 ECL(추출, 인지, 로드) 파이프라인을 통해 AI 애플리케이션 개발을 간소화하는 개발자 우선 플랫폼입니다. Milvus와 원활하게 통합되어 대화, 문서, 트랜스크립션을 효율적으로 연결하고 검색할 수 있는 Cognee는 착각을 줄이고 운영 비용을 최적화합니다.
+Milvus와 같은 벡터 저장소, 그래프 데이터베이스, LLM을 강력하게 지원하는 Cognee는 검색 증강 생성(RAG) 시스템 구축을 위한 유연하고 사용자 정의 가능한 프레임워크를 제공합니다. 프로덕션에 바로 사용할 수 있는 아키텍처는 AI 기반 애플리케이션의 정확성과 효율성을 향상시킵니다.
+이 튜토리얼에서는 Milvus 및 Cognee를 사용하여 검색 증강 생성(RAG) 파이프라인을 구축하는 방법을 보여드리겠습니다.
+$ pip install pymilvus git+https://github.com/topoteretes/cognee.git
+
+
+Google Colab을 사용하는 경우 방금 설치한 종속 요소를 사용하려면 런타임을 다시 시작해야 할 수 있습니다(화면 상단의 '런타임' 메뉴를 클릭하고 드롭다운 메뉴에서 '세션 다시 시작'을 선택).
+
+이 예제에서는 기본적으로 OpenAI를 LLM으로 사용합니다. API 키를 준비하여 set_llm_api_key() 함수에서 설정해야 합니다.
+Milvus를 벡터 데이터베이스로 구성하려면 VECTOR_DB_PROVIDER 을 milvus 으로 설정하고 VECTOR_DB_URL 및 VECTOR_DB_KEY 을 지정합니다. 이 데모에서는 Milvus Lite를 사용하여 데이터를 저장하기 때문에 VECTOR_DB_URL 만 제공하면 됩니다.
+import os
+
+import cognee
+
+cognee.config.set_llm_api_key("YOUR_OPENAI_API_KEY")
+
+
+os.environ["VECTOR_DB_PROVIDER"] = "milvus"
+os.environ["VECTOR_DB_URL"] = "./milvus.db"
+
+
+환경 변수는 VECTOR_DB_URL 와 VECTOR_DB_KEY 입니다:
+
+VECTOR_DB_URL 를 로컬 파일(예:./milvus.db)로 설정하는 것이 가장 편리한 방법인데, Milvus Lite를 자동으로 활용하여 모든 데이터를 이 파일에 저장하기 때문입니다.- 데이터 규모가 큰 경우, 도커나 쿠버네티스에 더 고성능의 Milvus 서버를 설정할 수 있습니다. 이 설정에서는 서버 URL(예:
http://localhost:19530)을 VECTOR_DB_URL 으로 사용하세요.
+- 밀버스의 완전 관리형 클라우드 서비스인 질리즈 클라우드를 사용하려면, 질리즈 클라우드의 퍼블릭 엔드포인트와 API 키에 해당하는
VECTOR_DB_URL 와 VECTOR_DB_KEY 을 조정하세요.
+
+
+데이터 준비
Milvus 문서 2.4.x의 FAQ 페이지를 RAG의 비공개 지식으로 사용하며, 이는 간단한 RAG 파이프라인을 위한 좋은 데이터 소스입니다.
+zip 파일을 다운로드하고 문서를 milvus_docs 폴더에 압축을 풉니다.
+$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
+
+milvus_docs/en/faq 폴더에서 모든 마크다운 파일을 로드합니다. 각 문서에 대해 "#"를 사용하여 파일의 내용을 구분하기만 하면 마크다운 파일의 각 주요 부분의 내용을 대략적으로 구분할 수 있습니다.
+from glob import glob
+
+text_lines = []
+
+for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
+ with open(file_path, "r") as file:
+ file_text = file.read()
+
+ text_lines += file_text.split("# ")
+
+RAG 빌드
코그니 데이터 재설정
await cognee.prune.prune_data()
+await cognee.prune.prune_system(metadata=True)
+
+이제 깨끗한 슬레이트가 준비되었으므로 데이터 집합을 추가하여 지식 그래프로 처리할 수 있습니다.
+데이터 추가 및 인지하기
await cognee.add(data=text_lines, dataset_name="milvus_faq")
+await cognee.cognify()
+
+# [DocumentChunk(id=UUID('6889e7ef-3670-555c-bb16-3eb50d1d30b0'), updated_at=datetime.datetime(2024, 12, 4, 6, 29, 46, 472907, tzinfo=datetime.timezone.utc), text='Does the query perform in memory? What are incremental data and historical data?\n\nYes. When ...
+# ...
+
+add 메서드는 데이터 세트(Milvus FAQ)를 Cognee에 로드하고 cognify 메서드는 데이터를 처리하여 엔티티, 관계 및 요약을 추출하여 지식 그래프를 구성합니다.
+요약 쿼리하기
이제 데이터가 처리되었으므로 지식 그래프를 쿼리해 보겠습니다.
+from cognee.api.v1.search import SearchType
+
+query_text = "How is data stored in milvus?"
+search_results = await cognee.search(SearchType.SUMMARIES, query_text=query_text)
+
+print(search_results[0])
+
+{'id': 'de5c6713-e079-5d0b-b11d-e9bacd1e0d73', 'text': 'Milvus stores two data types: inserted data and metadata.'}
+
+이 쿼리는 지식 그래프에서 쿼리 텍스트와 관련된 요약을 검색하고 가장 관련성이 높은 후보가 인쇄됩니다.
+청크 쿼리하기
요약은 개괄적인 인사이트를 제공하지만, 더 세분화된 세부 정보를 얻으려면 처리된 데이터 세트에서 직접 특정 데이터 청크를 쿼리할 수 있습니다. 이러한 청크는 지식 그래프를 만드는 동안 추가되고 분석된 원본 데이터에서 파생됩니다.
+from cognee.api.v1.search import SearchType
+
+query_text = "How is data stored in milvus?"
+search_results = await cognee.search(SearchType.CHUNKS, query_text=query_text)
+
+가독성을 높이기 위해 서식을 지정하고 표시해 보겠습니다!
+def format_and_print(data):
+ print("ID:", data["id"])
+ print("\nText:\n")
+ paragraphs = data["text"].split("\n\n")
+ for paragraph in paragraphs:
+ print(paragraph.strip())
+ print()
+
+
+format_and_print(search_results[0])
+
+ID: 4be01c4b-9ee5-541c-9b85-297883934ab3
+
+Text:
+
+Where does Milvus store data?
+
+Milvus deals with two types of data, inserted data and metadata.
+
+Inserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).
+
+Metadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.
+
+###
+
+이전 단계에서는 요약과 특정 데이터 청크 모두에 대해 Milvus FAQ 데이터 집합을 쿼리했습니다. 이를 통해 상세한 인사이트와 세분화된 정보를 얻을 수 있었지만 데이터 세트가 너무 커서 지식 그래프 내에서 종속성을 명확하게 시각화하기가 어려웠습니다.
+이 문제를 해결하기 위해 코그니 환경을 재설정하고 더 작고 집중된 데이터 세트로 작업할 것입니다. 이렇게 하면 코그니파이 프로세스 중에 추출된 관계와 종속성을 더 잘 보여줄 수 있습니다. 데이터를 단순화함으로써 Cognee가 지식 그래프에서 정보를 어떻게 구성하고 구조화하는지를 명확하게 확인할 수 있습니다.
+코그니 초기화
await cognee.prune.prune_data()
+await cognee.prune.prune_system(metadata=True)
+
+집중 데이터 세트 추가하기
여기서는 한 줄의 텍스트만 있는 작은 데이터 집합을 추가하고 처리하여 집중적이고 쉽게 해석할 수 있는 지식 그래프가 되도록 합니다.
+# We only use one line of text as the dataset, which simplifies the output later
+text = """
+ Natural language processing (NLP) is an interdisciplinary
+ subfield of computer science and information retrieval.
+ """
+
+await cognee.add(text)
+await cognee.cognify()
+
+인사이트 쿼리하기
이 작은 데이터 세트에 집중함으로써 이제 지식 그래프 내의 관계와 구조를 명확하게 분석할 수 있습니다.
+query_text = "Tell me about NLP"
+search_results = await cognee.search(SearchType.INSIGHTS, query_text=query_text)
+
+for result_text in search_results:
+ print(result_text)
+
+# Example output:
+# ({'id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'natural language processing', 'description': 'An interdisciplinary subfield of computer science and information retrieval.'}, {'relationship_name': 'is_a_subfield_of', 'source_node_id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'target_node_id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 15, 473137, tzinfo=datetime.timezone.utc)}, {'id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'computer science', 'description': 'The study of computation and information processing.'})
+# (...)
+#
+# It represents nodes and relationships in the knowledge graph:
+# - The first element is the source node (e.g., 'natural language processing').
+# - The second element is the relationship between nodes (e.g., 'is_a_subfield_of').
+# - The third element is the target node (e.g., 'computer science').
+
+이 출력은 지식 그래프 쿼리의 결과를 나타내며, 처리된 데이터 세트에서 추출된 엔티티(노드)와 그 관계(에지)를 보여줍니다. 각 튜플에는 소스 엔티티, 관계 유형, 대상 엔티티와 함께 고유 ID, 설명, 타임스탬프와 같은 메타데이터가 포함됩니다. 그래프는 주요 개념과 그 의미론적 연결을 강조하여 데이터 집합을 구조적으로 이해할 수 있도록 해줍니다.
+Milvus를 통해 코그니의 기본 사용법을 배웠습니다. 코그니의 고급 사용법을 더 알고 싶으시다면 공식 페이지 를 참조하세요.
diff --git a/localization/v2.4.x/site/ko/integrations/build_RAG_with_milvus_and_gemini.json b/localization/v2.4.x/site/ko/integrations/build_RAG_with_milvus_and_gemini.json
new file mode 100644
index 000000000..a26d74cf2
--- /dev/null
+++ b/localization/v2.4.x/site/ko/integrations/build_RAG_with_milvus_and_gemini.json
@@ -0,0 +1 @@
+{"codeList":["$ pip install --upgrade pymilvus google-generativeai requests tqdm\n","import os\n\nos.environ[\"GEMINI_API_KEY\"] = \"***********\"\n","$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip\n$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs\n","from glob import glob\n\ntext_lines = []\n\nfor file_path in glob(\"milvus_docs/en/faq/*.md\", recursive=True):\n with open(file_path, \"r\") as file:\n file_text = file.read()\n\n text_lines += file_text.split(\"# \")\n","import google.generativeai as genai\n\ngenai.configure(api_key=os.environ[\"GEMINI_API_KEY\"])\n\ngemini_model = genai.GenerativeModel(\"gemini-1.5-flash\")\n\nresponse = gemini_model.generate_content(\"who are you\")\nprint(response.text)\n","test_embeddings = genai.embed_content(\n model=\"models/text-embedding-004\", content=[\"This is a test1\", \"This is a test2\"]\n)[\"embedding\"]\n\nembedding_dim = len(test_embeddings[0])\nprint(embedding_dim)\nprint(test_embeddings[0][:10])\n","from pymilvus import MilvusClient\n\nmilvus_client = MilvusClient(uri=\"./milvus_demo.db\")\n\ncollection_name = \"my_rag_collection\"\n","if milvus_client.has_collection(collection_name):\n milvus_client.drop_collection(collection_name)\n","milvus_client.create_collection(\n collection_name=collection_name,\n dimension=embedding_dim,\n metric_type=\"IP\", # Inner product distance\n consistency_level=\"Strong\", # Strong consistency level\n)\n","from tqdm import tqdm\n\ndata = []\n\ndoc_embeddings = genai.embed_content(\n model=\"models/text-embedding-004\", content=text_lines\n)[\"embedding\"]\n\nfor i, line in enumerate(tqdm(text_lines, desc=\"Creating embeddings\")):\n data.append({\"id\": i, \"vector\": doc_embeddings[i], \"text\": line})\n\nmilvus_client.insert(collection_name=collection_name, data=data)\n","question = \"How is data stored in milvus?\"\n","question_embedding = genai.embed_content(\n model=\"models/text-embedding-004\", content=question\n)[\"embedding\"]\n\nsearch_res = milvus_client.search(\n collection_name=collection_name,\n data=[question_embedding],\n limit=3, # Return top 3 results\n search_params={\"metric_type\": \"IP\", \"params\": {}}, # Inner product distance\n output_fields=[\"text\"], # Return the text field\n)\n","import json\n\nretrieved_lines_with_distances = [\n (res[\"entity\"][\"text\"], res[\"distance\"]) for res in search_res[0]\n]\nprint(json.dumps(retrieved_lines_with_distances, indent=4))\n","context = \"\\n\".join(\n [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]\n)\n","SYSTEM_PROMPT = \"\"\"\nHuman: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.\n\"\"\"\nUSER_PROMPT = f\"\"\"\nUse the following pieces of information enclosed in tags to provide an answer to the question enclosed in tags.\n\n{context}\n \n\n{question}\n \n\"\"\"\n","gemini_model = genai.GenerativeModel(\n \"gemini-1.5-flash\", system_instruction=SYSTEM_PROMPT\n)\nresponse = gemini_model.generate_content(USER_PROMPT)\nprint(response.text)\n"],"headingContent":"Build RAG with Milvus and Gemini","anchorList":[{"label":"Milvus 및 Gemini로 RAG 구축하기","href":"Build-RAG-with-Milvus-and-Gemini","type":1,"isActive":false},{"label":"준비","href":"Preparation","type":2,"isActive":false},{"label":"Milvus에 데이터 로드","href":"Load-data-into-Milvus","type":2,"isActive":false},{"label":"RAG 구축","href":"Build-RAG","type":2,"isActive":false}]}
\ No newline at end of file
diff --git a/localization/v2.4.x/site/ko/integrations/build_RAG_with_milvus_and_gemini.md b/localization/v2.4.x/site/ko/integrations/build_RAG_with_milvus_and_gemini.md
new file mode 100644
index 000000000..669250e4e
--- /dev/null
+++ b/localization/v2.4.x/site/ko/integrations/build_RAG_with_milvus_and_gemini.md
@@ -0,0 +1,244 @@
+---
+id: build_RAG_with_milvus_and_gemini.md
+summary: >-
+ 이 튜토리얼에서는 Milvus와 Gemini를 사용하여 RAG(검색 증강 생성) 파이프라인을 구축하는 방법을 보여드리겠습니다. Gemini
+ 모델을 사용하여 주어진 쿼리를 기반으로 텍스트를 생성합니다. 또한 생성된 텍스트를 저장하고 검색하는 데 Milvus를 사용합니다.
+title: Milvus 및 Gemini로 RAG 구축하기
+---
+
+Milvus 및 Gemini로 RAG 구축하기
Gemini API와 Google AI Studio를 사용하면 Google의 최신 모델로 작업을 시작하고 아이디어를 확장 가능한 애플리케이션으로 전환할 수 있습니다. Gemini는 텍스트 생성, 문서 처리, 시각, 오디오 분석 등과 같은 작업을 위해 Gemini-1.5-Flash, Gemini-1.5-Flash-8B, Gemini-1.5-Pro 과 같은 강력한 언어 모델에 대한 액세스를 제공합니다. API를 사용하면 수백만 개의 토큰으로 긴 컨텍스트를 입력하고, 특정 작업에 맞게 모델을 미세 조정하고, JSON과 같은 구조화된 출력을 생성하고, 시맨틱 검색 및 코드 실행과 같은 기능을 활용할 수 있습니다.
+이 튜토리얼에서는 Milvus와 Gemini를 사용하여 RAG(검색 증강 생성) 파이프라인을 구축하는 방법을 보여드리겠습니다. Gemini 모델을 사용하여 주어진 쿼리를 기반으로 텍스트를 생성합니다. 또한 생성된 텍스트를 저장하고 검색하는 데 Milvus를 사용하겠습니다.
+준비
종속성 및 환경
$ pip install --upgrade pymilvus google-generativeai requests tqdm
+
+
+Google Colab을 사용하는 경우 방금 설치한 종속 요소를 사용하려면 런타임을 다시 시작해야 할 수 있습니다(화면 상단의 "런타임" 메뉴를 클릭하고 드롭다운 메뉴에서 "세션 다시 시작"을 선택).
+
+먼저 Google AI Studio 플랫폼에 로그인하여 환경 변수로 GEMINI_API_KEY API 키를 준비해야 합니다.
+import os
+
+os.environ["GEMINI_API_KEY"] = "***********"
+
+데이터 준비
Milvus 문서 2.4.x의 FAQ 페이지를 RAG의 비공개 지식으로 사용하며, 이는 간단한 RAG 파이프라인을 위한 좋은 데이터 소스입니다.
+zip 파일을 다운로드하고 문서를 milvus_docs 폴더에 압축을 풉니다.
+$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
+
+milvus_docs/en/faq 폴더에서 모든 마크다운 파일을 로드합니다. 각 문서에 대해 "#"를 사용하여 파일의 내용을 구분하면 마크다운 파일의 각 주요 부분의 내용을 대략적으로 구분할 수 있습니다.
+from glob import glob
+
+text_lines = []
+
+for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
+ with open(file_path, "r") as file:
+ file_text = file.read()
+
+ text_lines += file_text.split("# ")
+
+LLM 및 임베딩 모델 준비
여기서는 gemini-1.5-flash 을 LLM으로, text-embedding-004 을 임베딩 모델로 사용합니다.
+LLM에서 테스트 응답을 생성해 보겠습니다:
+import google.generativeai as genai
+
+genai.configure(api_key=os.environ["GEMINI_API_KEY"])
+
+gemini_model = genai.GenerativeModel("gemini-1.5-flash")
+
+response = gemini_model.generate_content("who are you")
+print(response.text)
+
+I am a large language model, trained by Google. I am an AI and don't have a personal identity or consciousness. My purpose is to process information and respond to a wide range of prompts and questions in a helpful and informative way.
+
+테스트 임베딩을 생성하고 해당 차원과 처음 몇 개의 요소를 인쇄합니다.
+test_embeddings = genai.embed_content(
+ model="models/text-embedding-004", content=["This is a test1", "This is a test2"]
+)["embedding"]
+
+embedding_dim = len(test_embeddings[0])
+print(embedding_dim)
+print(test_embeddings[0][:10])
+
+768
+[0.013588584, -0.004361838, -0.08481652, -0.039724775, 0.04723794, -0.0051557426, 0.026071774, 0.045514572, -0.016867816, 0.039378334]
+
+Milvus에 데이터 로드
컬렉션 생성
from pymilvus import MilvusClient
+
+milvus_client = MilvusClient(uri="./milvus_demo.db")
+
+collection_name = "my_rag_collection"
+
+
+MilvusClient 의 인수를 사용합니다:
+
+uri 을 로컬 파일(예:./milvus.db)로 설정하는 것이 가장 편리한 방법인데, Milvus Lite를 자동으로 활용하여 모든 데이터를 이 파일에 저장하기 때문입니다.- 데이터의 규모가 큰 경우, 도커나 쿠버네티스에 더 성능이 좋은 Milvus 서버를 설정할 수 있습니다. 이 설정에서는 서버 URL(예:
http://localhost:19530)을 uri 으로 사용하세요.
+- 밀버스의 완전 관리형 클라우드 서비스인 질리즈 클라우드를 사용하려면, 질리즈 클라우드의 퍼블릭 엔드포인트와 API 키에 해당하는
uri 와 token 을 조정하세요.
+
+
+컬렉션이 이미 존재하는지 확인하고 존재한다면 삭제합니다.
+if milvus_client.has_collection(collection_name):
+ milvus_client.drop_collection(collection_name)
+
+지정된 파라미터로 새 컬렉션을 생성합니다.
+필드 정보를 지정하지 않으면 기본 키로 id 필드와 벡터 데이터를 저장할 vector 필드가 자동으로 생성됩니다. 예약된 JSON 필드는 스키마에 정의되지 않은 필드와 그 값을 저장하는 데 사용됩니다.
+milvus_client.create_collection(
+ collection_name=collection_name,
+ dimension=embedding_dim,
+ metric_type="IP", # Inner product distance
+ consistency_level="Strong", # Strong consistency level
+)
+
+데이터 삽입
텍스트 줄을 반복하여 임베딩을 만든 다음 데이터를 Milvus에 삽입합니다.
+다음은 컬렉션 스키마에 정의되지 않은 필드인 새 필드 text 입니다. 이 필드는 상위 수준에서 일반 필드로 취급할 수 있는 예약된 JSON 동적 필드에 자동으로 추가됩니다.
+from tqdm import tqdm
+
+data = []
+
+doc_embeddings = genai.embed_content(
+ model="models/text-embedding-004", content=text_lines
+)["embedding"]
+
+for i, line in enumerate(tqdm(text_lines, desc="Creating embeddings")):
+ data.append({"id": i, "vector": doc_embeddings[i], "text": line})
+
+milvus_client.insert(collection_name=collection_name, data=data)
+
+Creating embeddings: 100%|██████████| 72/72 [00:00<00:00, 468201.38it/s]
+
+
+
+
+
+{'insert_count': 72, 'ids': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71], 'cost': 0}
+
+RAG 구축
쿼리에 대한 데이터 검색
Milvus에 대해 자주 묻는 질문을 지정해 보겠습니다.
+question = "How is data stored in milvus?"
+
+컬렉션에서 해당 질문을 검색하고 시맨틱 상위 3개 일치 항목을 검색합니다.
+question_embedding = genai.embed_content(
+ model="models/text-embedding-004", content=question
+)["embedding"]
+
+search_res = milvus_client.search(
+ collection_name=collection_name,
+ data=[question_embedding],
+ limit=3, # Return top 3 results
+ search_params={"metric_type": "IP", "params": {}}, # Inner product distance
+ output_fields=["text"], # Return the text field
+)
+
+쿼리의 검색 결과를 살펴봅시다.
+import json
+
+retrieved_lines_with_distances = [
+ (res["entity"]["text"], res["distance"]) for res in search_res[0]
+]
+print(json.dumps(retrieved_lines_with_distances, indent=4))
+
+[
+ [
+ " Where does Milvus store data?\n\nMilvus deals with two types of data, inserted data and metadata. \n\nInserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).\n\nMetadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.\n\n###",
+ 0.8048275113105774
+ ],
+ [
+ "Does the query perform in memory? What are incremental data and historical data?\n\nYes. When a query request comes, Milvus searches both incremental data and historical data by loading them into memory. Incremental data are in the growing segments, which are buffered in memory before they reach the threshold to be persisted in storage engine, while historical data are from the sealed segments that are stored in the object storage. Incremental data and historical data together constitute the whole dataset to search.\n\n###",
+ 0.7574886679649353
+ ],
+ [
+ "What is the maximum dataset size Milvus can handle?\n\n \nTheoretically, the maximum dataset size Milvus can handle is determined by the hardware it is run on, specifically system memory and storage:\n\n- Milvus loads all specified collections and partitions into memory before running queries. Therefore, memory size determines the maximum amount of data Milvus can query.\n- When new entities and and collection-related schema (currently only MinIO is supported for data persistence) are added to Milvus, system storage determines the maximum allowable size of inserted data.\n\n###",
+ 0.7453608512878418
+ ]
+]
+
+LLM을 사용하여 RAG 응답 얻기
검색된 문서를 문자열 형식으로 변환합니다.
+context = "\n".join(
+ [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]
+)
+
+Lanage 모델에 대한 시스템 및 사용자 프롬프트를 정의합니다. 이 프롬프트는 Milvus에서 검색된 문서로 조립됩니다.
+SYSTEM_PROMPT = """
+Human: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.
+"""
+USER_PROMPT = f"""
+Use the following pieces of information enclosed in <context> tags to provide an answer to the question enclosed in <question> tags.
+<context>
+{context}
+</context>
+<question>
+{question}
+</question>
+"""
+
+Gemini를 사용하여 프롬프트에 따라 응답을 생성합니다.
+gemini_model = genai.GenerativeModel(
+ "gemini-1.5-flash", system_instruction=SYSTEM_PROMPT
+)
+response = gemini_model.generate_content(USER_PROMPT)
+print(response.text)
+
+Milvus stores data in two ways: Inserted data (vector data, scalar data, and collection-specific schema) is stored as an incremental log in persistent storage using object storage backends such as MinIO, AWS S3, Google Cloud Storage, Azure Blob Storage, Alibaba Cloud OSS, and Tencent Cloud Object Storage. Metadata, generated by each Milvus module, is stored in etcd.
+
+훌륭합니다! Milvus와 Gemini로 RAG 파이프라인을 성공적으로 구축했습니다.
diff --git a/localization/v2.4.x/site/ko/integrations/build_RAG_with_milvus_and_ollama.json b/localization/v2.4.x/site/ko/integrations/build_RAG_with_milvus_and_ollama.json
new file mode 100644
index 000000000..18040f76c
--- /dev/null
+++ b/localization/v2.4.x/site/ko/integrations/build_RAG_with_milvus_and_ollama.json
@@ -0,0 +1 @@
+{"codeList":["$ pip install pymilvus ollama\n","$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip\n$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs\n","from glob import glob\n\ntext_lines = []\n\nfor file_path in glob(\"milvus_docs/en/faq/*.md\", recursive=True):\n with open(file_path, \"r\") as file:\n file_text = file.read()\n\n text_lines += file_text.split(\"# \")\n","! ollama pull mxbai-embed-large\n","! ollama pull llama3.2\n","import ollama\n\n\ndef emb_text(text):\n response = ollama.embeddings(model=\"mxbai-embed-large\", prompt=text)\n return response[\"embedding\"]\n","test_embedding = emb_text(\"This is a test\")\nembedding_dim = len(test_embedding)\nprint(embedding_dim)\nprint(test_embedding[:10])\n","from pymilvus import MilvusClient\n\nmilvus_client = MilvusClient(uri=\"./milvus_demo.db\")\n\ncollection_name = \"my_rag_collection\"\n","if milvus_client.has_collection(collection_name):\n milvus_client.drop_collection(collection_name)\n","milvus_client.create_collection(\n collection_name=collection_name,\n dimension=embedding_dim,\n metric_type=\"IP\", # Inner product distance\n consistency_level=\"Strong\", # Strong consistency level\n)\n","from tqdm import tqdm\n\ndata = []\n\nfor i, line in enumerate(tqdm(text_lines, desc=\"Creating embeddings\")):\n data.append({\"id\": i, \"vector\": emb_text(line), \"text\": line})\n\nmilvus_client.insert(collection_name=collection_name, data=data)\n","question = \"How is data stored in milvus?\"\n","search_res = milvus_client.search(\n collection_name=collection_name,\n data=[\n emb_text(question)\n ], # Use the `emb_text` function to convert the question to an embedding vector\n limit=3, # Return top 3 results\n search_params={\"metric_type\": \"IP\", \"params\": {}}, # Inner product distance\n output_fields=[\"text\"], # Return the text field\n)\n","import json\n\nretrieved_lines_with_distances = [\n (res[\"entity\"][\"text\"], res[\"distance\"]) for res in search_res[0]\n]\nprint(json.dumps(retrieved_lines_with_distances, indent=4))\n","context = \"\\n\".join(\n [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]\n)\n","SYSTEM_PROMPT = \"\"\"\nHuman: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.\n\"\"\"\nUSER_PROMPT = f\"\"\"\nUse the following pieces of information enclosed in tags to provide an answer to the question enclosed in tags.\n\n{context}\n \n\n{question}\n \n\"\"\"\n","from ollama import chat\nfrom ollama import ChatResponse\n\nresponse: ChatResponse = chat(\n model=\"llama3.2\",\n messages=[\n {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n {\"role\": \"user\", \"content\": USER_PROMPT},\n ],\n)\nprint(response[\"message\"][\"content\"])\n"],"headingContent":"Build RAG with Milvus and Ollama","anchorList":[{"label":"Milvus 및 Ollama로 RAG 구축하기","href":"Build-RAG-with-Milvus-and-Ollama","type":1,"isActive":false},{"label":"준비","href":"Preparation","type":2,"isActive":false},{"label":"Milvus에 데이터 로드","href":"Load-data-into-Milvus","type":2,"isActive":false},{"label":"RAG 구축","href":"Build-RAG","type":2,"isActive":false}]}
\ No newline at end of file
diff --git a/localization/v2.4.x/site/ko/integrations/build_RAG_with_milvus_and_ollama.md b/localization/v2.4.x/site/ko/integrations/build_RAG_with_milvus_and_ollama.md
new file mode 100644
index 000000000..52ff7efad
--- /dev/null
+++ b/localization/v2.4.x/site/ko/integrations/build_RAG_with_milvus_and_ollama.md
@@ -0,0 +1,279 @@
+---
+id: build_RAG_with_milvus_and_ollama.md
+summary: 이 가이드에서는 Ollama와 Milvus를 활용하여 RAG(검색 증강 생성) 파이프라인을 효율적이고 안전하게 구축하는 방법을 보여드립니다.
+title: Milvus 및 Ollama로 RAG 구축하기
+---
+
+Milvus 및 Ollama로 RAG 구축하기
Ollama는 대규모 언어 모델(LLM)을 로컬에서 실행하고 사용자 지정하는 작업을 간소화하는 오픈 소스 플랫폼입니다. 사용자 친화적이고 클라우드가 필요 없는 환경을 제공하여 고급 기술 없이도 손쉽게 모델을 다운로드, 설치 및 상호 작용할 수 있습니다. 범용에서 도메인별에 이르기까지 사전 학습된 LLM 라이브러리가 계속 늘어나고 있어 다양한 애플리케이션에 맞게 모델을 쉽게 관리하고 사용자 지정할 수 있습니다. 또한 데이터 개인정보 보호와 유연성을 보장하여 사용자가 자신의 머신에서 AI 기반 솔루션을 완전히 미세 조정, 최적화 및 배포할 수 있도록 지원합니다.
+이 가이드에서는 Ollama와 Milvus를 활용하여 RAG(검색 증강 세대) 파이프라인을 효율적이고 안전하게 구축하는 방법을 보여드립니다.
+준비
종속성 및 환경
$ pip install pymilvus ollama
+
+
+Google Colab을 사용하는 경우 방금 설치한 종속 요소를 사용하려면 런타임을 다시 시작해야 할 수 있습니다(화면 상단의 '런타임' 메뉴를 클릭하고 드롭다운 메뉴에서 '세션 다시 시작'을 선택).
+
+데이터 준비
저희는 Milvus 문서 2.4.x의 FAQ 페이지를 RAG의 비공개 지식으로 사용하며, 이는 간단한 RAG 파이프라인을 위한 좋은 데이터 소스입니다.
+zip 파일을 다운로드하고 문서를 milvus_docs 폴더에 압축을 풉니다.
+$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
+
+--2024-11-26 21:47:19-- https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+Resolving github.com (github.com)... 140.82.112.4
+Connecting to github.com (github.com)|140.82.112.4|:443... connected.
+HTTP request sent, awaiting response... 302 Found
+Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/267273319/c52902a0-e13c-4ca7-92e0-086751098a05?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241127%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241127T024720Z&X-Amz-Expires=300&X-Amz-Signature=7808b77cbdaa7e122196bcd75a73f29f2540333a350c4830bbdf5f286e876304&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dmilvus_docs_2.4.x_en.zip&response-content-type=application%2Foctet-stream [following]
+--2024-11-26 21:47:20-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/267273319/c52902a0-e13c-4ca7-92e0-086751098a05?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241127%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241127T024720Z&X-Amz-Expires=300&X-Amz-Signature=7808b77cbdaa7e122196bcd75a73f29f2540333a350c4830bbdf5f286e876304&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dmilvus_docs_2.4.x_en.zip&response-content-type=application%2Foctet-stream
+Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.109.133, 185.199.111.133, 185.199.108.133, ...
+Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.109.133|:443... connected.
+HTTP request sent, awaiting response... 200 OK
+Length: 613094 (599K) [application/octet-stream]
+Saving to: ‘milvus_docs_2.4.x_en.zip’
+
+milvus_docs_2.4.x_e 100%[===================>] 598.72K 1.20MB/s in 0.5s
+
+2024-11-26 21:47:20 (1.20 MB/s) - ‘milvus_docs_2.4.x_en.zip’ saved [613094/613094]
+
+milvus_docs/en/faq 폴더에서 모든 마크다운 파일을 로드합니다. 각 문서에 대해 "#"를 사용하여 파일의 내용을 구분하기만 하면 마크다운 파일의 각 주요 부분의 내용을 대략적으로 구분할 수 있습니다.
+from glob import glob
+
+text_lines = []
+
+for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
+ with open(file_path, "r") as file:
+ file_text = file.read()
+
+ text_lines += file_text.split("# ")
+
+LLM 및 임베딩 모델 준비
Ollama는 LLM 기반 작업과 임베딩 생성 모두에 대해 여러 모델을 지원하므로 검색 증강 생성(RAG) 애플리케이션을 쉽게 개발할 수 있습니다. 이 설정의 경우:
+
+- 텍스트 생성 작업에는 Llama 3.2(3B)를 LLM으로 사용합니다.
+- 임베딩 생성에는 의미론적 유사성에 최적화된 334M 파라미터 모델인 mxbai-embed-large를 사용하겠습니다.
+
+시작하기 전에 두 모델을 모두 로컬로 가져왔는지 확인하세요:
+! ollama pull mxbai-embed-large
+
+[?25lpulling manifest ⠋ [?25h[?25l[2K[1Gpulling manifest ⠙ [?25h[?25l[2K[1Gpulling manifest ⠹ [?25h[?25l[2K[1Gpulling manifest ⠸ [?25h[?25l[2K[1Gpulling manifest ⠼ [?25h[?25l[2K[1Gpulling manifest ⠴ [?25h[?25l[2K[1Gpulling manifest
+pulling 819c2adf5ce6... 100% ▕████████████████▏ 669 MB
+pulling c71d239df917... 100% ▕████████████████▏ 11 KB
+pulling b837481ff855... 100% ▕████████████████▏ 16 B
+pulling 38badd946f91... 100% ▕████████████████▏ 408 B
+verifying sha256 digest
+writing manifest
+success [?25h
+
+! ollama pull llama3.2
+
+[?25lpulling manifest ⠋ [?25h[?25l[2K[1Gpulling manifest ⠙ [?25h[?25l[2K[1Gpulling manifest ⠹ [?25h[?25l[2K[1Gpulling manifest ⠸ [?25h[?25l[2K[1Gpulling manifest ⠼ [?25h[?25l[2K[1Gpulling manifest ⠴ [?25h[?25l[2K[1Gpulling manifest
+pulling dde5aa3fc5ff... 100% ▕████████████████▏ 2.0 GB
+pulling 966de95ca8a6... 100% ▕████████████████▏ 1.4 KB
+pulling fcc5a6bec9da... 100% ▕████████████████▏ 7.7 KB
+pulling a70ff7e570d9... 100% ▕████████████████▏ 6.0 KB
+pulling 56bb8bd477a5... 100% ▕████████████████▏ 96 B
+pulling 34bb5ab01051... 100% ▕████████████████▏ 561 B
+verifying sha256 digest
+writing manifest
+success [?25h
+
+이 모델이 준비되면 LLM 기반 생성 및 임베딩 기반 검색 워크플로우를 구현할 수 있습니다.
+import ollama
+
+
+def emb_text(text):
+ response = ollama.embeddings(model="mxbai-embed-large", prompt=text)
+ return response["embedding"]
+
+테스트 임베딩을 생성하고 해당 차원과 처음 몇 개의 요소를 인쇄합니다.
+test_embedding = emb_text("This is a test")
+embedding_dim = len(test_embedding)
+print(embedding_dim)
+print(test_embedding[:10])
+
+1024
+[0.23276396095752716, 0.4257211685180664, 0.19724100828170776, 0.46120673418045044, -0.46039995551109314, -0.1413791924715042, -0.18261606991291046, -0.07602324336767197, 0.39991313219070435, 0.8337644338607788]
+
+Milvus에 데이터 로드
컬렉션 생성
from pymilvus import MilvusClient
+
+milvus_client = MilvusClient(uri="./milvus_demo.db")
+
+collection_name = "my_rag_collection"
+
+
+MilvusClient 의 인수를 사용합니다:
+
+uri 을 로컬 파일(예:./milvus.db)로 설정하는 것이 가장 편리한 방법인데, Milvus Lite를 자동으로 활용하여 모든 데이터를 이 파일에 저장하기 때문입니다.- 데이터의 규모가 큰 경우, 도커나 쿠버네티스에 더 성능이 좋은 Milvus 서버를 설정할 수 있습니다. 이 설정에서는 서버 URL(예:
http://localhost:19530)을 uri 으로 사용하세요.
+- 밀버스의 완전 관리형 클라우드 서비스인 질리즈 클라우드를 사용하려면, 질리즈 클라우드의 퍼블릭 엔드포인트와 API 키에 해당하는
uri 와 token 을 조정하세요.
+
+
+컬렉션이 이미 존재하는지 확인하고 존재한다면 삭제합니다.
+if milvus_client.has_collection(collection_name):
+ milvus_client.drop_collection(collection_name)
+
+지정된 파라미터로 새 컬렉션을 생성합니다.
+필드 정보를 지정하지 않으면 기본 키인 id 필드와 벡터 데이터를 저장할 vector 필드가 자동으로 생성됩니다. 예약된 JSON 필드는 스키마에 정의되지 않은 필드와 그 값을 저장하는 데 사용됩니다.
+milvus_client.create_collection(
+ collection_name=collection_name,
+ dimension=embedding_dim,
+ metric_type="IP", # Inner product distance
+ consistency_level="Strong", # Strong consistency level
+)
+
+데이터 삽입
텍스트 줄을 반복하여 임베딩을 만든 다음 데이터를 Milvus에 삽입합니다.
+다음은 컬렉션 스키마에 정의되지 않은 필드인 새 필드 text 입니다. 이 필드는 상위 수준에서 일반 필드로 취급할 수 있는 예약된 JSON 동적 필드에 자동으로 추가됩니다.
+from tqdm import tqdm
+
+data = []
+
+for i, line in enumerate(tqdm(text_lines, desc="Creating embeddings")):
+ data.append({"id": i, "vector": emb_text(line), "text": line})
+
+milvus_client.insert(collection_name=collection_name, data=data)
+
+Creating embeddings: 100%|██████████| 72/72 [00:03<00:00, 22.56it/s]
+
+
+
+
+
+{'insert_count': 72, 'ids': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71], 'cost': 0}
+
+RAG 구축
쿼리에 대한 데이터 검색
Milvus에 대해 자주 묻는 질문을 지정해 보겠습니다.
+question = "How is data stored in milvus?"
+
+컬렉션에서 해당 질문을 검색하고 시맨틱 상위 3개 일치 항목을 검색합니다.
+search_res = milvus_client.search(
+ collection_name=collection_name,
+ data=[
+ emb_text(question)
+ ], # Use the `emb_text` function to convert the question to an embedding vector
+ limit=3, # Return top 3 results
+ search_params={"metric_type": "IP", "params": {}}, # Inner product distance
+ output_fields=["text"], # Return the text field
+)
+
+쿼리의 검색 결과를 살펴봅시다.
+import json
+
+retrieved_lines_with_distances = [
+ (res["entity"]["text"], res["distance"]) for res in search_res[0]
+]
+print(json.dumps(retrieved_lines_with_distances, indent=4))
+
+[
+ [
+ " Where does Milvus store data?\n\nMilvus deals with two types of data, inserted data and metadata. \n\nInserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).\n\nMetadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.\n\n###",
+ 231.9398193359375
+ ],
+ [
+ "How does Milvus flush data?\n\nMilvus returns success when inserted data are loaded to the message queue. However, the data are not yet flushed to the disk. Then Milvus' data node writes the data in the message queue to persistent storage as incremental logs. If `flush()` is called, the data node is forced to write all data in the message queue to persistent storage immediately.\n\n###",
+ 226.48316955566406
+ ],
+ [
+ "What is the maximum dataset size Milvus can handle?\n\n \nTheoretically, the maximum dataset size Milvus can handle is determined by the hardware it is run on, specifically system memory and storage:\n\n- Milvus loads all specified collections and partitions into memory before running queries. Therefore, memory size determines the maximum amount of data Milvus can query.\n- When new entities and and collection-related schema (currently only MinIO is supported for data persistence) are added to Milvus, system storage determines the maximum allowable size of inserted data.\n\n###",
+ 210.60745239257812
+ ]
+]
+
+LLM을 사용하여 RAG 응답 얻기
검색된 문서를 문자열 형식으로 변환합니다.
+context = "\n".join(
+ [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]
+)
+
+Lanage 모델에 대한 시스템 및 사용자 프롬프트를 정의합니다. 이 프롬프트는 Milvus에서 검색된 문서로 조립됩니다.
+SYSTEM_PROMPT = """
+Human: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.
+"""
+USER_PROMPT = f"""
+Use the following pieces of information enclosed in <context> tags to provide an answer to the question enclosed in <question> tags.
+<context>
+{context}
+</context>
+<question>
+{question}
+</question>
+"""
+
+Ollama에서 제공하는 llama3.2 모델을 사용하여 프롬프트에 따라 응답을 생성합니다.
+from ollama import chat
+from ollama import ChatResponse
+
+response: ChatResponse = chat(
+ model="llama3.2",
+ messages=[
+ {"role": "system", "content": SYSTEM_PROMPT},
+ {"role": "user", "content": USER_PROMPT},
+ ],
+)
+print(response["message"]["content"])
+
+According to the provided context, data in Milvus is stored in two types:
+
+1. **Inserted data**: Storing data in persistent storage as incremental log. It supports multiple object storage backends such as MinIO, AWS S3, Google Cloud Storage (GCS), Azure Blob Storage, Alibaba Cloud OSS, and Tencent Cloud Object Storage.
+
+2. **Metadata**: Generated within Milvus and stored in etcd.
+
+훌륭합니다! Milvus와 Ollama를 사용하여 RAG 파이프라인을 성공적으로 구축했습니다.
diff --git a/localization/v2.4.x/site/ko/integrations/milvus_rag_with_dynamiq.json b/localization/v2.4.x/site/ko/integrations/milvus_rag_with_dynamiq.json
new file mode 100644
index 000000000..73f0099de
--- /dev/null
+++ b/localization/v2.4.x/site/ko/integrations/milvus_rag_with_dynamiq.json
@@ -0,0 +1 @@
+{"codeList":["$ pip install dynamiq pymilvus\n","import os\n\nos.environ[\"OPENAI_API_KEY\"] = \"sk-***********\"\n","# Importing necessary libraries for the workflow\nfrom io import BytesIO\nfrom dynamiq import Workflow\nfrom dynamiq.nodes import InputTransformer\nfrom dynamiq.connections import (\n OpenAI as OpenAIConnection,\n Milvus as MilvusConnection,\n MilvusDeploymentType,\n)\nfrom dynamiq.nodes.converters import PyPDFConverter\nfrom dynamiq.nodes.splitters.document import DocumentSplitter\nfrom dynamiq.nodes.embedders import OpenAIDocumentEmbedder\nfrom dynamiq.nodes.writers import MilvusDocumentWriter\n\n# Initialize the workflow\nrag_wf = Workflow()\n","converter = PyPDFConverter(document_creation_mode=\"one-doc-per-page\")\nconverter_added = rag_wf.flow.add_nodes(\n converter\n) # Add node to the DAG (Directed Acyclic Graph)\n","document_splitter = DocumentSplitter(\n split_by=\"sentence\", # Splits documents into sentences\n split_length=10,\n split_overlap=1,\n input_transformer=InputTransformer(\n selector={\n \"documents\": f\"${[converter.id]}.output.documents\",\n },\n ),\n).depends_on(\n converter\n) # Set dependency on the PDF converter\nsplitter_added = rag_wf.flow.add_nodes(document_splitter) # Add to the DAG\n","embedder = OpenAIDocumentEmbedder(\n connection=OpenAIConnection(api_key=os.environ[\"OPENAI_API_KEY\"]),\n input_transformer=InputTransformer(\n selector={\n \"documents\": f\"${[document_splitter.id]}.output.documents\",\n },\n ),\n).depends_on(\n document_splitter\n) # Set dependency on the splitter\ndocument_embedder_added = rag_wf.flow.add_nodes(embedder) # Add to the DAG\n","vector_store = (\n MilvusDocumentWriter(\n connection=MilvusConnection(\n deployment_type=MilvusDeploymentType.FILE, uri=\"./milvus.db\"\n ),\n index_name=\"my_milvus_collection\",\n dimension=1536,\n create_if_not_exist=True,\n metric_type=\"COSINE\",\n )\n .inputs(documents=embedder.outputs.documents) # Connect to embedder output\n .depends_on(embedder) # Set dependency on the embedder\n)\nmilvus_writer_added = rag_wf.flow.add_nodes(vector_store) # Add to the DAG\n","file_paths = [\"./pdf_files/WhatisMilvus.pdf\"]\ninput_data = {\n \"files\": [BytesIO(open(path, \"rb\").read()) for path in file_paths],\n \"metadata\": [{\"filename\": path} for path in file_paths],\n}\n\n# Run the workflow with the prepared input data\ninserted_data = rag_wf.run(input_data=input_data)\n","from dynamiq import Workflow\nfrom dynamiq.connections import (\n OpenAI as OpenAIConnection,\n Milvus as MilvusConnection,\n MilvusDeploymentType,\n)\nfrom dynamiq.nodes.embedders import OpenAITextEmbedder\nfrom dynamiq.nodes.retrievers import MilvusDocumentRetriever\nfrom dynamiq.nodes.llms import OpenAI\nfrom dynamiq.prompts import Message, Prompt\n\n# Initialize the workflow\nretrieval_wf = Workflow()\n","# Establish OpenAI connection\nopenai_connection = OpenAIConnection(api_key=os.environ[\"OPENAI_API_KEY\"])\n\n# Define the text embedder node\nembedder = OpenAITextEmbedder(\n connection=openai_connection,\n model=\"text-embedding-3-small\",\n)\n\n# Add the embedder node to the workflow\nembedder_added = retrieval_wf.flow.add_nodes(embedder)\n","document_retriever = (\n MilvusDocumentRetriever(\n connection=MilvusConnection(\n deployment_type=MilvusDeploymentType.FILE, uri=\"./milvus.db\"\n ),\n index_name=\"my_milvus_collection\",\n dimension=1536,\n top_k=5,\n )\n .inputs(embedding=embedder.outputs.embedding) # Connect to embedder output\n .depends_on(embedder) # Dependency on the embedder node\n)\n\n# Add the retriever node to the workflow\nmilvus_retriever_added = retrieval_wf.flow.add_nodes(document_retriever)\n","# Define the prompt template for the LLM\nprompt_template = \"\"\"\nPlease answer the question based on the provided context.\n\nQuestion: {{ query }}\n\nContext:\n{% for document in documents %}\n- {{ document.content }}\n{% endfor %}\n\"\"\"\n\n# Create the prompt object\nprompt = Prompt(messages=[Message(content=prompt_template, role=\"user\")])\n","answer_generator = (\n OpenAI(\n connection=openai_connection,\n model=\"gpt-4o\",\n prompt=prompt,\n )\n .inputs(\n documents=document_retriever.outputs.documents,\n query=embedder.outputs.query,\n )\n .depends_on(\n [document_retriever, embedder]\n ) # Dependencies on retriever and embedder\n)\n\n# Add the answer generator node to the workflow\nanswer_generator_added = retrieval_wf.flow.add_nodes(answer_generator)\n","# Run the workflow with a sample query\nsample_query = \"What is the Advanced Search Algorithms in Milvus?\"\n\nresult = retrieval_wf.run(input_data={\"query\": sample_query})\n\nanswer = result.output.get(answer_generator.id).get(\"output\", {}).get(\"content\")\nprint(answer)\n"],"headingContent":"Getting Started with Dynamiq and Milvus","anchorList":[{"label":"Dynamiq 및 Milvus 시작하기","href":"Getting-Started-with-Dynamiq-and-Milvus","type":1,"isActive":false},{"label":"준비 사항","href":"Preparation","type":2,"isActive":false},{"label":"RAG - 문서 색인 흐름","href":"RAG---Document-Indexing-Flow","type":2,"isActive":false},{"label":"RAG 문서 검색 흐름","href":"RAG-Document-Retrieval-Flow","type":2,"isActive":false}]}
\ No newline at end of file
diff --git a/localization/v2.4.x/site/ko/integrations/milvus_rag_with_dynamiq.md b/localization/v2.4.x/site/ko/integrations/milvus_rag_with_dynamiq.md
new file mode 100644
index 000000000..982ac45e1
--- /dev/null
+++ b/localization/v2.4.x/site/ko/integrations/milvus_rag_with_dynamiq.md
@@ -0,0 +1,337 @@
+---
+id: milvus_rag_with_dynamiq.md
+summary: >-
+ 이 튜토리얼에서는 RAG 워크플로우를 위해 특별히 제작된 고성능 벡터 데이터베이스인 Milvus와 함께 Dynamiq을 원활하게 사용하는
+ 방법을 살펴봅니다. Milvus는 벡터 임베딩의 효율적인 저장, 인덱싱 및 검색에 탁월하여 빠르고 정확한 컨텍스트 데이터 액세스가 필요한
+ AI 시스템에 없어서는 안 될 구성 요소입니다.
+title: Dynamiq 및 Milvus 시작하기
+---
+
+Dynamiq 및 Milvus 시작하기
Dynamiq은 AI 기반 애플리케이션의 개발을 간소화하는 강력한 Gen AI 프레임워크입니다. 검색 증강 생성(RAG) 및 대규모 언어 모델(LLM) 에이전트에 대한 강력한 지원을 통해 개발자가 쉽고 효율적으로 지능적이고 동적인 시스템을 만들 수 있도록 도와주는 Dynamiq입니다.
+이 튜토리얼에서는 RAG 워크플로우를 위해 특별히 제작된 고성능 벡터 데이터베이스인 Milvus와 함께 Dynamiq을 원활하게 사용하는 방법을 살펴봅니다. Milvus는 벡터 임베딩의 효율적인 저장, 인덱싱 및 검색에 탁월하여 빠르고 정확한 컨텍스트 데이터 액세스가 필요한 AI 시스템에 없어서는 안 될 구성 요소입니다.
+이 단계별 가이드에서는 두 가지 핵심 RAG 워크플로우를 다룹니다:
+
+문서 색인 흐름: 입력 파일(예: PDF)을 처리하고, 그 콘텐츠를 벡터 임베딩으로 변환하여 Milvus에 저장하는 방법을 알아보세요. Milvus의 고성능 인덱싱 기능을 활용하면 데이터를 신속하게 검색할 수 있습니다.
문서 검색 흐름: Milvus에서 관련 문서 임베딩을 쿼리하고 이를 사용하여 Dynamiq의 LLM 에이전트로 통찰력 있는 컨텍스트 인식 응답을 생성하여 원활한 AI 기반 사용자 환경을 구축하는 방법을 알아보세요.
+이 튜토리얼을 마치면 Milvus와 Dynamiq이 어떻게 협력하여 필요에 맞게 확장 가능한 상황 인식 AI 시스템을 구축하는지 확실히 이해할 수 있을 것입니다.
+준비 사항
필요한 라이브러리 다운로드
$ pip install dynamiq pymilvus
+
+
+Google Colab을 사용하는 경우 방금 설치한 종속 요소를 사용하려면 런타임을 다시 시작해야 할 수 있습니다(화면 상단의 '런타임' 메뉴를 클릭하고 드롭다운 메뉴에서 '세션 다시 시작'을 선택).
+
+LLM 에이전트 구성
이 예제에서는 OpenAI를 LLM으로 사용하겠습니다. 환경 변수로 OPENAI_API_KEY API 키를 준비해야 합니다.
+import os
+
+os.environ["OPENAI_API_KEY"] = "sk-***********"
+
+RAG - 문서 색인 흐름
이 튜토리얼에서는 Milvus를 벡터 데이터베이스로 사용하여 문서를 색인하는 검색 증강 생성(RAG) 워크플로우를 보여드립니다. 이 워크플로에서는 입력 PDF 파일을 가져와서 더 작은 덩어리로 처리하고, OpenAI의 임베딩 모델을 사용해 벡터 임베딩을 생성한 다음, 효율적인 검색을 위해 임베딩을 Milvus 컬렉션에 저장합니다.
+이 워크플로우가 끝나면 시맨틱 검색과 질문 답변과 같은 향후 RAG 작업을 지원하는 확장 가능하고 효율적인 문서 색인 시스템을 갖추게 됩니다.
+필요한 라이브러리 가져오기 및 워크플로우 초기화
# Importing necessary libraries for the workflow
+from io import BytesIO
+from dynamiq import Workflow
+from dynamiq.nodes import InputTransformer
+from dynamiq.connections import (
+ OpenAI as OpenAIConnection,
+ Milvus as MilvusConnection,
+ MilvusDeploymentType,
+)
+from dynamiq.nodes.converters import PyPDFConverter
+from dynamiq.nodes.splitters.document import DocumentSplitter
+from dynamiq.nodes.embedders import OpenAIDocumentEmbedder
+from dynamiq.nodes.writers import MilvusDocumentWriter
+
+# Initialize the workflow
+rag_wf = Workflow()
+
+PDF 변환기 노드 정의
converter = PyPDFConverter(document_creation_mode="one-doc-per-page")
+converter_added = rag_wf.flow.add_nodes(
+ converter
+) # Add node to the DAG (Directed Acyclic Graph)
+
+문서 분할기 노드 정의
document_splitter = DocumentSplitter(
+ split_by="sentence", # Splits documents into sentences
+ split_length=10,
+ split_overlap=1,
+ input_transformer=InputTransformer(
+ selector={
+ "documents": f"${[converter.id]}.output.documents",
+ },
+ ),
+).depends_on(
+ converter
+) # Set dependency on the PDF converter
+splitter_added = rag_wf.flow.add_nodes(document_splitter) # Add to the DAG
+
+임베딩 노드 정의
embedder = OpenAIDocumentEmbedder(
+ connection=OpenAIConnection(api_key=os.environ["OPENAI_API_KEY"]),
+ input_transformer=InputTransformer(
+ selector={
+ "documents": f"${[document_splitter.id]}.output.documents",
+ },
+ ),
+).depends_on(
+ document_splitter
+) # Set dependency on the splitter
+document_embedder_added = rag_wf.flow.add_nodes(embedder) # Add to the DAG
+
+Milvus 벡터 저장소 노드 정의하기
vector_store = (
+ MilvusDocumentWriter(
+ connection=MilvusConnection(
+ deployment_type=MilvusDeploymentType.FILE, uri="./milvus.db"
+ ),
+ index_name="my_milvus_collection",
+ dimension=1536,
+ create_if_not_exist=True,
+ metric_type="COSINE",
+ )
+ .inputs(documents=embedder.outputs.documents) # Connect to embedder output
+ .depends_on(embedder) # Set dependency on the embedder
+)
+milvus_writer_added = rag_wf.flow.add_nodes(vector_store) # Add to the DAG
+
+2024-11-19 22:14:03 - WARNING - Environment variable 'MILVUS_API_TOKEN' not found
+2024-11-19 22:14:03 - INFO - Pass in the local path ./milvus.db, and run it using milvus-lite
+2024-11-19 22:14:04 - DEBUG - Created new connection using: 0bef2849fdb1458a85df8bb9dd27f51d
+2024-11-19 22:14:04 - INFO - Collection my_milvus_collection does not exist. Creating a new collection.
+2024-11-19 22:14:04 - DEBUG - Successfully created collection: my_milvus_collection
+2024-11-19 22:14:05 - DEBUG - Successfully created an index on collection: my_milvus_collection
+2024-11-19 22:14:05 - DEBUG - Successfully created an index on collection: my_milvus_collection
+
+
+Milvus는 다양한 사용 사례에 맞는 두 가지 배포 유형을 제공합니다:
+
+- MilvusDeploymentType.FILE
+
+
+- 로컬 프로토타이핑 또는 소규모 데이터 저장에 이상적입니다.
+uri 을 로컬 파일 경로(예: ./milvus.db)로 설정하면 지정된 파일에 모든 데이터를 자동으로 저장하는 Milvus Lite를 활용할 수 있습니다.- 이 옵션은 빠른 설정과 실험을 위한 편리한 옵션입니다.
+
+
+- MilvusDeploymentType.HOST
+
+
+백만 개 이상의 벡터 관리와 같은 대규모 데이터 시나리오를 위해 설계되었습니다.
+자체 호스팅 서버
+
+- 도커 또는 쿠버네티스를 사용하여 고성능 Milvus 서버를 배포합니다.
+- 서버의 주소와 포트를
uri (예: http://localhost:19530)로 구성합니다.
+- 인증이 활성화된 경우:
+<your_username>:<your_password> 을 token 으로 입력합니다.- 인증을 사용하지 않는 경우:
+token 을 설정하지 않은 상태로 둡니다.
+질리즈 클라우드(관리형 서비스)
+
+- 클라우드 기반의 완전 관리형 밀버스 환경을 사용하려면 Zilliz Cloud를 사용하세요.
+- 질리즈 클라우드 콘솔에서 제공하는 퍼블릭 엔드포인트와 API 키에 따라
uri 및 token 을 설정합니다.
+
+
+입력 데이터 정의 및 워크플로우 실행
file_paths = ["./pdf_files/WhatisMilvus.pdf"]
+input_data = {
+ "files": [BytesIO(open(path, "rb").read()) for path in file_paths],
+ "metadata": [{"filename": path} for path in file_paths],
+}
+
+# Run the workflow with the prepared input data
+inserted_data = rag_wf.run(input_data=input_data)
+
+/var/folders/09/d0hx80nj35sb5hxb5cpc1q180000gn/T/ipykernel_31319/3145804345.py:4: ResourceWarning: unclosed file <_io.BufferedReader name='./pdf_files/WhatisMilvus.pdf'>
+ BytesIO(open(path, "rb").read()) for path in file_paths
+ResourceWarning: Enable tracemalloc to get the object allocation traceback
+2024-11-19 22:14:09 - INFO - Workflow 87878444-6a3d-43f3-ae32-0127564a959f: execution started.
+2024-11-19 22:14:09 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution started.
+2024-11-19 22:14:09 - INFO - Node PyPDF File Converter - 6eb42b1f-7637-407b-a3ac-4167bcf3b5c4: execution started.
+2024-11-19 22:14:09 - INFO - Node PyPDF File Converter - 6eb42b1f-7637-407b-a3ac-4167bcf3b5c4: execution succeeded in 58ms.
+2024-11-19 22:14:09 - INFO - Node DocumentSplitter - 5baed580-6de0-4dcd-bace-d7d947ab6c7f: execution started.
+/Users/jinhonglin/anaconda3/envs/myenv/lib/python3.11/site-packages/websockets/legacy/__init__.py:6: DeprecationWarning: websockets.legacy is deprecated; see https://websockets.readthedocs.io/en/stable/howto/upgrade.html for upgrade instructions
+ warnings.warn( # deprecated in 14.0 - 2024-11-09
+/Users/jinhonglin/anaconda3/envs/myenv/lib/python3.11/site-packages/pydantic/fields.py:804: PydanticDeprecatedSince20: Using extra keyword arguments on `Field` is deprecated and will be removed. Use `json_schema_extra` instead. (Extra keys: 'is_accessible_to_agent'). Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/
+ warn(
+2024-11-19 22:14:09 - INFO - Node DocumentSplitter - 5baed580-6de0-4dcd-bace-d7d947ab6c7f: execution succeeded in 104ms.
+2024-11-19 22:14:09 - INFO - Node OpenAIDocumentEmbedder - 91928f67-a00f-48f6-a864-f6e21672ec7e: execution started.
+2024-11-19 22:14:09 - INFO - Node OpenAIDocumentEmbedder - d30a4cdc-0fab-4aff-b2e5-6161a62cb6fd: execution started.
+2024-11-19 22:14:10 - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
+2024-11-19 22:14:10 - INFO - Node OpenAIDocumentEmbedder - d30a4cdc-0fab-4aff-b2e5-6161a62cb6fd: execution succeeded in 724ms.
+2024-11-19 22:14:10 - INFO - Node MilvusDocumentWriter - dddab4cc-1dae-4e7e-9101-1ec353f530da: execution started.
+2024-11-19 22:14:10 - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
+2024-11-19 22:14:10 - INFO - Node MilvusDocumentWriter - dddab4cc-1dae-4e7e-9101-1ec353f530da: execution succeeded in 66ms.
+2024-11-19 22:14:10 - INFO - Node OpenAIDocumentEmbedder - 91928f67-a00f-48f6-a864-f6e21672ec7e: execution succeeded in 961ms.
+2024-11-19 22:14:10 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution succeeded in 1.3s.
+2024-11-19 22:14:10 - INFO - Workflow 87878444-6a3d-43f3-ae32-0127564a959f: execution succeeded in 1.3s.
+
+이 워크플로우를 통해 Milvus를 벡터 데이터베이스로, OpenAI의 의미 표현을 위한 임베딩 모델을 사용하여 문서 인덱싱 파이프라인을 성공적으로 구현했습니다. 이 설정은 빠르고 정확한 벡터 기반 검색을 가능하게 하여 시맨틱 검색, 문서 검색, 문맥 AI 기반 상호 작용과 같은 RAG 워크플로우의 토대를 형성합니다.
+Milvus의 확장 가능한 스토리지 기능과 Dynamiq의 오케스트레이션을 통해 이 솔루션은 프로토타이핑과 대규모 프로덕션 배포에 모두 사용할 수 있습니다. 이제 이 파이프라인을 확장하여 검색 기반 질문 답변 또는 AI 기반 콘텐츠 생성과 같은 추가 작업을 포함할 수 있습니다.
+RAG 문서 검색 흐름
이 튜토리얼에서는 검색 증강 생성(RAG) 문서 검색 워크플로우를 구현합니다. 이 워크플로에서는 사용자 쿼리를 받아 벡터 임베딩을 생성하고, Milvus 벡터 데이터베이스에서 가장 관련성이 높은 문서를 검색한 다음, 검색된 문서를 기반으로 LLM(대규모 언어 모델)을 사용하여 문맥을 인식하는 상세한 답변을 생성합니다.
+이 워크플로우를 따르면 벡터 기반 문서 검색의 강력한 기능과 OpenAI의 고급 LLM의 기능을 결합한 시맨틱 검색 및 질문 답변을 위한 엔드투엔드 솔루션을 만들 수 있습니다. 이 접근 방식을 사용하면 문서 데이터베이스에 저장된 지식을 활용하여 사용자 쿼리에 효율적이고 지능적으로 응답할 수 있습니다.
+필요한 라이브러리 가져오기 및 워크플로우 초기화
from dynamiq import Workflow
+from dynamiq.connections import (
+ OpenAI as OpenAIConnection,
+ Milvus as MilvusConnection,
+ MilvusDeploymentType,
+)
+from dynamiq.nodes.embedders import OpenAITextEmbedder
+from dynamiq.nodes.retrievers import MilvusDocumentRetriever
+from dynamiq.nodes.llms import OpenAI
+from dynamiq.prompts import Message, Prompt
+
+# Initialize the workflow
+retrieval_wf = Workflow()
+
+OpenAI 연결 및 텍스트 임베더 정의
# Establish OpenAI connection
+openai_connection = OpenAIConnection(api_key=os.environ["OPENAI_API_KEY"])
+
+# Define the text embedder node
+embedder = OpenAITextEmbedder(
+ connection=openai_connection,
+ model="text-embedding-3-small",
+)
+
+# Add the embedder node to the workflow
+embedder_added = retrieval_wf.flow.add_nodes(embedder)
+
+Milvus 문서 리트리버 정의
document_retriever = (
+ MilvusDocumentRetriever(
+ connection=MilvusConnection(
+ deployment_type=MilvusDeploymentType.FILE, uri="./milvus.db"
+ ),
+ index_name="my_milvus_collection",
+ dimension=1536,
+ top_k=5,
+ )
+ .inputs(embedding=embedder.outputs.embedding) # Connect to embedder output
+ .depends_on(embedder) # Dependency on the embedder node
+)
+
+# Add the retriever node to the workflow
+milvus_retriever_added = retrieval_wf.flow.add_nodes(document_retriever)
+
+2024-11-19 22:14:19 - WARNING - Environment variable 'MILVUS_API_TOKEN' not found
+2024-11-19 22:14:19 - INFO - Pass in the local path ./milvus.db, and run it using milvus-lite
+2024-11-19 22:14:19 - DEBUG - Created new connection using: 98d1132773af4298a894ad5925845fd2
+2024-11-19 22:14:19 - INFO - Collection my_milvus_collection already exists. Skipping creation.
+
+프롬프트 템플릿 정의
# Define the prompt template for the LLM
+prompt_template = """
+Please answer the question based on the provided context.
+
+Question: {{ query }}
+
+Context:
+{% for document in documents %}
+- {{ document.content }}
+{% endfor %}
+"""
+
+# Create the prompt object
+prompt = Prompt(messages=[Message(content=prompt_template, role="user")])
+
+답변 생성기 정의하기
answer_generator = (
+ OpenAI(
+ connection=openai_connection,
+ model="gpt-4o",
+ prompt=prompt,
+ )
+ .inputs(
+ documents=document_retriever.outputs.documents,
+ query=embedder.outputs.query,
+ )
+ .depends_on(
+ [document_retriever, embedder]
+ ) # Dependencies on retriever and embedder
+)
+
+# Add the answer generator node to the workflow
+answer_generator_added = retrieval_wf.flow.add_nodes(answer_generator)
+
+워크플로 실행하기
# Run the workflow with a sample query
+sample_query = "What is the Advanced Search Algorithms in Milvus?"
+
+result = retrieval_wf.run(input_data={"query": sample_query})
+
+answer = result.output.get(answer_generator.id).get("output", {}).get("content")
+print(answer)
+
+2024-11-19 22:14:22 - INFO - Workflow f4a073fb-dfb6-499c-8cac-5710a7ad6d47: execution started.
+2024-11-19 22:14:22 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution started.
+2024-11-19 22:14:22 - INFO - Node OpenAITextEmbedder - 47afb0bc-cf96-429d-b58f-11b6c935fec3: execution started.
+2024-11-19 22:14:23 - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
+2024-11-19 22:14:23 - INFO - Node OpenAITextEmbedder - 47afb0bc-cf96-429d-b58f-11b6c935fec3: execution succeeded in 474ms.
+2024-11-19 22:14:23 - INFO - Node MilvusDocumentRetriever - 51c8311b-4837-411f-ba42-21e28239a2ee: execution started.
+2024-11-19 22:14:23 - INFO - Node MilvusDocumentRetriever - 51c8311b-4837-411f-ba42-21e28239a2ee: execution succeeded in 23ms.
+2024-11-19 22:14:23 - INFO - Node LLM - ac722325-bece-453f-a2ed-135b0749ee7a: execution started.
+2024-11-19 22:14:24 - INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
+2024-11-19 22:14:24 - INFO - Node LLM - ac722325-bece-453f-a2ed-135b0749ee7a: execution succeeded in 1.8s.
+2024-11-19 22:14:25 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution succeeded in 2.4s.
+2024-11-19 22:14:25 - INFO - Workflow f4a073fb-dfb6-499c-8cac-5710a7ad6d47: execution succeeded in 2.4s.
+
+
+The advanced search algorithms in Milvus include a variety of in-memory and on-disk indexing/search algorithms such as IVF (Inverted File), HNSW (Hierarchical Navigable Small World), and DiskANN. These algorithms have been deeply optimized to enhance performance, delivering 30%-70% better performance compared to popular implementations like FAISS and HNSWLib. These optimizations are part of Milvus's design to ensure high efficiency and scalability in handling vector data.
+
diff --git a/localization/v2.4.x/site/ko/menuStructure/ko.json b/localization/v2.4.x/site/ko/menuStructure/ko.json
index 4111d32a3..66afcfbbc 100644
--- a/localization/v2.4.x/site/ko/menuStructure/ko.json
+++ b/localization/v2.4.x/site/ko/menuStructure/ko.json
@@ -1344,6 +1344,24 @@
"id": "build_RAG_with_milvus_and_siliconflow.md",
"order": 4,
"children": []
+ },
+ {
+ "label": "삼바노바",
+ "id": "use_milvus_with_sambanova.md",
+ "order": 5,
+ "children": []
+ },
+ {
+ "label": "쌍둥이자리",
+ "id": "build_RAG_with_milvus_and_gemini.md",
+ "order": 6,
+ "children": []
+ },
+ {
+ "label": "Ollama",
+ "id": "build_RAG_with_milvus_and_ollama.md",
+ "order": 7,
+ "children": []
}
]
},
@@ -1433,15 +1451,15 @@
"children": []
},
{
- "label": "삼바노바",
- "id": "use_milvus_with_sambanova.md",
+ "label": "PrivateGPT",
+ "id": "use_milvus_in_private_gpt.md",
"order": 10,
"children": []
},
{
- "label": "PrivateGPT",
- "id": "use_milvus_in_private_gpt.md",
- "order": 8,
+ "label": "Dynamiq",
+ "id": "milvus_rag_with_dynamiq.md",
+ "order": 11,
"children": []
}
]
@@ -1493,6 +1511,12 @@
"id": "knowledge_table_with_milvus.md",
"order": 2,
"children": []
+ },
+ {
+ "label": "Cognee",
+ "id": "build_RAG_with_milvus_and_cognee.md",
+ "order": 3,
+ "children": []
}
]
},
diff --git a/localization/v2.4.x/site/pt/integrations/build_RAG_with_milvus_and_cognee.json b/localization/v2.4.x/site/pt/integrations/build_RAG_with_milvus_and_cognee.json
new file mode 100644
index 000000000..4d9e8ceff
--- /dev/null
+++ b/localization/v2.4.x/site/pt/integrations/build_RAG_with_milvus_and_cognee.json
@@ -0,0 +1 @@
+{"codeList":["$ pip install pymilvus git+https://github.com/topoteretes/cognee.git\n","import os\n\nimport cognee\n\ncognee.config.set_llm_api_key(\"YOUR_OPENAI_API_KEY\")\n\n\nos.environ[\"VECTOR_DB_PROVIDER\"] = \"milvus\"\nos.environ[\"VECTOR_DB_URL\"] = \"./milvus.db\"\n","$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip\n$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs\n","from glob import glob\n\ntext_lines = []\n\nfor file_path in glob(\"milvus_docs/en/faq/*.md\", recursive=True):\n with open(file_path, \"r\") as file:\n file_text = file.read()\n\n text_lines += file_text.split(\"# \")\n","await cognee.prune.prune_data()\nawait cognee.prune.prune_system(metadata=True)\n","await cognee.add(data=text_lines, dataset_name=\"milvus_faq\")\nawait cognee.cognify()\n\n# [DocumentChunk(id=UUID('6889e7ef-3670-555c-bb16-3eb50d1d30b0'), updated_at=datetime.datetime(2024, 12, 4, 6, 29, 46, 472907, tzinfo=datetime.timezone.utc), text='Does the query perform in memory? What are incremental data and historical data?\\n\\nYes. When ...\n# ...\n","from cognee.api.v1.search import SearchType\n\nquery_text = \"How is data stored in milvus?\"\nsearch_results = await cognee.search(SearchType.SUMMARIES, query_text=query_text)\n\nprint(search_results[0])\n","from cognee.api.v1.search import SearchType\n\nquery_text = \"How is data stored in milvus?\"\nsearch_results = await cognee.search(SearchType.CHUNKS, query_text=query_text)\n","def format_and_print(data):\n print(\"ID:\", data[\"id\"])\n print(\"\\nText:\\n\")\n paragraphs = data[\"text\"].split(\"\\n\\n\")\n for paragraph in paragraphs:\n print(paragraph.strip())\n print()\n\n\nformat_and_print(search_results[0])\n","await cognee.prune.prune_data()\nawait cognee.prune.prune_system(metadata=True)\n","# We only use one line of text as the dataset, which simplifies the output later\ntext = \"\"\"\n Natural language processing (NLP) is an interdisciplinary\n subfield of computer science and information retrieval.\n \"\"\"\n\nawait cognee.add(text)\nawait cognee.cognify()\n","query_text = \"Tell me about NLP\"\nsearch_results = await cognee.search(SearchType.INSIGHTS, query_text=query_text)\n\nfor result_text in search_results:\n print(result_text)\n\n# Example output:\n# ({'id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'natural language processing', 'description': 'An interdisciplinary subfield of computer science and information retrieval.'}, {'relationship_name': 'is_a_subfield_of', 'source_node_id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'target_node_id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 15, 473137, tzinfo=datetime.timezone.utc)}, {'id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'computer science', 'description': 'The study of computation and information processing.'})\n# (...)\n#\n# It represents nodes and relationships in the knowledge graph:\n# - The first element is the source node (e.g., 'natural language processing').\n# - The second element is the relationship between nodes (e.g., 'is_a_subfield_of').\n# - The third element is the target node (e.g., 'computer science').\n"],"headingContent":"","anchorList":[{"label":"Construir o RAG","href":"Build-RAG","type":2,"isActive":false}]}
\ No newline at end of file
diff --git a/localization/v2.4.x/site/pt/integrations/build_RAG_with_milvus_and_cognee.md b/localization/v2.4.x/site/pt/integrations/build_RAG_with_milvus_and_cognee.md
new file mode 100644
index 000000000..6757879ec
--- /dev/null
+++ b/localization/v2.4.x/site/pt/integrations/build_RAG_with_milvus_and_cognee.md
@@ -0,0 +1,159 @@
+---
+id: build_RAG_with_milvus_and_cognee.md
+summary: >-
+ Neste tutorial, vamos mostrar-lhe como construir um pipeline RAG
+ (Retrieval-Augmented Generation) com o Milvus e o Cognee.
+title: Criar RAG com Milvus e Cognee
+---
+
+Criar RAG com Milvus e Cognee
O Cognee é uma plataforma que dá prioridade ao programador e que simplifica o desenvolvimento de aplicações de IA com pipelines ECL (Extract, Cognify, Load) modulares e dimensionáveis. Ao integrar-se perfeitamente com o Milvus, o Cognee permite a ligação e a recuperação eficientes de conversas, documentos e transcrições, reduzindo as alucinações e optimizando os custos operacionais.
+Com um forte suporte para armazenamentos vectoriais como o Milvus, bases de dados de grafos e LLMs, o Cognee fornece uma estrutura flexível e personalizável para a criação de sistemas de geração aumentada de recuperação (RAG). Sua arquitetura pronta para produção garante maior precisão e eficiência para aplicativos alimentados por IA.
+Neste tutorial, vamos mostrar-lhe como construir um pipeline RAG (Retrieval-Augmented Generation) com o Milvus e o Cognee.
+$ pip install pymilvus git+https://github.com/topoteretes/cognee.git
+
+
+Se estiver a utilizar o Google Colab, para ativar as dependências acabadas de instalar, poderá ter de reiniciar o tempo de execução (clique no menu "Tempo de execução" na parte superior do ecrã e selecione "Reiniciar sessão" no menu pendente).
+
+Por predefinição, neste exemplo, é utilizado o OpenAI como LLM. É necessário preparar a chave api e defini-la na função config set_llm_api_key().
+Para configurar o Milvus como base de dados vetorial, defina VECTOR_DB_PROVIDER para milvus e especifique VECTOR_DB_URL e VECTOR_DB_KEY. Uma vez que estamos a utilizar o Milvus Lite para armazenar dados nesta demonstração, apenas é necessário fornecer VECTOR_DB_URL.
+import os
+
+import cognee
+
+cognee.config.set_llm_api_key("YOUR_OPENAI_API_KEY")
+
+
+os.environ["VECTOR_DB_PROVIDER"] = "milvus"
+os.environ["VECTOR_DB_URL"] = "./milvus.db"
+
+
+Quanto às variáveis de ambiente VECTOR_DB_URL e VECTOR_DB_KEY:
+
+- Definir o
VECTOR_DB_URL como um arquivo local, por exemplo,./milvus.db, é o método mais conveniente, pois utiliza automaticamente o Milvus Lite para armazenar todos os dados neste arquivo.
+- Se tiver uma grande escala de dados, pode configurar um servidor Milvus mais eficiente em docker ou kubernetes. Nesta configuração, utilize o uri do servidor, por exemplo,
http://localhost:19530, como o seu VECTOR_DB_URL.
+- Se pretender utilizar o Zilliz Cloud, o serviço de nuvem totalmente gerido para o Milvus, ajuste os endereços
VECTOR_DB_URL e VECTOR_DB_KEY, que correspondem ao Public Endpoint e à chave Api no Zilliz Cloud.
+
+
+Preparar os dados
Utilizamos as páginas de FAQ da Documentação do Milvus 2.4.x como conhecimento privado no nosso RAG, que é uma boa fonte de dados para um pipeline RAG simples.
+Descarregue o ficheiro zip e extraia os documentos para a pasta milvus_docs.
+$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
+
+Carregamos todos os ficheiros markdown da pasta milvus_docs/en/faq. Para cada documento, utilizamos simplesmente "#" para separar o conteúdo do ficheiro, o que pode separar aproximadamente o conteúdo de cada parte principal do ficheiro markdown.
+from glob import glob
+
+text_lines = []
+
+for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
+ with open(file_path, "r") as file:
+ file_text = file.read()
+
+ text_lines += file_text.split("# ")
+
+Construir o RAG
Redefinição dos dados do Cognee
await cognee.prune.prune_data()
+await cognee.prune.prune_system(metadata=True)
+
+Com um quadro limpo pronto, podemos agora adicionar o nosso conjunto de dados e processá-lo num gráfico de conhecimento.
+Adicionar dados e reconhecer
await cognee.add(data=text_lines, dataset_name="milvus_faq")
+await cognee.cognify()
+
+# [DocumentChunk(id=UUID('6889e7ef-3670-555c-bb16-3eb50d1d30b0'), updated_at=datetime.datetime(2024, 12, 4, 6, 29, 46, 472907, tzinfo=datetime.timezone.utc), text='Does the query perform in memory? What are incremental data and historical data?\n\nYes. When ...
+# ...
+
+O método add carrega o conjunto de dados (Milvus FAQs) no Cognee e o método cognify processa os dados para extrair entidades, relações e resumos, construindo um gráfico de conhecimento.
+Consulta de resumos
Agora que os dados foram processados, vamos consultar o gráfico de conhecimento.
+from cognee.api.v1.search import SearchType
+
+query_text = "How is data stored in milvus?"
+search_results = await cognee.search(SearchType.SUMMARIES, query_text=query_text)
+
+print(search_results[0])
+
+{'id': 'de5c6713-e079-5d0b-b11d-e9bacd1e0d73', 'text': 'Milvus stores two data types: inserted data and metadata.'}
+
+Esta consulta procura no gráfico de conhecimento um resumo relacionado com o texto da consulta e o candidato mais relacionado é impresso.
+Consultar por partes
Os resumos oferecem informações de alto nível, mas para obter detalhes mais granulares, podemos consultar partes específicas de dados diretamente a partir do conjunto de dados processado. Esses pedaços são derivados dos dados originais que foram adicionados e analisados durante a criação do gráfico de conhecimento.
+from cognee.api.v1.search import SearchType
+
+query_text = "How is data stored in milvus?"
+search_results = await cognee.search(SearchType.CHUNKS, query_text=query_text)
+
+Vamos formatá-los e exibi-los para facilitar a leitura!
+def format_and_print(data):
+ print("ID:", data["id"])
+ print("\nText:\n")
+ paragraphs = data["text"].split("\n\n")
+ for paragraph in paragraphs:
+ print(paragraph.strip())
+ print()
+
+
+format_and_print(search_results[0])
+
+ID: 4be01c4b-9ee5-541c-9b85-297883934ab3
+
+Text:
+
+Where does Milvus store data?
+
+Milvus deals with two types of data, inserted data and metadata.
+
+Inserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).
+
+Metadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.
+
+###
+
+Nos passos anteriores, consultámos o conjunto de dados das FAQ do Milvus para obter resumos e partes específicas de dados. Embora isso tenha fornecido insights detalhados e informações granulares, o conjunto de dados era grande, o que dificultava a visualização clara das dependências no gráfico de conhecimento.
+Para resolver este problema, vamos reiniciar o ambiente Cognee e trabalhar com um conjunto de dados mais pequeno e mais específico. Isto permitir-nos-á demonstrar melhor as relações e dependências extraídas durante o processo cognify. Ao simplificar os dados, podemos ver claramente como o Cognee organiza e estrutura as informações no gráfico de conhecimento.
+Redefinir o Cognee
await cognee.prune.prune_data()
+await cognee.prune.prune_system(metadata=True)
+
+Adicionando o conjunto de dados focado
Aqui, um conjunto de dados menor com apenas uma linha de texto é adicionado e processado para garantir um gráfico de conhecimento focado e facilmente interpretável.
+# We only use one line of text as the dataset, which simplifies the output later
+text = """
+ Natural language processing (NLP) is an interdisciplinary
+ subfield of computer science and information retrieval.
+ """
+
+await cognee.add(text)
+await cognee.cognify()
+
+Consulta de informações
Ao concentrarmo-nos neste conjunto de dados mais pequeno, podemos agora analisar claramente as relações e a estrutura do gráfico de conhecimento.
+query_text = "Tell me about NLP"
+search_results = await cognee.search(SearchType.INSIGHTS, query_text=query_text)
+
+for result_text in search_results:
+ print(result_text)
+
+# Example output:
+# ({'id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'natural language processing', 'description': 'An interdisciplinary subfield of computer science and information retrieval.'}, {'relationship_name': 'is_a_subfield_of', 'source_node_id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'target_node_id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 15, 473137, tzinfo=datetime.timezone.utc)}, {'id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'computer science', 'description': 'The study of computation and information processing.'})
+# (...)
+#
+# It represents nodes and relationships in the knowledge graph:
+# - The first element is the source node (e.g., 'natural language processing').
+# - The second element is the relationship between nodes (e.g., 'is_a_subfield_of').
+# - The third element is the target node (e.g., 'computer science').
+
+Este resultado representa os resultados de uma consulta do gráfico de conhecimento, mostrando as entidades (nós) e as suas relações (arestas) extraídas do conjunto de dados processado. Cada tupla inclui uma entidade de origem, um tipo de relação e uma entidade de destino, juntamente com metadados como IDs únicos, descrições e carimbos de data/hora. O gráfico destaca conceitos-chave e suas conexões semânticas, fornecendo uma compreensão estruturada do conjunto de dados.
+Parabéns, aprendeu a utilização básica do cognee com o Milvus. Se quiser saber mais sobre a utilização avançada do cognee, consulte a sua página oficial.
diff --git a/localization/v2.4.x/site/pt/integrations/build_RAG_with_milvus_and_gemini.json b/localization/v2.4.x/site/pt/integrations/build_RAG_with_milvus_and_gemini.json
new file mode 100644
index 000000000..43f95e424
--- /dev/null
+++ b/localization/v2.4.x/site/pt/integrations/build_RAG_with_milvus_and_gemini.json
@@ -0,0 +1 @@
+{"codeList":["$ pip install --upgrade pymilvus google-generativeai requests tqdm\n","import os\n\nos.environ[\"GEMINI_API_KEY\"] = \"***********\"\n","$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip\n$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs\n","from glob import glob\n\ntext_lines = []\n\nfor file_path in glob(\"milvus_docs/en/faq/*.md\", recursive=True):\n with open(file_path, \"r\") as file:\n file_text = file.read()\n\n text_lines += file_text.split(\"# \")\n","import google.generativeai as genai\n\ngenai.configure(api_key=os.environ[\"GEMINI_API_KEY\"])\n\ngemini_model = genai.GenerativeModel(\"gemini-1.5-flash\")\n\nresponse = gemini_model.generate_content(\"who are you\")\nprint(response.text)\n","test_embeddings = genai.embed_content(\n model=\"models/text-embedding-004\", content=[\"This is a test1\", \"This is a test2\"]\n)[\"embedding\"]\n\nembedding_dim = len(test_embeddings[0])\nprint(embedding_dim)\nprint(test_embeddings[0][:10])\n","from pymilvus import MilvusClient\n\nmilvus_client = MilvusClient(uri=\"./milvus_demo.db\")\n\ncollection_name = \"my_rag_collection\"\n","if milvus_client.has_collection(collection_name):\n milvus_client.drop_collection(collection_name)\n","milvus_client.create_collection(\n collection_name=collection_name,\n dimension=embedding_dim,\n metric_type=\"IP\", # Inner product distance\n consistency_level=\"Strong\", # Strong consistency level\n)\n","from tqdm import tqdm\n\ndata = []\n\ndoc_embeddings = genai.embed_content(\n model=\"models/text-embedding-004\", content=text_lines\n)[\"embedding\"]\n\nfor i, line in enumerate(tqdm(text_lines, desc=\"Creating embeddings\")):\n data.append({\"id\": i, \"vector\": doc_embeddings[i], \"text\": line})\n\nmilvus_client.insert(collection_name=collection_name, data=data)\n","question = \"How is data stored in milvus?\"\n","question_embedding = genai.embed_content(\n model=\"models/text-embedding-004\", content=question\n)[\"embedding\"]\n\nsearch_res = milvus_client.search(\n collection_name=collection_name,\n data=[question_embedding],\n limit=3, # Return top 3 results\n search_params={\"metric_type\": \"IP\", \"params\": {}}, # Inner product distance\n output_fields=[\"text\"], # Return the text field\n)\n","import json\n\nretrieved_lines_with_distances = [\n (res[\"entity\"][\"text\"], res[\"distance\"]) for res in search_res[0]\n]\nprint(json.dumps(retrieved_lines_with_distances, indent=4))\n","context = \"\\n\".join(\n [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]\n)\n","SYSTEM_PROMPT = \"\"\"\nHuman: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.\n\"\"\"\nUSER_PROMPT = f\"\"\"\nUse the following pieces of information enclosed in tags to provide an answer to the question enclosed in tags.\n\n{context}\n \n\n{question}\n \n\"\"\"\n","gemini_model = genai.GenerativeModel(\n \"gemini-1.5-flash\", system_instruction=SYSTEM_PROMPT\n)\nresponse = gemini_model.generate_content(USER_PROMPT)\nprint(response.text)\n"],"headingContent":"Build RAG with Milvus and Gemini","anchorList":[{"label":"Criar RAG com o Milvus e o Gemini","href":"Build-RAG-with-Milvus-and-Gemini","type":1,"isActive":false},{"label":"Preparação","href":"Preparation","type":2,"isActive":false},{"label":"Carregar dados no Milvus","href":"Load-data-into-Milvus","type":2,"isActive":false},{"label":"Construir RAG","href":"Build-RAG","type":2,"isActive":false}]}
\ No newline at end of file
diff --git a/localization/v2.4.x/site/pt/integrations/build_RAG_with_milvus_and_gemini.md b/localization/v2.4.x/site/pt/integrations/build_RAG_with_milvus_and_gemini.md
new file mode 100644
index 000000000..6545caf02
--- /dev/null
+++ b/localization/v2.4.x/site/pt/integrations/build_RAG_with_milvus_and_gemini.md
@@ -0,0 +1,246 @@
+---
+id: build_RAG_with_milvus_and_gemini.md
+summary: >-
+ Neste tutorial, mostraremos como construir um pipeline RAG
+ (Retrieval-Augmented Generation) com o Milvus e o Gemini. Utilizaremos o
+ modelo Gemini para gerar texto com base numa determinada consulta. Também
+ usaremos o Milvus para armazenar e recuperar o texto gerado.
+title: Criar RAG com o Milvus e o Gemini
+---
+
+Criar RAG com o Milvus e o Gemini
A API Gemini e o Google AI Studio ajudam-no a começar a trabalhar com os modelos mais recentes da Google e a transformar as suas ideias em aplicações escaláveis. O Gemini fornece acesso a modelos de linguagem poderosos como Gemini-1.5-Flash, Gemini-1.5-Flash-8B e Gemini-1.5-Pro para tarefas como geração de texto, processamento de documentos, visão, análise de áudio e muito mais. A API permite-lhe introduzir contextos longos com milhões de tokens, afinar modelos para tarefas específicas, gerar resultados estruturados como JSON e tirar partido de capacidades como a recuperação semântica e a execução de código.
+Neste tutorial, mostraremos como construir um pipeline RAG (Retrieval-Augmented Generation) com o Milvus e o Gemini. Utilizaremos o modelo Gemini para gerar texto com base numa determinada consulta. Também usaremos o Milvus para armazenar e recuperar o texto gerado.
+Preparação
Dependências e ambiente
$ pip install --upgrade pymilvus google-generativeai requests tqdm
+
+
+Se estiver a utilizar o Google Colab, para ativar as dependências que acabou de instalar, poderá ter de reiniciar o tempo de execução (clique no menu "Tempo de execução" na parte superior do ecrã e selecione "Reiniciar sessão" no menu pendente).
+
+Em primeiro lugar, deve iniciar sessão na plataforma Google AI Studio e preparar a chave api GEMINI_API_KEY como uma variável de ambiente.
+import os
+
+os.environ["GEMINI_API_KEY"] = "***********"
+
+Preparar os dados
Utilizamos as páginas de FAQ da Documentação Milvus 2.4.x como conhecimento privado no nosso RAG, que é uma boa fonte de dados para um pipeline RAG simples.
+Descarregue o ficheiro zip e extraia os documentos para a pasta milvus_docs.
+$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
+
+Carregamos todos os ficheiros markdown da pasta milvus_docs/en/faq. Para cada documento, basta utilizar "#" para separar o conteúdo do ficheiro, o que pode separar aproximadamente o conteúdo de cada parte principal do ficheiro markdown.
+from glob import glob
+
+text_lines = []
+
+for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
+ with open(file_path, "r") as file:
+ file_text = file.read()
+
+ text_lines += file_text.split("# ")
+
+Preparar o LLM e o modelo de incorporação
Utilizamos o gemini-1.5-flash como LLM e o text-embedding-004 como modelo de incorporação.
+Vamos tentar gerar uma resposta de teste a partir do LLM:
+import google.generativeai as genai
+
+genai.configure(api_key=os.environ["GEMINI_API_KEY"])
+
+gemini_model = genai.GenerativeModel("gemini-1.5-flash")
+
+response = gemini_model.generate_content("who are you")
+print(response.text)
+
+I am a large language model, trained by Google. I am an AI and don't have a personal identity or consciousness. My purpose is to process information and respond to a wide range of prompts and questions in a helpful and informative way.
+
+Gerar uma incorporação de teste e imprimir a sua dimensão e os primeiros elementos.
+test_embeddings = genai.embed_content(
+ model="models/text-embedding-004", content=["This is a test1", "This is a test2"]
+)["embedding"]
+
+embedding_dim = len(test_embeddings[0])
+print(embedding_dim)
+print(test_embeddings[0][:10])
+
+768
+[0.013588584, -0.004361838, -0.08481652, -0.039724775, 0.04723794, -0.0051557426, 0.026071774, 0.045514572, -0.016867816, 0.039378334]
+
+Carregar dados no Milvus
Criar a coleção
from pymilvus import MilvusClient
+
+milvus_client = MilvusClient(uri="./milvus_demo.db")
+
+collection_name = "my_rag_collection"
+
+
+Quanto ao argumento de MilvusClient:
+
+- Definir o
uri como um ficheiro local, por exemplo./milvus.db, é o método mais conveniente, uma vez que utiliza automaticamente o Milvus Lite para armazenar todos os dados neste ficheiro.
+- Se tiver uma grande escala de dados, pode configurar um servidor Milvus mais eficiente em docker ou kubernetes. Nesta configuração, utilize o uri do servidor, por exemplo,
http://localhost:19530, como o seu uri.
+- Se pretender utilizar o Zilliz Cloud, o serviço de nuvem totalmente gerido para o Milvus, ajuste os endereços
uri e token, que correspondem ao Public Endpoint e à chave Api no Zilliz Cloud.
+
+
+Verificar se a coleção já existe e eliminá-la se existir.
+if milvus_client.has_collection(collection_name):
+ milvus_client.drop_collection(collection_name)
+
+Criar uma nova coleção com os parâmetros especificados.
+Se não especificarmos qualquer informação de campo, o Milvus criará automaticamente um campo id por defeito para a chave primária e um campo vector para armazenar os dados vectoriais. Um campo JSON reservado é utilizado para armazenar campos não definidos pelo esquema e os respectivos valores.
+milvus_client.create_collection(
+ collection_name=collection_name,
+ dimension=embedding_dim,
+ metric_type="IP", # Inner product distance
+ consistency_level="Strong", # Strong consistency level
+)
+
+Inserir dados
Itere pelas linhas de texto, crie embeddings e, em seguida, insira os dados no Milvus.
+Aqui está um novo campo text, que é um campo não definido no esquema da coleção. Será automaticamente adicionado ao campo dinâmico JSON reservado, que pode ser tratado como um campo normal a um nível elevado.
+from tqdm import tqdm
+
+data = []
+
+doc_embeddings = genai.embed_content(
+ model="models/text-embedding-004", content=text_lines
+)["embedding"]
+
+for i, line in enumerate(tqdm(text_lines, desc="Creating embeddings")):
+ data.append({"id": i, "vector": doc_embeddings[i], "text": line})
+
+milvus_client.insert(collection_name=collection_name, data=data)
+
+Creating embeddings: 100%|██████████| 72/72 [00:00<00:00, 468201.38it/s]
+
+
+
+
+
+{'insert_count': 72, 'ids': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71], 'cost': 0}
+
+Construir RAG
Recuperar dados para uma consulta
Vamos especificar uma pergunta frequente sobre o Milvus.
+question = "How is data stored in milvus?"
+
+Pesquise a pergunta na coleção e obtenha as 3 principais correspondências semânticas.
+question_embedding = genai.embed_content(
+ model="models/text-embedding-004", content=question
+)["embedding"]
+
+search_res = milvus_client.search(
+ collection_name=collection_name,
+ data=[question_embedding],
+ limit=3, # Return top 3 results
+ search_params={"metric_type": "IP", "params": {}}, # Inner product distance
+ output_fields=["text"], # Return the text field
+)
+
+Vejamos os resultados da pesquisa da consulta
+import json
+
+retrieved_lines_with_distances = [
+ (res["entity"]["text"], res["distance"]) for res in search_res[0]
+]
+print(json.dumps(retrieved_lines_with_distances, indent=4))
+
+[
+ [
+ " Where does Milvus store data?\n\nMilvus deals with two types of data, inserted data and metadata. \n\nInserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).\n\nMetadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.\n\n###",
+ 0.8048275113105774
+ ],
+ [
+ "Does the query perform in memory? What are incremental data and historical data?\n\nYes. When a query request comes, Milvus searches both incremental data and historical data by loading them into memory. Incremental data are in the growing segments, which are buffered in memory before they reach the threshold to be persisted in storage engine, while historical data are from the sealed segments that are stored in the object storage. Incremental data and historical data together constitute the whole dataset to search.\n\n###",
+ 0.7574886679649353
+ ],
+ [
+ "What is the maximum dataset size Milvus can handle?\n\n \nTheoretically, the maximum dataset size Milvus can handle is determined by the hardware it is run on, specifically system memory and storage:\n\n- Milvus loads all specified collections and partitions into memory before running queries. Therefore, memory size determines the maximum amount of data Milvus can query.\n- When new entities and and collection-related schema (currently only MinIO is supported for data persistence) are added to Milvus, system storage determines the maximum allowable size of inserted data.\n\n###",
+ 0.7453608512878418
+ ]
+]
+
+Utilizar o LLM para obter uma resposta RAG
Converter os documentos recuperados num formato de cadeia de caracteres.
+context = "\n".join(
+ [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]
+)
+
+Definir avisos do sistema e do utilizador para o Modelo de Linguagem. Este prompt é montado com os documentos recuperados do Milvus.
+SYSTEM_PROMPT = """
+Human: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.
+"""
+USER_PROMPT = f"""
+Use the following pieces of information enclosed in <context> tags to provide an answer to the question enclosed in <question> tags.
+<context>
+{context}
+</context>
+<question>
+{question}
+</question>
+"""
+
+Utilize o Gemini para gerar uma resposta com base nos prompts.
+gemini_model = genai.GenerativeModel(
+ "gemini-1.5-flash", system_instruction=SYSTEM_PROMPT
+)
+response = gemini_model.generate_content(USER_PROMPT)
+print(response.text)
+
+Milvus stores data in two ways: Inserted data (vector data, scalar data, and collection-specific schema) is stored as an incremental log in persistent storage using object storage backends such as MinIO, AWS S3, Google Cloud Storage, Azure Blob Storage, Alibaba Cloud OSS, and Tencent Cloud Object Storage. Metadata, generated by each Milvus module, is stored in etcd.
+
+Ótimo! Construímos com sucesso um pipeline RAG com o Milvus e o Gemini.
diff --git a/localization/v2.4.x/site/pt/integrations/build_RAG_with_milvus_and_ollama.json b/localization/v2.4.x/site/pt/integrations/build_RAG_with_milvus_and_ollama.json
new file mode 100644
index 000000000..bff8833ad
--- /dev/null
+++ b/localization/v2.4.x/site/pt/integrations/build_RAG_with_milvus_and_ollama.json
@@ -0,0 +1 @@
+{"codeList":["$ pip install pymilvus ollama\n","$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip\n$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs\n","from glob import glob\n\ntext_lines = []\n\nfor file_path in glob(\"milvus_docs/en/faq/*.md\", recursive=True):\n with open(file_path, \"r\") as file:\n file_text = file.read()\n\n text_lines += file_text.split(\"# \")\n","! ollama pull mxbai-embed-large\n","! ollama pull llama3.2\n","import ollama\n\n\ndef emb_text(text):\n response = ollama.embeddings(model=\"mxbai-embed-large\", prompt=text)\n return response[\"embedding\"]\n","test_embedding = emb_text(\"This is a test\")\nembedding_dim = len(test_embedding)\nprint(embedding_dim)\nprint(test_embedding[:10])\n","from pymilvus import MilvusClient\n\nmilvus_client = MilvusClient(uri=\"./milvus_demo.db\")\n\ncollection_name = \"my_rag_collection\"\n","if milvus_client.has_collection(collection_name):\n milvus_client.drop_collection(collection_name)\n","milvus_client.create_collection(\n collection_name=collection_name,\n dimension=embedding_dim,\n metric_type=\"IP\", # Inner product distance\n consistency_level=\"Strong\", # Strong consistency level\n)\n","from tqdm import tqdm\n\ndata = []\n\nfor i, line in enumerate(tqdm(text_lines, desc=\"Creating embeddings\")):\n data.append({\"id\": i, \"vector\": emb_text(line), \"text\": line})\n\nmilvus_client.insert(collection_name=collection_name, data=data)\n","question = \"How is data stored in milvus?\"\n","search_res = milvus_client.search(\n collection_name=collection_name,\n data=[\n emb_text(question)\n ], # Use the `emb_text` function to convert the question to an embedding vector\n limit=3, # Return top 3 results\n search_params={\"metric_type\": \"IP\", \"params\": {}}, # Inner product distance\n output_fields=[\"text\"], # Return the text field\n)\n","import json\n\nretrieved_lines_with_distances = [\n (res[\"entity\"][\"text\"], res[\"distance\"]) for res in search_res[0]\n]\nprint(json.dumps(retrieved_lines_with_distances, indent=4))\n","context = \"\\n\".join(\n [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]\n)\n","SYSTEM_PROMPT = \"\"\"\nHuman: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.\n\"\"\"\nUSER_PROMPT = f\"\"\"\nUse the following pieces of information enclosed in tags to provide an answer to the question enclosed in tags.\n\n{context}\n \n\n{question}\n \n\"\"\"\n","from ollama import chat\nfrom ollama import ChatResponse\n\nresponse: ChatResponse = chat(\n model=\"llama3.2\",\n messages=[\n {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n {\"role\": \"user\", \"content\": USER_PROMPT},\n ],\n)\nprint(response[\"message\"][\"content\"])\n"],"headingContent":"Build RAG with Milvus and Ollama","anchorList":[{"label":"Construir RAG com Milvus e Ollama","href":"Build-RAG-with-Milvus-and-Ollama","type":1,"isActive":false},{"label":"Preparação","href":"Preparation","type":2,"isActive":false},{"label":"Carregar dados no Milvus","href":"Load-data-into-Milvus","type":2,"isActive":false},{"label":"Construir RAG","href":"Build-RAG","type":2,"isActive":false}]}
\ No newline at end of file
diff --git a/localization/v2.4.x/site/pt/integrations/build_RAG_with_milvus_and_ollama.md b/localization/v2.4.x/site/pt/integrations/build_RAG_with_milvus_and_ollama.md
new file mode 100644
index 000000000..98dbc5cd8
--- /dev/null
+++ b/localization/v2.4.x/site/pt/integrations/build_RAG_with_milvus_and_ollama.md
@@ -0,0 +1,281 @@
+---
+id: build_RAG_with_milvus_and_ollama.md
+summary: >-
+ Neste guia, mostraremos como aproveitar o Ollama e o Milvus para criar um
+ pipeline RAG (Retrieval-Augmented Generation) de forma eficiente e segura.
+title: Construir RAG com Milvus e Ollama
+---
+
+Construir RAG com Milvus e Ollama
Ollama é uma plataforma de código aberto que simplifica a execução e a personalização de grandes modelos de linguagem (LLMs) localmente. Fornece uma experiência fácil de usar e sem nuvem, permitindo downloads, instalação e interação de modelos sem esforço, sem exigir habilidades técnicas avançadas. Com uma biblioteca crescente de LLMs pré-treinados - de uso geral a domínio específico - a Ollama facilita a gestão e a personalização de modelos para várias aplicações. Garante a privacidade e a flexibilidade dos dados, permitindo que os utilizadores afinem, optimizem e implementem soluções orientadas para a IA inteiramente nas suas máquinas.
+Neste guia, mostraremos como aproveitar o Ollama e o Milvus para criar um pipeline RAG (Retrieval-Augmented Generation) de forma eficiente e segura.
+Preparação
Dependências e ambiente
$ pip install pymilvus ollama
+
+
+Se estiver a utilizar o Google Colab, para ativar as dependências que acabou de instalar, poderá ter de reiniciar o tempo de execução (clique no menu "Tempo de execução" na parte superior do ecrã e selecione "Reiniciar sessão" no menu pendente).
+
+Preparar os dados
Utilizamos as páginas de FAQ da Documentação do Milvus 2.4.x como conhecimento privado no nosso RAG, que é uma boa fonte de dados para um pipeline RAG simples.
+Descarregue o ficheiro zip e extraia os documentos para a pasta milvus_docs.
+$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
+
+--2024-11-26 21:47:19-- https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+Resolving github.com (github.com)... 140.82.112.4
+Connecting to github.com (github.com)|140.82.112.4|:443... connected.
+HTTP request sent, awaiting response... 302 Found
+Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/267273319/c52902a0-e13c-4ca7-92e0-086751098a05?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241127%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241127T024720Z&X-Amz-Expires=300&X-Amz-Signature=7808b77cbdaa7e122196bcd75a73f29f2540333a350c4830bbdf5f286e876304&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dmilvus_docs_2.4.x_en.zip&response-content-type=application%2Foctet-stream [following]
+--2024-11-26 21:47:20-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/267273319/c52902a0-e13c-4ca7-92e0-086751098a05?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241127%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241127T024720Z&X-Amz-Expires=300&X-Amz-Signature=7808b77cbdaa7e122196bcd75a73f29f2540333a350c4830bbdf5f286e876304&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dmilvus_docs_2.4.x_en.zip&response-content-type=application%2Foctet-stream
+Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.109.133, 185.199.111.133, 185.199.108.133, ...
+Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.109.133|:443... connected.
+HTTP request sent, awaiting response... 200 OK
+Length: 613094 (599K) [application/octet-stream]
+Saving to: ‘milvus_docs_2.4.x_en.zip’
+
+milvus_docs_2.4.x_e 100%[===================>] 598.72K 1.20MB/s in 0.5s
+
+2024-11-26 21:47:20 (1.20 MB/s) - ‘milvus_docs_2.4.x_en.zip’ saved [613094/613094]
+
+Carregamos todos os ficheiros markdown da pasta milvus_docs/en/faq. Para cada documento, utilizamos simplesmente "#" para separar o conteúdo do ficheiro, o que permite separar aproximadamente o conteúdo de cada parte principal do ficheiro markdown.
+from glob import glob
+
+text_lines = []
+
+for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
+ with open(file_path, "r") as file:
+ file_text = file.read()
+
+ text_lines += file_text.split("# ")
+
+Preparar o LLM e o modelo de incorporação
O Ollama suporta vários modelos para tarefas baseadas em LLM e geração de incorporação, facilitando o desenvolvimento de aplicações de geração aumentada por recuperação (RAG). Para esta configuração:
+
+- Usaremos o Llama 3.2 (3B) como nosso LLM para tarefas de geração de texto.
+- Para a geração de embedding, usaremos o mxbai-embed-large, um modelo de 334M parâmetros optimizado para a similaridade semântica.
+
+Antes de começar, certifique-se de que ambos os modelos são extraídos localmente:
+! ollama pull mxbai-embed-large
+
+[?25lpulling manifest ⠋ [?25h[?25l[2K[1Gpulling manifest ⠙ [?25h[?25l[2K[1Gpulling manifest ⠹ [?25h[?25l[2K[1Gpulling manifest ⠸ [?25h[?25l[2K[1Gpulling manifest ⠼ [?25h[?25l[2K[1Gpulling manifest ⠴ [?25h[?25l[2K[1Gpulling manifest
+pulling 819c2adf5ce6... 100% ▕████████████████▏ 669 MB
+pulling c71d239df917... 100% ▕████████████████▏ 11 KB
+pulling b837481ff855... 100% ▕████████████████▏ 16 B
+pulling 38badd946f91... 100% ▕████████████████▏ 408 B
+verifying sha256 digest
+writing manifest
+success [?25h
+
+! ollama pull llama3.2
+
+[?25lpulling manifest ⠋ [?25h[?25l[2K[1Gpulling manifest ⠙ [?25h[?25l[2K[1Gpulling manifest ⠹ [?25h[?25l[2K[1Gpulling manifest ⠸ [?25h[?25l[2K[1Gpulling manifest ⠼ [?25h[?25l[2K[1Gpulling manifest ⠴ [?25h[?25l[2K[1Gpulling manifest
+pulling dde5aa3fc5ff... 100% ▕████████████████▏ 2.0 GB
+pulling 966de95ca8a6... 100% ▕████████████████▏ 1.4 KB
+pulling fcc5a6bec9da... 100% ▕████████████████▏ 7.7 KB
+pulling a70ff7e570d9... 100% ▕████████████████▏ 6.0 KB
+pulling 56bb8bd477a5... 100% ▕████████████████▏ 96 B
+pulling 34bb5ab01051... 100% ▕████████████████▏ 561 B
+verifying sha256 digest
+writing manifest
+success [?25h
+
+Com estes modelos prontos, podemos proceder à implementação de fluxos de trabalho de geração orientada por LLM e de recuperação baseada em embedding.
+import ollama
+
+
+def emb_text(text):
+ response = ollama.embeddings(model="mxbai-embed-large", prompt=text)
+ return response["embedding"]
+
+Gerar um embedding de teste e imprimir a sua dimensão e os primeiros elementos.
+test_embedding = emb_text("This is a test")
+embedding_dim = len(test_embedding)
+print(embedding_dim)
+print(test_embedding[:10])
+
+1024
+[0.23276396095752716, 0.4257211685180664, 0.19724100828170776, 0.46120673418045044, -0.46039995551109314, -0.1413791924715042, -0.18261606991291046, -0.07602324336767197, 0.39991313219070435, 0.8337644338607788]
+
+Carregar dados no Milvus
Criar a coleção
from pymilvus import MilvusClient
+
+milvus_client = MilvusClient(uri="./milvus_demo.db")
+
+collection_name = "my_rag_collection"
+
+
+Quanto ao argumento de MilvusClient:
+
+- Definir o
uri como um ficheiro local, por exemplo./milvus.db, é o método mais conveniente, uma vez que utiliza automaticamente o Milvus Lite para armazenar todos os dados neste ficheiro.
+- Se tiver uma grande escala de dados, pode configurar um servidor Milvus mais eficiente em docker ou kubernetes. Nesta configuração, utilize o uri do servidor, por exemplo,
http://localhost:19530, como o seu uri.
+- Se pretender utilizar o Zilliz Cloud, o serviço de nuvem totalmente gerido para o Milvus, ajuste os endereços
uri e token, que correspondem ao Public Endpoint e à chave Api no Zilliz Cloud.
+
+
+Verificar se a coleção já existe e eliminá-la se existir.
+if milvus_client.has_collection(collection_name):
+ milvus_client.drop_collection(collection_name)
+
+Criar uma nova coleção com os parâmetros especificados.
+Se não especificarmos qualquer informação de campo, o Milvus criará automaticamente um campo id por defeito para a chave primária e um campo vector para armazenar os dados vectoriais. Um campo JSON reservado é utilizado para armazenar campos não definidos pelo esquema e os respectivos valores.
+milvus_client.create_collection(
+ collection_name=collection_name,
+ dimension=embedding_dim,
+ metric_type="IP", # Inner product distance
+ consistency_level="Strong", # Strong consistency level
+)
+
+Inserir dados
Itere pelas linhas de texto, crie embeddings e, em seguida, insira os dados no Milvus.
+Aqui está um novo campo text, que é um campo não definido no esquema da coleção. Será automaticamente adicionado ao campo dinâmico JSON reservado, que pode ser tratado como um campo normal a um nível elevado.
+from tqdm import tqdm
+
+data = []
+
+for i, line in enumerate(tqdm(text_lines, desc="Creating embeddings")):
+ data.append({"id": i, "vector": emb_text(line), "text": line})
+
+milvus_client.insert(collection_name=collection_name, data=data)
+
+Creating embeddings: 100%|██████████| 72/72 [00:03<00:00, 22.56it/s]
+
+
+
+
+
+{'insert_count': 72, 'ids': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71], 'cost': 0}
+
+Construir RAG
Recuperar dados para uma consulta
Vamos especificar uma pergunta frequente sobre o Milvus.
+question = "How is data stored in milvus?"
+
+Pesquise a pergunta na coleção e obtenha as 3 principais correspondências semânticas.
+search_res = milvus_client.search(
+ collection_name=collection_name,
+ data=[
+ emb_text(question)
+ ], # Use the `emb_text` function to convert the question to an embedding vector
+ limit=3, # Return top 3 results
+ search_params={"metric_type": "IP", "params": {}}, # Inner product distance
+ output_fields=["text"], # Return the text field
+)
+
+Vejamos os resultados da pesquisa da consulta
+import json
+
+retrieved_lines_with_distances = [
+ (res["entity"]["text"], res["distance"]) for res in search_res[0]
+]
+print(json.dumps(retrieved_lines_with_distances, indent=4))
+
+[
+ [
+ " Where does Milvus store data?\n\nMilvus deals with two types of data, inserted data and metadata. \n\nInserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).\n\nMetadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.\n\n###",
+ 231.9398193359375
+ ],
+ [
+ "How does Milvus flush data?\n\nMilvus returns success when inserted data are loaded to the message queue. However, the data are not yet flushed to the disk. Then Milvus' data node writes the data in the message queue to persistent storage as incremental logs. If `flush()` is called, the data node is forced to write all data in the message queue to persistent storage immediately.\n\n###",
+ 226.48316955566406
+ ],
+ [
+ "What is the maximum dataset size Milvus can handle?\n\n \nTheoretically, the maximum dataset size Milvus can handle is determined by the hardware it is run on, specifically system memory and storage:\n\n- Milvus loads all specified collections and partitions into memory before running queries. Therefore, memory size determines the maximum amount of data Milvus can query.\n- When new entities and and collection-related schema (currently only MinIO is supported for data persistence) are added to Milvus, system storage determines the maximum allowable size of inserted data.\n\n###",
+ 210.60745239257812
+ ]
+]
+
+Utilizar o LLM para obter uma resposta RAG
Converter os documentos recuperados num formato de cadeia de caracteres.
+context = "\n".join(
+ [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]
+)
+
+Definir avisos do sistema e do utilizador para o Modelo de Linguagem. Este prompt é montado com os documentos recuperados do Milvus.
+SYSTEM_PROMPT = """
+Human: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.
+"""
+USER_PROMPT = f"""
+Use the following pieces of information enclosed in <context> tags to provide an answer to the question enclosed in <question> tags.
+<context>
+{context}
+</context>
+<question>
+{question}
+</question>
+"""
+
+Utilizar o modelo llama3.2 fornecido por Ollama para gerar uma resposta com base nos avisos.
+from ollama import chat
+from ollama import ChatResponse
+
+response: ChatResponse = chat(
+ model="llama3.2",
+ messages=[
+ {"role": "system", "content": SYSTEM_PROMPT},
+ {"role": "user", "content": USER_PROMPT},
+ ],
+)
+print(response["message"]["content"])
+
+According to the provided context, data in Milvus is stored in two types:
+
+1. **Inserted data**: Storing data in persistent storage as incremental log. It supports multiple object storage backends such as MinIO, AWS S3, Google Cloud Storage (GCS), Azure Blob Storage, Alibaba Cloud OSS, and Tencent Cloud Object Storage.
+
+2. **Metadata**: Generated within Milvus and stored in etcd.
+
+Ótimo! Criámos com êxito uma conduta RAG com Milvus e Ollama.
diff --git a/localization/v2.4.x/site/pt/integrations/milvus_rag_with_dynamiq.json b/localization/v2.4.x/site/pt/integrations/milvus_rag_with_dynamiq.json
new file mode 100644
index 000000000..e9505b7ca
--- /dev/null
+++ b/localization/v2.4.x/site/pt/integrations/milvus_rag_with_dynamiq.json
@@ -0,0 +1 @@
+{"codeList":["$ pip install dynamiq pymilvus\n","import os\n\nos.environ[\"OPENAI_API_KEY\"] = \"sk-***********\"\n","# Importing necessary libraries for the workflow\nfrom io import BytesIO\nfrom dynamiq import Workflow\nfrom dynamiq.nodes import InputTransformer\nfrom dynamiq.connections import (\n OpenAI as OpenAIConnection,\n Milvus as MilvusConnection,\n MilvusDeploymentType,\n)\nfrom dynamiq.nodes.converters import PyPDFConverter\nfrom dynamiq.nodes.splitters.document import DocumentSplitter\nfrom dynamiq.nodes.embedders import OpenAIDocumentEmbedder\nfrom dynamiq.nodes.writers import MilvusDocumentWriter\n\n# Initialize the workflow\nrag_wf = Workflow()\n","converter = PyPDFConverter(document_creation_mode=\"one-doc-per-page\")\nconverter_added = rag_wf.flow.add_nodes(\n converter\n) # Add node to the DAG (Directed Acyclic Graph)\n","document_splitter = DocumentSplitter(\n split_by=\"sentence\", # Splits documents into sentences\n split_length=10,\n split_overlap=1,\n input_transformer=InputTransformer(\n selector={\n \"documents\": f\"${[converter.id]}.output.documents\",\n },\n ),\n).depends_on(\n converter\n) # Set dependency on the PDF converter\nsplitter_added = rag_wf.flow.add_nodes(document_splitter) # Add to the DAG\n","embedder = OpenAIDocumentEmbedder(\n connection=OpenAIConnection(api_key=os.environ[\"OPENAI_API_KEY\"]),\n input_transformer=InputTransformer(\n selector={\n \"documents\": f\"${[document_splitter.id]}.output.documents\",\n },\n ),\n).depends_on(\n document_splitter\n) # Set dependency on the splitter\ndocument_embedder_added = rag_wf.flow.add_nodes(embedder) # Add to the DAG\n","vector_store = (\n MilvusDocumentWriter(\n connection=MilvusConnection(\n deployment_type=MilvusDeploymentType.FILE, uri=\"./milvus.db\"\n ),\n index_name=\"my_milvus_collection\",\n dimension=1536,\n create_if_not_exist=True,\n metric_type=\"COSINE\",\n )\n .inputs(documents=embedder.outputs.documents) # Connect to embedder output\n .depends_on(embedder) # Set dependency on the embedder\n)\nmilvus_writer_added = rag_wf.flow.add_nodes(vector_store) # Add to the DAG\n","file_paths = [\"./pdf_files/WhatisMilvus.pdf\"]\ninput_data = {\n \"files\": [BytesIO(open(path, \"rb\").read()) for path in file_paths],\n \"metadata\": [{\"filename\": path} for path in file_paths],\n}\n\n# Run the workflow with the prepared input data\ninserted_data = rag_wf.run(input_data=input_data)\n","from dynamiq import Workflow\nfrom dynamiq.connections import (\n OpenAI as OpenAIConnection,\n Milvus as MilvusConnection,\n MilvusDeploymentType,\n)\nfrom dynamiq.nodes.embedders import OpenAITextEmbedder\nfrom dynamiq.nodes.retrievers import MilvusDocumentRetriever\nfrom dynamiq.nodes.llms import OpenAI\nfrom dynamiq.prompts import Message, Prompt\n\n# Initialize the workflow\nretrieval_wf = Workflow()\n","# Establish OpenAI connection\nopenai_connection = OpenAIConnection(api_key=os.environ[\"OPENAI_API_KEY\"])\n\n# Define the text embedder node\nembedder = OpenAITextEmbedder(\n connection=openai_connection,\n model=\"text-embedding-3-small\",\n)\n\n# Add the embedder node to the workflow\nembedder_added = retrieval_wf.flow.add_nodes(embedder)\n","document_retriever = (\n MilvusDocumentRetriever(\n connection=MilvusConnection(\n deployment_type=MilvusDeploymentType.FILE, uri=\"./milvus.db\"\n ),\n index_name=\"my_milvus_collection\",\n dimension=1536,\n top_k=5,\n )\n .inputs(embedding=embedder.outputs.embedding) # Connect to embedder output\n .depends_on(embedder) # Dependency on the embedder node\n)\n\n# Add the retriever node to the workflow\nmilvus_retriever_added = retrieval_wf.flow.add_nodes(document_retriever)\n","# Define the prompt template for the LLM\nprompt_template = \"\"\"\nPlease answer the question based on the provided context.\n\nQuestion: {{ query }}\n\nContext:\n{% for document in documents %}\n- {{ document.content }}\n{% endfor %}\n\"\"\"\n\n# Create the prompt object\nprompt = Prompt(messages=[Message(content=prompt_template, role=\"user\")])\n","answer_generator = (\n OpenAI(\n connection=openai_connection,\n model=\"gpt-4o\",\n prompt=prompt,\n )\n .inputs(\n documents=document_retriever.outputs.documents,\n query=embedder.outputs.query,\n )\n .depends_on(\n [document_retriever, embedder]\n ) # Dependencies on retriever and embedder\n)\n\n# Add the answer generator node to the workflow\nanswer_generator_added = retrieval_wf.flow.add_nodes(answer_generator)\n","# Run the workflow with a sample query\nsample_query = \"What is the Advanced Search Algorithms in Milvus?\"\n\nresult = retrieval_wf.run(input_data={\"query\": sample_query})\n\nanswer = result.output.get(answer_generator.id).get(\"output\", {}).get(\"content\")\nprint(answer)\n"],"headingContent":"Getting Started with Dynamiq and Milvus","anchorList":[{"label":"Introdução ao Dynamiq e ao Milvus","href":"Getting-Started-with-Dynamiq-and-Milvus","type":1,"isActive":false},{"label":"Preparação","href":"Preparation","type":2,"isActive":false},{"label":"RAG - Fluxo de indexação de documentos","href":"RAG---Document-Indexing-Flow","type":2,"isActive":false},{"label":"Fluxo de recuperação de documentos RAG","href":"RAG-Document-Retrieval-Flow","type":2,"isActive":false}]}
\ No newline at end of file
diff --git a/localization/v2.4.x/site/pt/integrations/milvus_rag_with_dynamiq.md b/localization/v2.4.x/site/pt/integrations/milvus_rag_with_dynamiq.md
new file mode 100644
index 000000000..1eee0e921
--- /dev/null
+++ b/localization/v2.4.x/site/pt/integrations/milvus_rag_with_dynamiq.md
@@ -0,0 +1,340 @@
+---
+id: milvus_rag_with_dynamiq.md
+summary: >-
+ Neste tutorial, exploraremos como usar perfeitamente o Dynamiq com o Milvus, o
+ banco de dados vetorial de alto desempenho criado especificamente para fluxos
+ de trabalho RAG. O Milvus é excelente em armazenamento, indexação e
+ recuperação eficientes de embeddings vetoriais, tornando-o um componente
+ indispensável para sistemas de IA que exigem acesso rápido e preciso a dados
+ contextuais.
+title: Introdução ao Dynamiq e ao Milvus
+---
+
+Introdução ao Dynamiq e ao Milvus
O Dynamiq é uma poderosa estrutura de IA de geração que simplifica o desenvolvimento de aplicativos alimentados por IA. Com suporte robusto para agentes de geração aumentada por recuperação (RAG) e modelo de linguagem grande (LLM), o Dynamiq permite que os desenvolvedores criem sistemas inteligentes e dinâmicos com facilidade e eficiência.
+Neste tutorial, exploraremos como usar o Dynamiq com o Milvus, o banco de dados vetorial de alto desempenho criado especificamente para fluxos de trabalho RAG. O Milvus é excelente em armazenamento, indexação e recuperação eficientes de embeddings vetoriais, tornando-o um componente indispensável para sistemas de IA que exigem acesso rápido e preciso a dados contextuais.
+Este guia passo a passo abordará dois fluxos de trabalho principais do RAG:
+
+Fluxo de indexação de documentos: Saiba como processar ficheiros de entrada (por exemplo, PDFs), transformar o seu conteúdo em embeddings vectoriais e armazená-los no Milvus. A utilização das capacidades de indexação de alto desempenho do Milvus garante que os seus dados estão prontos para uma rápida recuperação.
Fluxo de recuperação de documentos: descubra como consultar o Milvus para obter embeddings de documentos relevantes e usá-los para gerar respostas perspicazes e conscientes do contexto com os agentes LLM da Dynamiq, criando uma experiência de utilizador com IA perfeita.
+No final deste tutorial, obterá uma sólida compreensão de como o Milvus e o Dynamiq trabalham em conjunto para criar sistemas de IA escaláveis e sensíveis ao contexto, adaptados às suas necessidades.
+Preparação
Descarregar as bibliotecas necessárias
$ pip install dynamiq pymilvus
+
+
+Se estiver a utilizar o Google Colab, para ativar as dependências que acabou de instalar, poderá ter de reiniciar o tempo de execução (clique no menu "Tempo de execução" na parte superior do ecrã e selecione "Reiniciar sessão" no menu pendente).
+
+Configurar o agente LLM
Neste exemplo, vamos utilizar o OpenAI como LLM. Deve preparar a chave api OPENAI_API_KEY como uma variável de ambiente.
+import os
+
+os.environ["OPENAI_API_KEY"] = "sk-***********"
+
+RAG - Fluxo de indexação de documentos
Este tutorial demonstra um fluxo de trabalho RAG (Retrieval-Augmented Generation) para indexar documentos com o Milvus como base de dados vetorial. O fluxo de trabalho recebe arquivos PDF de entrada, processa-os em pedaços menores, gera embeddings de vetor usando o modelo de embedding do OpenAI e armazena os embeddings em uma coleção do Milvus para recuperação eficiente.
+No final deste fluxo de trabalho, terá um sistema de indexação de documentos escalável e eficiente que suporta futuras tarefas RAG, como pesquisa semântica e resposta a perguntas.
+Importar as bibliotecas necessárias e inicializar o fluxo de trabalho
# Importing necessary libraries for the workflow
+from io import BytesIO
+from dynamiq import Workflow
+from dynamiq.nodes import InputTransformer
+from dynamiq.connections import (
+ OpenAI as OpenAIConnection,
+ Milvus as MilvusConnection,
+ MilvusDeploymentType,
+)
+from dynamiq.nodes.converters import PyPDFConverter
+from dynamiq.nodes.splitters.document import DocumentSplitter
+from dynamiq.nodes.embedders import OpenAIDocumentEmbedder
+from dynamiq.nodes.writers import MilvusDocumentWriter
+
+# Initialize the workflow
+rag_wf = Workflow()
+
+Definir nó do conversor de PDF
converter = PyPDFConverter(document_creation_mode="one-doc-per-page")
+converter_added = rag_wf.flow.add_nodes(
+ converter
+) # Add node to the DAG (Directed Acyclic Graph)
+
+Definir nó divisor de documentos
document_splitter = DocumentSplitter(
+ split_by="sentence", # Splits documents into sentences
+ split_length=10,
+ split_overlap=1,
+ input_transformer=InputTransformer(
+ selector={
+ "documents": f"${[converter.id]}.output.documents",
+ },
+ ),
+).depends_on(
+ converter
+) # Set dependency on the PDF converter
+splitter_added = rag_wf.flow.add_nodes(document_splitter) # Add to the DAG
+
+Definir nó de incorporação
embedder = OpenAIDocumentEmbedder(
+ connection=OpenAIConnection(api_key=os.environ["OPENAI_API_KEY"]),
+ input_transformer=InputTransformer(
+ selector={
+ "documents": f"${[document_splitter.id]}.output.documents",
+ },
+ ),
+).depends_on(
+ document_splitter
+) # Set dependency on the splitter
+document_embedder_added = rag_wf.flow.add_nodes(embedder) # Add to the DAG
+
+Definir nó de armazenamento de vetor Milvus
vector_store = (
+ MilvusDocumentWriter(
+ connection=MilvusConnection(
+ deployment_type=MilvusDeploymentType.FILE, uri="./milvus.db"
+ ),
+ index_name="my_milvus_collection",
+ dimension=1536,
+ create_if_not_exist=True,
+ metric_type="COSINE",
+ )
+ .inputs(documents=embedder.outputs.documents) # Connect to embedder output
+ .depends_on(embedder) # Set dependency on the embedder
+)
+milvus_writer_added = rag_wf.flow.add_nodes(vector_store) # Add to the DAG
+
+2024-11-19 22:14:03 - WARNING - Environment variable 'MILVUS_API_TOKEN' not found
+2024-11-19 22:14:03 - INFO - Pass in the local path ./milvus.db, and run it using milvus-lite
+2024-11-19 22:14:04 - DEBUG - Created new connection using: 0bef2849fdb1458a85df8bb9dd27f51d
+2024-11-19 22:14:04 - INFO - Collection my_milvus_collection does not exist. Creating a new collection.
+2024-11-19 22:14:04 - DEBUG - Successfully created collection: my_milvus_collection
+2024-11-19 22:14:05 - DEBUG - Successfully created an index on collection: my_milvus_collection
+2024-11-19 22:14:05 - DEBUG - Successfully created an index on collection: my_milvus_collection
+
+
+O Milvus oferece dois tipos de implantação, que atendem a diferentes casos de uso:
+
+- MilvusDeploymentType.FILE
+
+
+- Ideal para prototipagem local ou armazenamento de dados em pequena escala.
+- Defina o
uri para um caminho de ficheiro local (por exemplo, ./milvus.db) para tirar partido do Milvus Lite, que armazena automaticamente todos os dados no ficheiro especificado.
+- Esta é uma opção conveniente para configuração e experimentação rápidas.
+
+
+- MilvusDeploymentType.HOST
+
+
+Concebido para cenários de dados em grande escala, como a gestão de mais de um milhão de vectores.
+Servidor auto-hospedado
+
+- Implante um servidor Milvus de alto desempenho usando Docker ou Kubernetes.
+- Configure o endereço e a porta do servidor como
uri (por exemplo, http://localhost:19530).
+- Se a autenticação estiver activada:
+- Forneça
<your_username>:<your_password> como o token.
+- Se a autenticação estiver desativada:
+- Deixar o endereço
token não definido.
+
+Zilliz Cloud (serviço gerido)
+
+- Para uma experiência Milvus totalmente gerida e baseada na nuvem, utilize o Zilliz Cloud.
+- Defina
uri e token de acordo com o Public Endpoint e a chave API fornecidos na consola do Zilliz Cloud.
+
+
+Definir dados de entrada e executar o fluxo de trabalho
file_paths = ["./pdf_files/WhatisMilvus.pdf"]
+input_data = {
+ "files": [BytesIO(open(path, "rb").read()) for path in file_paths],
+ "metadata": [{"filename": path} for path in file_paths],
+}
+
+# Run the workflow with the prepared input data
+inserted_data = rag_wf.run(input_data=input_data)
+
+/var/folders/09/d0hx80nj35sb5hxb5cpc1q180000gn/T/ipykernel_31319/3145804345.py:4: ResourceWarning: unclosed file <_io.BufferedReader name='./pdf_files/WhatisMilvus.pdf'>
+ BytesIO(open(path, "rb").read()) for path in file_paths
+ResourceWarning: Enable tracemalloc to get the object allocation traceback
+2024-11-19 22:14:09 - INFO - Workflow 87878444-6a3d-43f3-ae32-0127564a959f: execution started.
+2024-11-19 22:14:09 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution started.
+2024-11-19 22:14:09 - INFO - Node PyPDF File Converter - 6eb42b1f-7637-407b-a3ac-4167bcf3b5c4: execution started.
+2024-11-19 22:14:09 - INFO - Node PyPDF File Converter - 6eb42b1f-7637-407b-a3ac-4167bcf3b5c4: execution succeeded in 58ms.
+2024-11-19 22:14:09 - INFO - Node DocumentSplitter - 5baed580-6de0-4dcd-bace-d7d947ab6c7f: execution started.
+/Users/jinhonglin/anaconda3/envs/myenv/lib/python3.11/site-packages/websockets/legacy/__init__.py:6: DeprecationWarning: websockets.legacy is deprecated; see https://websockets.readthedocs.io/en/stable/howto/upgrade.html for upgrade instructions
+ warnings.warn( # deprecated in 14.0 - 2024-11-09
+/Users/jinhonglin/anaconda3/envs/myenv/lib/python3.11/site-packages/pydantic/fields.py:804: PydanticDeprecatedSince20: Using extra keyword arguments on `Field` is deprecated and will be removed. Use `json_schema_extra` instead. (Extra keys: 'is_accessible_to_agent'). Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/
+ warn(
+2024-11-19 22:14:09 - INFO - Node DocumentSplitter - 5baed580-6de0-4dcd-bace-d7d947ab6c7f: execution succeeded in 104ms.
+2024-11-19 22:14:09 - INFO - Node OpenAIDocumentEmbedder - 91928f67-a00f-48f6-a864-f6e21672ec7e: execution started.
+2024-11-19 22:14:09 - INFO - Node OpenAIDocumentEmbedder - d30a4cdc-0fab-4aff-b2e5-6161a62cb6fd: execution started.
+2024-11-19 22:14:10 - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
+2024-11-19 22:14:10 - INFO - Node OpenAIDocumentEmbedder - d30a4cdc-0fab-4aff-b2e5-6161a62cb6fd: execution succeeded in 724ms.
+2024-11-19 22:14:10 - INFO - Node MilvusDocumentWriter - dddab4cc-1dae-4e7e-9101-1ec353f530da: execution started.
+2024-11-19 22:14:10 - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
+2024-11-19 22:14:10 - INFO - Node MilvusDocumentWriter - dddab4cc-1dae-4e7e-9101-1ec353f530da: execution succeeded in 66ms.
+2024-11-19 22:14:10 - INFO - Node OpenAIDocumentEmbedder - 91928f67-a00f-48f6-a864-f6e21672ec7e: execution succeeded in 961ms.
+2024-11-19 22:14:10 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution succeeded in 1.3s.
+2024-11-19 22:14:10 - INFO - Workflow 87878444-6a3d-43f3-ae32-0127564a959f: execution succeeded in 1.3s.
+
+Através deste fluxo de trabalho, implementámos com êxito um pipeline de indexação de documentos utilizando o Milvus como base de dados vetorial e o modelo de incorporação do OpenAI para representação semântica. Esta configuração permite uma recuperação rápida e precisa baseada em vectores, formando a base para fluxos de trabalho RAG, como a pesquisa semântica, a recuperação de documentos e as interações contextuais orientadas para a IA.
+Com os recursos de armazenamento escalonável do Milvus e a orquestração do Dynamiq, essa solução está pronta para implantações de prototipagem e produção em grande escala. Pode agora alargar este pipeline para incluir tarefas adicionais, como a resposta a perguntas baseadas na recuperação ou a geração de conteúdos orientados para a IA.
+Fluxo de recuperação de documentos RAG
Neste tutorial, implementamos um fluxo de trabalho de recuperação de documentos RAG (Retrieval-Augmented Generation). Este fluxo de trabalho recebe uma consulta do utilizador, gera uma incorporação de vetor para a mesma, recupera os documentos mais relevantes de uma base de dados de vectores Milvus e utiliza um modelo de linguagem de grande dimensão (LLM) para gerar uma resposta detalhada e contextualizada com base nos documentos recuperados.
+Ao seguir este fluxo de trabalho, criará uma solução completa para pesquisa semântica e resposta a perguntas, combinando o poder da recuperação de documentos baseada em vectores com as capacidades dos LLMs avançados da OpenAI. Esta abordagem permite respostas eficientes e inteligentes às perguntas dos utilizadores, tirando partido do conhecimento armazenado na sua base de dados de documentos.
+Importar as bibliotecas necessárias e inicializar o fluxo de trabalho
from dynamiq import Workflow
+from dynamiq.connections import (
+ OpenAI as OpenAIConnection,
+ Milvus as MilvusConnection,
+ MilvusDeploymentType,
+)
+from dynamiq.nodes.embedders import OpenAITextEmbedder
+from dynamiq.nodes.retrievers import MilvusDocumentRetriever
+from dynamiq.nodes.llms import OpenAI
+from dynamiq.prompts import Message, Prompt
+
+# Initialize the workflow
+retrieval_wf = Workflow()
+
+Definir a ligação OpenAI e o Text Embedder
# Establish OpenAI connection
+openai_connection = OpenAIConnection(api_key=os.environ["OPENAI_API_KEY"])
+
+# Define the text embedder node
+embedder = OpenAITextEmbedder(
+ connection=openai_connection,
+ model="text-embedding-3-small",
+)
+
+# Add the embedder node to the workflow
+embedder_added = retrieval_wf.flow.add_nodes(embedder)
+
+Definir o Milvus Document Retriever
document_retriever = (
+ MilvusDocumentRetriever(
+ connection=MilvusConnection(
+ deployment_type=MilvusDeploymentType.FILE, uri="./milvus.db"
+ ),
+ index_name="my_milvus_collection",
+ dimension=1536,
+ top_k=5,
+ )
+ .inputs(embedding=embedder.outputs.embedding) # Connect to embedder output
+ .depends_on(embedder) # Dependency on the embedder node
+)
+
+# Add the retriever node to the workflow
+milvus_retriever_added = retrieval_wf.flow.add_nodes(document_retriever)
+
+2024-11-19 22:14:19 - WARNING - Environment variable 'MILVUS_API_TOKEN' not found
+2024-11-19 22:14:19 - INFO - Pass in the local path ./milvus.db, and run it using milvus-lite
+2024-11-19 22:14:19 - DEBUG - Created new connection using: 98d1132773af4298a894ad5925845fd2
+2024-11-19 22:14:19 - INFO - Collection my_milvus_collection already exists. Skipping creation.
+
+Definir o modelo de prompt
# Define the prompt template for the LLM
+prompt_template = """
+Please answer the question based on the provided context.
+
+Question: {{ query }}
+
+Context:
+{% for document in documents %}
+- {{ document.content }}
+{% endfor %}
+"""
+
+# Create the prompt object
+prompt = Prompt(messages=[Message(content=prompt_template, role="user")])
+
+Definir o gerador de respostas
answer_generator = (
+ OpenAI(
+ connection=openai_connection,
+ model="gpt-4o",
+ prompt=prompt,
+ )
+ .inputs(
+ documents=document_retriever.outputs.documents,
+ query=embedder.outputs.query,
+ )
+ .depends_on(
+ [document_retriever, embedder]
+ ) # Dependencies on retriever and embedder
+)
+
+# Add the answer generator node to the workflow
+answer_generator_added = retrieval_wf.flow.add_nodes(answer_generator)
+
+Executar o fluxo de trabalho
# Run the workflow with a sample query
+sample_query = "What is the Advanced Search Algorithms in Milvus?"
+
+result = retrieval_wf.run(input_data={"query": sample_query})
+
+answer = result.output.get(answer_generator.id).get("output", {}).get("content")
+print(answer)
+
+2024-11-19 22:14:22 - INFO - Workflow f4a073fb-dfb6-499c-8cac-5710a7ad6d47: execution started.
+2024-11-19 22:14:22 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution started.
+2024-11-19 22:14:22 - INFO - Node OpenAITextEmbedder - 47afb0bc-cf96-429d-b58f-11b6c935fec3: execution started.
+2024-11-19 22:14:23 - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
+2024-11-19 22:14:23 - INFO - Node OpenAITextEmbedder - 47afb0bc-cf96-429d-b58f-11b6c935fec3: execution succeeded in 474ms.
+2024-11-19 22:14:23 - INFO - Node MilvusDocumentRetriever - 51c8311b-4837-411f-ba42-21e28239a2ee: execution started.
+2024-11-19 22:14:23 - INFO - Node MilvusDocumentRetriever - 51c8311b-4837-411f-ba42-21e28239a2ee: execution succeeded in 23ms.
+2024-11-19 22:14:23 - INFO - Node LLM - ac722325-bece-453f-a2ed-135b0749ee7a: execution started.
+2024-11-19 22:14:24 - INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
+2024-11-19 22:14:24 - INFO - Node LLM - ac722325-bece-453f-a2ed-135b0749ee7a: execution succeeded in 1.8s.
+2024-11-19 22:14:25 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution succeeded in 2.4s.
+2024-11-19 22:14:25 - INFO - Workflow f4a073fb-dfb6-499c-8cac-5710a7ad6d47: execution succeeded in 2.4s.
+
+
+The advanced search algorithms in Milvus include a variety of in-memory and on-disk indexing/search algorithms such as IVF (Inverted File), HNSW (Hierarchical Navigable Small World), and DiskANN. These algorithms have been deeply optimized to enhance performance, delivering 30%-70% better performance compared to popular implementations like FAISS and HNSWLib. These optimizations are part of Milvus's design to ensure high efficiency and scalability in handling vector data.
+
diff --git a/localization/v2.4.x/site/pt/menuStructure/pt.json b/localization/v2.4.x/site/pt/menuStructure/pt.json
index 5f47b3a8a..a2ca2a8ff 100644
--- a/localization/v2.4.x/site/pt/menuStructure/pt.json
+++ b/localization/v2.4.x/site/pt/menuStructure/pt.json
@@ -1344,6 +1344,24 @@
"id": "build_RAG_with_milvus_and_siliconflow.md",
"order": 4,
"children": []
+ },
+ {
+ "label": "SambaNova",
+ "id": "use_milvus_with_sambanova.md",
+ "order": 5,
+ "children": []
+ },
+ {
+ "label": "Gémeos",
+ "id": "build_RAG_with_milvus_and_gemini.md",
+ "order": 6,
+ "children": []
+ },
+ {
+ "label": "Ollama",
+ "id": "build_RAG_with_milvus_and_ollama.md",
+ "order": 7,
+ "children": []
}
]
},
@@ -1433,15 +1451,15 @@
"children": []
},
{
- "label": "SambaNova",
- "id": "use_milvus_with_sambanova.md",
+ "label": "PrivadoGPT",
+ "id": "use_milvus_in_private_gpt.md",
"order": 10,
"children": []
},
{
- "label": "PrivadoGPT",
- "id": "use_milvus_in_private_gpt.md",
- "order": 8,
+ "label": "Dynamiq",
+ "id": "milvus_rag_with_dynamiq.md",
+ "order": 11,
"children": []
}
]
@@ -1493,6 +1511,12 @@
"id": "knowledge_table_with_milvus.md",
"order": 2,
"children": []
+ },
+ {
+ "label": "Cognee",
+ "id": "build_RAG_with_milvus_and_cognee.md",
+ "order": 3,
+ "children": []
}
]
},
diff --git a/localization/v2.4.x/site/zh/integrations/build_RAG_with_milvus_and_cognee.json b/localization/v2.4.x/site/zh/integrations/build_RAG_with_milvus_and_cognee.json
new file mode 100644
index 000000000..ebcfd27d9
--- /dev/null
+++ b/localization/v2.4.x/site/zh/integrations/build_RAG_with_milvus_and_cognee.json
@@ -0,0 +1 @@
+{"codeList":["$ pip install pymilvus git+https://github.com/topoteretes/cognee.git\n","import os\n\nimport cognee\n\ncognee.config.set_llm_api_key(\"YOUR_OPENAI_API_KEY\")\n\n\nos.environ[\"VECTOR_DB_PROVIDER\"] = \"milvus\"\nos.environ[\"VECTOR_DB_URL\"] = \"./milvus.db\"\n","$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip\n$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs\n","from glob import glob\n\ntext_lines = []\n\nfor file_path in glob(\"milvus_docs/en/faq/*.md\", recursive=True):\n with open(file_path, \"r\") as file:\n file_text = file.read()\n\n text_lines += file_text.split(\"# \")\n","await cognee.prune.prune_data()\nawait cognee.prune.prune_system(metadata=True)\n","await cognee.add(data=text_lines, dataset_name=\"milvus_faq\")\nawait cognee.cognify()\n\n# [DocumentChunk(id=UUID('6889e7ef-3670-555c-bb16-3eb50d1d30b0'), updated_at=datetime.datetime(2024, 12, 4, 6, 29, 46, 472907, tzinfo=datetime.timezone.utc), text='Does the query perform in memory? What are incremental data and historical data?\\n\\nYes. When ...\n# ...\n","from cognee.api.v1.search import SearchType\n\nquery_text = \"How is data stored in milvus?\"\nsearch_results = await cognee.search(SearchType.SUMMARIES, query_text=query_text)\n\nprint(search_results[0])\n","from cognee.api.v1.search import SearchType\n\nquery_text = \"How is data stored in milvus?\"\nsearch_results = await cognee.search(SearchType.CHUNKS, query_text=query_text)\n","def format_and_print(data):\n print(\"ID:\", data[\"id\"])\n print(\"\\nText:\\n\")\n paragraphs = data[\"text\"].split(\"\\n\\n\")\n for paragraph in paragraphs:\n print(paragraph.strip())\n print()\n\n\nformat_and_print(search_results[0])\n","await cognee.prune.prune_data()\nawait cognee.prune.prune_system(metadata=True)\n","# We only use one line of text as the dataset, which simplifies the output later\ntext = \"\"\"\n Natural language processing (NLP) is an interdisciplinary\n subfield of computer science and information retrieval.\n \"\"\"\n\nawait cognee.add(text)\nawait cognee.cognify()\n","query_text = \"Tell me about NLP\"\nsearch_results = await cognee.search(SearchType.INSIGHTS, query_text=query_text)\n\nfor result_text in search_results:\n print(result_text)\n\n# Example output:\n# ({'id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'natural language processing', 'description': 'An interdisciplinary subfield of computer science and information retrieval.'}, {'relationship_name': 'is_a_subfield_of', 'source_node_id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'target_node_id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 15, 473137, tzinfo=datetime.timezone.utc)}, {'id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'computer science', 'description': 'The study of computation and information processing.'})\n# (...)\n#\n# It represents nodes and relationships in the knowledge graph:\n# - The first element is the source node (e.g., 'natural language processing').\n# - The second element is the relationship between nodes (e.g., 'is_a_subfield_of').\n# - The third element is the target node (e.g., 'computer science').\n"],"headingContent":"","anchorList":[{"label":"构建 RAG","href":"Build-RAG","type":2,"isActive":false}]}
\ No newline at end of file
diff --git a/localization/v2.4.x/site/zh/integrations/build_RAG_with_milvus_and_cognee.md b/localization/v2.4.x/site/zh/integrations/build_RAG_with_milvus_and_cognee.md
new file mode 100644
index 000000000..85fa339c4
--- /dev/null
+++ b/localization/v2.4.x/site/zh/integrations/build_RAG_with_milvus_and_cognee.md
@@ -0,0 +1,157 @@
+---
+id: build_RAG_with_milvus_and_cognee.md
+summary: 在本教程中，我们将向您展示如何使用 Milvus 和 Cognee 构建 RAG（检索-增强生成）管道。
+title: 与 Milvus 和 Cognee 一起构建 RAG
+---
+
+使用 Milvus 和 Cognee 构建 RAG
Cognee是一个以开发人员为先的平台，可通过可扩展的模块化 ECL（提取、认知、加载）管道简化人工智能应用程序开发。通过与 Milvus 无缝集成，Cognee 可实现对话、文档和转录的高效连接和检索，减少幻觉并优化操作符。
+凭借对 Milvus、图数据库和 LLMs 等向量存储的强大支持，Cognee 为构建检索增强生成（RAG）系统提供了一个灵活且可定制的框架。其生产就绪的架构可确保提高人工智能应用的准确性和效率。
+在本教程中，我们将向您展示如何使用 Milvus 和 Cognee 构建 RAG（检索增强生成）管道。
+$ pip install pymilvus git+https://github.com/topoteretes/cognee.git
+
+
+如果您使用的是 Google Colab，要启用刚刚安装的依赖项，可能需要重启运行时（点击屏幕上方的 "运行时 "菜单，从下拉菜单中选择 "重启会话"）。
+
+默认情况下，本例中使用 OpenAI 作为 LLM。您应准备好api 密钥，并在配置set_llm_api_key() 函数中进行设置。
+要将 Milvus 配置为向量数据库，请将VECTOR_DB_PROVIDER 设置为milvus ，并指定VECTOR_DB_URL 和VECTOR_DB_KEY 。由于我们在本演示中使用 Milvus Lite 存储数据，因此只需提供VECTOR_DB_URL 。
+import os
+
+import cognee
+
+cognee.config.set_llm_api_key("YOUR_OPENAI_API_KEY")
+
+
+os.environ["VECTOR_DB_PROVIDER"] = "milvus"
+os.environ["VECTOR_DB_URL"] = "./milvus.db"
+
+
+至于环境变量VECTOR_DB_URL 和VECTOR_DB_KEY ：
+
+- 将
VECTOR_DB_URL 设置为本地文件，如./milvus.db ，是最方便的方法，因为它会自动利用Milvus Lite将所有数据存储在此文件中。
+- 如果数据规模较大，可以在docker 或 kubernetes 上设置性能更强的 Milvus 服务器。在此设置中，请使用服务器 uri，例如
http://localhost:19530 ，作为您的VECTOR_DB_URL 。
+- 如果你想使用Zilliz Cloud（Milvus 的全托管云服务），请调整
VECTOR_DB_URL 和VECTOR_DB_KEY ，它们与 Zilliz Cloud 中的公共端点和 Api 密钥相对应。
+
+
+准备数据
我们使用Milvus 文档 2.4.x中的常见问题页面作为 RAG 中的私有知识，这对于简单的 RAG 管道来说是一个很好的数据源。
+下载 zip 文件并将文档解压缩到milvus_docs 文件夹中。
+$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
+
+我们从milvus_docs/en/faq 文件夹中加载所有标记文件。对于每个文档，我们只需简单地使用 "#"来分隔文件中的内容，这样就能大致分隔出 markdown 文件中每个主要部分的内容。
+from glob import glob
+
+text_lines = []
+
+for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
+ with open(file_path, "r") as file:
+ file_text = file.read()
+
+ text_lines += file_text.split("# ")
+
+构建 RAG
重置 Cognee 数据
await cognee.prune.prune_data()
+await cognee.prune.prune_system(metadata=True)
+
+现在，我们可以添加数据集并将其处理成知识图谱。
+添加数据和认知
await cognee.add(data=text_lines, dataset_name="milvus_faq")
+await cognee.cognify()
+
+# [DocumentChunk(id=UUID('6889e7ef-3670-555c-bb16-3eb50d1d30b0'), updated_at=datetime.datetime(2024, 12, 4, 6, 29, 46, 472907, tzinfo=datetime.timezone.utc), text='Does the query perform in memory? What are incremental data and historical data?\n\nYes. When ...
+# ...
+
+add 方法将数据集（Milvus 常见问题解答）加载到 Cognee 中，cognify 方法对数据进行处理，提取实体、关系和摘要，构建知识图谱。
+查询摘要
数据处理完成后，我们来查询知识图谱。
+from cognee.api.v1.search import SearchType
+
+query_text = "How is data stored in milvus?"
+search_results = await cognee.search(SearchType.SUMMARIES, query_text=query_text)
+
+print(search_results[0])
+
+{'id': 'de5c6713-e079-5d0b-b11d-e9bacd1e0d73', 'text': 'Milvus stores two data types: inserted data and metadata.'}
+
+该查询会在知识图谱中搜索与查询文本相关的摘要，并打印出最相关的候选摘要。
+查询摘要块
摘要提供了高层次的见解，但对于更细化的细节，我们可以直接从处理过的数据集中查询特定的数据块。这些数据块来自知识图谱创建过程中添加和分析的原始数据。
+from cognee.api.v1.search import SearchType
+
+query_text = "How is data stored in milvus?"
+search_results = await cognee.search(SearchType.CHUNKS, query_text=query_text)
+
+让我们将其格式化并显示出来，以提高可读性！
+def format_and_print(data):
+ print("ID:", data["id"])
+ print("\nText:\n")
+ paragraphs = data["text"].split("\n\n")
+ for paragraph in paragraphs:
+ print(paragraph.strip())
+ print()
+
+
+format_and_print(search_results[0])
+
+ID: 4be01c4b-9ee5-541c-9b85-297883934ab3
+
+Text:
+
+Where does Milvus store data?
+
+Milvus deals with two types of data, inserted data and metadata.
+
+Inserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).
+
+Metadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.
+
+###
+
+在前面的步骤中，我们查询了 Milvus FAQ 数据集的摘要和特定数据块。虽然这提供了详细的见解和细粒度信息，但由于数据集很大，要清晰可视化知识图谱中的依赖关系很有难度。
+为了解决这个问题，我们将重新设置 Cognee 环境，并使用更小、更集中的数据集。这将使我们能够更好地展示在认知过程中提取的关系和依赖性。通过简化数据，我们可以清楚地看到 Cognee 如何组织和构建知识图谱中的信息。
+重置 Cognee
await cognee.prune.prune_data()
+await cognee.prune.prune_system(metadata=True)
+
+添加重点数据集
在这里，我们添加并处理了一个只有一行文本的较小数据集，以确保知识图谱重点突出、易于解读。
+# We only use one line of text as the dataset, which simplifies the output later
+text = """
+ Natural language processing (NLP) is an interdisciplinary
+ subfield of computer science and information retrieval.
+ """
+
+await cognee.add(text)
+await cognee.cognify()
+
+查询洞察
通过关注这个较小的数据集，我们现在可以清楚地分析知识图谱中的关系和结构。
+query_text = "Tell me about NLP"
+search_results = await cognee.search(SearchType.INSIGHTS, query_text=query_text)
+
+for result_text in search_results:
+ print(result_text)
+
+# Example output:
+# ({'id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'natural language processing', 'description': 'An interdisciplinary subfield of computer science and information retrieval.'}, {'relationship_name': 'is_a_subfield_of', 'source_node_id': UUID('bc338a39-64d6-549a-acec-da60846dd90d'), 'target_node_id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 15, 473137, tzinfo=datetime.timezone.utc)}, {'id': UUID('6218dbab-eb6a-5759-a864-b3419755ffe0'), 'updated_at': datetime.datetime(2024, 11, 21, 12, 23, 1, 211808, tzinfo=datetime.timezone.utc), 'name': 'computer science', 'description': 'The study of computation and information processing.'})
+# (...)
+#
+# It represents nodes and relationships in the knowledge graph:
+# - The first element is the source node (e.g., 'natural language processing').
+# - The second element is the relationship between nodes (e.g., 'is_a_subfield_of').
+# - The third element is the target node (e.g., 'computer science').
+
+该输出代表了知识图谱查询的结果，展示了从处理过的数据集中提取的实体（节点）及其关系（边）。每个元组包括源实体、关系类型和目标实体，以及唯一 ID、描述和时间戳等元数据。图突出显示了关键概念及其语义联系，提供了对数据集的结构化理解。
+恭喜您，您已经学会了 cognee 与 Milvus 的基本用法。如果想了解更多 cognee 的高级用法，请参阅其官方页面。
diff --git a/localization/v2.4.x/site/zh/integrations/build_RAG_with_milvus_and_gemini.json b/localization/v2.4.x/site/zh/integrations/build_RAG_with_milvus_and_gemini.json
new file mode 100644
index 000000000..fdecb5839
--- /dev/null
+++ b/localization/v2.4.x/site/zh/integrations/build_RAG_with_milvus_and_gemini.json
@@ -0,0 +1 @@
+{"codeList":["$ pip install --upgrade pymilvus google-generativeai requests tqdm\n","import os\n\nos.environ[\"GEMINI_API_KEY\"] = \"***********\"\n","$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip\n$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs\n","from glob import glob\n\ntext_lines = []\n\nfor file_path in glob(\"milvus_docs/en/faq/*.md\", recursive=True):\n with open(file_path, \"r\") as file:\n file_text = file.read()\n\n text_lines += file_text.split(\"# \")\n","import google.generativeai as genai\n\ngenai.configure(api_key=os.environ[\"GEMINI_API_KEY\"])\n\ngemini_model = genai.GenerativeModel(\"gemini-1.5-flash\")\n\nresponse = gemini_model.generate_content(\"who are you\")\nprint(response.text)\n","test_embeddings = genai.embed_content(\n model=\"models/text-embedding-004\", content=[\"This is a test1\", \"This is a test2\"]\n)[\"embedding\"]\n\nembedding_dim = len(test_embeddings[0])\nprint(embedding_dim)\nprint(test_embeddings[0][:10])\n","from pymilvus import MilvusClient\n\nmilvus_client = MilvusClient(uri=\"./milvus_demo.db\")\n\ncollection_name = \"my_rag_collection\"\n","if milvus_client.has_collection(collection_name):\n milvus_client.drop_collection(collection_name)\n","milvus_client.create_collection(\n collection_name=collection_name,\n dimension=embedding_dim,\n metric_type=\"IP\", # Inner product distance\n consistency_level=\"Strong\", # Strong consistency level\n)\n","from tqdm import tqdm\n\ndata = []\n\ndoc_embeddings = genai.embed_content(\n model=\"models/text-embedding-004\", content=text_lines\n)[\"embedding\"]\n\nfor i, line in enumerate(tqdm(text_lines, desc=\"Creating embeddings\")):\n data.append({\"id\": i, \"vector\": doc_embeddings[i], \"text\": line})\n\nmilvus_client.insert(collection_name=collection_name, data=data)\n","question = \"How is data stored in milvus?\"\n","question_embedding = genai.embed_content(\n model=\"models/text-embedding-004\", content=question\n)[\"embedding\"]\n\nsearch_res = milvus_client.search(\n collection_name=collection_name,\n data=[question_embedding],\n limit=3, # Return top 3 results\n search_params={\"metric_type\": \"IP\", \"params\": {}}, # Inner product distance\n output_fields=[\"text\"], # Return the text field\n)\n","import json\n\nretrieved_lines_with_distances = [\n (res[\"entity\"][\"text\"], res[\"distance\"]) for res in search_res[0]\n]\nprint(json.dumps(retrieved_lines_with_distances, indent=4))\n","context = \"\\n\".join(\n [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]\n)\n","SYSTEM_PROMPT = \"\"\"\nHuman: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.\n\"\"\"\nUSER_PROMPT = f\"\"\"\nUse the following pieces of information enclosed in tags to provide an answer to the question enclosed in tags.\n\n{context}\n \n\n{question}\n \n\"\"\"\n","gemini_model = genai.GenerativeModel(\n \"gemini-1.5-flash\", system_instruction=SYSTEM_PROMPT\n)\nresponse = gemini_model.generate_content(USER_PROMPT)\nprint(response.text)\n"],"headingContent":"Build RAG with Milvus and Gemini","anchorList":[{"label":"使用 Milvus 和 Gemini 构建 RAG","href":"Build-RAG-with-Milvus-and-Gemini","type":1,"isActive":false},{"label":"准备工作","href":"Preparation","type":2,"isActive":false},{"label":"将数据载入 Milvus","href":"Load-data-into-Milvus","type":2,"isActive":false},{"label":"构建 RAG","href":"Build-RAG","type":2,"isActive":false}]}
\ No newline at end of file
diff --git a/localization/v2.4.x/site/zh/integrations/build_RAG_with_milvus_and_gemini.md b/localization/v2.4.x/site/zh/integrations/build_RAG_with_milvus_and_gemini.md
new file mode 100644
index 000000000..928c83d6f
--- /dev/null
+++ b/localization/v2.4.x/site/zh/integrations/build_RAG_with_milvus_and_gemini.md
@@ -0,0 +1,244 @@
+---
+id: build_RAG_with_milvus_and_gemini.md
+summary: >-
+ 在本教程中，我们将向您展示如何使用 Milvus 和 Gemini 构建 RAG（检索-增强生成）管道。我们将使用 Gemini
+ 模型根据给定的查询生成文本。我们还将使用 Milvus 来存储和检索生成的文本。
+title: 使用 Milvus 和 Gemini 构建 RAG
+---
+
+使用 Milvus 和 Gemini 构建 RAG
Gemini API和Google AI Studio可帮助您开始使用 Google 的最新模型，并将您的想法转化为可扩展的应用程序。Gemini 可为文本生成、文档处理、视觉、音频分析等任务提供强大的语言模型，如Gemini-1.5-Flash 、Gemini-1.5-Flash-8B 和Gemini-1.5-Pro 。通过 API，您可以输入包含数百万个标记的长语境，针对特定任务对模型进行微调，生成 JSON 等结构化输出，并利用语义检索和代码执行等功能。
+在本教程中，我们将向您展示如何使用 Milvus 和 Gemini 构建 RAG（检索-增强生成）管道。我们将使用 Gemini 模型根据给定查询生成文本。我们还将使用 Milvus 来存储和检索生成的文本。
+准备工作
依赖和环境
$ pip install --upgrade pymilvus google-generativeai requests tqdm
+
+
+如果使用的是 Google Colab，要启用刚刚安装的依赖项，可能需要重启运行时（点击屏幕上方的 "运行时 "菜单，从下拉菜单中选择 "重启会话"）。
+
+首先应登录 Google AI Studio 平台，并将api key GEMINI_API_KEY 作为环境变量。
+import os
+
+os.environ["GEMINI_API_KEY"] = "***********"
+
+准备数据
我们使用Milvus 文档 2.4.x中的常见问题页面作为 RAG 中的私有知识，这对于简单的 RAG 管道来说是一个很好的数据源。
+下载 zip 文件并将文档解压缩到milvus_docs 文件夹中。
+$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
+
+我们从milvus_docs/en/faq 文件夹中加载所有标记文件。对于每个文档，我们只需简单地使用 "#"来分隔文件中的内容，这样就能大致分隔出 markdown 文件中每个主要部分的内容。
+from glob import glob
+
+text_lines = []
+
+for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
+ with open(file_path, "r") as file:
+ file_text = file.read()
+
+ text_lines += file_text.split("# ")
+
+准备 LLM 和嵌入模型
我们使用gemini-1.5-flash 作为 LLM，使用text-embedding-004 作为 Embeddings 模型。
+让我们尝试从 LLM 生成一个测试响应：
+import google.generativeai as genai
+
+genai.configure(api_key=os.environ["GEMINI_API_KEY"])
+
+gemini_model = genai.GenerativeModel("gemini-1.5-flash")
+
+response = gemini_model.generate_content("who are you")
+print(response.text)
+
+I am a large language model, trained by Google. I am an AI and don't have a personal identity or consciousness. My purpose is to process information and respond to a wide range of prompts and questions in a helpful and informative way.
+
+生成测试嵌入并打印其维度和前几个元素。
+test_embeddings = genai.embed_content(
+ model="models/text-embedding-004", content=["This is a test1", "This is a test2"]
+)["embedding"]
+
+embedding_dim = len(test_embeddings[0])
+print(embedding_dim)
+print(test_embeddings[0][:10])
+
+768
+[0.013588584, -0.004361838, -0.08481652, -0.039724775, 0.04723794, -0.0051557426, 0.026071774, 0.045514572, -0.016867816, 0.039378334]
+
+将数据载入 Milvus
创建 Collections
from pymilvus import MilvusClient
+
+milvus_client = MilvusClient(uri="./milvus_demo.db")
+
+collection_name = "my_rag_collection"
+
+
+至于MilvusClient 的参数：
+
+- 将
uri 设置为本地文件，如./milvus.db ，是最方便的方法，因为它会自动利用Milvus Lite将所有数据存储在此文件中。
+- 如果数据规模较大，可以在docker 或 kubernetes 上设置性能更强的 Milvus 服务器。在此设置中，请使用服务器 uri，例如
http://localhost:19530 ，作为您的uri 。
+- 如果你想使用Zilliz Cloud（Milvus 的全托管云服务），请调整
uri 和token ，它们与 Zilliz Cloud 中的公共端点和 Api 密钥相对应。
+
+
+检查 Collections 是否已存在，如果已存在，则将其删除。
+if milvus_client.has_collection(collection_name):
+ milvus_client.drop_collection(collection_name)
+
+使用指定参数创建新 Collections。
+如果我们不指定任何字段信息，Milvus 会自动创建一个主键的默认id 字段，以及一个存储向量数据的vector 字段。保留的 JSON 字段用于存储非 Schema 定义的字段及其值。
+milvus_client.create_collection(
+ collection_name=collection_name,
+ dimension=embedding_dim,
+ metric_type="IP", # Inner product distance
+ consistency_level="Strong", # Strong consistency level
+)
+
+插入数据
遍历文本行，创建 Embeddings，然后将数据插入 Milvus。
+这里有一个新字段text ，它是 Collections Schema 中的一个非定义字段。它将自动添加到保留的 JSON 动态字段中，在高层次上可将其视为普通字段。
+from tqdm import tqdm
+
+data = []
+
+doc_embeddings = genai.embed_content(
+ model="models/text-embedding-004", content=text_lines
+)["embedding"]
+
+for i, line in enumerate(tqdm(text_lines, desc="Creating embeddings")):
+ data.append({"id": i, "vector": doc_embeddings[i], "text": line})
+
+milvus_client.insert(collection_name=collection_name, data=data)
+
+Creating embeddings: 100%|██████████| 72/72 [00:00<00:00, 468201.38it/s]
+
+
+
+
+
+{'insert_count': 72, 'ids': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71], 'cost': 0}
+
+构建 RAG
为查询检索数据
让我们指定一个关于 Milvus 的常见问题。
+question = "How is data stored in milvus?"
+
+在 Collections 中搜索该问题并检索语义前 3 个匹配项。
+question_embedding = genai.embed_content(
+ model="models/text-embedding-004", content=question
+)["embedding"]
+
+search_res = milvus_client.search(
+ collection_name=collection_name,
+ data=[question_embedding],
+ limit=3, # Return top 3 results
+ search_params={"metric_type": "IP", "params": {}}, # Inner product distance
+ output_fields=["text"], # Return the text field
+)
+
+让我们看看查询的搜索结果
+import json
+
+retrieved_lines_with_distances = [
+ (res["entity"]["text"], res["distance"]) for res in search_res[0]
+]
+print(json.dumps(retrieved_lines_with_distances, indent=4))
+
+[
+ [
+ " Where does Milvus store data?\n\nMilvus deals with two types of data, inserted data and metadata. \n\nInserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).\n\nMetadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.\n\n###",
+ 0.8048275113105774
+ ],
+ [
+ "Does the query perform in memory? What are incremental data and historical data?\n\nYes. When a query request comes, Milvus searches both incremental data and historical data by loading them into memory. Incremental data are in the growing segments, which are buffered in memory before they reach the threshold to be persisted in storage engine, while historical data are from the sealed segments that are stored in the object storage. Incremental data and historical data together constitute the whole dataset to search.\n\n###",
+ 0.7574886679649353
+ ],
+ [
+ "What is the maximum dataset size Milvus can handle?\n\n \nTheoretically, the maximum dataset size Milvus can handle is determined by the hardware it is run on, specifically system memory and storage:\n\n- Milvus loads all specified collections and partitions into memory before running queries. Therefore, memory size determines the maximum amount of data Milvus can query.\n- When new entities and and collection-related schema (currently only MinIO is supported for data persistence) are added to Milvus, system storage determines the maximum allowable size of inserted data.\n\n###",
+ 0.7453608512878418
+ ]
+]
+
+使用 LLM 获取 RAG 响应
将检索到的文档转换为字符串格式。
+context = "\n".join(
+ [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]
+)
+
+为 Lanage 模型定义系统和用户提示。该提示与从 Milvus 检索到的文件组装在一起。
+SYSTEM_PROMPT = """
+Human: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.
+"""
+USER_PROMPT = f"""
+Use the following pieces of information enclosed in <context> tags to provide an answer to the question enclosed in <question> tags.
+<context>
+{context}
+</context>
+<question>
+{question}
+</question>
+"""
+
+使用 Gemini 根据提示生成响应。
+gemini_model = genai.GenerativeModel(
+ "gemini-1.5-flash", system_instruction=SYSTEM_PROMPT
+)
+response = gemini_model.generate_content(USER_PROMPT)
+print(response.text)
+
+Milvus stores data in two ways: Inserted data (vector data, scalar data, and collection-specific schema) is stored as an incremental log in persistent storage using object storage backends such as MinIO, AWS S3, Google Cloud Storage, Azure Blob Storage, Alibaba Cloud OSS, and Tencent Cloud Object Storage. Metadata, generated by each Milvus module, is stored in etcd.
+
+好极了！我们利用 Milvus 和 Gemini 成功构建了一个 RAG 管道。
diff --git a/localization/v2.4.x/site/zh/integrations/build_RAG_with_milvus_and_ollama.json b/localization/v2.4.x/site/zh/integrations/build_RAG_with_milvus_and_ollama.json
new file mode 100644
index 000000000..b4b24f75a
--- /dev/null
+++ b/localization/v2.4.x/site/zh/integrations/build_RAG_with_milvus_and_ollama.json
@@ -0,0 +1 @@
+{"codeList":["$ pip install pymilvus ollama\n","$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip\n$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs\n","from glob import glob\n\ntext_lines = []\n\nfor file_path in glob(\"milvus_docs/en/faq/*.md\", recursive=True):\n with open(file_path, \"r\") as file:\n file_text = file.read()\n\n text_lines += file_text.split(\"# \")\n","! ollama pull mxbai-embed-large\n","! ollama pull llama3.2\n","import ollama\n\n\ndef emb_text(text):\n response = ollama.embeddings(model=\"mxbai-embed-large\", prompt=text)\n return response[\"embedding\"]\n","test_embedding = emb_text(\"This is a test\")\nembedding_dim = len(test_embedding)\nprint(embedding_dim)\nprint(test_embedding[:10])\n","from pymilvus import MilvusClient\n\nmilvus_client = MilvusClient(uri=\"./milvus_demo.db\")\n\ncollection_name = \"my_rag_collection\"\n","if milvus_client.has_collection(collection_name):\n milvus_client.drop_collection(collection_name)\n","milvus_client.create_collection(\n collection_name=collection_name,\n dimension=embedding_dim,\n metric_type=\"IP\", # Inner product distance\n consistency_level=\"Strong\", # Strong consistency level\n)\n","from tqdm import tqdm\n\ndata = []\n\nfor i, line in enumerate(tqdm(text_lines, desc=\"Creating embeddings\")):\n data.append({\"id\": i, \"vector\": emb_text(line), \"text\": line})\n\nmilvus_client.insert(collection_name=collection_name, data=data)\n","question = \"How is data stored in milvus?\"\n","search_res = milvus_client.search(\n collection_name=collection_name,\n data=[\n emb_text(question)\n ], # Use the `emb_text` function to convert the question to an embedding vector\n limit=3, # Return top 3 results\n search_params={\"metric_type\": \"IP\", \"params\": {}}, # Inner product distance\n output_fields=[\"text\"], # Return the text field\n)\n","import json\n\nretrieved_lines_with_distances = [\n (res[\"entity\"][\"text\"], res[\"distance\"]) for res in search_res[0]\n]\nprint(json.dumps(retrieved_lines_with_distances, indent=4))\n","context = \"\\n\".join(\n [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]\n)\n","SYSTEM_PROMPT = \"\"\"\nHuman: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.\n\"\"\"\nUSER_PROMPT = f\"\"\"\nUse the following pieces of information enclosed in tags to provide an answer to the question enclosed in tags.\n\n{context}\n \n\n{question}\n \n\"\"\"\n","from ollama import chat\nfrom ollama import ChatResponse\n\nresponse: ChatResponse = chat(\n model=\"llama3.2\",\n messages=[\n {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n {\"role\": \"user\", \"content\": USER_PROMPT},\n ],\n)\nprint(response[\"message\"][\"content\"])\n"],"headingContent":"Build RAG with Milvus and Ollama","anchorList":[{"label":"使用 Milvus 和 Ollama 构建 RAG","href":"Build-RAG-with-Milvus-and-Ollama","type":1,"isActive":false},{"label":"准备工作","href":"Preparation","type":2,"isActive":false},{"label":"将数据加载到 Milvus 中","href":"Load-data-into-Milvus","type":2,"isActive":false},{"label":"构建 RAG","href":"Build-RAG","type":2,"isActive":false}]}
\ No newline at end of file
diff --git a/localization/v2.4.x/site/zh/integrations/build_RAG_with_milvus_and_ollama.md b/localization/v2.4.x/site/zh/integrations/build_RAG_with_milvus_and_ollama.md
new file mode 100644
index 000000000..cc6ce3f7d
--- /dev/null
+++ b/localization/v2.4.x/site/zh/integrations/build_RAG_with_milvus_and_ollama.md
@@ -0,0 +1,279 @@
+---
+id: build_RAG_with_milvus_and_ollama.md
+summary: 在本指南中，我们将向您展示如何利用 Ollama 和 Milvus 高效、安全地构建 RAG（检索-增强生成）管道。
+title: 使用 Milvus 和 Ollama 构建 RAG
+---
+
+使用 Milvus 和 Ollama 构建 RAG
Ollama是一个开源平台，可简化大型语言模型 (LLM) 的本地运行和定制。它提供了用户友好的无云体验，无需高级技术技能即可轻松实现模型下载、安装和交互。凭借不断增长的预训练 LLMs 库（从通用型到特定领域型），Ollama 可以轻松管理和定制各种应用的模型。它确保了数据的私密性和灵活性，使用户能够完全在自己的机器上微调、优化和部署人工智能驱动的解决方案。
+在本指南中，我们将向您展示如何利用 Ollama 和 Milvus 高效、安全地构建 RAG（检索-增强生成）管道。
+准备工作
依赖关系和环境
$ pip install pymilvus ollama
+
+
+如果使用的是 Google Colab，要启用刚刚安装的依赖项，可能需要重启运行时（点击屏幕上方的 "运行时 "菜单，从下拉菜单中选择 "重启会话"）。
+
+准备数据
我们使用Milvus 文档 2.4.x中的常见问题页面作为 RAG 中的私有知识，这对于简单的 RAG 管道来说是一个很好的数据源。
+下载 zip 文件并将文档解压缩到milvus_docs 文件夹中。
+$ wget https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+$ unzip -q milvus_docs_2.4.x_en.zip -d milvus_docs
+
+--2024-11-26 21:47:19-- https://github.com/milvus-io/milvus-docs/releases/download/v2.4.6-preview/milvus_docs_2.4.x_en.zip
+Resolving github.com (github.com)... 140.82.112.4
+Connecting to github.com (github.com)|140.82.112.4|:443... connected.
+HTTP request sent, awaiting response... 302 Found
+Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/267273319/c52902a0-e13c-4ca7-92e0-086751098a05?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241127%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241127T024720Z&X-Amz-Expires=300&X-Amz-Signature=7808b77cbdaa7e122196bcd75a73f29f2540333a350c4830bbdf5f286e876304&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dmilvus_docs_2.4.x_en.zip&response-content-type=application%2Foctet-stream [following]
+--2024-11-26 21:47:20-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/267273319/c52902a0-e13c-4ca7-92e0-086751098a05?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241127%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241127T024720Z&X-Amz-Expires=300&X-Amz-Signature=7808b77cbdaa7e122196bcd75a73f29f2540333a350c4830bbdf5f286e876304&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dmilvus_docs_2.4.x_en.zip&response-content-type=application%2Foctet-stream
+Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.109.133, 185.199.111.133, 185.199.108.133, ...
+Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.109.133|:443... connected.
+HTTP request sent, awaiting response... 200 OK
+Length: 613094 (599K) [application/octet-stream]
+Saving to: ‘milvus_docs_2.4.x_en.zip’
+
+milvus_docs_2.4.x_e 100%[===================>] 598.72K 1.20MB/s in 0.5s
+
+2024-11-26 21:47:20 (1.20 MB/s) - ‘milvus_docs_2.4.x_en.zip’ saved [613094/613094]
+
+我们从milvus_docs/en/faq 文件夹中加载所有标记文件。对于每个文档，我们只需简单地使用 "#"来分隔文件中的内容，这样就能大致分隔出 markdown 文件中每个主要部分的内容。
+from glob import glob
+
+text_lines = []
+
+for file_path in glob("milvus_docs/en/faq/*.md", recursive=True):
+ with open(file_path, "r") as file:
+ file_text = file.read()
+
+ text_lines += file_text.split("# ")
+
+准备 LLM 和 Embeddings 模型
Ollama 支持基于 LLM 任务和嵌入生成的多种模型，这使得开发检索增强生成（RAG）应用变得非常容易。在此设置中
+
+- 我们将使用Llama 3.2 (3B)作为文本生成任务的 LLM。
+- 对于嵌入生成，我们将使用mxbai-embed-large，这是一个针对语义相似性优化的 334M 参数模型。
+
+在开始之前，请确保这两个模型都已拉到本地：
+! ollama pull mxbai-embed-large
+
+[?25lpulling manifest ⠋ [?25h[?25l[2K[1Gpulling manifest ⠙ [?25h[?25l[2K[1Gpulling manifest ⠹ [?25h[?25l[2K[1Gpulling manifest ⠸ [?25h[?25l[2K[1Gpulling manifest ⠼ [?25h[?25l[2K[1Gpulling manifest ⠴ [?25h[?25l[2K[1Gpulling manifest
+pulling 819c2adf5ce6... 100% ▕████████████████▏ 669 MB
+pulling c71d239df917... 100% ▕████████████████▏ 11 KB
+pulling b837481ff855... 100% ▕████████████████▏ 16 B
+pulling 38badd946f91... 100% ▕████████████████▏ 408 B
+verifying sha256 digest
+writing manifest
+success [?25h
+
+! ollama pull llama3.2
+
+[?25lpulling manifest ⠋ [?25h[?25l[2K[1Gpulling manifest ⠙ [?25h[?25l[2K[1Gpulling manifest ⠹ [?25h[?25l[2K[1Gpulling manifest ⠸ [?25h[?25l[2K[1Gpulling manifest ⠼ [?25h[?25l[2K[1Gpulling manifest ⠴ [?25h[?25l[2K[1Gpulling manifest
+pulling dde5aa3fc5ff... 100% ▕████████████████▏ 2.0 GB
+pulling 966de95ca8a6... 100% ▕████████████████▏ 1.4 KB
+pulling fcc5a6bec9da... 100% ▕████████████████▏ 7.7 KB
+pulling a70ff7e570d9... 100% ▕████████████████▏ 6.0 KB
+pulling 56bb8bd477a5... 100% ▕████████████████▏ 96 B
+pulling 34bb5ab01051... 100% ▕████████████████▏ 561 B
+verifying sha256 digest
+writing manifest
+success [?25h
+
+准备好这些模型后，我们就可以着手实施 LLM 驱动的生成和基于嵌入的检索工作流程了。
+import ollama
+
+
+def emb_text(text):
+ response = ollama.embeddings(model="mxbai-embed-large", prompt=text)
+ return response["embedding"]
+
+生成一个测试嵌入并打印其维度和前几个元素。
+test_embedding = emb_text("This is a test")
+embedding_dim = len(test_embedding)
+print(embedding_dim)
+print(test_embedding[:10])
+
+1024
+[0.23276396095752716, 0.4257211685180664, 0.19724100828170776, 0.46120673418045044, -0.46039995551109314, -0.1413791924715042, -0.18261606991291046, -0.07602324336767197, 0.39991313219070435, 0.8337644338607788]
+
+将数据加载到 Milvus 中
创建 Collections
from pymilvus import MilvusClient
+
+milvus_client = MilvusClient(uri="./milvus_demo.db")
+
+collection_name = "my_rag_collection"
+
+
+至于MilvusClient 的参数：
+
+- 将
uri 设置为本地文件，如./milvus.db ，是最方便的方法，因为它会自动利用Milvus Lite将所有数据存储在此文件中。
+- 如果数据规模较大，可以在docker 或 kubernetes 上设置性能更强的 Milvus 服务器。在此设置中，请使用服务器 uri，例如
http://localhost:19530 ，作为您的uri 。
+- 如果你想使用Zilliz Cloud（Milvus 的全托管云服务），请调整
uri 和token ，它们与 Zilliz Cloud 中的公共端点和 Api 密钥相对应。
+
+
+检查 Collections 是否已存在，如果已存在，则将其删除。
+if milvus_client.has_collection(collection_name):
+ milvus_client.drop_collection(collection_name)
+
+使用指定参数创建新 Collections。
+如果我们不指定任何字段信息，Milvus 会自动创建一个主键的默认id 字段，以及一个存储向量数据的vector 字段。保留的 JSON 字段用于存储非 Schema 定义的字段及其值。
+milvus_client.create_collection(
+ collection_name=collection_name,
+ dimension=embedding_dim,
+ metric_type="IP", # Inner product distance
+ consistency_level="Strong", # Strong consistency level
+)
+
+插入数据
遍历文本行，创建 Embeddings，然后将数据插入 Milvus。
+这里有一个新字段text ，它是 Collections Schema 中的一个非定义字段。它将自动添加到保留的 JSON 动态字段中，在高层次上可将其视为普通字段。
+from tqdm import tqdm
+
+data = []
+
+for i, line in enumerate(tqdm(text_lines, desc="Creating embeddings")):
+ data.append({"id": i, "vector": emb_text(line), "text": line})
+
+milvus_client.insert(collection_name=collection_name, data=data)
+
+Creating embeddings: 100%|██████████| 72/72 [00:03<00:00, 22.56it/s]
+
+
+
+
+
+{'insert_count': 72, 'ids': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71], 'cost': 0}
+
+构建 RAG
为查询检索数据
让我们指定一个关于 Milvus 的常见问题。
+question = "How is data stored in milvus?"
+
+在 Collections 中搜索该问题并检索语义前 3 个匹配项。
+search_res = milvus_client.search(
+ collection_name=collection_name,
+ data=[
+ emb_text(question)
+ ], # Use the `emb_text` function to convert the question to an embedding vector
+ limit=3, # Return top 3 results
+ search_params={"metric_type": "IP", "params": {}}, # Inner product distance
+ output_fields=["text"], # Return the text field
+)
+
+让我们看看查询的搜索结果
+import json
+
+retrieved_lines_with_distances = [
+ (res["entity"]["text"], res["distance"]) for res in search_res[0]
+]
+print(json.dumps(retrieved_lines_with_distances, indent=4))
+
+[
+ [
+ " Where does Milvus store data?\n\nMilvus deals with two types of data, inserted data and metadata. \n\nInserted data, including vector data, scalar data, and collection-specific schema, are stored in persistent storage as incremental log. Milvus supports multiple object storage backends, including [MinIO](https://min.io/), [AWS S3](https://aws.amazon.com/s3/?nc1=h_ls), [Google Cloud Storage](https://cloud.google.com/storage?hl=en#object-storage-for-companies-of-all-sizes) (GCS), [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service), and [Tencent Cloud Object Storage](https://www.tencentcloud.com/products/cos) (COS).\n\nMetadata are generated within Milvus. Each Milvus module has its own metadata that are stored in etcd.\n\n###",
+ 231.9398193359375
+ ],
+ [
+ "How does Milvus flush data?\n\nMilvus returns success when inserted data are loaded to the message queue. However, the data are not yet flushed to the disk. Then Milvus' data node writes the data in the message queue to persistent storage as incremental logs. If `flush()` is called, the data node is forced to write all data in the message queue to persistent storage immediately.\n\n###",
+ 226.48316955566406
+ ],
+ [
+ "What is the maximum dataset size Milvus can handle?\n\n \nTheoretically, the maximum dataset size Milvus can handle is determined by the hardware it is run on, specifically system memory and storage:\n\n- Milvus loads all specified collections and partitions into memory before running queries. Therefore, memory size determines the maximum amount of data Milvus can query.\n- When new entities and and collection-related schema (currently only MinIO is supported for data persistence) are added to Milvus, system storage determines the maximum allowable size of inserted data.\n\n###",
+ 210.60745239257812
+ ]
+]
+
+使用 LLM 获取 RAG 响应
将检索到的文档转换为字符串格式。
+context = "\n".join(
+ [line_with_distance[0] for line_with_distance in retrieved_lines_with_distances]
+)
+
+为 Lanage 模型定义系统和用户提示。该提示与从 Milvus 检索到的文档组装在一起。
+SYSTEM_PROMPT = """
+Human: You are an AI assistant. You are able to find answers to the questions from the contextual passage snippets provided.
+"""
+USER_PROMPT = f"""
+Use the following pieces of information enclosed in <context> tags to provide an answer to the question enclosed in <question> tags.
+<context>
+{context}
+</context>
+<question>
+{question}
+</question>
+"""
+
+使用 Ollama 提供的llama3.2 模型，根据提示生成响应。
+from ollama import chat
+from ollama import ChatResponse
+
+response: ChatResponse = chat(
+ model="llama3.2",
+ messages=[
+ {"role": "system", "content": SYSTEM_PROMPT},
+ {"role": "user", "content": USER_PROMPT},
+ ],
+)
+print(response["message"]["content"])
+
+According to the provided context, data in Milvus is stored in two types:
+
+1. **Inserted data**: Storing data in persistent storage as incremental log. It supports multiple object storage backends such as MinIO, AWS S3, Google Cloud Storage (GCS), Azure Blob Storage, Alibaba Cloud OSS, and Tencent Cloud Object Storage.
+
+2. **Metadata**: Generated within Milvus and stored in etcd.
+
+很好！我们利用 Milvus 和 Ollama 成功构建了 RAG 管道。
diff --git a/localization/v2.4.x/site/zh/integrations/milvus_rag_with_dynamiq.json b/localization/v2.4.x/site/zh/integrations/milvus_rag_with_dynamiq.json
new file mode 100644
index 000000000..2aa46d9a1
--- /dev/null
+++ b/localization/v2.4.x/site/zh/integrations/milvus_rag_with_dynamiq.json
@@ -0,0 +1 @@
+{"codeList":["$ pip install dynamiq pymilvus\n","import os\n\nos.environ[\"OPENAI_API_KEY\"] = \"sk-***********\"\n","# Importing necessary libraries for the workflow\nfrom io import BytesIO\nfrom dynamiq import Workflow\nfrom dynamiq.nodes import InputTransformer\nfrom dynamiq.connections import (\n OpenAI as OpenAIConnection,\n Milvus as MilvusConnection,\n MilvusDeploymentType,\n)\nfrom dynamiq.nodes.converters import PyPDFConverter\nfrom dynamiq.nodes.splitters.document import DocumentSplitter\nfrom dynamiq.nodes.embedders import OpenAIDocumentEmbedder\nfrom dynamiq.nodes.writers import MilvusDocumentWriter\n\n# Initialize the workflow\nrag_wf = Workflow()\n","converter = PyPDFConverter(document_creation_mode=\"one-doc-per-page\")\nconverter_added = rag_wf.flow.add_nodes(\n converter\n) # Add node to the DAG (Directed Acyclic Graph)\n","document_splitter = DocumentSplitter(\n split_by=\"sentence\", # Splits documents into sentences\n split_length=10,\n split_overlap=1,\n input_transformer=InputTransformer(\n selector={\n \"documents\": f\"${[converter.id]}.output.documents\",\n },\n ),\n).depends_on(\n converter\n) # Set dependency on the PDF converter\nsplitter_added = rag_wf.flow.add_nodes(document_splitter) # Add to the DAG\n","embedder = OpenAIDocumentEmbedder(\n connection=OpenAIConnection(api_key=os.environ[\"OPENAI_API_KEY\"]),\n input_transformer=InputTransformer(\n selector={\n \"documents\": f\"${[document_splitter.id]}.output.documents\",\n },\n ),\n).depends_on(\n document_splitter\n) # Set dependency on the splitter\ndocument_embedder_added = rag_wf.flow.add_nodes(embedder) # Add to the DAG\n","vector_store = (\n MilvusDocumentWriter(\n connection=MilvusConnection(\n deployment_type=MilvusDeploymentType.FILE, uri=\"./milvus.db\"\n ),\n index_name=\"my_milvus_collection\",\n dimension=1536,\n create_if_not_exist=True,\n metric_type=\"COSINE\",\n )\n .inputs(documents=embedder.outputs.documents) # Connect to embedder output\n .depends_on(embedder) # Set dependency on the embedder\n)\nmilvus_writer_added = rag_wf.flow.add_nodes(vector_store) # Add to the DAG\n","file_paths = [\"./pdf_files/WhatisMilvus.pdf\"]\ninput_data = {\n \"files\": [BytesIO(open(path, \"rb\").read()) for path in file_paths],\n \"metadata\": [{\"filename\": path} for path in file_paths],\n}\n\n# Run the workflow with the prepared input data\ninserted_data = rag_wf.run(input_data=input_data)\n","from dynamiq import Workflow\nfrom dynamiq.connections import (\n OpenAI as OpenAIConnection,\n Milvus as MilvusConnection,\n MilvusDeploymentType,\n)\nfrom dynamiq.nodes.embedders import OpenAITextEmbedder\nfrom dynamiq.nodes.retrievers import MilvusDocumentRetriever\nfrom dynamiq.nodes.llms import OpenAI\nfrom dynamiq.prompts import Message, Prompt\n\n# Initialize the workflow\nretrieval_wf = Workflow()\n","# Establish OpenAI connection\nopenai_connection = OpenAIConnection(api_key=os.environ[\"OPENAI_API_KEY\"])\n\n# Define the text embedder node\nembedder = OpenAITextEmbedder(\n connection=openai_connection,\n model=\"text-embedding-3-small\",\n)\n\n# Add the embedder node to the workflow\nembedder_added = retrieval_wf.flow.add_nodes(embedder)\n","document_retriever = (\n MilvusDocumentRetriever(\n connection=MilvusConnection(\n deployment_type=MilvusDeploymentType.FILE, uri=\"./milvus.db\"\n ),\n index_name=\"my_milvus_collection\",\n dimension=1536,\n top_k=5,\n )\n .inputs(embedding=embedder.outputs.embedding) # Connect to embedder output\n .depends_on(embedder) # Dependency on the embedder node\n)\n\n# Add the retriever node to the workflow\nmilvus_retriever_added = retrieval_wf.flow.add_nodes(document_retriever)\n","# Define the prompt template for the LLM\nprompt_template = \"\"\"\nPlease answer the question based on the provided context.\n\nQuestion: {{ query }}\n\nContext:\n{% for document in documents %}\n- {{ document.content }}\n{% endfor %}\n\"\"\"\n\n# Create the prompt object\nprompt = Prompt(messages=[Message(content=prompt_template, role=\"user\")])\n","answer_generator = (\n OpenAI(\n connection=openai_connection,\n model=\"gpt-4o\",\n prompt=prompt,\n )\n .inputs(\n documents=document_retriever.outputs.documents,\n query=embedder.outputs.query,\n )\n .depends_on(\n [document_retriever, embedder]\n ) # Dependencies on retriever and embedder\n)\n\n# Add the answer generator node to the workflow\nanswer_generator_added = retrieval_wf.flow.add_nodes(answer_generator)\n","# Run the workflow with a sample query\nsample_query = \"What is the Advanced Search Algorithms in Milvus?\"\n\nresult = retrieval_wf.run(input_data={\"query\": sample_query})\n\nanswer = result.output.get(answer_generator.id).get(\"output\", {}).get(\"content\")\nprint(answer)\n"],"headingContent":"Getting Started with Dynamiq and Milvus","anchorList":[{"label":"开始使用 Dynamiq 和 Milvus","href":"Getting-Started-with-Dynamiq-and-Milvus","type":1,"isActive":false},{"label":"准备工作","href":"Preparation","type":2,"isActive":false},{"label":"RAG - 文档索引流程","href":"RAG---Document-Indexing-Flow","type":2,"isActive":false},{"label":"RAG 文档检索流程","href":"RAG-Document-Retrieval-Flow","type":2,"isActive":false}]}
\ No newline at end of file
diff --git a/localization/v2.4.x/site/zh/integrations/milvus_rag_with_dynamiq.md b/localization/v2.4.x/site/zh/integrations/milvus_rag_with_dynamiq.md
new file mode 100644
index 000000000..33e0cafbd
--- /dev/null
+++ b/localization/v2.4.x/site/zh/integrations/milvus_rag_with_dynamiq.md
@@ -0,0 +1,336 @@
+---
+id: milvus_rag_with_dynamiq.md
+summary: >-
+ 在本教程中，我们将探讨如何将 Dynamiq 与 Milvus（专为 RAG 工作流打造的高性能向量数据库）无缝结合使用。Milvus
+ 擅长向量嵌入的高效存储、索引和检索，是要求快速、精确访问上下文数据的人工智能系统不可或缺的组件。
+title: 开始使用 Dynamiq 和 Milvus
+---
+
+开始使用 Dynamiq 和 Milvus
Dynamiq是一个功能强大的 Gen AI 框架，可简化人工智能应用的开发。凭借对检索增强生成（RAG）和大型语言模型（LLM）Agent 的强大支持，Dynamiq 使开发人员能够轻松高效地创建智能动态系统。
+在本教程中，我们将探讨如何将 Dynamiq 与Milvus（专为 RAG 工作流打造的高性能向量数据库）无缝结合使用。Milvus 擅长向量嵌入的高效存储、索引和检索，是要求快速、精确访问上下文数据的人工智能系统不可或缺的组件。
+本分步指南将涵盖两个核心 RAG 工作流程：
+
+文档索引流程：了解如何处理输入文件（如 PDF），将其内容转换为向量嵌入，并存储到 Milvus 中。利用 Milvus 的高性能索引功能，可确保您的数据随时可供快速检索。
文档检索流程：了解如何查询 Milvus 中的相关文档嵌入，并使用它们与 Dynamiq 的 LLM Agents 一起生成有洞察力的、上下文感知的响应，从而创建无缝的人工智能驱动的用户体验。
+本教程结束时，您将对 Milvus 和 Dynamiq 如何协同工作，根据您的需求量身打造可扩展的上下文感知人工智能系统有一个扎实的了解。
+准备工作
下载所需程序库
$ pip install dynamiq pymilvus
+
+
+如果使用的是 Google Colab，要启用刚刚安装的依赖项，可能需要重启运行时（点击屏幕上方的 "运行时 "菜单，从下拉菜单中选择 "重启会话"）。
+
+配置 LLM Agent
在本例中，我们将使用 OpenAI 作为 LLM。您应将api key OPENAI_API_KEY 设置为环境变量。
+import os
+
+os.environ["OPENAI_API_KEY"] = "sk-***********"
+
+RAG - 文档索引流程
本教程演示了以 Milvus 为向量数据库编制文档索引的检索增强生成（RAG）工作流程。该工作流程接收输入的 PDF 文件，将其处理成较小的块，使用 OpenAI 的嵌入模型生成向量嵌入，并将嵌入存储在 Milvus Collections 中，以便高效检索。
+本工作流程结束时，您将拥有一个可扩展的高效文档索引系统，可支持语义搜索和问题解答等未来的 RAG 任务。
+导入所需库并初始化工作流
# Importing necessary libraries for the workflow
+from io import BytesIO
+from dynamiq import Workflow
+from dynamiq.nodes import InputTransformer
+from dynamiq.connections import (
+ OpenAI as OpenAIConnection,
+ Milvus as MilvusConnection,
+ MilvusDeploymentType,
+)
+from dynamiq.nodes.converters import PyPDFConverter
+from dynamiq.nodes.splitters.document import DocumentSplitter
+from dynamiq.nodes.embedders import OpenAIDocumentEmbedder
+from dynamiq.nodes.writers import MilvusDocumentWriter
+
+# Initialize the workflow
+rag_wf = Workflow()
+
+定义 PDF 转换器节点
converter = PyPDFConverter(document_creation_mode="one-doc-per-page")
+converter_added = rag_wf.flow.add_nodes(
+ converter
+) # Add node to the DAG (Directed Acyclic Graph)
+
+定义文档分割器节点
document_splitter = DocumentSplitter(
+ split_by="sentence", # Splits documents into sentences
+ split_length=10,
+ split_overlap=1,
+ input_transformer=InputTransformer(
+ selector={
+ "documents": f"${[converter.id]}.output.documents",
+ },
+ ),
+).depends_on(
+ converter
+) # Set dependency on the PDF converter
+splitter_added = rag_wf.flow.add_nodes(document_splitter) # Add to the DAG
+
+定义 Embeddings 节点
embedder = OpenAIDocumentEmbedder(
+ connection=OpenAIConnection(api_key=os.environ["OPENAI_API_KEY"]),
+ input_transformer=InputTransformer(
+ selector={
+ "documents": f"${[document_splitter.id]}.output.documents",
+ },
+ ),
+).depends_on(
+ document_splitter
+) # Set dependency on the splitter
+document_embedder_added = rag_wf.flow.add_nodes(embedder) # Add to the DAG
+
+定义 Milvus 向量存储节点
vector_store = (
+ MilvusDocumentWriter(
+ connection=MilvusConnection(
+ deployment_type=MilvusDeploymentType.FILE, uri="./milvus.db"
+ ),
+ index_name="my_milvus_collection",
+ dimension=1536,
+ create_if_not_exist=True,
+ metric_type="COSINE",
+ )
+ .inputs(documents=embedder.outputs.documents) # Connect to embedder output
+ .depends_on(embedder) # Set dependency on the embedder
+)
+milvus_writer_added = rag_wf.flow.add_nodes(vector_store) # Add to the DAG
+
+2024-11-19 22:14:03 - WARNING - Environment variable 'MILVUS_API_TOKEN' not found
+2024-11-19 22:14:03 - INFO - Pass in the local path ./milvus.db, and run it using milvus-lite
+2024-11-19 22:14:04 - DEBUG - Created new connection using: 0bef2849fdb1458a85df8bb9dd27f51d
+2024-11-19 22:14:04 - INFO - Collection my_milvus_collection does not exist. Creating a new collection.
+2024-11-19 22:14:04 - DEBUG - Successfully created collection: my_milvus_collection
+2024-11-19 22:14:05 - DEBUG - Successfully created an index on collection: my_milvus_collection
+2024-11-19 22:14:05 - DEBUG - Successfully created an index on collection: my_milvus_collection
+
+
+Milvus 提供两种部署类型，以满足不同的使用情况：
+
+- MilvusDeploymentType.FILE
+
+
+- 本地原型或小规模数据存储的理想选择。
+- 将
uri 设置为本地文件路径（如./milvus.db ），以利用Milvus Lite，它可自动将所有数据存储到指定文件中。
+- 这是快速设置和实验的便捷选项。
+
+
+- MilvusDeploymentType.HOST
+
+
+专为大规模数据场景设计，例如管理超过一百万个向量。
+自托管服务器
+
+- 使用Docker 或 Kubernetes 部署高性能 Milvus 服务器。
+- 将服务器地址和端口配置为
uri （如http://localhost:19530 ）。
+- 如果启用了身份验证：
+- 提供
<your_username>:<your_password> 作为token 。
+- 如果禁用了身份验证：
+- 不设置
token 。
+
+Zilliz Cloud（托管服务）
+
+- 要获得全面管理、基于云的 Milvus 体验，请使用Zilliz Cloud。
+- 根据 Zilliz Cloud 控制台提供的公共端点和 API 密钥设置
uri 和token 。
+
+
+定义输入数据并运行工作流
file_paths = ["./pdf_files/WhatisMilvus.pdf"]
+input_data = {
+ "files": [BytesIO(open(path, "rb").read()) for path in file_paths],
+ "metadata": [{"filename": path} for path in file_paths],
+}
+
+# Run the workflow with the prepared input data
+inserted_data = rag_wf.run(input_data=input_data)
+
+/var/folders/09/d0hx80nj35sb5hxb5cpc1q180000gn/T/ipykernel_31319/3145804345.py:4: ResourceWarning: unclosed file <_io.BufferedReader name='./pdf_files/WhatisMilvus.pdf'>
+ BytesIO(open(path, "rb").read()) for path in file_paths
+ResourceWarning: Enable tracemalloc to get the object allocation traceback
+2024-11-19 22:14:09 - INFO - Workflow 87878444-6a3d-43f3-ae32-0127564a959f: execution started.
+2024-11-19 22:14:09 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution started.
+2024-11-19 22:14:09 - INFO - Node PyPDF File Converter - 6eb42b1f-7637-407b-a3ac-4167bcf3b5c4: execution started.
+2024-11-19 22:14:09 - INFO - Node PyPDF File Converter - 6eb42b1f-7637-407b-a3ac-4167bcf3b5c4: execution succeeded in 58ms.
+2024-11-19 22:14:09 - INFO - Node DocumentSplitter - 5baed580-6de0-4dcd-bace-d7d947ab6c7f: execution started.
+/Users/jinhonglin/anaconda3/envs/myenv/lib/python3.11/site-packages/websockets/legacy/__init__.py:6: DeprecationWarning: websockets.legacy is deprecated; see https://websockets.readthedocs.io/en/stable/howto/upgrade.html for upgrade instructions
+ warnings.warn( # deprecated in 14.0 - 2024-11-09
+/Users/jinhonglin/anaconda3/envs/myenv/lib/python3.11/site-packages/pydantic/fields.py:804: PydanticDeprecatedSince20: Using extra keyword arguments on `Field` is deprecated and will be removed. Use `json_schema_extra` instead. (Extra keys: 'is_accessible_to_agent'). Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.7/migration/
+ warn(
+2024-11-19 22:14:09 - INFO - Node DocumentSplitter - 5baed580-6de0-4dcd-bace-d7d947ab6c7f: execution succeeded in 104ms.
+2024-11-19 22:14:09 - INFO - Node OpenAIDocumentEmbedder - 91928f67-a00f-48f6-a864-f6e21672ec7e: execution started.
+2024-11-19 22:14:09 - INFO - Node OpenAIDocumentEmbedder - d30a4cdc-0fab-4aff-b2e5-6161a62cb6fd: execution started.
+2024-11-19 22:14:10 - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
+2024-11-19 22:14:10 - INFO - Node OpenAIDocumentEmbedder - d30a4cdc-0fab-4aff-b2e5-6161a62cb6fd: execution succeeded in 724ms.
+2024-11-19 22:14:10 - INFO - Node MilvusDocumentWriter - dddab4cc-1dae-4e7e-9101-1ec353f530da: execution started.
+2024-11-19 22:14:10 - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
+2024-11-19 22:14:10 - INFO - Node MilvusDocumentWriter - dddab4cc-1dae-4e7e-9101-1ec353f530da: execution succeeded in 66ms.
+2024-11-19 22:14:10 - INFO - Node OpenAIDocumentEmbedder - 91928f67-a00f-48f6-a864-f6e21672ec7e: execution succeeded in 961ms.
+2024-11-19 22:14:10 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution succeeded in 1.3s.
+2024-11-19 22:14:10 - INFO - Workflow 87878444-6a3d-43f3-ae32-0127564a959f: execution succeeded in 1.3s.
+
+通过这个工作流程，我们成功实现了一个文档索引管道，使用 Milvus 作为向量数据库，并使用 OpenAI 的 Embeddings 模型进行语义表示。这一设置实现了快速、准确的基于向量的检索，为语义搜索、文档检索和上下文人工智能驱动的交互等 RAG 工作流程奠定了基础。
+借助 Milvus 的可扩展存储功能和 Dynamiq 的协调功能，该解决方案既可用于原型开发，也可用于大规模生产部署。现在，您可以扩展这一管道，将基于检索的问题解答或人工智能驱动的内容生成等额外任务纳入其中。
+RAG 文档检索流程
在本教程中，我们将实现一个检索-增强生成（RAG）文档检索工作流。该工作流接收用户查询，为其生成向量嵌入，从 Milvus 向量数据库中检索最相关的文档，并使用大型语言模型（LLM）根据检索到的文档生成详细的、上下文感知的答案。
+通过遵循这一工作流程，您将创建一个用于语义搜索和问题解答的端到端解决方案，将基于向量的文档检索功能与 OpenAI 高级 LLMs 的能力结合起来。这种方法可利用文档数据库中存储的知识，对用户查询做出高效、智能的响应。
+导入所需库并初始化工作流程
from dynamiq import Workflow
+from dynamiq.connections import (
+ OpenAI as OpenAIConnection,
+ Milvus as MilvusConnection,
+ MilvusDeploymentType,
+)
+from dynamiq.nodes.embedders import OpenAITextEmbedder
+from dynamiq.nodes.retrievers import MilvusDocumentRetriever
+from dynamiq.nodes.llms import OpenAI
+from dynamiq.prompts import Message, Prompt
+
+# Initialize the workflow
+retrieval_wf = Workflow()
+
+定义 OpenAI 连接和文本嵌入器
# Establish OpenAI connection
+openai_connection = OpenAIConnection(api_key=os.environ["OPENAI_API_KEY"])
+
+# Define the text embedder node
+embedder = OpenAITextEmbedder(
+ connection=openai_connection,
+ model="text-embedding-3-small",
+)
+
+# Add the embedder node to the workflow
+embedder_added = retrieval_wf.flow.add_nodes(embedder)
+
+定义 Milvus 文档检索器
document_retriever = (
+ MilvusDocumentRetriever(
+ connection=MilvusConnection(
+ deployment_type=MilvusDeploymentType.FILE, uri="./milvus.db"
+ ),
+ index_name="my_milvus_collection",
+ dimension=1536,
+ top_k=5,
+ )
+ .inputs(embedding=embedder.outputs.embedding) # Connect to embedder output
+ .depends_on(embedder) # Dependency on the embedder node
+)
+
+# Add the retriever node to the workflow
+milvus_retriever_added = retrieval_wf.flow.add_nodes(document_retriever)
+
+2024-11-19 22:14:19 - WARNING - Environment variable 'MILVUS_API_TOKEN' not found
+2024-11-19 22:14:19 - INFO - Pass in the local path ./milvus.db, and run it using milvus-lite
+2024-11-19 22:14:19 - DEBUG - Created new connection using: 98d1132773af4298a894ad5925845fd2
+2024-11-19 22:14:19 - INFO - Collection my_milvus_collection already exists. Skipping creation.
+
+定义提示模板
# Define the prompt template for the LLM
+prompt_template = """
+Please answer the question based on the provided context.
+
+Question: {{ query }}
+
+Context:
+{% for document in documents %}
+- {{ document.content }}
+{% endfor %}
+"""
+
+# Create the prompt object
+prompt = Prompt(messages=[Message(content=prompt_template, role="user")])
+
+定义答案生成器
answer_generator = (
+ OpenAI(
+ connection=openai_connection,
+ model="gpt-4o",
+ prompt=prompt,
+ )
+ .inputs(
+ documents=document_retriever.outputs.documents,
+ query=embedder.outputs.query,
+ )
+ .depends_on(
+ [document_retriever, embedder]
+ ) # Dependencies on retriever and embedder
+)
+
+# Add the answer generator node to the workflow
+answer_generator_added = retrieval_wf.flow.add_nodes(answer_generator)
+
+运行工作流
# Run the workflow with a sample query
+sample_query = "What is the Advanced Search Algorithms in Milvus?"
+
+result = retrieval_wf.run(input_data={"query": sample_query})
+
+answer = result.output.get(answer_generator.id).get("output", {}).get("content")
+print(answer)
+
+2024-11-19 22:14:22 - INFO - Workflow f4a073fb-dfb6-499c-8cac-5710a7ad6d47: execution started.
+2024-11-19 22:14:22 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution started.
+2024-11-19 22:14:22 - INFO - Node OpenAITextEmbedder - 47afb0bc-cf96-429d-b58f-11b6c935fec3: execution started.
+2024-11-19 22:14:23 - INFO - HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"
+2024-11-19 22:14:23 - INFO - Node OpenAITextEmbedder - 47afb0bc-cf96-429d-b58f-11b6c935fec3: execution succeeded in 474ms.
+2024-11-19 22:14:23 - INFO - Node MilvusDocumentRetriever - 51c8311b-4837-411f-ba42-21e28239a2ee: execution started.
+2024-11-19 22:14:23 - INFO - Node MilvusDocumentRetriever - 51c8311b-4837-411f-ba42-21e28239a2ee: execution succeeded in 23ms.
+2024-11-19 22:14:23 - INFO - Node LLM - ac722325-bece-453f-a2ed-135b0749ee7a: execution started.
+2024-11-19 22:14:24 - INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
+2024-11-19 22:14:24 - INFO - Node LLM - ac722325-bece-453f-a2ed-135b0749ee7a: execution succeeded in 1.8s.
+2024-11-19 22:14:25 - INFO - Flow b30b48ec-d5d2-4e4c-8e25-d6976c8a9c17: execution succeeded in 2.4s.
+2024-11-19 22:14:25 - INFO - Workflow f4a073fb-dfb6-499c-8cac-5710a7ad6d47: execution succeeded in 2.4s.
+
+
+The advanced search algorithms in Milvus include a variety of in-memory and on-disk indexing/search algorithms such as IVF (Inverted File), HNSW (Hierarchical Navigable Small World), and DiskANN. These algorithms have been deeply optimized to enhance performance, delivering 30%-70% better performance compared to popular implementations like FAISS and HNSWLib. These optimizations are part of Milvus's design to ensure high efficiency and scalability in handling vector data.
+
diff --git a/localization/v2.4.x/site/zh/menuStructure/zh.json b/localization/v2.4.x/site/zh/menuStructure/zh.json
index b8d288c2c..efa326c3d 100644
--- a/localization/v2.4.x/site/zh/menuStructure/zh.json
+++ b/localization/v2.4.x/site/zh/menuStructure/zh.json
@@ -677,7 +677,7 @@
"children": []
},
{
- "label": "From Milvus 2.3.x",
+ "label": "来自 Milvus 2.3.x",
"id": "from-m2x.md",
"order": 5,
"children": []
@@ -1119,7 +1119,7 @@
]
},
{
- "label": "Milvus 备份",
+ "label": "Milvus 后备电源",
"id": "milvus_backup",
"order": 1,
"children": [
@@ -1344,6 +1344,24 @@
"id": "build_RAG_with_milvus_and_siliconflow.md",
"order": 4,
"children": []
+ },
+ {
+ "label": "桑巴诺瓦",
+ "id": "use_milvus_with_sambanova.md",
+ "order": 5,
+ "children": []
+ },
+ {
+ "label": "双子座",
+ "id": "build_RAG_with_milvus_and_gemini.md",
+ "order": 6,
+ "children": []
+ },
+ {
+ "label": "奥拉玛",
+ "id": "build_RAG_with_milvus_and_ollama.md",
+ "order": 7,
+ "children": []
}
]
},
@@ -1433,15 +1451,15 @@
"children": []
},
{
- "label": "桑巴诺瓦",
- "id": "use_milvus_with_sambanova.md",
+ "label": "PrivateGPT",
+ "id": "use_milvus_in_private_gpt.md",
"order": 10,
"children": []
},
{
- "label": "PrivateGPT",
- "id": "use_milvus_in_private_gpt.md",
- "order": 8,
+ "label": "Dynamiq",
+ "id": "milvus_rag_with_dynamiq.md",
+ "order": 11,
"children": []
}
]
@@ -1493,6 +1511,12 @@
"id": "knowledge_table_with_milvus.md",
"order": 2,
"children": []
+ },
+ {
+ "label": "Cognee",
+ "id": "build_RAG_with_milvus_and_cognee.md",
+ "order": 3,
+ "children": []
}
]
},
diff --git a/localization/v2.5.x/site/de/tutorials/use_ColPali_with_milvus.md b/localization/v2.5.x/site/de/tutorials/use_ColPali_with_milvus.md
index 046aae259..1071e1783 100644
--- a/localization/v2.5.x/site/de/tutorials/use_ColPali_with_milvus.md
+++ b/localization/v2.5.x/site/de/tutorials/use_ColPali_with_milvus.md
@@ -1,10 +1,10 @@
---
id: use_ColPali_with_milvus.md
summary: >-
- In diesem Notizbuch bezeichnen wir diese Art der Multi-Vektor-Darstellung der
- Allgemeinheit halber als "ColBERT-Einbettungen". Das tatsächliche Modell, das
- verwendet wird, ist jedoch das ColPali-Modell. Wir werden demonstrieren, wie
- Milvus für Multi-Vektor-Retrieval verwendet werden kann. Darauf aufbauend
+ In diesem Notizbuch bezeichnen wir diese Art der Multi-Vektor-Darstellung aus
+ Gründen der Allgemeinheit als "ColBERT-Einbettungen". Das tatsächliche Modell,
+ das verwendet wird, ist jedoch das ColPali-Modell. Wir werden demonstrieren,
+ wie Milvus für Multi-Vektor-Retrieval verwendet werden kann. Darauf aufbauend
stellen wir vor, wie ColPali für das Abrufen von Seiten auf der Grundlage
einer gegebenen Abfrage verwendet werden kann.
title: ColPali für multimodales Retrieval mit Milvus verwenden
@@ -33,7 +33,7 @@ title: ColPali für multimodales Retrieval mit Milvus verwenden
Moderne Retrieval-Modelle verwenden in der Regel eine einzige Einbettung, um Text oder Bilder darzustellen. ColBERT hingegen ist ein neuronales Modell, das eine Liste von Einbettungen für jede Dateninstanz verwendet und eine "MaxSim"-Operation zur Berechnung der Ähnlichkeit zwischen zwei Texten einsetzt. Neben Textdaten enthalten auch Abbildungen, Tabellen und Diagramme reichhaltige Informationen, die beim textbasierten Information Retrieval oft unberücksichtigt bleiben.
- 
+ 
diff --git a/localization/v2.5.x/site/en/tutorials/use_ColPali_with_milvus.md b/localization/v2.5.x/site/en/tutorials/use_ColPali_with_milvus.md
index 8d2beb6ce..5a39cacc6 100644
--- a/localization/v2.5.x/site/en/tutorials/use_ColPali_with_milvus.md
+++ b/localization/v2.5.x/site/en/tutorials/use_ColPali_with_milvus.md
@@ -32,7 +32,7 @@ title: Use ColPali for Multi-Modal Retrieval with Milvus
Modern retrieval models typically use a single embedding to represent text or images. ColBERT, however, is a neural model that utilizes a list of embeddings for each data instance and employs a “MaxSim” operation to calculate the similarity between two texts. Beyond textual data, figures, tables, and diagrams also contain rich information, which is often disregarded in text-based information retrieval.
-
+
diff --git a/localization/v2.5.x/site/es/menuStructure/es.json b/localization/v2.5.x/site/es/menuStructure/es.json
index 21c587854..3b3a5fd48 100644
--- a/localization/v2.5.x/site/es/menuStructure/es.json
+++ b/localization/v2.5.x/site/es/menuStructure/es.json
@@ -1098,7 +1098,7 @@
"children": []
},
{
- "label": "Configurar la caché de trozos",
+ "label": "Configurar Chunk Cache",
"id": "chunk_cache.md",
"order": 3,
"children": []
diff --git a/localization/v2.5.x/site/es/tutorials/use_ColPali_with_milvus.md b/localization/v2.5.x/site/es/tutorials/use_ColPali_with_milvus.md
index 91b86f73b..3a6200692 100644
--- a/localization/v2.5.x/site/es/tutorials/use_ColPali_with_milvus.md
+++ b/localization/v2.5.x/site/es/tutorials/use_ColPali_with_milvus.md
@@ -33,7 +33,7 @@ title: Utilizar ColPali para la recuperación multimodal con Milvus
Los modelos de recuperación modernos suelen utilizar una única incrustación para representar texto o imágenes. ColBERT, sin embargo, es un modelo neuronal que utiliza una lista de incrustaciones para cada instancia de datos y emplea una operación "MaxSim" para calcular la similitud entre dos textos. Más allá de los datos textuales, las figuras, tablas y diagramas también contienen abundante información, que a menudo no se tiene en cuenta en la recuperación de información basada en texto.
-
+
diff --git a/localization/v2.5.x/site/fr/tutorials/use_ColPali_with_milvus.md b/localization/v2.5.x/site/fr/tutorials/use_ColPali_with_milvus.md
index 8892352d1..2f03e63b1 100644
--- a/localization/v2.5.x/site/fr/tutorials/use_ColPali_with_milvus.md
+++ b/localization/v2.5.x/site/fr/tutorials/use_ColPali_with_milvus.md
@@ -32,7 +32,7 @@ title: Utiliser ColPali pour la recherche multimodale avec Milvus
Les modèles de recherche modernes utilisent généralement un seul ancrage pour représenter le texte ou les images. ColBERT, en revanche, est un modèle neuronal qui utilise une liste d'enchâssements pour chaque instance de données et emploie une opération "MaxSim" pour calculer la similarité entre deux textes. Au-delà des données textuelles, les figures, les tableaux et les diagrammes contiennent également des informations riches, qui sont souvent ignorées dans la recherche d'informations basée sur le texte.
-
+
@@ -313,7 +313,7 @@ torch.Size([1030, 128])
retriever.create_collection()
retriever.create_index()
-Nous allons insérer des listes d'encastrements dans la base de données Milvus.
+Nous insérerons les listes d'encastrements dans la base de données Milvus.
filepaths = ["./pages/" + name for name in os.listdir("./pages")]
for i in range(len(filepaths)):
data = {
diff --git a/localization/v2.5.x/site/it/menuStructure/it.json b/localization/v2.5.x/site/it/menuStructure/it.json
index c3fe16b8e..0b8c8e268 100644
--- a/localization/v2.5.x/site/it/menuStructure/it.json
+++ b/localization/v2.5.x/site/it/menuStructure/it.json
@@ -1986,7 +1986,7 @@
"children": []
},
{
- "label": "Ricerca a imbuto con le matrici Matryoshka",
+ "label": "Ricerca a imbuto con matrioske embeddings",
"id": "funnel_search_with_matryoshka.md",
"order": 10,
"children": []
diff --git a/localization/v2.5.x/site/it/tutorials/use_ColPali_with_milvus.md b/localization/v2.5.x/site/it/tutorials/use_ColPali_with_milvus.md
index 408482beb..c7742e37d 100644
--- a/localization/v2.5.x/site/it/tutorials/use_ColPali_with_milvus.md
+++ b/localization/v2.5.x/site/it/tutorials/use_ColPali_with_milvus.md
@@ -33,7 +33,7 @@ title: Utilizzare ColPali per il recupero multimodale con Milvus
I moderni modelli di recupero utilizzano in genere un singolo embedding per rappresentare il testo o le immagini. ColBERT, invece, è un modello neurale che utilizza un elenco di incorporazioni per ogni istanza di dati e impiega un'operazione "MaxSim" per calcolare la somiglianza tra due testi. Oltre ai dati testuali, anche le figure, le tabelle e i diagrammi contengono informazioni ricche, che spesso non vengono prese in considerazione nel recupero delle informazioni basato sul testo.
-
+
diff --git a/localization/v2.5.x/site/ja/menuStructure/ja.json b/localization/v2.5.x/site/ja/menuStructure/ja.json
index 7be1a0171..fc2a36de3 100644
--- a/localization/v2.5.x/site/ja/menuStructure/ja.json
+++ b/localization/v2.5.x/site/ja/menuStructure/ja.json
@@ -2009,7 +2009,7 @@
"children": []
},
{
- "label": "推薦システム",
+ "label": "レコメンダー・システム",
"id": "recommendation_system.md",
"order": 6,
"children": []
diff --git a/localization/v2.5.x/site/ja/tutorials/use_ColPali_with_milvus.md b/localization/v2.5.x/site/ja/tutorials/use_ColPali_with_milvus.md
index 0c7b1bdee..00f4c470e 100644
--- a/localization/v2.5.x/site/ja/tutorials/use_ColPali_with_milvus.md
+++ b/localization/v2.5.x/site/ja/tutorials/use_ColPali_with_milvus.md
@@ -28,7 +28,7 @@ title: Milvusでマルチモーダル検索にColPaliを使う
最新の検索モデルでは、テキストや画像を表現するために単一の埋め込みを使用するのが一般的です。しかしColBERTは、各データインスタンスに対して埋め込みリストを利用するニューラルモデルであり、2つのテキスト間の類似度を計算するために「MaxSim」演算を採用しています。テキストデータだけでなく、図、表、ダイアグラムにも豊富な情報が含まれているが、テキストベースの情報検索では軽視されがちである。
-
+
@@ -71,7 +71,7 @@ $ pip instal pillow
d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"
>
- PDF RAGを例として使用する。ColBERT論文をダウンロードし、./pdf 。ColPaliはテキストを直接処理するのではなく、ページ全体を画像にラスタライズする。ColPali モデルは、これらの画像に含まれるテキスト情報を理解することに優れています。したがって、各 PDF ページを画像に変換して処理します。
+ PDF RAGを例として使用する。ColBERT論文をダウンロードし、./pdf 。ColPaliはテキストを直接処理するのではなく、ページ全体を画像にラスタライズする。ColPali モデルは、これらの画像に含まれるテキスト情報を理解することに優れています。そのため、各 PDF ページを画像に変換して処理します。
from pdf2image import convert_from_path
pdf_path = "pdfs/2004.12832v2.pdf"
@@ -91,10 +91,10 @@ client = MilvusClient(uri=./milvus.db に設定するのが最も便利な方法です。
もし、100万ベクトルを超えるような大規模なデータがある場合は、DockerやKubernetes上に、よりパフォーマンスの高いMilvusサーバを構築することができます。このセットアップでは、サーバのアドレスとポートをURIとして使用してください（例：http://localhost:19530 ）。Milvusで認証機能を有効にしている場合、トークンには"<your_username>:<your_password>"を使用します。
-MilvusのフルマネージドクラウドサービスであるMilvus Cloudを利用する場合は、Milvus CloudのPublic EndpointとAPI Keyに対応するuri とtoken を調整します。
+MilvusのフルマネージドクラウドサービスであるMilvus Cloudを利用する場合は、Milvus CloudのPublic EndpointとAPI keyに対応するuri とtoken を調整してください。
-MilvusColbertRetrieverクラスを定義し、Milvusクライアントをラップしてマルチベクターデータを取得できるようにします。この実装は、ColBERT埋め込みを平坦化してコレクションに挿入し、各行がColBERT埋め込みリストの個々の埋め込みを表す。また、各埋め込みの出所を追跡するために、doc_id と seq_id を記録する。
+MilvusColbertRetrieverクラスを定義し、Milvusクライアントをラップしてマルチベクターデータを取得できるようにします。この実装は、ColBERT 埋め込みデータを平坦化し、コレクションに挿入する。また、各埋め込みの出所を追跡するために、doc_id と seq_id を記録する。
ColBERT 埋め込みリストで検索する場合、複数の検索が行われる。検索された doc_id は、重複排除される。再ランク付けプロセスが実行され、各 doc_id の完全な埋め込みが取得され、MaxSim スコアが計算され、最終的なランク付け結果が生成される。
class MilvusColbertRetriever:
def __init__(self, milvus_client, collection_name, dim=128):
diff --git a/localization/v2.5.x/site/ko/tutorials/use_ColPali_with_milvus.md b/localization/v2.5.x/site/ko/tutorials/use_ColPali_with_milvus.md
index 3821ce0f2..a2e3df545 100644
--- a/localization/v2.5.x/site/ko/tutorials/use_ColPali_with_milvus.md
+++ b/localization/v2.5.x/site/ko/tutorials/use_ColPali_with_milvus.md
@@ -30,7 +30,7 @@ title: Milvus로 다중 모달 검색에 ColPali 사용
최신 검색 모델은 일반적으로 텍스트나 이미지를 표현하기 위해 단일 임베딩을 사용합니다. 그러나 ColBERT는 각 데이터 인스턴스에 대한 임베딩 목록을 활용하고 "MaxSim" 연산을 사용하여 두 텍스트 간의 유사성을 계산하는 신경 모델입니다. 텍스트 데이터 외에도 그림, 표, 다이어그램에는 텍스트 기반 정보 검색에서 종종 무시되는 풍부한 정보가 포함되어 있습니다.
-
+
diff --git a/localization/v2.5.x/site/pt/tutorials/use_ColPali_with_milvus.md b/localization/v2.5.x/site/pt/tutorials/use_ColPali_with_milvus.md
index 9f4445ae2..c817fac69 100644
--- a/localization/v2.5.x/site/pt/tutorials/use_ColPali_with_milvus.md
+++ b/localization/v2.5.x/site/pt/tutorials/use_ColPali_with_milvus.md
@@ -32,12 +32,12 @@ title: Use ColPali para recuperação multimodal com Milvus
Os modelos de recuperação modernos utilizam normalmente um único embedding para representar texto ou imagens. O ColBERT, no entanto, é um modelo neural que utiliza uma lista de embeddings para cada instância de dados e emprega uma operação "MaxSim" para calcular a semelhança entre dois textos. Para além dos dados textuais, as figuras, tabelas e diagramas também contêm informações ricas, que são frequentemente ignoradas na recuperação de informações baseada em texto.
-
+
A função MaxSim compara uma consulta com um documento (o que está a ser pesquisado) através da análise dos seus token embeddings. Para cada palavra na consulta, seleciona a palavra mais semelhante do documento (utilizando a semelhança de cosseno ou a distância L2 ao quadrado) e soma estas semelhanças máximas entre todas as palavras na consulta
-O ColPali é um método que combina a representação multi-vetorial do ColBERT com o PaliGemma (um modelo de linguagem multimodal de grande dimensão) para tirar partido das suas fortes capacidades de compreensão. Esta abordagem permite que uma página com texto e imagens seja representada utilizando uma incorporação multi-vetorial unificada. As incorporações dentro desta representação multi-vetorial podem capturar informações detalhadas, melhorando o desempenho da geração aumentada de recuperação (RAG) para dados multimodais.
+O ColPali é um método que combina a representação multi-vetorial do ColBERT com o PaliGemma (um modelo de linguagem multimodal de grande dimensão) para tirar partido das suas fortes capacidades de compreensão. Esta abordagem permite que uma página com texto e imagens seja representada utilizando uma incorporação multi-vetorial unificada. As incorporações dentro desta representação multi-vetorial podem captar informações detalhadas, melhorando o desempenho da geração aumentada de recuperação (RAG) para dados multimodais.
Neste caderno, referimo-nos a este tipo de representação multi-vetorial como "ColBERT embeddings" por uma questão de generalidade. No entanto, o modelo real que está a ser utilizado é o modelo ColPali. Vamos demonstrar como utilizar o Milvus para a recuperação multi-vetorial. Com base nisso, apresentaremos como utilizar o ColPali para recuperar páginas com base numa determinada consulta.
Preparação
Utilizaremos o PDF RAG como exemplo. Pode descarregar o documento ColBERT e colocá-lo em ./pdf. O ColPali não processa diretamente o texto; em vez disso, toda a página é rasterizada numa imagem. O modelo ColPali é excelente para compreender a informação textual contida nestas imagens. Portanto, converteremos cada página do PDF em uma imagem para processamento.
+ Utilizaremos o PDF RAG como exemplo. Pode descarregar o documento ColBERT e colocá-lo em ./pdf. O ColPali não processa o texto diretamente; em vez disso, toda a página é rasterizada numa imagem. O modelo ColPali é excelente para compreender a informação textual contida nestas imagens. Portanto, converteremos cada página do PDF em uma imagem para processamento.
from pdf2image import convert_from_path
pdf_path = "pdfs/2004.12832v2.pdf"
@@ -332,4 +332,4 @@ retriever.create_index()
./pages/page_5.png
./pages/page_7.png
-Por fim, recuperamos o nome da página original. Com o ColPali, podemos recuperar documentos multimodais sem a necessidade de técnicas de processamento complexas para extrair texto e imagens dos documentos. Ao tirar partido de modelos de visão de grande dimensão, podem ser analisadas mais informações, como tabelas e figuras, sem perdas significativas de informação.
+Por fim, recuperamos o nome da página original. Com o ColPali, podemos recuperar documentos multimodais sem a necessidade de técnicas de processamento complexas para extrair texto e imagens dos documentos. Ao tirar partido de modelos de visão de grande dimensão, é possível analisar mais informações - como tabelas e figuras - sem perdas significativas de informação.
diff --git a/localization/v2.5.x/site/zh/menuStructure/zh.json b/localization/v2.5.x/site/zh/menuStructure/zh.json
index 85fef5b10..6c6d5e7fb 100644
--- a/localization/v2.5.x/site/zh/menuStructure/zh.json
+++ b/localization/v2.5.x/site/zh/menuStructure/zh.json
@@ -672,7 +672,7 @@
"children": []
},
{
- "label": "Filtered Search",
+ "label": "过滤搜索",
"id": "filtered-search.md",
"order": 1,
"children": []
diff --git a/localization/v2.5.x/site/zh/tutorials/use_ColPali_with_milvus.md b/localization/v2.5.x/site/zh/tutorials/use_ColPali_with_milvus.md
index b62870616..5538643af 100644
--- a/localization/v2.5.x/site/zh/tutorials/use_ColPali_with_milvus.md
+++ b/localization/v2.5.x/site/zh/tutorials/use_ColPali_with_milvus.md
@@ -29,7 +29,7 @@ title: 使用 ColPali 与 Milvus 一起进行多模式检索
现代检索模型通常使用单一嵌入来表示文本或图像。而 ColBERT 是一种神经模型，它利用每个数据实例的嵌入列表，并采用 "MaxSim "操作来计算两个文本之间的相似度。除文本数据外，图、表和图表也包含丰富的信息，而这些信息在基于文本的信息检索中往往被忽略。
-
+
diff --git a/tools/cache.json b/tools/cache.json
index 080e7c5ef..4a2eb2ff2 100644
--- a/tools/cache.json
+++ b/tools/cache.json
@@ -212,7 +212,7 @@
"v2.4.x/site/en/reference/array_data_type.md": "2024-12-04T08:00:16.529Z",
"v2.4.x/site/en/userGuide/search-query-get/single-vector-search.md": "2024-12-04T08:01:10.984Z",
"v2.4.x/site/en/userGuide/use-json-fields.md": "2024-12-06T07:05:02.401Z",
- "v2.4.x/site/en/menuStructure/en.json": "2024-12-06T07:16:50.798Z",
+ "v2.4.x/site/en/menuStructure/en.json": "2024-12-06T07:41:11.186Z",
"v2.4.x/site/en/userGuide/manage-collections.md": "2024-11-18T03:34:16.027Z",
"v2.4.x/site/en/getstarted/milvus_lite.md": "2024-09-06T02:54:42.351Z",
"v2.4.x/site/en/release_notes.md": "2024-11-25T08:33:06.376Z",
@@ -248,7 +248,11 @@
"v2.4.x/site/en/tutorials/funnel_search_with_matryoshka.md": "2024-11-18T03:33:50.166Z",
"v2.4.x/site/en/tutorials/use_ColPali_with_milvus.md": "2024-12-06T07:04:45.155Z",
"v2.4.x/site/en/userGuide/manage-indexes/index-with-gpu.md": "2024-11-28T09:51:07.679Z",
- "v2.4.x/site/en/getstarted/run-milvus-docker/install_standalone-windows.md": "2024-12-06T07:03:58.705Z"
+ "v2.4.x/site/en/getstarted/run-milvus-docker/install_standalone-windows.md": "2024-12-06T07:03:58.705Z",
+ "v2.4.x/site/en/integrations/build_RAG_with_milvus_and_cognee.md": "2024-12-06T07:22:13.410Z",
+ "v2.4.x/site/en/integrations/build_RAG_with_milvus_and_gemini.md": "2024-12-06T07:22:21.204Z",
+ "v2.4.x/site/en/integrations/build_RAG_with_milvus_and_ollama.md": "2024-12-06T07:22:28.916Z",
+ "v2.4.x/site/en/integrations/milvus_rag_with_dynamiq.md": "2024-12-06T07:22:38.050Z"
},
"v2.3.x": {
"v2.3.x/site/en/about/limitations.md": "2024-08-28T10:40:29.333Z",
@@ -1155,7 +1159,7 @@
"v2.5.x/site/en/tutorials/text_image_search.md": "2024-11-27T07:24:00.232Z",
"v2.5.x/site/en/tutorials/text_search_engine.md": "2024-11-27T07:24:01.460Z",
"v2.5.x/site/en/tutorials/tutorials-overview.md": "2024-12-06T07:02:35.435Z",
- "v2.5.x/site/en/tutorials/use_ColPali_with_milvus.md": "2024-12-06T07:02:42.279Z",
+ "v2.5.x/site/en/tutorials/use_ColPali_with_milvus.md": "2024-12-06T07:21:09.184Z",
"v2.5.x/site/en/tutorials/vector_visualization.md": "2024-11-27T07:24:05.450Z",
"v2.5.x/site/en/tutorials/video_similarity_search.md": "2024-11-27T07:24:06.574Z",
"v2.5.x/site/en/userGuide/collections/create-collection-instantly.md": "2024-11-27T07:24:07.802Z",
@@ -1234,7 +1238,7 @@
"v2.5.x/site/en/userGuide/tools/milvus_backup_api.md": "2024-11-27T07:25:46.624Z",
"v2.5.x/site/en/userGuide/tools/milvus_backup_cli.md": "2024-11-27T07:25:47.846Z",
"v2.5.x/site/en/userGuide/tools/milvus_backup_overview.md": "2024-11-27T07:25:49.181Z",
- "v2.5.x/site/en/menuStructure/en.json": "2024-12-06T07:11:25.563Z",
+ "v2.5.x/site/en/menuStructure/en.json": "2024-12-06T07:32:52.850Z",
"v2.5.x/site/en/getstarted/run-milvus-docker/install_standalone-windows.md": "2024-12-06T07:01:20.487Z",
"v2.5.x/site/en/integrations/build_RAG_with_milvus_and_cognee.md": "2024-12-06T07:01:27.204Z",
"v2.5.x/site/en/integrations/build_RAG_with_milvus_and_gemini.md": "2024-12-06T07:01:34.066Z",