Skip to content

Latest commit

 

History

History
84 lines (64 loc) · 3.87 KB

Session 4 Lab.md

File metadata and controls

84 lines (64 loc) · 3.87 KB
Raw

GraphRAG - State of the Art Approach

GraphRAG is a state-of-the-art approach to perform Retrieval Augmented Generation utilizing graph database connections to uncover the context and information required to answer user questions. More Info on GraphRAG: Microsoft Research blog post GraphRAG repo

Deploy GraphRAG accelerator

GraphRAG Accelerator

(Optional) use Service Principal in GH codespaces

Steps for creating Service Principal:

  • Use create new secret option and record secret for later step.

Ensure the SP has sufficient permissions to deploy the GraphRAG solution.

Use the following az login command using SP:

az login --service-principal -u <app-id> -p <password-or-cert> --tenant <tenant>

Upload files and Build Index

Once GraphRAG accelerator is deployed use the following notebook to upload files and create GraphRAG Index

Setup Notebook

Submit questions to GraphRAG flow

Open the graphrag_flow, and configure connection to graphrag service.

Create Promptflow GraphRag Connection

  • Fill in endpoint with the GraphRag APIM URL (example: https://apim.azure-api.net)
  • Do not fill in api_key

Run following commands

cd src/flows/graphrag_flow
pf connection create -f graphrag_connection.yaml

Configure flow settings and submit question to graphrag.

Note - Questions submitted to GraphRAG may take an extended amount of time which may be affected by both size of graph and graph query settings.

(Optional) Perform evaluation against GraphRAG flow

Use the ground truth questions and answers to evaluate the performance of GraphRAG. This dataset should span the breadth of topics that your copilot is expected to handle.

Execute batch run of test data

  • Open graphrag_flow/flow.dag.yaml in the Visual Editor.
  • Create a batch run experiment to run the copilot on a set of questions.
  • Configure the inputs to the batch_run step:
    • data: Name of the input jsonl file to be used. File should be saved in the '/input' directory.
    • [fill in fields]

Run the batch run experiment

  • Save the batch execution yaml file.
  • Run the batch execution file.

Perform evaluation of the GraphRAG against ground truth data

  • Open the model_as_a_judge/flow.dag.yaml in the Visual Editor.
  • Create a batch run experiment to evaluate the GraphRAG responses against the ground truth data.
  • Select 'Existing Run' to perform the evaluation against the generated answers in step 5
    • Uncomment the # data to provide the qna data set and provide ground_truth inputs to the evaluation promptflow.
  • Configure the inputs to the batch run job:
    • data: Name of the test_data jsonl file used.
    • [fill in fields]

Step 6b: Run the evaluation

  • Save the batch execution yaml file.
  • Run the batch execution file.
  • Inspect the output to evaluate the performance of your GraphRAG.
  • Check the evaluation metrics to determine the quality of the responses generated by your GraphRAG.
    • Output Example Metrics: 
      •   gpt_coherence: 3.8
  gpt_similarity: 3.65
  gpt_fluency: 3.55
  gpt_relevance: 4
  gpt_groundedness: 3.3
  f1_score: 0.47