diff --git a/docs/open_source/testset_generation/rag_evaluation/index.md b/docs/open_source/testset_generation/rag_evaluation/index.md index a3a3a8c45c..9b25d75734 100644 --- a/docs/open_source/testset_generation/rag_evaluation/index.md +++ b/docs/open_source/testset_generation/rag_evaluation/index.md @@ -4,7 +4,7 @@ After automatically generating a test set for your RAG agent using RAGET, you ca of the agent's answers** compared to the reference answers (using a LLM-as-a-judge approach). The main purpose of this evaluation is to help you **identify the weakest components in your RAG agent**. -> ℹ️ You can find a [tutorial](../../../reference/notebooks/RAGET.ipynb) where we demonstrate the capabilities of RAGET +> ℹ️ You can find a [tutorial](../../../reference/notebooks/RAGET_IPCC.ipynb) where we demonstrate the capabilities of RAGET > with a simple RAG agent build with LlamaIndex > on the IPCC report. diff --git a/docs/open_source/testset_generation/testset_generation/index.md b/docs/open_source/testset_generation/testset_generation/index.md index e73449b15b..fdbb38be7a 100644 --- a/docs/open_source/testset_generation/testset_generation/index.md +++ b/docs/open_source/testset_generation/testset_generation/index.md @@ -6,7 +6,7 @@ an in-house evaluation dataset is a painful task that requires manual curation a To help with this, the Giskard python library provides **RAGET: RAG Evaluation Toolkit**, a toolkit to evaluate RAG agents **automatically**. -> ℹ️ You can find a [tutorial](../../../reference/notebooks/RAGET.ipynb) where we demonstrate the capabilities of RAGET +> ℹ️ You can find a [tutorial](../../../reference/notebooks/RAGET_IPCC.ipynb) where we demonstrate the capabilities of RAGET > with a simple RAG agent build with LlamaIndex > on the IPCC report. diff --git a/docs/reference/notebooks/RAGET_Banking_Supervision.ipynb b/docs/reference/notebooks/RAGET_Banking_Supervision.ipynb new file mode 100644 index 0000000000..cc6616c3ef --- /dev/null +++ b/docs/reference/notebooks/RAGET_Banking_Supervision.ipynb @@ -0,0 +1,2368 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# RAG Evaluation Toolkit on a Banking Supervisory Process Agent" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Install dependencies and download the Banking Supervision report" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!pip install \"giskard[llm]\" --upgrade\n", + "!pip install llama-index PyMuPDF" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!wget \"https://www.bankingsupervision.europa.eu/ecb/pub/pdf/ssm.supervisory_guides202401_manual.en.pdf\" -O \"banking_supervision_report.pdf\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Build RAG Agent on the Banking Supervision report" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "import pandas as pd\n", + "import warnings\n", + "pd.set_option(\"display.max_colwidth\", 400)\n", + "warnings.filterwarnings('ignore')" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "from llama_index.core import VectorStoreIndex\n", + "from llama_index.core.node_parser import SentenceSplitter\n", + "from llama_index.readers.file import PyMuPDFReader\n", + "from llama_index.core.base.llms.types import ChatMessage, MessageRole\n", + "\n", + "loader = PyMuPDFReader()\n", + "documents = loader.load(file_path=\"./banking_supervision_report.pdf\")" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "splitter = SentenceSplitter(chunk_size=512)\n", + "index = VectorStoreIndex.from_documents(documents, transformations=[splitter]) \n", + "chat_engine = index.as_chat_engine()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Let's test the Agent" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "'SSM stands for Single Supervisory Mechanism.'" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "str(chat_engine.chat(\"What is SSM?\"))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Generate a test set on the Banking Supervision report" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "from giskard.rag import KnowledgeBase, generate_testset, QATestset\n", + "\n", + "text_nodes = splitter(documents)\n", + "knowledge_base_df = pd.DataFrame([node.text for node in text_nodes], columns=[\"text\"])\n", + "knowledge_base = KnowledgeBase(knowledge_base_df)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "testset = generate_testset(knowledge_base, \n", + " num_questions=100,\n", + " agent_description=\"A chatbot answering questions about banking supervision procedures and methodologies.\",\n", + " language=\"en\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Save the testset\n", + "testset.save(\"banking_supervision_testset.jsonl\")\n", + "\n", + "# Load the testset\n", + "testset = QATestset.load(\"banking_supervision_testset.jsonl\")" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
questionreference_answerreference_contextconversation_historymetadata
id
68320700-52c1-4490-b85b-0fb459ad7e96What must a significant institution obtain before including interim or year-end profits in CET1 capital?A significant institution must obtain prior approval before including interim or year-end profits in its CET1 capital.Document 162: If necessary, \\nadditional information is requested. Next, the JST assesses whether the relevant \\nregulations are complied with and establishes whether the capital instrument is listed \\nin the EBA’s public list of CET1 instruments. If the instrument is not in the EBA’s \\npublic list, before adopting any decision, the ECB consults the EBA. \\nSubsequent issuances of instruments f...[]{'question_type': 'simple', 'seed_document_id': 162, 'topic': 'ECB Banking Supervision'}
fbd171ae-5438-4080-8910-a6b55af63d3aWhat is the process for an SI wishing to establish a branch in a non-participating Member State?An SI wishing to establish a branch in a non-participating Member State has to notify the relevant NCA of its intention. Upon receipt of this notification, the NCA informs the ECB, which exercises the powers of the competent authority of the home Member State. The JST assesses whether the requirements for establishing a branch are met. If the requirements are met, the JST prepares a Supervisor...Document 99: Supervisory Manual – Supervision of all supervised entities \\n \\n59 \\nprovide services can be exercised, subject to national law and in the interests of the \\ngeneral good. The ECB carries out the tasks of the competent authority of the host \\nMember State for institutions established in non-participating Member States which \\nexercise the freedom to provide services in participat...[]{'question_type': 'simple', 'seed_document_id': 99, 'topic': 'Banking Supervision and Passporting'}
4f6d0dc5-4aed-4c78-ba40-c3e2422cbae9What are the main channels of political accountability for the ECB?The main channels of political accountability for the ECB include: 1. The Chair of the Supervisory Board attending regular hearings and ad hoc exchanges of views in the European Parliament and the Eurogroup. National parliaments can also invite the Chair or another member of the Supervisory Board, along with a representative from the respective NCA. 2. The ECB provides the European Parliament’...Document 8: There are several main channels of \\npolitical accountability for the ECB: \\n1. \\nThe Chair of the Supervisory Board attends regular hearings and ad hoc \\nexchanges of views in the European Parliament and the Eurogroup. National \\nparliaments can also invite the Chair or another member of the Supervisory \\nBoard, along with a representative from the respective NCA. \\n2. \\nThe ECB p...[]{'question_type': 'simple', 'seed_document_id': 8, 'topic': 'Others'}
9367137e-42a0-4052-8700-5126ef4c4b1eWhat are the responsibilities of the Joint Supervisory Teams (JSTs) in day-to-day supervision?The day-to-day supervision of SIs is primarily conducted off-site by the JSTs, which comprise staff from NCAs and the ECB and are supported by the horizontal and specialised expertise divisions of DG/HOL and similar staff at the NCAs. The JST analyses the supervisory reporting, financial statements and internal documentation of supervised entities. They hold regular and ad hoc meetings with th...Document 75: Supervisory Manual – Supervisory cycle \\n \\n45 \\nThe SREP is applied proportionately to both SIs and LSIs, ensuring that the highest \\nand most consistent supervisory standards are upheld. \\nIn addition to ongoing activities, the ECB takes ad hoc supervisory actions through \\nthe Directorate General SSM Governance & Operations (DG/SGO); the \\nAuthorisation Division of DG/SGO grant...[]{'question_type': 'simple', 'seed_document_id': 75, 'topic': 'European Banking Supervision'}
0b3f0ff0-9c5c-4704-a884-64e7d720953fWhat role does the ECB play in the supervision of significant institutions?The ECB acts as a gatekeeper to ensure that significant supervised entities comply with robust governance arrangements, including fit and proper requirements for management. It also has direct authority to exercise supervisory powers regarding the approval of key function holders and branch managers in significant institutions under national law.Document 128: Supervisory Manual – Supervision of significant institutions \\n \\n75 \\nThe responsibility of the ECB is to act as a gatekeeper. Its task is to ensure that \\nsignificant supervised entities comply with the requirements to have in place robust \\ngovernance arrangements, including the fit and proper requirements for the persons \\nresponsible for the management of institutions. The E...[]{'question_type': 'simple', 'seed_document_id': 128, 'topic': 'Others'}
\n", + "
" + ], + "text/plain": [ + " question \\\n", + "id \n", + "68320700-52c1-4490-b85b-0fb459ad7e96 What must a significant institution obtain before including interim or year-end profits in CET1 capital? \n", + "fbd171ae-5438-4080-8910-a6b55af63d3a What is the process for an SI wishing to establish a branch in a non-participating Member State? \n", + "4f6d0dc5-4aed-4c78-ba40-c3e2422cbae9 What are the main channels of political accountability for the ECB? \n", + "9367137e-42a0-4052-8700-5126ef4c4b1e What are the responsibilities of the Joint Supervisory Teams (JSTs) in day-to-day supervision? \n", + "0b3f0ff0-9c5c-4704-a884-64e7d720953f What role does the ECB play in the supervision of significant institutions? \n", + "\n", + " reference_answer \\\n", + "id \n", + "68320700-52c1-4490-b85b-0fb459ad7e96 A significant institution must obtain prior approval before including interim or year-end profits in its CET1 capital. \n", + "fbd171ae-5438-4080-8910-a6b55af63d3a An SI wishing to establish a branch in a non-participating Member State has to notify the relevant NCA of its intention. Upon receipt of this notification, the NCA informs the ECB, which exercises the powers of the competent authority of the home Member State. The JST assesses whether the requirements for establishing a branch are met. If the requirements are met, the JST prepares a Supervisor... \n", + "4f6d0dc5-4aed-4c78-ba40-c3e2422cbae9 The main channels of political accountability for the ECB include: 1. The Chair of the Supervisory Board attending regular hearings and ad hoc exchanges of views in the European Parliament and the Eurogroup. National parliaments can also invite the Chair or another member of the Supervisory Board, along with a representative from the respective NCA. 2. The ECB provides the European Parliament’... \n", + "9367137e-42a0-4052-8700-5126ef4c4b1e The day-to-day supervision of SIs is primarily conducted off-site by the JSTs, which comprise staff from NCAs and the ECB and are supported by the horizontal and specialised expertise divisions of DG/HOL and similar staff at the NCAs. The JST analyses the supervisory reporting, financial statements and internal documentation of supervised entities. They hold regular and ad hoc meetings with th... \n", + "0b3f0ff0-9c5c-4704-a884-64e7d720953f The ECB acts as a gatekeeper to ensure that significant supervised entities comply with robust governance arrangements, including fit and proper requirements for management. It also has direct authority to exercise supervisory powers regarding the approval of key function holders and branch managers in significant institutions under national law. \n", + "\n", + " reference_context \\\n", + "id \n", + "68320700-52c1-4490-b85b-0fb459ad7e96 Document 162: If necessary, \\nadditional information is requested. Next, the JST assesses whether the relevant \\nregulations are complied with and establishes whether the capital instrument is listed \\nin the EBA’s public list of CET1 instruments. If the instrument is not in the EBA’s \\npublic list, before adopting any decision, the ECB consults the EBA. \\nSubsequent issuances of instruments f... \n", + "fbd171ae-5438-4080-8910-a6b55af63d3a Document 99: Supervisory Manual – Supervision of all supervised entities \\n \\n59 \\nprovide services can be exercised, subject to national law and in the interests of the \\ngeneral good. The ECB carries out the tasks of the competent authority of the host \\nMember State for institutions established in non-participating Member States which \\nexercise the freedom to provide services in participat... \n", + "4f6d0dc5-4aed-4c78-ba40-c3e2422cbae9 Document 8: There are several main channels of \\npolitical accountability for the ECB: \\n1. \\nThe Chair of the Supervisory Board attends regular hearings and ad hoc \\nexchanges of views in the European Parliament and the Eurogroup. National \\nparliaments can also invite the Chair or another member of the Supervisory \\nBoard, along with a representative from the respective NCA. \\n2. \\nThe ECB p... \n", + "9367137e-42a0-4052-8700-5126ef4c4b1e Document 75: Supervisory Manual – Supervisory cycle \\n \\n45 \\nThe SREP is applied proportionately to both SIs and LSIs, ensuring that the highest \\nand most consistent supervisory standards are upheld. \\nIn addition to ongoing activities, the ECB takes ad hoc supervisory actions through \\nthe Directorate General SSM Governance & Operations (DG/SGO); the \\nAuthorisation Division of DG/SGO grant... \n", + "0b3f0ff0-9c5c-4704-a884-64e7d720953f Document 128: Supervisory Manual – Supervision of significant institutions \\n \\n75 \\nThe responsibility of the ECB is to act as a gatekeeper. Its task is to ensure that \\nsignificant supervised entities comply with the requirements to have in place robust \\ngovernance arrangements, including the fit and proper requirements for the persons \\nresponsible for the management of institutions. The E... \n", + "\n", + " conversation_history \\\n", + "id \n", + "68320700-52c1-4490-b85b-0fb459ad7e96 [] \n", + "fbd171ae-5438-4080-8910-a6b55af63d3a [] \n", + "4f6d0dc5-4aed-4c78-ba40-c3e2422cbae9 [] \n", + "9367137e-42a0-4052-8700-5126ef4c4b1e [] \n", + "0b3f0ff0-9c5c-4704-a884-64e7d720953f [] \n", + "\n", + " metadata \n", + "id \n", + "68320700-52c1-4490-b85b-0fb459ad7e96 {'question_type': 'simple', 'seed_document_id': 162, 'topic': 'ECB Banking Supervision'} \n", + "fbd171ae-5438-4080-8910-a6b55af63d3a {'question_type': 'simple', 'seed_document_id': 99, 'topic': 'Banking Supervision and Passporting'} \n", + "4f6d0dc5-4aed-4c78-ba40-c3e2422cbae9 {'question_type': 'simple', 'seed_document_id': 8, 'topic': 'Others'} \n", + "9367137e-42a0-4052-8700-5126ef4c4b1e {'question_type': 'simple', 'seed_document_id': 75, 'topic': 'European Banking Supervision'} \n", + "0b3f0ff0-9c5c-4704-a884-64e7d720953f {'question_type': 'simple', 'seed_document_id': 128, 'topic': 'Others'} " + ] + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "testset.to_pandas().head(5)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Evaluate and Diagnose the Agent" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [], + "source": [ + "from giskard.rag import evaluate, RAGReport\n", + "from giskard.rag.metrics.ragas_metrics import ragas_context_recall, ragas_context_precision" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "def answer_fn(question, history=None):\n", + " if history:\n", + " answer = chat_engine.chat(question, chat_history=[ChatMessage(role=MessageRole.USER if msg[\"role\"] ==\"user\" else MessageRole.ASSISTANT,\n", + " content=msg[\"content\"]) for msg in history])\n", + " else:\n", + " answer = chat_engine.chat(question, chat_history=[])\n", + " return str(answer)\n", + "\n", + "report = evaluate(answer_fn, \n", + " testset=testset, \n", + " knowledge_base=knowledge_base,\n", + " metrics=[ragas_context_recall, ragas_context_precision])" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": {}, + "outputs": [], + "source": [ + "# Save the report\n", + "report.save(\"banking_supervision_report\")\n", + "\n", + "# Load the report\n", + "report = RAGReport.load(\"banking_supervision_report\")" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "display(report.to_html(embed=True))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### RAGET question types\n", + "\n", + "Each question type assesses a few RAG components. This makes it possible to localize weaknesses in the RAG Agent and give feedback to the developers.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "|Question type | Description | Example | Targeted RAG components |\n", + "|---|---|---|---|\n", + "| **Simple** | Simple questions generated from an excerpt of the knowledge base | *What is the purpose of the holistic approach in the SREP?* | `Generator`, `Retriever` | \n", + "| **Complex** | Questions made more complex by paraphrasing | *In what capacity and with what frequency do NCAs contribute to the formulation and scheduling of supervisory activities, especially concerning the organization of on-site missions?* | `Generator` | \n", + "| **Distracting** | Questions made to confuse the retrieval part of the RAG with a distracting element from the knowledge base but irrelevant to the question | *Under what conditions does the ECB levy fees to cover the costs of its supervisory tasks, particularly in the context of financial conglomerates requiring cross-sector supervision?* | `Generator`, `Retriever`, `Rewriter` |\n", + "| **Situational** | Questions including user context to evaluate the ability of the generation to produce relevant answer according to the context | *As a bank manager looking to understand the appeal process for a regulatory decision made by the ECB, could you explain what role the ABoR plays in the supervisory decision review process?* |`Generator` |\n", + "| **Double** | Questions with two distinct parts to evaluate the capabilities of the query rewriter of the RAG | *What role does the SSM Secretariat Division play in the decision-making process of the ECB's supervisory tasks, and which directorates general are involved in the preparation of draft decisions for supervised entities in the ECB Banking Supervision?* | `Generator`, `Rewriter` |\n", + "| **Conversational** |Questions made as part of a conversation, first message describe the context of the question that is ask in the last message, also tests the rewriter | - *I am interested in the sources used for the assessment of risks and vulnerabilities in ECB Banking Supervision.*
- *What are these sources?* | `Rewriter`, `Routing` |\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": ".venv", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.4" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/docs/reference/notebooks/RAGET.ipynb b/docs/reference/notebooks/RAGET_IPCC.ipynb similarity index 100% rename from docs/reference/notebooks/RAGET.ipynb rename to docs/reference/notebooks/RAGET_IPCC.ipynb diff --git a/docs/tutorials/rag_tutorials/index.md b/docs/tutorials/rag_tutorials/index.md index e8d06a2594..542eca4e20 100644 --- a/docs/tutorials/rag_tutorials/index.md +++ b/docs/tutorials/rag_tutorials/index.md @@ -1,19 +1,27 @@ # RAG Tutorials + ```{toctree} :caption: Table of Contents :maxdepth: 1 :hidden: -../../reference/notebooks/RAGET.ipynb +../../reference/notebooks/RAGET_IPCC.ipynb +../../reference/notebooks/RAGET_Banking_Supervision.ipynb + ``` + :::::{grid} 1 1 2 2 ::::{grid-item-card}

IPCC Climate Change Report

RAGET Demo with LlamaIndex RAG :text-align: center -:link: ../../reference/notebooks/RAGET.ipynb +:link: ../../reference/notebooks/RAGET_IPCC.ipynb :::: +::::{grid-item-card}

ECB Banking Supervision Report

RAGET Demo with LlamaIndex RAG +:text-align: center +:link: ../../reference/notebooks/RAGET_Banking_Supervision.ipynb +:::: -::::: \ No newline at end of file +:::::