RAGAS evaluation data generation

RAGAS, the RAG evaluation framework, also support their advanced evaluation data generation. You can learn more about their evaluation data generation method at here.

Generate QA set from Corpus data using RAGAS

You can generate QA set from corpus data using RAGAS.

If you didn’t make corpus data yet, you can make it by following the tutorial.

import pandas as pd

from autorag.data.qacreation.ragas import generate_qa_ragas

corpus_df = pd.read_parquet('path/to/corpus.parquet')
qa_df = generate_qa_ragas(corpus_df, test_size=50)

This will make QA set with 50 questions using the RAGAS evaluation data generation method.

You can use output qa_df directly for AutoRAG optimization.

RAGAS question types

• simple

• reasoning

• multi_context

• conditional

You can set the distribution of question types by making a distribution list. You can access each question type by importing RAGAS evolution types.

from ragas.testset.evolutions import simple, reasoning, multi_context, conditional
from autorag.data.qacreation.ragas import generate_qa_ragas

distributions = {  # uniform distribution
    simple: 0.25,
    reasoning: 0.25,
    multi_context: 0.25,
    conditional: 0.25
}
qa_df = generate_qa_ragas(corpus_df, test_size=50, distributions=distributions)

Now you can pass this distribution dictionary as input.

Use custom models

RAGAS supports custom models using Langchain. You can get a support list of models from Langchain at here.

Plus, you might need to use ChatModel class from Langchain.

from autorag.data.qacreation.ragas import generate_qa_ragas
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

generator_llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0.4)
critic_llm = ChatOpenAI(model="gpt-4", temperature=0.0)
embedding_model = OpenAIEmbeddings()

qa_df = generate_qa_ragas(corpus_df, test_size=50, distributions=distributions,
                          generator_llm=generator_llm, critic_llm=critic_llm, embedding_model=embedding_model)