autorag.data.beta package

Subpackages

• autorag.data.beta.filter package
  • Submodules
  • autorag.data.beta.filter.dontknow module
    • Response
      • Response.is_dont_know
      • Response.model_computed_fields
      • Response.model_config
      • Response.model_fields
    • dontknow_filter_llama_index()
    • dontknow_filter_openai()
    • dontknow_filter_rule_based()
  • autorag.data.beta.filter.prompt module
  • Module contents
• autorag.data.beta.generation_gt package
  • Submodules
  • autorag.data.beta.generation_gt.base module
    • add_gen_gt()
  • autorag.data.beta.generation_gt.llama_index_gen_gt module
    • make_basic_gen_gt()
    • make_concise_gen_gt()
    • make_gen_gt_llama_index()
  • autorag.data.beta.generation_gt.openai_gen_gt module
    • Response
      • Response.answer
      • Response.model_computed_fields
      • Response.model_config
      • Response.model_fields
    • make_basic_gen_gt()
    • make_concise_gen_gt()
    • make_gen_gt_openai()
  • autorag.data.beta.generation_gt.prompt module
  • Module contents
• autorag.data.beta.query package
  • Submodules
  • autorag.data.beta.query.llama_gen_query module
    • concept_completion_query_gen()
    • factoid_query_gen()
    • llama_index_generate_base()
    • two_hop_incremental()
  • autorag.data.beta.query.openai_gen_query module
    • Response
      • Response.model_computed_fields
      • Response.model_config
      • Response.model_fields
      • Response.query
    • TwoHopIncrementalResponse
      • TwoHopIncrementalResponse.answer
      • TwoHopIncrementalResponse.model_computed_fields
      • TwoHopIncrementalResponse.model_config
      • TwoHopIncrementalResponse.model_fields
      • TwoHopIncrementalResponse.one_hop_question
      • TwoHopIncrementalResponse.two_hop_question
    • concept_completion_query_gen()
    • factoid_query_gen()
    • query_gen_openai_base()
    • two_hop_incremental()
  • autorag.data.beta.query.prompt module
  • Module contents

Submodules

autorag.data.beta.extract_evidence module

autorag.data.beta.sample module

autorag.data.beta.sample.random_single_hop(corpus_df: DataFrame, n: int, random_state: int = 42) → DataFrame

autorag.data.beta.sample.range_single_hop(corpus_df: DataFrame, idx_range: Iterable)

autorag.data.beta.schema module

class autorag.data.beta.schema.Corpus(corpus_df: DataFrame | None = None, linked_raw: Raw | None = None)

Bases: object

The Corpus class that stored chunked passages. It can generate qa set, linked with Raw instance.

batch_apply(fn: Callable[[Dict, Any], Awaitable[Dict]], batch_size: int = 32, **kwargs) → Corpus

property linked_raw: Raw

map(fn: Callable[[DataFrame, Any], DataFrame], **kwargs) → Corpus

sample(fn: Callable[[DataFrame, Any], DataFrame], **kwargs) → QA

Sample the corpus for making QA. It selects the subset of the corpus and makes QA set from it. You can generate questions from the created question. It is the first step to make QA set from the corpus. If you select just one passage from each passage, it will be a single-hop QA set. If you select multiple passages from each passage, it will be a multi-hop QA set.

Parameters:

fn – The select function to perform.

It returns QA dataframe. :return: QA instance that is selected. It contains qid and retrieval_gt columns.

to_parquet(save_path: str)

Save the corpus to the AutoRAG compatible parquet file. It is not for the data creation, for running AutoRAG. If you want to save it directly, use the below code. corpus.data.to_parquet(save_path)

Parameters:

save_path – The path to save the corpus.

class autorag.data.beta.schema.QA(qa_df: DataFrame | None = None, linked_corpus: Corpus | None = None)

Bases: object

batch_apply(fn: Callable[[Dict, Any], Awaitable[Dict]], batch_size: int = 32, **kwargs) → QA

batch_filter(fn: Callable[[Dict, Any], Awaitable[bool]], batch_size: int = 32, **kwargs) → QA

filter(fn: Callable[[Dict, Any], bool], **kwargs) → QA

property linked_corpus: Corpus

make_retrieval_gt_contents() → QA

Make retrieval_gt_contents column from retrieval_gt column. :return: The QA instance that has a retrieval_gt_contents column.

map(fn: Callable[[DataFrame, Any], DataFrame], **kwargs) → QA

to_parquet(qa_save_path: str, corpus_save_path: str)

Save the qa and corpus to the AutoRAG compatible parquet file. It is not for the data creation, for running AutoRAG. If you want to save it directly, use the below code. qa.data.to_parquet(save_path)

Parameters:

• qa_save_path – The path to save the qa dataset.

• corpus_save_path – The path to save the corpus.

update_corpus(new_corpus: Corpus) → QA

Update linked corpus. Not just replace linked_corpus to the new Corpus, it replaces the whole retrieval_gt to the new corpus using linked_raw. The QA data must have a retrieval_gt column.

Parameters:

new_corpus – Corpus that you want to replace. Must have valid linked_raw and raw_id, raw_start_idx, raw_end_idx columns.

Returns:

The QA instance that updated linked corpus.

class autorag.data.beta.schema.Raw(raw_df: DataFrame | None = None)

Bases: object

The Raw class that stored document parsing results. It can do chunking. It has two column names, ‘raw_id’ and ‘contents’.

batch_apply(fn: Callable[[Dict, Any], Awaitable[Dict]], batch_size: int = 32, **kwargs) → Raw

chunk(module_name: str, **module_params) → Corpus

flatmap(fn: Callable, **kwargs) → Raw

map(fn: Callable[[DataFrame, Any], DataFrame], **kwargs) → Raw

Module contents