autorag.nodes.passagecompressor package

Submodules

autorag.nodes.passagecompressor.base module

class autorag.nodes.passagecompressor.base.BasePassageCompressor(project_dir: str, *args, **kwargs)[source]

Bases: BaseModule

cast_to_run(previous_result: DataFrame, *args, **kwargs)[source]

This function is for cast function (a.k.a decorator) only for pure function in the whole node.

class autorag.nodes.passagecompressor.base.LlamaIndexCompressor(project_dir: str, **kwargs)[source]

Bases: BasePassageCompressor

param_list = ['prompt', 'chat_prompt', 'batch']
pure(previous_result: DataFrame, *args, **kwargs)[source]
autorag.nodes.passagecompressor.base.make_llm(llm_name: str, kwargs: Dict) → LLM[source]

autorag.nodes.passagecompressor.longllmlingua module

class autorag.nodes.passagecompressor.longllmlingua.LongLLMLingua(project_dir: str, model_name: str = 'NousResearch/Llama-2-7b-hf', **kwargs)[source]

Bases: BasePassageCompressor

pure(previous_result: DataFrame, *args, **kwargs)[source]
autorag.nodes.passagecompressor.longllmlingua.llmlingua_pure(query: str, contents: List[str], llm_lingua, instructions: str, target_token: int = 300, **kwargs) → str[source]

Return the compressed text.

Parameters:

  • query – The query for retrieved passages.

  • contents – The contents of retrieved passages.

  • llm_lingua – The llm instance, that will be used to compress.

  • instructions – The instructions for compression.

  • target_token – The target token for compression. Default is 300.

  • kwargs – Additional keyword arguments.

Returns:

The compressed text.

autorag.nodes.passagecompressor.pass_compressor module

class autorag.nodes.passagecompressor.pass_compressor.PassCompressor(project_dir: str, *args, **kwargs)[source]

Bases: BasePassageCompressor

pure(previous_result: DataFrame, *args, **kwargs)[source]

autorag.nodes.passagecompressor.refine module

class autorag.nodes.passagecompressor.refine.Refine(project_dir: str, **kwargs)[source]

Bases: LlamaIndexCompressor

llm: LLM

autorag.nodes.passagecompressor.run module

autorag.nodes.passagecompressor.run.evaluate_passage_compressor_node(result_df: DataFrame, metric_inputs: List[MetricInput], metrics: List[str])[source]
autorag.nodes.passagecompressor.run.run_passage_compressor_node(modules: List, module_params: List[Dict], previous_result: DataFrame, node_line_dir: str, strategies: Dict) → DataFrame[source]

Run evaluation and select the best module among passage compressor modules.

Parameters:

  • modules – Passage compressor modules to run.

  • module_params – Passage compressor module parameters.

  • previous_result – Previous result dataframe. Could be retrieval, reranker modules result. It means it must contain ‘query’, ‘retrieved_contents’, ‘retrieved_ids’, ‘retrieve_scores’ columns.

  • node_line_dir – This node line’s directory.

  • strategies – Strategies for passage compressor node. In this node, we use You can skip evaluation when you use only one module and a module parameter.

Returns:

The best result dataframe with previous result columns. This node will replace ‘retrieved_contents’ to compressed passages, so its length will be one.

autorag.nodes.passagecompressor.tree_summarize module

class autorag.nodes.passagecompressor.tree_summarize.TreeSummarize(project_dir: str, **kwargs)[source]

Bases: LlamaIndexCompressor

llm: LLM

Module contents