Percentile Cutoff

This module is inspired by our similarity percentile cutoff module.

Filter out the contents that are below the content’s length times percentile.

Module Parameters

  • percentile : The percentile value to filter out the contents. This is essential to run the module, so you have to set this parameter.

  • reverse : If True, the lower the score, the better. Default is False.

Example config.yaml

modules:
  - module_type: percentile_cutoff
    percentile: 0.6