Skip to content

ManifestFilterManager rebuilds metrics evaluators for every data file #1117

Open

Description

@dino-saurabh

ManifestFilterManager::ShouldDelete builds a new StrictMetricsEvaluator and InclusiveMetricsEvaluator for every data file it checks manifest_filter_manager.cc#L366-L375). Make() rewrites and binds the expression each time, even though the residual only changes per partition. The existing TODO already notes this:

TODO(Guotao): Cache strict/inclusive metrics evaluators per partition residual.

Measured with the benchmarks from #1116, building and evaluating per file costs about 6x as much as evaluating with a prebuilt evaluator (16 predicates: ~9 µs vs ~1.5 µs per file).

Proposal: cache the evaluators per partition residual, and add a benchmark for the build-per-file case to measure the change.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions