Skip to content

feat(fluentbit): add the type_converter filter - #2132

Open
NopeCharon wants to merge 1 commit into
fluent:masterfrom
NopeCharon:feat/2131-fluentbit-type-converter-filter
Open

NopeCharon wants to merge 1 commit into
fluent:masterfrom
NopeCharon:feat/2131-fluentbit-type-converter-filter

Conversation

@NopeCharon

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

Adds a typeConverter filter to ClusterFilter and Filter, for Fluent Bit's type_converter plugin. Until now the only way to use it was customPlugin.

There is one list for each source type Fluent Bit supports: intKeys, uintKeys, floatKeys and strKeys. Each entry has a fromKey, a toKey and the target type:

spec:
  match: "*"
  filters:
    - typeConverter:
        uintKeys:
          - fromKey: Mem.total
            toKey: Mem.total_str
            type: string
        strKeys:
          - fromKey: code
            toKey: code_int
            type: int

This renders as:

[Filter]
    Name    type_converter
    Match    *
    Uint_Key    Mem.total Mem.total_str string
    Str_Key    code code_int int

In the YAML config format, a key with several conversions becomes a list.

I used structured entries rather than raw "FROM TO TYPE" strings so the CRD can check them. Fluent Bit wants exactly three tokens per entry and won't start without at least one rule ([filter:type_converter] no rules). So the API server rejects:

  • keys that are empty or contain whitespace
  • types other than int, uint, float and string
  • empty lists, and a typeConverter with no conversions (a CEL rule)

The deepcopy code, CRDs, setup.yaml and API docs are regenerated, and the filter is listed in docs/plugins/fluentbit/index.md.

Which issue(s) this PR fixes:

Fixes #2131

Does this PR introduced a user-facing change?

Add the type_converter filter plugin to the Fluent Bit ClusterFilter and Filter.

Additional documentation, usage docs, etc.:

New docs/plugins/fluentbit/filter/type_converter.md

Testing

  • TestFilter_TypeConverter_Params checks the rendered parameters, including several entries for one key and an empty filter.
  • TestClusterFilterList_Load_TypeConverter checks the classic and YAML output of a ClusterFilter.
  • make test, make lint, make verify and make generate manifests docs-update pass locally, with no diff left over.
  • I ran both rendered configs (classic and YAML) on fluent/fluent-bit:5.2.0 with a dummy input, and the converted fields showed up with the right types, e.g. "code_int":404,"ratio_float":0.5,"n_str":"42".
  • I applied the generated CRD to an envtest API server. A valid ClusterFilter is accepted. An empty typeConverter, an empty list, an unsupported type, an empty key and a key with a space are all rejected with the messages above.

@cvas22 cvas22 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked the classic and YAML output against the Fluent Bit type_converter docs: one Int_Key/Uint_Key/Float_Key/Str_Key line per conversion in classic, and the repeated keys turn into a list in YAML, which is what the YAML loader expects. The CEL rule for at least one list makes sense too. Looks good to me.

@marcofranssen

Copy link
Copy Markdown
Collaborator

Can you rebase to resolve the conflicts?

Fluent Bit's type_converter filter converts the value of a key to int,
uint, float or string and appends it under a new key. It's commonly used
to make a numeric field matchable by grep or modify, or to turn a numeric
string into a number before it reaches a metrics or search backend.
FilterItem had no field for it, so ClusterFilter and Filter users had to
fall back to customPlugin.

Add a typeConverter filter with one list for each source type Fluent Bit
supports (intKeys, uintKeys, floatKeys, strKeys). Each entry is a
TypeConversion with fromKey, toKey and type, rendered as
"<Src>_Key    FROM TO TYPE" in the classic format and as a list in the
YAML format.

Fluent Bit splits each entry into exactly three tokens and won't start
when the filter has no rules, so the CRD rejects:
- keys that are empty or contain whitespace
- types other than int, uint, float and string
- empty lists, and a typeConverter with no conversions (CEL rule)

The deepcopy code, CRDs, setup.yaml and API docs are regenerated, and
the filter is listed in docs/plugins/fluentbit/index.md.

Fixes fluent#2131

Signed-off-by: NopeCharon <115794254+NopeCharon@users.noreply.github.com>
@NopeCharon

Copy link
Copy Markdown
Contributor Author

Rebased onto master and resolved the conflicts (kept both sysinfo and typeConverter).

@NopeCharon
NopeCharon force-pushed the feat/2131-fluentbit-type-converter-filter branch from b72e70f to 0ffe540 Compare November 12, 2026 13:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: support the Fluent Bit type_converter filter

3 participants