[CharlieR-o-o-t]

What this PR does / why we need it

After a config change, fluent-bit pods often keep running the previous config while logging a successful reload. They stay one version behind until the config changes again. With namespaced FluentBitConfig this means a newly created route is missing on part of the nodes and the logs of that namespace are not delivered from them.

Cause. fluent-bit-watcher sends SIGHUP on the first Create/Write event in /fluent-bit/config. A single update of the mounted Secret is a burst of events, because the kubelet updates the volume atomically:

1. create a new ..<timestamp> directory - first event
2. write the files into it
3. create ..data_tmp and rename it to ..data - more events
4. remove the old directory

The first event arrives before ..data points to the new directory. fluent-bit starts the hot reload right away and reads the old content. The events of step 3 come a fraction of a millisecond later and are dropped, because a reload is already running:

level=info msg="Config file changed, reloading..."
[engine] caught signal (SIGHUP)
[ info] reloading instance pid=14 tid=0x7ff8ee586e80
[engine] caught signal (SIGHUP)
[engine] caught signal (SIGHUP)
level=info msg="Config file changed, reloading..."
level=info msg="Config file changed, reloading..."
[error] reloading in progress, aborting.
[error] reloading in progress, aborting.
...
[ info] [reload] successful reload done.

Nothing triggers another reload afterwards. When the old main config is read together with files that are already switched (a Lua script of a removed filter, for example), the same race shows up as [filter_lua] filter cannot be loaded / reloaded config is invalid. Reloading is halted.

Seen on a 120-node cluster with fluent-operator v3.10.0 and the fluent-bit:5.1.0 image: after one route was re-created, about 70% of the pods that reloaded were running the config from before the change.

Fix. The watcher now waits until the watch path has been unchanged for -reload-delay (default 1s) and then sends one SIGHUP per update. This also removes the reloading in progress, aborting errors that every config change produced.

Which issue(s) this PR fixes

Fixes #2141

Testing

New tests in cmd/fluent-watcher/fluentbit/main_test.go update a directory the way the kubelet does and check which content is visible through ..data at the moment of the reload. Without the delay all three fail with reloaded config "v1", want "v2".

Not covered: a change that lands while fluent-bit is still reloading (the reload takes up to the grace period) is still dropped by fluent-bit itself. That needs feedback from fluent-bit, for example the hot-reload counter of the HTTP API, and is left for a follow-up.

Does this PR introduce a user-facing change?

fluent-bit-watcher: reload fluent-bit only after the mounted config has been switched, so pods no longer stay on the previous config after a change. New flag `-reload-delay` (default 1s).