feat: add raw format and rawLogKey support for Fluent Bit Kafka by Vaibhav-C-S · Pull Request #2000 · fluent/fluent-operator

Vaibhav-C-S · 2026-06-20T02:48:33Z

What this PR does / why we need it:

Adds support for Fluent Bit Kafka raw output mode in fluent-operator.

Fluent Bit's Kafka output supports Format raw together with Raw_Log_Key, where the value of the configured record key is sent as the Kafka message payload. However, fluent-operator's Kafka output spec did not expose rawLogKey, and the documented format options did not include raw.

This PR adds support for:

kafka.format: raw
kafka.rawLogKey

Which issue(s) this PR fixes:

Fixes #1878

Does this PR introduced a user-facing change?

Added support for Fluent Bit Kafka raw output mode, allowing users to configure `format: raw` with `rawLogKey` to send a selected record field as the Kafka message payload.

Additional documentation, usage docs, etc.:

Testing

Ran make test
Tested locally on Minikube with Fluent Operator and Kafka
Verified ClusterOutput accepts kafka.rawLogKey
Verified generated Fluent Bit config contains

[Output]
    Name    kafka
    Match    dummy.*
    Format    raw
    Raw_Log_Key    log
    Brokers    kafka.kafka-test.svc.cluster.local:9092
    Topics    test

Message received at kafka consumer:
Input

[Input]
    Name    dummy
    Tag    dummy.test
    Dummy    {"log":"hello raw kafka from fluent-operator"}

Message received at kafka consumer

hello raw kafka from fluent-operator

Signed-off-by: Vaibhav-C-S <vaibhavsharma.c@gmail.com>

Copilot

Pull request overview

This PR extends Fluent Operator’s Fluent Bit Kafka output plugin surface area to support Fluent Bit’s raw output mode, enabling users to send a selected record field as the Kafka message payload via Raw_Log_Key.

Changes:

Add rawLogKey to the Kafka output plugin API and render it as Raw_Log_Key in generated Fluent Bit config.
Document format: raw and rawLogKey in the Kafka output plugin docs.
Regenerate CRDs and bundled manifests/Helm CRDs to include the new field and updated format description.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
manifests/setup/setup.yaml	Regenerated install manifest to include `raw` in format description and add `rawLogKey` schema/docs.
docs/plugins/fluentbit/output/kafka.md	Document `format: raw` and `rawLogKey` in the Kafka output plugin doc table.
config/crd/bases/fluentbit.fluent.io_outputs.yaml	Regenerated CRD base for Output to include `rawLogKey` and updated `format` description.
config/crd/bases/fluentbit.fluent.io_clusteroutputs.yaml	Regenerated CRD base for ClusterOutput to include `rawLogKey` and updated `format` description.
charts/fluent-operator/crds/fluentbit.fluent.io_outputs.yaml	Regenerated Helm CRD for Output with `rawLogKey` and updated `format` description.
charts/fluent-operator/crds/fluentbit.fluent.io_clusteroutputs.yaml	Regenerated Helm CRD for ClusterOutput with `rawLogKey` and updated `format` description.
charts/fluent-operator-fluent-bit-crds/templates/fluentbit.fluent.io_outputs.yaml	Regenerated Fluent Bit CRD chart template for Output with `rawLogKey` and updated `format` description.
charts/fluent-operator-fluent-bit-crds/templates/fluentbit.fluent.io_clusteroutputs.yaml	Regenerated Fluent Bit CRD chart template for ClusterOutput with `rawLogKey` and updated `format` description.
apis/fluentbit/v1alpha2/plugins/output/kafka_types.go	Add `RawLogKey` to the Kafka output API and emit `Raw_Log_Key` in Params.

+	// Specify data format, options available: json, msgpack, raw.
 	Format string `json:"format,omitempty"`
+	// When using the raw format, the value of raw_log_key in the record is sent to Kafka as the payload.
+	RawLogKey string `json:"rawLogKey,omitempty"`


 | ----- | ----------- | ------ |
-| format | Specify data format, options available: json, msgpack. | string |
+| format | Specify data format, options available: json, msgpack, raw. | string |
+| rawLogKey | When using the raw format, the value of raw_log_key in the record is sent to Kafka as the payload. | string |


Signed-off-by: Vaibhav-C-S <vaibhavsharma.c@gmail.com>

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Vaibhav Cheruvu <123824888+Vaibhav-C-S@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.

 | ----- | ----------- | ------ |
-| format | Specify data format, options available: json, msgpack. | string |
+| format | Specify data format, options available: json, msgpack, raw. | string |
+| rawLogKey | When using the `raw` format, the value of `raw_log_key` in the record is sent to Kafka as the payload. | string |


+                  rawLogKey:
+                    description: When using the raw format, the value of raw_log_key
+                      in the record is sent to Kafka as the payload.


+                  rawLogKey:
+                    description: When using the raw format, the value of raw_log_key
+                      in the record is sent to Kafka as the payload.


+	// Specify data format, options available: json, msgpack, raw.
 	Format string `json:"format,omitempty"`


Signed-off-by: Vaibhav-C-S <vaibhavsharma.c@gmail.com>

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.

-	// Specify data format, options available: json, msgpack.
+	// Specify data format, options available: json, msgpack, raw.
 	Format string `json:"format,omitempty"`
+	// When using the raw format, the value of the record field specified by RawLogKey is sent to Kafka as the payload.


+	if k.RawLogKey != "" {
+		kvs.Insert("Raw_Log_Key", k.RawLogKey)
+	}


+	if k.RawLogKey != "" {
+		kvs.Insert("Raw_Log_Key", k.RawLogKey)
+	}


 | ----- | ----------- | ------ |
-| format | Specify data format, options available: json, msgpack. | string |
+| format | Specify data format, options available: json, msgpack, raw. | string |
+| rawLogKey | When using the raw format, the value of the record field specified by RawLogKey is sent to Kafka as the payload. | string |


fluentbit(kafka): add raw format and rawLogKey support

727d7aa

Signed-off-by: Vaibhav-C-S <vaibhavsharma.c@gmail.com>

Copilot AI review requested due to automatic review settings June 20, 2026 02:48

Copilot started reviewing on behalf of Vaibhav-C-S June 20, 2026 02:48 View session

Copilot AI reviewed Jun 20, 2026

View reviewed changes

Vaibhav-C-S and others added 2 commits June 20, 2026 08:38

fluentbit(kafka): add raw format and rawLogKey support

2937c43

Signed-off-by: Vaibhav-C-S <vaibhavsharma.c@gmail.com>

Merge branch 'master' into fluentbit-kafka-raw-log-key

82217a4

Copilot AI review requested due to automatic review settings June 22, 2026 17:30

Copilot started reviewing on behalf of Vaibhav-C-S June 22, 2026 17:30 View session

Copilot AI reviewed Jun 22, 2026

View reviewed changes

Comment thread apis/fluentbit/v1alpha2/plugins/output/kafka_types.go Outdated

Potential fix for pull request finding

279964f

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Vaibhav Cheruvu <123824888+Vaibhav-C-S@users.noreply.github.com>

Copilot AI review requested due to automatic review settings June 22, 2026 17:45

Copilot started reviewing on behalf of Vaibhav-C-S June 22, 2026 17:46 View session

Copilot AI reviewed Jun 22, 2026

View reviewed changes

Vaibhav-C-S added 2 commits June 22, 2026 23:24

fluentbit(kafka): add raw format support

9ffafba

Signed-off-by: Vaibhav-C-S <vaibhavsharma.c@gmail.com>

fluentbit(kafka): add raw format support

f128444

Signed-off-by: Vaibhav-C-S <vaibhavsharma.c@gmail.com>

Copilot AI review requested due to automatic review settings June 22, 2026 18:02

Copilot started reviewing on behalf of Vaibhav-C-S June 22, 2026 18:03 View session

Copilot AI reviewed Jun 22, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add raw format and rawLogKey support for Fluent Bit Kafka#2000

feat: add raw format and rawLogKey support for Fluent Bit Kafka#2000
Vaibhav-C-S wants to merge 6 commits into
fluent:masterfrom
Vaibhav-C-S:fluentbit-kafka-raw-log-key

Vaibhav-C-S commented Jun 20, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		// Specify data format, options available: json, msgpack, raw.
		Format string `json:"format,omitempty"`

Conversation

Vaibhav-C-S commented Jun 20, 2026

What this PR does / why we need it:

Which issue(s) this PR fixes:

Does this PR introduced a user-facing change?

Additional documentation, usage docs, etc.:

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants