claude_code: add Claude Code OTel integration#19765
Conversation
efd6
added
enhancement
Elastic Docs Style Checker (Vale)Summary: 3 suggestions found 💡 Suggestions (3): Optional style improvements. Apply when helpful.
The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
🚀 Benchmarks reportTo see the full report comment with |
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
|
👀 I have started reviewing the PR |
![vera-review-bot[bot]](https://avatars.githubusercontent.com/in/3297022?s=60&v=4){kind=small}
| if: ctx.error_detail != null | ||
|
|
||
| - append: | ||
| tag: append_related_user_email |
There was a problem hiding this comment.
🟡 MEDIUM data_stream/events/.../default.yml:260
related.user enrichment never fires (nested access on flat key)
The append_related_user_email and append_related_user_id processors guard on ctx.user?.email and ctx.user?.id (nested map access) and template the value as {{{user.email}}} / {{{user.id}}}. But the earlier flatten_attributes_to_root script copies OTel attributes to the root with ctx[key] = val, so user.email and user.id arrive as literal flat dotted keys (ctx['user.email']), not a nested user object. ctx.user is therefore always null, both if conditions are always false, and the processors never run. This is confirmed by every expected fixture that carries user data (e.g. test-tool-result-bash.json-expected.json, test-api-request.json-expected.json, test-plugin-loaded.json-expected.json): each has flat user.email/user.id but no related.user field at all. By contrast append_related_hosts works because it reads the single-token key ctx.server_name (test-mcp-server-connection.json-expected.json shows related.hosts populated). For a security-focused integration, related.user is the user-correlation pivot used by the dashboards, so it is silently empty on every event.
Recommendation:
Read the flat dotted keys explicitly instead of nested access. A script processor handles the array append with de-duplication cleanly:
- script:
tag: set_related_user
lang: painless
description: Populate related.user from the flattened user identifiers.
source: >
def vals = new ArrayList();
def email = ctx['user.email'];
def id = ctx['user.id'];
if (email != null) { vals.add(email); }
if (id != null && !vals.contains(id)) { vals.add(id); }
if (!vals.isEmpty()) {
ctx.related = ctx.related ?: new HashMap();
ctx.related.user = vals;
}Alternatively, have flatten_attributes_to_root reconstruct a nested user map (as it already does for event) so the existing append processors' ctx.user?.email checks resolve.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
|
👀 I have started reviewing the PR |
Vera Review BotFor the current commit state, I did not find any issues. 🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
|
Collect telemetry from Claude Code CLI sessions via OpenTelemetry. The package includes an ingest pipeline that flattens OTel attributes to root-level fields, enriches ECS fields (event.action, event.outcome, related.user, related.hosts), and six Kibana dashboards covering session timeline, security overview, cost/usage, MCP server access, tool calls, and permission decisions. Test samples were collected from live sessions and sanitised.
|
👀 I have started reviewing the PR |
Vera Review BotFor the current commit state, I did not find any issues. 🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
|
TL;DRBuild #45096 failed because Remediation
Investigation detailsRoot CauseThe failed build ran on commit The current head Evidence
-| enabled_via | How the plugin was enabled (e.g. user-install). | keyword |
+| enabled_via | How the plugin was enabled (for example user-install). | keyword |
...
-| tool_name | Name of the tool invoked (e.g. Bash, Read, Write, mcp_tool). | keyword |
+| tool_name | Name of the tool invoked (for example Bash, Read, Write, mcp_tool). | keyword |
Error: checking package failed: checking readme files are up-to-date failed: files do not matchVerification
Follow-up
What is this? | From workflow: PR Buildkite Detective Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not. |
andrewkroh
added
New Integration
|
👀 I have started reviewing the PR |
Vera Review BotFor the current commit state, I did not find any issues. 🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
|
|
👀 I have started reviewing the PR |
1 similar comment
|
👀 I have started reviewing the PR |
| type: keyword | ||
| description: Name of the marketplace entry. | ||
| # OTel instrumentation scope | ||
| - name: scope.name |
There was a problem hiding this comment.
🔵 LOW data_stream/events/fields/fields.yml:272
scope. fields declared under wrong path*
These entries are nested inside the claude_code.events group, so they declare claude_code.events.scope.name and claude_code.events.scope.version. The pipeline only renames attributes to claude_code.events; it never moves the top-level OTel scope object. As sample_event.json shows, the actual fields land at the document root as scope.name / scope.version. The result is that the declared claude_code.events.scope.* fields are never populated, and the real scope.* fields are undeclared (working only via dynamic mapping). The integration ingests fine, but the field definitions do not match where the data lands.
Recommendation:
Declare scope at the document root, outside the claude_code.events group, so the mappings match the emitted fields:
- name: scope.name
type: keyword
description: OTel instrumentation scope name.
- name: scope.version
type: keyword
description: OTel instrumentation scope version.🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
There was a problem hiding this comment.
This is half right. The claude_code.events.scope fields should be removed, but the scope fields are defined due to OTel.
| } | ||
| handleMap(ctx); | ||
|
|
||
| - set: |
There was a problem hiding this comment.
🔵 LOW data_stream/events/.../default.yml:425
No @custom pipeline hook for user extension
The pipeline ends with the error-handling processors and has no call to a @custom pipeline. Without that hook, an operator who needs to add their own enrichment or field tweaks has to edit the managed pipeline directly (lost on upgrade). The @custom pipeline is the standard extensibility mechanism for integration ingest pipelines and is conventionally included as the final processing step on new data streams.
Recommendation:
Add a @custom pipeline call as the last processing step (before the null-removal/error-tagging block, or at the end of the main processing), so user customizations survive upgrades:
- pipeline:
tag: pipeline_custom
name: '{{ IngestPipeline "@custom" }}'
ignore_missing_pipeline: true🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
There was a problem hiding this comment.
This is not correct. @custom calls are injected by fleet at install time.
Replace the Painless flattening script with dot_expander + rename to move all OTel custom attributes into a claude_code.* namespace. This prevents semantically different vendor fields from colliding with ECS or other integration fields at the root level. Other changes in this commit: - Add type conversion processors for numeric/boolean attributes. - Map event_name to event.action (ECS) and remove from root. - Update all dashboards to filter on event.action. - Add optional event.original capture via Json.dump (gated on preserve_original_event tag). - Move scope.name/scope.version into claude_code.scope.*. - Remove redundant OTel infrastructure fields (body, observed_timestamp). - Retain resource.attributes for OTel passthrough mapping.
|
✅ All changelog entries have the correct PR link. |
|
👀 I have started reviewing the PR |
💚 Build Succeeded
History
cc @efd6 |
|
|
||
| Run the [Elastic Distribution of the OpenTelemetry Collector](https://github.com/elastic/elastic-agent) with an `otlp` receiver and an `elasticsearch` exporter. Configure the `data_stream.dataset` resource attribute as above. The collector routes events to `logs-claude_code.events.otel-*`. | ||
|
|
||
| ### Option C: Managed OTLP (mOTLP) |
There was a problem hiding this comment.
@efd6 in the interest of giving users an opinionated "best" option, could mOTLP be positioned as the recommended approach, since it avoids users having to deploy/manage agents? Regarding the user flow for this option - do they need to install the integration assets in Fleet first (for the mappings/pipeline/dashboards) before pointing Claude Code at the mOTLP endpoint?
3 participants
Proposed commit message
Note
Apologies for the size. The OTel system tests need a mock server and the inputs to all pipeline tests are JSON, rather than lines.
Also, ref elastic/elastic-package#3711.
Checklist
changelog.ymlfile.Author's Checklist
How to test this PR locally
Related issues
Screenshots