feat(mcp): add SemanticToolFilter for embedding-based tool filtering#6454
Open
C0d3N1nja97342 wants to merge 3 commits into
Open
feat(mcp): add SemanticToolFilter for embedding-based tool filtering#6454C0d3N1nja97342 wants to merge 3 commits into
C0d3N1nja97342 wants to merge 3 commits into
Conversation
Add SemanticToolFilter, a dynamic MCP tool filter that includes a tool only when its embedding (name + description) is semantically similar to a query derived from the requesting agent's role/goal/backstory, or from run_context[run_context_key] when populated. It implements the existing per-tool ToolFilter protocol (2-arg __call__), so it plugs into tool_resolver with no resolver changes. Reuses embedders from crewai.rag.embeddings (build_embedder) and computes cosine similarity in pure Python (no new dependency). Embeddings are cached per text. Fails open: when no query can be derived or the embedder raises, the tool is included so a misconfigured embedder never silently strips all tools. Also adds a create_semantic_tool_filter factory and exports both from crewai.mcp.
Add tests for SemanticToolFilter: relevant-vs-irrelevant streaming by agent profile, fail-open when no query is available, fail-open when the embedder raises, run_context query taking precedence over the agent profile, threshold boundary, per-text embedding caching, the factory, and empty tool name/description failing open. Uses a deterministic bag-of-words embedder so no model or network is required.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
📝 WalkthroughWalkthroughAdds semantic, embedding-based MCP tool filtering with a new filter class and factory in ChangesSemantic Tool Filter
Sequence Diagram(s)sequenceDiagram
participant Agent
participant SemanticToolFilter
participant EmbeddingFunction
participant Tool
Agent->>SemanticToolFilter: filter(context, tool)
SemanticToolFilter->>SemanticToolFilter: derive query from run_context or agent role/goal/backstory
SemanticToolFilter->>Tool: build embedding text from name/description
SemanticToolFilter->>EmbeddingFunction: embed(query)
SemanticToolFilter->>EmbeddingFunction: embed(tool text)
EmbeddingFunction-->>SemanticToolFilter: embeddings
SemanticToolFilter->>SemanticToolFilter: compute cosine similarity
SemanticToolFilter-->>Agent: include tool if similarity >= threshold
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
lib/crewai/src/crewai/mcp/filters.py (1)
182-182: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value
zip(..., strict=False)silently masks dimension mismatches.If
aandbever differ in length (e.g., query/tool embedded by different models), the extra components are dropped and a wrong similarity is returned instead of surfacing the misconfiguration. Since both vectors here come from the same embedder this is unlikely, butstrict=Truewould turn a latent config error into an explicit failure (which then fails open via theexceptin__call__).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/crewai/src/crewai/mcp/filters.py` at line 182, The cosine-similarity calculation in the dot-product helper currently uses zip(..., strict=False), which can hide embedding length mismatches and return an incorrect score. Update the vector pairing in the similarity logic in filters.py to enforce equal lengths with strict=True (or otherwise validate lengths before summing) so mismatched embeddings fail explicitly and are handled by the existing fallback path in __call__.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@lib/crewai/src/crewai/mcp/filters.py`:
- Around line 261-267: The fail-open path in `ToolFilter` is bypassed when
`_embed()` returns an empty vector instead of raising, causing
`_cosine_similarity()` to exclude tools silently. Update the `ToolFilter` logic
around the `try`/`except` block to treat empty `query_vec` or `tool_vec` the
same as an embedding failure, and return `True` in that case so the class-level
fail-open guarantee holds. Use the `_embed`, `_cosine_similarity`, and
`threshold` flow in `filters.py` to locate the fix.
---
Nitpick comments:
In `@lib/crewai/src/crewai/mcp/filters.py`:
- Line 182: The cosine-similarity calculation in the dot-product helper
currently uses zip(..., strict=False), which can hide embedding length
mismatches and return an incorrect score. Update the vector pairing in the
similarity logic in filters.py to enforce equal lengths with strict=True (or
otherwise validate lengths before summing) so mismatched embeddings fail
explicitly and are handled by the existing fallback path in __call__.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 81bc4c4c-36ed-424d-b93d-0841761fcf60
📒 Files selected for processing (3)
lib/crewai/src/crewai/mcp/__init__.pylib/crewai/src/crewai/mcp/filters.pylib/crewai/tests/mcp/test_semantic_tool_filter.py
Address CodeRabbit review: treat empty embedding vectors as a failure (return True, fail open) instead of letting _cosine_similarity return 0.0 and silently exclude the tool; move the cosine call inside the try/except so it is covered by the fail-open path; and use zip(strict=True) so mismatched embedding dimensions raise explicitly and are caught by the same fallback. Add tests for empty-vector and mismatched-length fail-open.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
MCP servers can expose many tools, and CrewAI injects all of them into the agent. When a server exposes more than ~20 tools, the full tool schema bloats the prompt and the LLM is more likely to pick a wrong or semantically-similar tool. The existing
tool_filteron MCP server configs only supports static allow/block lists (StaticToolFilter) or a hand-written callable — there is no way to filter tools by relevance to the requesting agent.Solution
Add
SemanticToolFilter, a dynamic filter that implements the existing per-toolToolFilterprotocol (__call__(context, tool) -> bool). It embeds a query derived from the requesting agent'srole/goal/backstory(orrun_context["query"]when populated) and each tool'sname+description, including the tool only when cosine similarity is at leastthreshold.It plugs into
tool_resolverwith no resolver changes — the resolver already constructsToolFilterContext(agent=self._agent, ...)and callstool_filter(context, tool)for the 2-arg dynamic form. It reuses embedders fromcrewai.rag.embeddings(build_embedder), computes cosine similarity in pure Python (no new dependency), and caches embeddings per text.It is threshold-based rather than top-K because the
ToolFilterprotocol decides each tool independently (it has no view of the full set or a budget). Fail-open: when no query can be derived or the embedder raises, the tool is included — a misconfigured embedder never silently strips all tools.Changes
lib/crewai/src/crewai/mcp/filters.py:SemanticToolFilterclass (__init__(embedder, threshold=0.3, run_context_key="query"), 2-arg__call__, per-text embedding cache, fail-open),_cosine_similarityhelper,create_semantic_tool_filterfactory.lib/crewai/src/crewai/mcp/__init__.py: exportSemanticToolFilterandcreate_semantic_tool_filter.lib/crewai/tests/mcp/test_semantic_tool_filter.py: 8 tests covering relevant-vs-irrelevant filtering by agent profile, fail-open (no query / embedder raises / empty tool),run_contextquery precedence, threshold boundary, embedding caching, and the factory.Testing
uv run pytest lib/crewai/tests/mcp/test_semantic_tool_filter.py→ 8 passed.ruff checkandruff format --checkclean on changed files.web_searchwhile a Coder agent keepsrun_codefrom the same shared tool list.Notes for Reviewer
.github/CONTRIBUTING.md, thellm-generatedlabel is requested (applied if permissions allow).ToolFilterprotocol (MCP-only, load-time) rather than a top-K retrieval over all tools at injection time. The latter would require changing the core tool-injection path and is a larger, separate piece of work; this PR is additive and non-invasive.run_context-based query is supported in the API for forward compatibility, though the current resolver passesrun_context=None.