Read Tool & Reasoning Tokens from GenAI API instead of Catalog Metadata#838
Read Tool & Reasoning Tokens from GenAI API instead of Catalog Metadata#838sayanshaw24 wants to merge 2 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
| tool_ctx.supports_tool_calling = true; | ||
| } | ||
|
|
||
| // Fall back to GenAI model API (reads genai_config.json + model-family fallback map) |
There was a problem hiding this comment.
This feels a little clunky to have a bunch of metadata coming from the catalog and some from the genai_config.json but we only merge those when creating a session.
Should we instead when populating the catalog models update the ModelInfo for a cached model to incorporate metadata from genai_config.json at that point? Same process post-download of a new model. That way all the other code doesn't need to know/care about the multiple sources of metadata - the just use the ModelInfo as-is.
There was a problem hiding this comment.
Good call. Refactored in the latest commit. Model::Load() now populates ModelInfo with tool/reasoning tags from the GenAI model immediately after loading. BuildToolCallContext() is unchanged; it just reads from ModelInfo as before. Downstream code doesn't need to know about multiple metadata sources anymore.
| return *oga_model_; | ||
| } | ||
|
|
||
| std::string GenAIModelInstance::GetGenerationTag(const char* tag_name) const { |
There was a problem hiding this comment.
nit: GetTag would be more generic unless there's an expected use-case where we would need to separate 'generation' tags from others. I assume the tag name could provide any separation required though.
There was a problem hiding this comment.
agreed, renamed to GetTag in both GenAI and FL. just need to update the GenAI version here in FL once the PR is merged and included.
…uildToolCallContext
2 participants
Read tool calling and reasoning tokens from GenAI model API
Summary
Updates
BuildToolCallContext()to read tool call and reasoning marker tokens from GenAI'sOgaModel::GetGenerationTag()API introduced in this PR, instead of catalog metadata.Changes
GenAIModelInstance(genai_model_instance.h/.cc)std::string GetGenerationTag(const char* tag_name) constmethodOgaModel::GetGenerationTag()— returns the tag value or empty stringChatSession::BuildToolCallContext()(chat_session.cc)CatalogModel().Info()for tool/reasoning tokens and support flagssupports_tool_calling/supports_reasoningare now inferred from non-empty start tokensResolution priority
FOUNDRY_LOCAL_MODEL_PROP_*paramsgenai_config.jsonwith model-family fallback mapCatalog metadata is no longer consulted for these tokens.
Dependency
Requires updated
onnxruntime-genainuget withOgaModelGetGenerationTagAPI.deps_versions.jsonbump needed once GenAI nuget publishesTag names supported by the GenAI API
"tool_call_start","tool_call_end"— tool call delimiters"reasoning_start","reasoning_end"— chain-of-thought reasoning delimitersTesting
Will be validated end-to-end once the GenAI nuget is available. The code change is minimal and mechanical — replaces catalog reads with GenAI API calls using the same field names.