Samuel100/sdkv2 rust by samuel100 · Pull Request #836 · microsoft/Foundry-Local

samuel100 · 2026-06-24T12:58:36Z

Rust SDK updated to new C++ Foundry Local Core ABI.

Replace the owned-Self FoundryLocalManager::create with an Arc<Self> that shares one process-wide instance via a static OnceLock<Mutex<Weak<...>>>. While any handle is alive, callers share the same instance; when the last Arc drops, NativeManager::Drop runs teardown (Manager_Shutdown + Manager_Release) while the ORT runtime is still alive and before the library's C++ static destructors -- restoring singleton semantics without the atexit hook that caused the WebGPU ReleaseEpFactory double-unregister (ORT #29206). Use OnceLock to wrap the Mutex (const Weak::new() needs Rust 1.73, above the crate's 1.70 MSRV). Update stale atexit/singleton doc comments in manager.rs, foundry_local_manager.rs, and docs/api.md. Keep the test helper's &'static signature by holding a process-lifetime OnceLock<Arc<...>>, so all existing call sites compile unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…elease Fix a use-after-free reachable from safe code: Model, the OpenAI clients, and sessions held only a raw flModel*/flSession* plus Arc<Api> (which keeps just the shared library loaded), so dropping the last Arc<FoundryLocalManager> released the native catalog and model handles while those derived objects were still alive. A natural factory pattern (create manager, get a client, return it) would dangle. This was latent under the old leaked &'static singleton and became reachable once teardown moved onto last-Arc Drop. Thread a strong Arc<NativeManager> keep-alive through NativeModel, NativeCatalog, and NativeSession so every derived handle keeps the native manager (and thus the catalog/model handles it owns) alive until the handle itself is dropped. The keep-alive targets NativeManager (a leaf that owns no Rust wrappers), so there is no reference cycle. Also add NATIVE_LIFECYCLE, a Mutex<()> held across both Manager_Create and Manager_Release, to close the create/teardown race: an Arc's strong count reaches zero slightly before Drop finishes Manager_Release, so a concurrent create() could otherwise observe no instance yet be rejected by the native single-instance guard. Validated: cargo fmt/check/clippy/doc clean (lib, tests, examples). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

origin/main #746 (Add speech result types) inserted GetSpeechSegment and GetSpeechResult into flItemApi between GetToolResult and GetMetadata. Because ffi.rs mirrors the vtable positionally via #[repr(C)], the old layout left GetMetadata/GetMutableMetadata/GetQueue and all ItemQueue_* slots shifted by two pointers — so calls like ItemQueue_Push/TryPop (used by streaming) would misdispatch to the wrong native function against a core built from the new header. Add the two function-pointer slots in the matching position, plus the supporting flSpeechWord/flSpeechSegmentData/flSpeechResultData structs, flSpeechSegmentKind, the FOUNDRY_LOCAL_ITEM_SPEECH_SEGMENT/RESULT item-type constants, and the DURATION/CONFIDENCE_UNSET sentinels. API version is unchanged (still 1). Validated: cargo fmt/check/clippy clean (lib, tests, examples). Vtable order now matches foundry_local_c.h flItemApi field-for-field (31 entries). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

origin/main #746 changed the native audio path to emit SPEECH_SEGMENT items (streaming) and a SPEECH_RESULT item (final), replacing the TextItem outputs. LiveAudioTranscriptionSession uses that native path but read results only via read_text_item / item_text (ITEM_TEXT only), so against the new core it silently produced empty streaming results and an empty final transcript. Add read_speech_segment / read_speech_result_text helpers (via the new GetSpeechSegment / GetSpeechResult accessors) and wire them into the streaming trampoline and final-transcript aggregation, with a TEXT fallback so the OpenAI-JSON path and older cores keep working. Segment timing (ms→s) and PARTIAL/FINAL state are mapped onto the existing response envelope. AudioClient::transcribe / transcribe_streaming are unaffected — they use the OpenAI-JSON path, which still returns OPENAI_JSON-tagged TEXT items. Validated: cargo fmt/check/clippy clean (lib, tests, examples). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

vercel · 2026-06-24T12:58:41Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
foundry-local	Ready	Preview, Comment	Jun 24, 2026 12:58pm

Copilot

Pull request overview

This PR ports the Foundry Local Rust SDK (sdk_v2/rust/) onto the new C++ Core ABI exposed through foundry_local_c.h. It rebuilds the public surface (FoundryLocalManager, Catalog, Model, OpenAI-style chat/embedding/audio clients, and a live-audio streaming session) on top of thin extern "system" FFI wrappers, with native-handle lifetimes anchored to a shared Arc<NativeManager>, and adds a single-binary integration test suite plus generated API docs.

Changes:

New FFI-backed core wrappers (detail/{api,native,manager,session,items,model,info,...}) that mirror the C ABI's ownership model (Manager owns Catalog/Models; handles keep the Manager alive; item-queue/request ownership transfer rules).
New OpenAI-compatible clients (openai/{chat_client,embedding_client,audio_client,live_audio_session}) and public types, re-exported from lib.rs.
A consolidated integration test binary (tests/integration/*), examples, Cargo.toml/build.rs updates, and API reference docs.

Reviewed changes

Copilot reviewed 43 out of 43 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
`sdk_v2/rust/src/openai/live_audio_session.rs`	Live audio streaming session; exposes `push_queue_capacity`/`bits_per_sample` options that aren't wired to the native layer (commented).
`sdk_v2/rust/src/detail/{api,native,manager,session,items,model,info}.rs`	Core FFI wrappers and native lifetime/ownership handling; verified against the C ABI and stored conventions.
`sdk_v2/rust/src/openai/{chat_client,embedding_client,audio_client,mod}.rs`	OpenAI-compatible clients and response/streaming handling.
`sdk_v2/rust/src/{lib,catalog,configuration,foundry_local_manager,types,error}.rs`	Public API surface and re-exports (`Model`, `DownloadBuilder`, etc.).
`sdk_v2/rust/tests/integration/common/mod.rs`	Shared test config; `logsDir` still points at the legacy `sdk/rust` tree (commented).
`sdk_v2/rust/tests/integration/{main,manager,model,web_service,live_audio,embedding_client,chat_client}_test.rs`	New consolidated integration tests; minor duplicate/mislabeled log line in chat test (commented).
`sdk_v2/rust/GENERATE-DOCS.md`, `sdk_v2/rust/docs/api.md`	Doc generation guide and API reference; reference the legacy `sdk/rust` path and a nonexistent `ModelVariant` type (commented).
`sdk_v2/rust/Cargo.toml`, `sdk_v2/rust/build.rs`	Package metadata and native-artifact build script (verified).

+    /// Maximum number of audio chunks buffered in the internal push queue.
+    /// If the queue is full, [`LiveAudioTranscriptionSession::append`] will
+    /// wait asynchronously.
+    /// Default: 100 (~3 seconds of audio at typical chunk sizes).
+    pub push_queue_capacity: usize,


+/// * `logsDir`        → `<repo-root>/sdk/rust/logs`
+/// * `logLevel`       → `Warn`
+pub fn test_config() -> FoundryLocalConfig {
+    let repo_root = get_git_repo_root();
+    let logs_dir = repo_root.join("sdk").join("rust").join("logs");


+To generate and open the API docs in your browser:
+
+```bash
+cd sdk/rust


+| `FoundryLocalConfig` | Configuration (app name, log level, service endpoint) |
+| `Catalog` | Model discovery and lookup |
+| `Model` | Grouped model (alias → best variant) |
+| `ModelVariant` | Single variant — download, load, unload |


@@ -0,0 +1,552 @@
+# Foundry Local Rust SDK — Public API Reference
+
+> Auto-generated from `sdk/rust/src` source files.


+    println!("Response: {content}");
+
+    println!("REST response: {content}");


nenad1002 · 2026-06-24T20:07:17Z

+    /// final results are delivered through the transcription stream before it
+    /// completes. The native stop always completes to avoid session leaks,
+    /// even if the provided [`CancellationToken`] fires.
+    pub async fn stop(&self, _ct: Option<CancellationToken>) -> Result<()> {


You need to have Drop trait for the object (LiveAudioTranscriptionStream) as well, I think it can just call stop() otherwise you have a leak

nenad1002 · 2026-06-24T20:10:42Z

+
+    /// Provide an application logger.
+    ///
+    /// *Stub* — the logger is stored but not yet wired into the native core.


Why not wired yet?

nenad1002 · 2026-06-24T20:16:12Z

+    let mut archive = zip::ZipArchive::new(io::Cursor::new(&bytes))
+        .map_err(|e| format!("open nupkg {}: {e}", pkg.name))?;
+
+    let mut extracted = 0usize;


nit: the downloaded .nupkg bytes are extracted directly without any integrity check, maybe add checksum check

nenad1002 · 2026-06-24T20:27:48Z

+
+    /// Create a [`ChatClient`](crate::openai::ChatClient) bound to the (selected) variant.
+    pub fn create_chat_client(&self) -> crate::openai::ChatClient {
+        crate::openai::ChatClient::new(self.id(), self.selected_native().clone())


pub fn create_chat_client(&self) -> ChatClient { let v = self.selected_variant(); ChatClient::new(&v.info.id, v.native.clone()) }

Prefer this structure, equally fast, but does not cause race condition if 2 threads do this.

nenad1002 · 2026-06-24T20:31:47Z

+    }
+
+    /// Unload the (selected) variant from memory.
+    pub async fn unload(&self) -> Result<String> {


This should not return String

nenad1002 · 2026-06-24T20:37:16Z

+    }
+
+    /// Remove the (selected) variant from the local cache.
+    pub async fn remove_from_cache(&self) -> Result<String> {


Sam here, no String

nenad1002 · 2026-06-24T20:40:55Z

+    ///
+    /// The native catalog manages its own caching and refresh, so this is a
+    /// no-op retained for API compatibility.
+    pub async fn update_models(&self) -> Result<()> {


This is confusing, maybe make it more clear it's NO-OP

nenad1002 · 2026-06-24T20:44:33Z

+        match &self.inner {
+            ModelKind::Variant(v) => {
+                vec![Arc::new(Model {
+                    inner: ModelKind::Variant(v.clone()),


Let's try to store Vec<Arc<VariantData>> instead of Vec<VariantData>, to avoid clones

// before enum ModelKind { Variant(VariantData), Group { alias: String, variants: Vec<VariantData>, selected: AtomicUsize }, } // after enum ModelKind { Variant(Arc<VariantData>), Group { alias: String, variants: Vec<Arc<VariantData>>, selected: AtomicUsize }, } ``

nenad1002 · 2026-06-24T21:02:31Z

+        match &self.inner {
+            ModelKind::Variant(v) => Err(FoundryLocalError::ModelOperation {
+                reason: format!(
+                    "select_variant is not supported on a single variant. \


select_variant_by_id is not supported...

Should we check all error messages are structured properly?

nenad1002 · 2026-06-24T21:06:55Z

All Copilot comments look actionable as well

samuel100 and others added 8 commits June 22, 2026 17:18

initial commit of rust v2 sdk

91eb27e

bug fixes

3dcb833

WebGpu error fix

b76c4bd

Merge remote-tracking branch 'origin/main' into samuel100/sdkv2-rust

3891878

Copilot AI review requested due to automatic review settings June 24, 2026 12:58

samuel100 requested a review from nenad1002 June 24, 2026 12:58

Copilot started reviewing on behalf of samuel100 June 24, 2026 12:59 View session

Copilot AI reviewed Jun 24, 2026

View reviewed changes

nenad1002 reviewed Jun 24, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Samuel100/sdkv2 rust#836

Samuel100/sdkv2 rust#836
samuel100 wants to merge 8 commits into
mainfrom
samuel100/sdkv2-rust

samuel100 commented Jun 24, 2026

Uh oh!

vercel Bot commented Jun 24, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

nenad1002 Jun 24, 2026 •

edited

Loading

Uh oh!

nenad1002 Jun 24, 2026

Uh oh!

nenad1002 Jun 24, 2026

Uh oh!

nenad1002 Jun 24, 2026

Uh oh!

nenad1002 Jun 24, 2026

Uh oh!

nenad1002 Jun 24, 2026

Uh oh!

nenad1002 Jun 24, 2026

Uh oh!

nenad1002 Jun 24, 2026

Uh oh!

nenad1002 Jun 24, 2026

Uh oh!

nenad1002 commented Jun 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -0,0 +1,552 @@
		# Foundry Local Rust SDK — Public API Reference

		> Auto-generated from `sdk/rust/src` source files.

		println!("Response: {content}");

		println!("REST response: {content}");

Uh oh!

Conversation

samuel100 commented Jun 24, 2026

Uh oh!

vercel Bot commented Jun 24, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

nenad1002 Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nenad1002 commented Jun 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nenad1002 Jun 24, 2026 •

edited

Loading