From 77c1a219bfb494c1b3a967c4748dad5db79a9c83 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Tue, 9 Jun 2026 02:00:08 +0800 Subject: [PATCH] feat(ui): live progress + speed for library/component downloads MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Library / component dependency downloads now show the same live progress bar, byte counts and transfer speed that toolchain downloads already show. Three paths fed download events; only toolchain and builtin-index installs rendered progress. Project / custom-index dependency installs went through the silenced direct xlings CLI and ran dark — "Downloading " then a long, feedback-free hang. - Route project/custom-index installs through the xlings NDJSON `interface install_packages` capability (the capability and the `install` CLI share `cmd_install`; install destination is chosen by package scope, so packages still land in the project-local data root). This restores the live bar on that path. - Add an indeterminate ("connecting…") render for the pre-sizing window where the downloader reports totalBytes==0 (DNS/TLS/redirect, or a body streamed with no Content-Length), so the line never freezes. - Centralize the download-progress state machine + render policy in `mcpp.ui` (DownloadProgress); toolchain, builtin-index and custom-index installs now share one UI. - Pin bundled xlings to 0.4.51. Bumps version to 0.0.53. Updates e2e 52/58/60 to expect the interface transport while keeping their project-local-install and hook-ordering invariants. Design: .agents/docs/2026-06-09-library-download-progress-design.md --- ...-06-09-library-download-progress-design.md | 162 ++++++++++++ .xlings.json | 2 +- CHANGELOG.md | 20 ++ mcpp.toml | 2 +- src/cli.cppm | 161 ++++-------- src/toolchain/fingerprint.cppm | 2 +- src/ui.cppm | 235 +++++++++++++++--- src/xlings.cppm | 2 +- tests/e2e/52_local_path_namespaced_index.sh | 35 ++- .../e2e/58_preinstall_mcpp_deps_for_hooks.sh | 48 ++-- tests/e2e/60_stale_xpkg_cache_reinstall.sh | 19 +- 11 files changed, 506 insertions(+), 182 deletions(-) create mode 100644 .agents/docs/2026-06-09-library-download-progress-design.md diff --git a/.agents/docs/2026-06-09-library-download-progress-design.md b/.agents/docs/2026-06-09-library-download-progress-design.md new file mode 100644 index 00000000..122cb498 --- /dev/null +++ b/.agents/docs/2026-06-09-library-download-progress-design.md @@ -0,0 +1,162 @@ +# Library / Component Download Progress — Design + +> Status: approved, in implementation +> Target release: mcpp 0.0.53 +> Scope: make library / component downloads show the same live progress + speed +> that toolchain downloads already show, across Linux / macOS / Windows. + +## 1. Problem + +`mcpp toolchain install` shows a live progress bar with percent, bytes and +transfer speed. Installing **library / component dependencies** (during +`mcpp build`, dependency resolution, or `mcpp new --template`) prints a single +`Downloading v` line and then **appears to hang** for a long time with +no percent, no speed and no sign of life — especially for large packages on slow +mirrors. + +## 2. Root cause (evidence-based) + +There are **three** download code paths. They were not equally instrumented. + +| Path | Trigger | Transport | Symptom | +|------|---------|-----------|---------| +| ① Toolchain | `mcpp toolchain install` | NDJSON `interface install_packages` + `CliInstallProgress` | progress OK | +| ② Builtin-index deps | dep resolves via builtin index (`useProjectEnv = false`) | same as ① | progress OK *once bytes flow* | +| ③ Custom/project-index deps | dep resolves via a project-added index (`useProjectEnv = true`) | **`install_direct(projEnv, target, quiet=true)`** | **fully silent** | + +Two independent defects fall out of this: + +### Defect A — path ③ is silenced + +`src/cli.cppm` routes project/custom-index dependency installs through +`mcpp::xlings::install_direct(projEnv, target, /*quiet=*/true)`. `quiet=true` +redirects xlings' stdout/stderr to the platform null device, so xlings' own +progress output is discarded and **no NDJSON events are parsed** (the direct CLI +does not speak NDJSON). The `Downloading ` line is printed by mcpp *before* +this call, then the whole download runs dark. + +This path was switched to the direct CLI in a prior change to guarantee that +project-index packages land in the **project-local** xlings data root (so a +package's install hook can find sibling packages from the same index). That +change traded away progress as a side effect. + +**Why the NDJSON interface is in fact safe here (verified against xlings +source):** in the pinned xlings, the `install_packages` *capability* and the +`install` *CLI* both call the same `xim::cmd_install(...)`. The install +destination is chosen by **package scope** — `storeRoot = (scope == Project ? +project_data_dir() : global_data_dir()) / "xpkgs"` — which is derived from +*which index the package came from*, not from interface-vs-CLI. `download_progress` +events are emitted unconditionally (no scope gate). So the NDJSON interface +installs project-scoped packages into the project-local data root **and** streams +progress. The earlier switch was working around index *exposure*, which is now +handled separately (the project index is symlinked/exposed into the project data +dir and targets are passed as `indexName:fqname@version`). + +### Defect B — the "connecting" phase freezes the line (all NDJSON paths) + +Captured NDJSON for a real package download shows the downloader emits +`download_progress` events with `totalBytes = 0` (and often `downloadedBytes = 0`) +during the initial connect / TLS / redirect / pre-sizing window, before the +transfer's total size is known: + +``` +elapsed 0.2s–1.4s : downloadedBytes=0 totalBytes=0 sizesReady=false (connecting) +elapsed 1.6s : downloadedBytes=57344 totalBytes=1754511 (bytes flow) +elapsed 2.0s : downloadedBytes=1754511 totalBytes=1754511 finished +``` + +Both `CliInstallProgress::on_data` and `make_bootstrap_progress_callback` only +call `ProgressBar::update_bytes` when `total > 0`, so during the connecting +window the line shows nothing. For a large file on a slow mirror this window can +last many seconds and reads as a hang. Toolchain downloads hide this because the +connect window is tiny relative to a multi-minute transfer; library downloads do +not. + +## 3. Design + +Three changes, all behind existing abstractions, all cross-platform. + +### 3.1 Fix 1 — path ③ uses the NDJSON interface (mcpp-style bar) + +In `src/cli.cppm`, the `useProjectEnv` branch of the dependency install lambda +calls the NDJSON interface with the project env and an `CliInstallProgress` +handler, instead of the silenced direct CLI: + +```cpp +if (useProjectEnv) { + auto projEnv = mcpp::config::make_project_xlings_env(**cfg, *root); + auto argsJson = std::format(R"({{"targets":["{}"],"yes":true}})", target); + CliInstallProgress progress; + auto r = mcpp::xlings::call(projEnv, "install_packages", argsJson, &progress); + if (!r) return std::unexpected(mcpp::pm::CallError{r.error()}); + return *r; +} +``` + +This yields the same cyan `Downloading … [bar] X/Y Z/s` UI on **all** three +paths. Package scope (and therefore install location) is unchanged — packages +from a project index still install into the project-local data root. + +### 3.2 Fix 2 — indeterminate rendering while size is unknown + +Add `ProgressBar::update_indeterminate(current_bytes, elapsed_sec)`: render a +swept/indeterminate bar plus an info suffix that ticks — `connecting… Ns`, or a +byte counter `X.Y MB Ns` once bytes arrive without a known total. This also +covers servers that stream without `Content-Length`. + +Wire both progress consumers (`CliInstallProgress::on_data` and +`make_bootstrap_progress_callback`) to call `update_indeterminate` when the +active file has `total == 0` (and `update_bytes` once `total > 0`). + +Refactor `render_line` so the bar string is produced by a small callable; the +percent path and the indeterminate path share the terminal-width budgeting. + +### 3.3 Fix 3 — pin xlings to the latest release + +Bump `pinned::kXlingsVersion` in `src/xlings.cppm` to the latest xlings release +(the version that ships the unified `cmd_install` + scope-routed install + +unconditional `download_progress`). This removes any dependence on older xlings +behavior for the path ③ install location. + +## 4. Cross-platform + +- Progress rendering already runs on Linux/macOS/Windows (toolchain path uses it + today); no new platform branches are introduced. +- `mcpp::xlings::call` / `build_interface_command` already handle the Windows vs + POSIX command shape and env propagation; path ③ inherits that. +- `update_indeterminate` uses the same `render_line` budgeting (TIOCGWINSZ / + `$COLUMNS` / 80-col fallback) as the existing bar; non-TTY / `--quiet` paths + remain silent via the existing `g_quiet` guard. + +## 5. Testing + +- **Unit-ish / build:** project builds clean with the existing warning gate. +- **E2E regression:** `tests/e2e/52_local_path_namespaced_index.sh` and + `tests/e2e/58_preinstall_mcpp_deps_for_hooks.sh` currently assert path ③ uses + the *direct* CLI. They are updated to assert the **interface** transport while + keeping their existing invariants (project-local install location, and + `mcpp.deps` installed before a dependent package's install hook runs). +- **Real integration check:** with a real project-local custom index and a real + installable package, confirm (a) the package installs into the project-local + data root, (b) the install hook still sees its `mcpp.deps`, and (c) the + `Downloading … [bar] X/Y Z/s` UI renders. This is the empirical gate before + merge. + +## 6. Rollout + +1. Implement Fix 1–3; update e2e tests. +2. Local build + e2e + real integration check. +3. Bump `MCPP_VERSION` (`src/toolchain/fingerprint.cppm`) and `mcpp.toml` to + `0.0.53`; CHANGELOG entry. +4. PR → CI green → squash merge → trigger release `0.0.53`. +5. Ecosystem: add mcpp `0.0.53` to the package index (mcpp xpkg descriptor: + version + per-platform sha256), publish release artifacts to the + release-asset mirrors, PR + merge, verify an end-to-end install of the new + version. + +## 7. Out of scope / follow-ups + +- No change to xlings' own native CLI progress UI. +- No change to the dependency-resolution / preinstall ordering logic (it already + installs deps before dependents; only the per-install transport on path ③ + changes). diff --git a/.xlings.json b/.xlings.json index 0d32bea3..806f890b 100644 --- a/.xlings.json +++ b/.xlings.json @@ -1,5 +1,5 @@ { "workspace": { - "mcpp": "0.0.52" + "mcpp": "0.0.53" } } diff --git a/CHANGELOG.md b/CHANGELOG.md index f5c34a37..b69a4284 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,26 @@ > 本文件追踪 `mcpp-community/mcpp` 公开仓的版本演进。 > 格式参考 [Keep a Changelog](https://keepachangelog.com/zh-CN/1.1.0/)。 +## [0.0.53] — 2026-06-09 + +### 新增 + +- 库 / 组件下载现在与工具链下载一样显示实时进度条、字节进度与速度。自定义 / + 项目索引依赖改经 xlings NDJSON `interface install_packages` 安装(仍落在项目 + 本地数据根,不改变安装位置与 install hook 顺序),不再静默卡住。 + +### 修复 + +- 下载连接 / 预取大小阶段(`totalBytes` 尚未知)进度行不再"冻结"无反馈: + 新增不确定态渲染,显示 `connecting…` + 已用时,流式无 `Content-Length` + 时显示已下载字节,直到拿到总大小再切换为百分比进度条。 + +### 其他 + +- 内置 xlings 版本上调至 `0.4.51`。 +- 下载进度的状态机与渲染集中到 `mcpp.ui`(`DownloadProgress`),工具链 / + 内置索引 / 自定义索引三条路径共用同一套 UI。 + ## [0.0.46] — 2026-06-03 ### 新增 diff --git a/mcpp.toml b/mcpp.toml index aecab7a8..7d856e24 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,6 +1,6 @@ [package] name = "mcpp" -version = "0.0.52" +version = "0.0.53" description = "Modern C++ build & package management tool" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/src/cli.cppm b/src/cli.cppm index 2831e63c..c9ae75cf 100644 --- a/src/cli.cppm +++ b/src/cli.cppm @@ -448,119 +448,52 @@ mcpp::ui::PathContext make_path_ctx(const mcpp::config::GlobalConfig* cfg, return ctx; } -// Stateless adapter from `mcpp::config::BootstrapProgress` (xlings -// download_progress event) to a sticky ProgressBar. Used by -// load_or_init() during the one-time sandbox bootstrap (xim:patchelf, -// xim:ninja, plus their transitive deps). -// -// Two xlings quirks the callback has to absorb: -// 1. Each file's `finished=true` event arrives twice in a row. -// 2. During multi-package installs the `files[]` array reshuffles -// between events (the active download isn't always at slot 0). -// The fix mirrors CliInstallProgress: dedupe via a `finished_` set and -// always pick "active first if still in event, else first -// started+unfinished" rather than reading slot 0 blindly. -mcpp::config::BootstrapProgressCallback make_bootstrap_progress_callback() { - auto bar = std::make_shared>(); - auto active = std::make_shared(); - auto finished = std::make_shared>(); - return [bar, active, finished](const mcpp::config::BootstrapProgress& ev) { - // Process newly-finished entries. - for (auto& f : ev.files) { - if (finished->contains(f.name)) continue; - if (!f.finished) continue; - if (*active == f.name) { - if (*bar) (*bar)->finish(); - bar->reset(); - active->clear(); - } - finished->insert(f.name); - } - - // Pick what to display: prefer continuing with `*active` if it's - // still in the array and not finished, otherwise the first - // started+unfinished entry. - const mcpp::config::BootstrapFile* current = nullptr; - for (auto& f : ev.files) { - if (f.name == *active && !f.finished - && !finished->contains(f.name)) { current = &f; break; } - } - if (!current) { - for (auto& f : ev.files) { - if (finished->contains(f.name)) continue; - if (f.started && !f.finished) { current = &f; break; } - } +// Map a decoded NDJSON `download_progress` files[] snapshot onto the neutral +// `mcpp::ui::DownloadFile` the centralized renderer consumes. +template +std::vector to_ui_download_files(const std::vector& files) { + std::vector out; + out.reserve(files.size()); + for (auto& f : files) { + if constexpr (requires { f.downloadedBytes; }) { + out.push_back({ f.name, + static_cast(f.downloadedBytes), + static_cast(f.totalBytes), + f.started, f.finished }); + } else { + out.push_back({ f.name, + static_cast(f.downloaded), + static_cast(f.total), + f.started, f.finished }); } - if (!current) return; + } + return out; +} - if (current->name != *active) { - if (*bar) (*bar)->finish(); - *active = current->name; - bar->emplace("Downloading", current->name); - } - if (current->totalBytes > 0) { - (*bar)->update_bytes(static_cast(current->downloadedBytes), - static_cast(current->totalBytes), - ev.elapsedSec); - } +// Adapter from `mcpp::config::BootstrapProgress` (xlings download_progress +// event) to the centralized download renderer. Used by load_or_init() during +// the one-time sandbox bootstrap (xim:patchelf, xim:ninja + transitive deps). +mcpp::config::BootstrapProgressCallback make_bootstrap_progress_callback() { + auto progress = std::make_shared(); + return [progress](const mcpp::config::BootstrapProgress& ev) { + auto files = to_ui_download_files(ev.files); + progress->update(files, ev.elapsedSec); }; } +// EventHandler that forwards xlings `download_progress` events to the same +// centralized renderer. Used for toolchain, builtin-index and custom-index +// installs alike, so all three show identical UI. struct CliInstallProgress : mcpp::fetcher::EventHandler { - std::optional bar_; - std::string active_; - std::unordered_set finished_; + mcpp::ui::DownloadProgress progress_; void on_data(const mcpp::fetcher::DataEvent& d) override { if (d.dataKind != "download_progress") return; auto files = parse_all_install_files(d.payloadJson); if (files.empty()) return; - - // 1. Process any newly-finished entries. Each file is reported - // twice with finished=true (xlings quirk); the `finished_` - // set dedupes both that AND the rotation case where the - // same file shows up at a different array slot in a later - // event. - for (auto& f : files) { - if (finished_.contains(f.name)) continue; - if (!f.finished) continue; - if (active_ == f.name) { - if (bar_) bar_->finish(); - bar_.reset(); - active_.clear(); - } - finished_.insert(f.name); - } - - // 2. Pick what to display. Prefer continuing with the current - // `active_` if it's still in the array and not finished — - // otherwise the first started+unfinished entry. This stops - // the bar from flickering between names when xlings reshuffles - // files[] across events during a multi-package install. - const InstallProgressFile* current = nullptr; - for (auto& f : files) { - if (f.name == active_ && !f.finished - && !finished_.contains(f.name)) { current = &f; break; } - } - if (!current) { - for (auto& f : files) { - if (finished_.contains(f.name)) continue; - if (f.started && !f.finished) { current = &f; break; } - } - } - if (!current) return; - - if (current->name != active_) { - if (bar_) bar_->finish(); - active_ = current->name; - bar_.emplace("Downloading", current->name); - } - if (current->total > 0) { - double elapsed = extract_payload_number(d.payloadJson, "elapsedSec"); - bar_->update_bytes(static_cast(current->downloaded), - static_cast(current->total), - elapsed); - } + double elapsed = extract_payload_number(d.payloadJson, "elapsedSec"); + auto ui_files = to_ui_download_files(files); + progress_.update(ui_files, elapsed); } void on_log(const mcpp::fetcher::LogEvent& e) override { @@ -579,7 +512,8 @@ struct CliInstallProgress : mcpp::fetcher::EventHandler { mcpp::log::info("xlings", std::format("hint: {}", e.hint)); } - ~CliInstallProgress() override { if (bar_) bar_->finish(); } + // progress_'s own destructor finishes the active bar. + ~CliInstallProgress() override = default; }; // Compose a stable canonical compile-flags string for fingerprinting. @@ -2312,11 +2246,24 @@ prepare_build(bool print_fingerprint, auto install_one = [&](std::string target) -> std::expected { if (useProjectEnv) { + // Project/custom-index deps install into the project-local + // xlings data root (so a package's install hook can find + // sibling packages from the same index). The NDJSON + // interface honors this: in the pinned xlings the + // `install_packages` capability and the `install` CLI share + // `xim::cmd_install`, and the install destination is chosen + // by package *scope* (project vs global), not by transport. + // Using the interface (rather than the silenced direct CLI) + // restores the live `Downloading … [bar] X/Y Z/s` UI here, + // matching the toolchain and builtin-index paths. auto projEnv = mcpp::config::make_project_xlings_env(**cfg, *root); - int directRc = mcpp::xlings::install_direct(projEnv, target, /*quiet=*/true); - mcpp::xlings::CallResult result; - result.exitCode = directRc; - return result; + auto argsJson = std::format( + R"({{"targets":["{}"],"yes":true}})", target); + CliInstallProgress progress; + auto r = mcpp::xlings::call( + projEnv, "install_packages", argsJson, &progress); + if (!r) return std::unexpected(mcpp::pm::CallError{r.error()}); + return *r; } std::vector targets{ std::move(target) }; CliInstallProgress progress; diff --git a/src/toolchain/fingerprint.cppm b/src/toolchain/fingerprint.cppm index 13d9202c..c746bb9b 100644 --- a/src/toolchain/fingerprint.cppm +++ b/src/toolchain/fingerprint.cppm @@ -18,7 +18,7 @@ import mcpp.toolchain.detect; export namespace mcpp::toolchain { -inline constexpr std::string_view MCPP_VERSION = "0.0.52"; +inline constexpr std::string_view MCPP_VERSION = "0.0.53"; struct FingerprintInputs { Toolchain toolchain; diff --git a/src/ui.cppm b/src/ui.cppm index 0c2a997f..6ca9cd61 100644 --- a/src/ui.cppm +++ b/src/ui.cppm @@ -82,12 +82,21 @@ public: void update_bytes(std::size_t current_bytes, std::size_t total_bytes, double elapsed_sec = 0.0); + // Connecting / pre-sizing phase: the total size isn't known yet (the + // downloader reports totalBytes==0 during DNS/TLS/redirect before the + // transfer's Content-Length is available, and some servers stream with no + // length at all). Renders a swept (indeterminate) bar plus a ticking + // `connecting… Ns` / `X.Y MB Ns` suffix so the line never freezes. + void update_indeterminate(std::size_t current_bytes, + double elapsed_sec = 0.0); + // Finish: replaces progress with final-state line. void finish(); void finish_with(std::string_view final_message); private: void render_line(std::size_t percent, const std::string& info_text); + void render_line_swept(std::size_t frame, const std::string& info_text); std::string verb_; std::string label_; @@ -95,6 +104,42 @@ private: bool finished_ = false; }; +// --- download progress (centralized) --- +// +// One file's download state, decoded from xlings' NDJSON `download_progress` +// `files[]` entries. A neutral struct so this UI module stays free of any +// fetcher / config dependency (those import each other and would cycle). +struct DownloadFile { + std::string name; + std::size_t downloaded = 0; + std::size_t total = 0; // 0 = size not known yet (connecting) + bool started = false; + bool finished = false; +}; + +// Centralized renderer for a streaming multi-file download. Owns the +// ProgressBar plus the "which file is active / which are done" bookkeeping, +// and decides per frame whether to draw a percentage bar (size known) or a +// swept/indeterminate bar (still connecting). Absorbs two xlings quirks: +// each file's `finished=true` is reported twice, and `files[]` reshuffles +// between events. Feed it each event's `files[]` snapshot + cumulative +// `elapsedSec`; it is the single place mcpp turns download events into UI. +class DownloadProgress { +public: + DownloadProgress() = default; + ~DownloadProgress(); + DownloadProgress(const DownloadProgress&) = delete; + DownloadProgress& operator=(const DownloadProgress&) = delete; + + void update(std::span files, double elapsed_sec); + void finish(); // finish the active bar if any (idempotent) + +private: + std::optional bar_; + std::string active_; + std::unordered_set finished_; +}; + // --- quiet flag (suppresses status / info / finished) --- void set_quiet(bool q); bool is_quiet(); @@ -334,31 +379,34 @@ std::string trunc_visible(std::string s, std::size_t max) { return s; } -} // namespace - -ProgressBar::ProgressBar(std::string_view verb, std::string_view label) - : verb_(verb), label_(label), - lastDraw_(std::chrono::steady_clock::now() - std::chrono::seconds(1)) -{} - -ProgressBar::~ProgressBar() { - if (!finished_) finish(); +// Indeterminate ("swept") bar: a fixed-width block bounces back and forth so +// the bar animates while the total size is still unknown. `frame` advances with +// elapsed time; the result includes the `[`/`]` brackets so it drops into the +// same layout budget as render_bar(). +std::string render_bar_swept(std::size_t frame, std::size_t width = 20) { + if (width == 0) return "[]"; + std::size_t block = std::min(3, width); + std::size_t span = width - block; // cells the block can travel + std::size_t pos = 0; + if (span > 0) { + std::size_t period = span * 2; + std::size_t p = frame % period; + pos = (p <= span) ? p : (period - p); // ping-pong + } + std::string inner(width, ' '); + for (std::size_t i = 0; i < block && pos + i < width; ++i) inner[pos + i] = '='; + return "[" + inner + "]"; } -// Render a single progress-bar frame. The verb is drawn separately (with -// optional color) so we can keep ANSI escapes out of the truncation budget. -// `cols` is the available terminal width; `info_text` is the trailing -// "%" / "X MB / Y MB / Z MB/s" suffix; `pct` drives the bar fill. -// -// Layout (visible chars only): +// Shared terminal-width budgeting for a one-line status: draws //