Mirror kernel archives in ci-mirrors to not depend on currently unavailable kernel mirror sources

[jieyouxu]

View all comments

Summary

(Tempoarily) closes #158718.
See #t-infra > kernel.org is borked.

Currently rust-lang/rust CI is borked because:

1. armhf-gnu directly tries to download kernel v4.x artifacts from kernel.org mirror, and
2. certain dist-* jobs tries to download kernel v3.x and v4.x artifacts from kernel.org mirror indirectly via crosstools-ng ("CT-NG") ¹.

However, kernel.org mirror sync infra has an active incident (ref: https://status.linuxfoundation.org/incidents/3y1k8b4ky71t) where not all files are available. In particular, v3.x and v4.x kernel artifacts are not recovered yet as of time of writing (check https://mirrors.edge.kernel.org/pub/linux/kernel/).

Mitigation

This PR tries to mitigate this by using our own mirrored kernel artifacts from ci-mirrors (see rust-lang/ci-mirrors#41, rust-lang/ci-mirrors#42, rust-lang/ci-mirrors#43), such that linux-$A.$B.$C.tar.gz artifacts become available as search locations to CT-NG.

We carry a patch to CT-NG to use our own ci-mirrors artifact SHA256 checksums, the checksum mismatches are as far as we know due to compression setting differences (ci-mirrors source archives from https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/snapshot/).

Notes

• armhf-gnu still has artifacts from third-party mirrors/sources (busybox, ubuntu-base-22.04.2). Not in scope of this PR.
• It looks like 3.* kernel series are considered obselete by CT-NG, and need CT_OBSOLETE=y config.
• CT-NG checks for known-good kernel versions and checksums against the provided mirror.

CT-NG known good versions

Re. latest kernel versions in each release versus what CT-NG 1.28.0 considers a known good version w/ checksum:

Kernel LTS release series	Last release in the series	CT-NG 1.28.0 known good kernel version w/ checksum
3.2.*	3.2.102	3.2.101
3.10.*	3.10.108	3.10.108
4.4.*	4.4.302	4.4.302
4.19.*	4.19.325	4.19.325
4.20.*	4.20.17	4.20.17
5.19.*	5.19.17	5.19.16

(Bold CT-NG known good version indicates not latest in release series.)

ci-mirrors kernal tarball SHA256 checksums

Refer to https://github.com/rust-lang/ci-mirrors/blob/b474b4bb35108dab668907172c858854f209c809/files/rustc/kernel.toml:

Kernel Version	SHA256
3.2.101	93e8391e029f131d5ba4b7ad76cc34b12f2c2244059604042f2297c4bde093f7
3.10.108	b1711610cf3faf7194156dacdb98c63c1b7ffd02377269d7f75df63d823ccbba
4.4.302	a22ceab143d40f511203265e5a70d6cc5ec39163cd54fa281346d19176f64451
4.19.325	8753443636e475b506e08abd40059ec9b84904a115d206014f0c856dfe13a25e
4.20.17	313b7bebb46084efbfcaf75f4ea6faf2e14c8cbc1711fcba483dc0a036c9acc1
5.19.16	bbf0ead65559250e0784c13d4f9716b7f917a1d3a4e00f671f3994cc6990bb02

---

try-job: armhf-gnu
try-job: dist-armhf-linux
try-job: dist-arm-linux-gnueabi
try-job: dist-powerpc64le-linux-musl
try-job: dist-loongarch64-musl
try-job: dist-riscv64-linux
try-job: dist-powerpc64le-linux-gnu
try-job: dist-powerpc64-linux-musl
try-job: dist-armv7-linux
try-job: dist-powerpc-linux
try-job: dist-loongarch64-linux
try-job: dist-s390x-linux
try-job: dist-i586-gnu-i586-i686-musl
try-job: dist-powerpc64-linux-gnu

Footnotes

1. Add v5.x but those are already recovered. ↩

[rust-log-analyzer]

The job dist-apple-various failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

[RUSTC-TIMING] yoke_derive test:false 0.859
   Compiling displaydoc v0.2.5
error: linking with `cc` failed: exit status: 1
  |
  = note:  "cc" "-Wl,-exported_symbols_list" "-Wl,/Users/runner/work/rust/rust/build/aarch64-apple-darwin/bootstrap-tools/release/build/zerofrom-derive/bd0d67e481534373/out/rustcUwtmU8/list" "/Users/runner/work/rust/rust/build/aarch64-apple-darwin/bootstrap-tools/release/build/zerofrom-derive/bd0d67e481534373/out/rustcUwtmU8/symbols.o" "<7 object files omitted>" "/Users/runner/work/rust/rust/build/aarch64-apple-darwin/bootstrap-tools/release/build/zerofrom-derive/bd0d67e481534373/out/rustcUwtmU8/rmeta.o" "<1 object files omitted>" "/Users/runner/work/rust/rust/build/aarch64-apple-darwin/bootstrap-tools/release/build/synstructure/971d5cb1c2bb7029/out/libsynstructure-971d5cb1c2bb7029.rlib" "/Users/runner/work/rust/rust/build/aarch64-apple-darwin/bootstrap-tools/release/build/syn/c096cb8fc5f8ec7c/out/libsyn-c096cb8fc5f8ec7c.rlib" "/Users/runner/work/rust/rust/build/aarch64-apple-darwin/bootstrap-tools/release/build/quote/f82bde6d5961fcb6/out/libquote-f82bde6d5961fcb6.rlib" "/Users/runner/work/rust/rust/build/aarch64-apple-darwin/bootstrap-tools/release/build/proc-macro2/b089cc3d8a1dc156/out/libproc_macro2-b089cc3d8a1dc156.rlib" "/Users/runner/work/rust/rust/build/aarch64-apple-darwin/bootstrap-tools/release/build/unicode-ident/59c09d8f5bb0c300/out/libunicode_ident-59c09d8f5bb0c300.rlib" "<sysroot>/lib/rustlib/aarch64-apple-darwin/lib/{libproc_macro-*,librustc_literal_escaper-*,libstd-*,libpanic_unwind-*,libobject-*,libmemchr-*,libaddr2line-*,libgimli-*,libcfg_if-*,librustc_demangle-*,libstd_detect-*,libhashbrown-*,librustc_std_workspace_alloc-*,libminiz_oxide-*,libadler2-*,libunwind-*,liblibc-*,librustc_std_workspace_core-*,liballoc-*,libcore-*,libcompiler_builtins-*}.rlib" "-lSystem" "-lc" "-lm" "-arch" "arm64" "-mmacosx-version-min=11.0.0" "-o" "/Users/runner/work/rust/rust/build/aarch64-apple-darwin/bootstrap-tools/release/build/zerofrom-derive/bd0d67e481534373/out/libzerofrom_derive-bd0d67e481534373.dylib" "-Wl,-dead_strip" "-dynamiclib" "-nodefaultlibs"
  = note: some arguments are omitted. use `--verbose` to show all linker arguments
  = note: clang: error: unable to execute command: Segmentation fault: 11
          clang: error: linker command failed due to signal (use -v to see invocation)
          Apple clang version 17.0.0 (clang-1700.6.3.2)
          Target: arm64-apple-darwin24.6.0
          Thread model: posix
          InstalledDir: /Applications/Xcode_26.2.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
          clang: note: diagnostic msg: 
          ********************
          
          PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
          Linker snapshot containing input(s) and associated run script(s) are located at:
          clang: note: diagnostic msg: /var/folders/k8/j7r3p6cx43xdqhzy2rmp6tqr0000gn/T/linker-crash-6955e4
          clang: note: diagnostic msg: 
          
          ********************
          

[RUSTC-TIMING] zerofrom_derive test:false 1.557

[rust-bors]

💔 Test for c8dc98e failed: CI. Failed job:

• auto - dist-apple-various (web logs, enhanced plaintext logs)

[jieyouxu]

AFAIK unrelated, but pls open issue about it (can't do it rn)
@bors retry

[rust-bors]

☀️ Try build successful (CI)
Build commit: 28afe60 (28afe6046ea763c71100169d6ac90fb452ae428e)
Base parent: c397dae (c397dae808f70caebab1fc4e11b3edf7e59f58c7)

[jhpratt]

This seems likely to succeed. Let's let the cargo submodule update run if it does. If this fails, limit the possibility of repeated failures by ensuring it's only a single PR.

@bors treeclosed=21

[rust-bors]

Tree closed for PRs with priority less than 21.

[rust-bors]

☀️ Test successful - CI
Approved by: Mark-Simulacrum
Duration: 3h 12m 58s
Pushing 96eb6d7 to main...

[jhpratt]

@bors treeclosed=5

[rust-bors]

Tree closed for PRs with priority less than 5.

[github-actions]

What is this?

This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing c397dae (parent) -> 96eb6d7 (this PR)

Test differences

Show 4 test diffs

4 doctest diffs were found. These are ignored, as they are noisy.

Test dashboard

Run

cargo run --manifest-path src/ci/citool/Cargo.toml -- \
    test-dashboard 96eb6d756bb98a82ae056c384575ff059bf80e8f --output-dir test-dashboard

And then open test-dashboard/index.html in your browser to see an overview of all executed tests.

Job duration changes

1. i686-msvc-2: 2h 4m -> 1h 16m (-38.4%)
2. dist-i686-linux: 1h 29m -> 2h (+35.0%)
3. dist-aarch64-apple: 2h 22m -> 1h 39m (-30.3%)
4. dist-x86_64-llvm-mingw: 1h 34m -> 2h (+27.7%)
5. x86_64-gnu-llvm-21-3: 1h 26m -> 1h 49m (+26.0%)
6. dist-i686-mingw: 2h 11m -> 2h 37m (+19.5%)
7. dist-arm-linux-musl: 1h 38m -> 1h 19m (-19.4%)
8. dist-x86_64-apple: 2h 1m -> 1h 39m (-18.5%)
9. dist-x86_64-msvc: 2h 6m -> 1h 44m (-17.8%)
10. aarch64-apple-macos-26: 3h 7m -> 2h 43m (-12.8%)

How to interpret the job duration changes?

Job durations can vary a lot, based on the actual runner instance
that executed the job, system noise, invalidated caches, etc. The table above is provided
mostly for t-infra members, for simpler debugging of potential CI slow-downs.

[rust-log-analyzer]

A job failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

  COMMIT_MESSAGE: cargo update 
##[endgroup]
Downloading single artifact
Preparing to download the following artifacts:
- Cargo-lock (ID: 8086846255, Size: 49060, Expected Digest: sha256:b16175bb8cf58898252fd9a540779338969c83a001443f7e1455d2c4e785d87c)
Redirecting to blob download url: https://productionresultssa18.blob.core.windows.net/actions-results/e2e1d469-341a-4700-8486-85f43a10ff06/workflow-job-run-6cdca086-467c-5fc3-9866-36f7e678943f/artifacts/858175980ba8726bab9f963fd06060d5199d874bbd735e523e2d990d354f8c0b.zip
Starting download of artifact to: /home/runner/work/rust/rust
(node:2293) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
SHA256 digest of downloaded artifact is b16175bb8cf58898252fd9a540779338969c83a001443f7e1455d2c4e785d87c
Artifact download completed successfully.
Total of 1 artifact(s) downloaded
Download artifact has finished successfully
---
  COMMIT_MESSAGE: cargo update 
##[endgroup]
Downloading single artifact
Preparing to download the following artifacts:
- cargo-updates (ID: 8086846398, Size: 2775, Expected Digest: sha256:f1fff7ac19f35c21713b6180871000706cfa2a4bdbf50965a92dd12cfbaa3659)
Redirecting to blob download url: https://productionresultssa18.blob.core.windows.net/actions-results/e2e1d469-341a-4700-8486-85f43a10ff06/workflow-job-run-6cdca086-467c-5fc3-9866-36f7e678943f/artifacts/ce9034d4568b3430f906da1368ae1cf3e14ef1542859eb2e8d635c0876371f53.zip
Starting download of artifact to: /home/runner/work/rust/rust
(node:2305) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
SHA256 digest of downloaded artifact is f1fff7ac19f35c21713b6180871000706cfa2a4bdbf50965a92dd12cfbaa3659
Artifact download completed successfully.
Total of 1 artifact(s) downloaded
Download artifact has finished successfully
##[group]Run echo "${COMMIT_MESSAGE}" > commit.txt
echo "${COMMIT_MESSAGE}" > commit.txt
cat cargo_update.log >> commit.txt

echo "${PR_MESSAGE}" > body.md
echo '```txt' >> body.md
cat cargo_update.log >> body.md
echo '```' >> body.md
shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
---
dep-bumps
following is the output from `cargo update`:
  COMMIT_MESSAGE: cargo update 
##[endgroup]
##[group]Run git config user.name github-actions
git config user.name github-actions
git config user.email github-actions@github.com
git switch --force-create cargo_update
git add ./Cargo.lock ./library/Cargo.lock ./src/tools/rustbook/Cargo.lock
git commit --no-verify --file=commit.txt
shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
---
dep-bumps
following is the output from `cargo update`:
  COMMIT_MESSAGE: cargo update 
##[endgroup]
remote: error: GH013: Repository rule violations found for refs/heads/cargo_update.        
remote: Review all repository rules at https://github.com/rust-lang/rust/rules?ref=refs%2Fheads%2Fcargo_update        
remote: 
remote: - Cannot update this protected ref.        
remote: 
remote: - Changes must be made through a pull request.        
remote: 
remote: - Cannot force-push to this branch        
remote: 
To https://github.com/rust-lang/rust
 ! [remote rejected]   cargo_update -> cargo_update (push declined due to repository rule violations)
error: failed to push some refs to 'https://github.com/rust-lang/rust'
##[error]Process completed with exit code 1.
Post job cleanup.

[rust-timer]

Finished benchmarking commit (96eb6d7): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This perf run didn't have relevant results for this metric.

Max RSS (memory usage)

This perf run didn't have relevant results for this metric.

Cycles

Results (secondary -2.0%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.6%	[2.6%, 2.6%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-4.3%	[-6.3%, -2.4%]	2
All ❌✅ (primary)	-	-	0

Binary size

This perf run didn't have relevant results for this metric.

Bootstrap: 485.944s -> 485.519s (-0.09%)
Artifact size: 393.37 MiB -> 393.84 MiB (0.12%)