feat(netprobe): HTTP(S) + DoH endpoint probes via libcurl (Refs #697) by leoparente · Pull Request #792 · netboxlabs/pktvisor

leoparente · 2026-06-27T12:24:42Z

Summary

Adds two new netprobe input test types, both built on libcurl (already a dependency), sharing one transport:

http — an HTTP(S) endpoint probe (peer of ping/tcp): periodically requests a target URL and reports availability, status-aware success, response time, and an optional per-phase timing breakdown.
doh — a DNS-over-HTTPS probe (RFC 8484): resolves a configured DNS query against a target DoH URL and reports DNS-aware availability (HTTP success and rcode NOERROR), an rcode breakdown, and response time.

Refs #697 — this supersedes that PR's hand-rolled HTTP/2-over-nghttp2 DoH transport with a complete, metric-emitting DoH probe on libcurl, and exceeds it (timings, status, rcode awareness).

Architecture

curl_multi driven by the existing netprobe uvw (libuv) event loop — no new threads:

HttpClient transport (libs/visor_http_client/): one curl_multi per stream, curl sockets bridged to uvw::poll_handle + one uvw::timer_handle; completions via curl_multi_info_read; timing/status from curl_easy_getinfo. Loop-quiescent teardown (handles closed on the loop thread before the loop stops). Supports a request body, custom headers, and optional response-body capture (the HTTP probe stays zero-copy).
HttpProbe and DohProbe (: NetProbe, peers of PingProbe/TcpProbe) issue requests through the shared per-stream HttpClient.
DoH builds/parses DNS messages with pcpp::DnsLayer and reuses visor_dns's rcode/qtype name maps — no new dependency, and not the DNS stream handler (it can't attach to a netprobe input and its rich metrics degenerate for a single repeated query).
HTTP/2: libcurl is built with libnghttp2 (with_nghttp2=True) so curl negotiates h2 via ALPN for both probes; falls back to HTTP/1.1 transparently.

Metrics

Status-aware, per-target, reusing the existing netprobe response-time histograms/quantiles + the opt-in http_response_phases group:

http: successes (2xx/3xx), http_status_failures (4xx/5xx), top_status_codes, transport failures by cause.
doh: successes (HTTP success and rcode NOERROR), dns_response_failures (HTTP success but non-NOERROR/unparseable), http_status_failures, top_rcodes (e.g. NOERROR/NXDOMAIN/SRVFAIL/PARSE_ERROR), transport failures by cause.

Config

# http
test_type: http
http_method: GET            # GET (default) | HEAD | POST | ...
targets: { health: { target: "https://api.example.com/ping" } }

# doh
test_type: doh
qname: "example.com"        # required
qtype: "A"                  # optional, default A
http_method: "POST"         # POST (default) | GET (?dns=<base64url>)
targets: { cloudflare: { target: "https://cloudflare-dns.com/dns-query" } }

Defaults: follow-redirects on (bounded to 10), TLS verification on, per-request timeout_msec.

Out of scope (v1)

DoH: DNSSEC validation, EDNS options, multiple queries per probe, rdata inspection beyond rcode. General: HTTP/3; custom auth; request bodies beyond DoH's. (HTTP/2 is enabled but not forced — curl negotiates.)

Testing

unit-tests-visor-http-client (30) — HttpClient: 200/404/refused/timeout/close-in-flight, POST body + headers + response capture, per-phase timing sanity.
unit-tests-handler-netprobe (84) — status-aware HTTP + DNS-aware DoH metric derivation, top_status_codes/top_rcodes, response-time, http_response_phases on/off, merge.
unit-tests-input-netprobe (41) — http/doh config validation + end-to-end tests (real stream → curl → in-process cpp-httplib server → handler) for HTTP and for DoH over both POST and GET, plus in-flight-stop teardown.

All pass locally (Release, HTTP/2-enabled libcurl).

Reviewer notes

The HTTP/2 enablement (libnghttp2) is an isolated commit; libnghttp2 builds on musl/Alpine (it powers libcurl there), so it should be fine in the static pipeline — flag if the musl job disagrees.
Two optional test-hardening ideas were noted and deliberately deferred (not defects): a full base64url round-trip check in the DoH GET e2e (the canned server ignores ?dns=; the encoder is verified by inspection), and a single-unambiguous-in-flight-request variant of the DoH teardown test.

🤖 Generated with Claude Code

… (Refs #697)

…osed state (Refs #697)

… (Refs #697)

…er capture (Refs #697)

…ponse_phases group (Refs #697)

…-phase timings (Refs #697)

github-actions · 2026-06-27T12:35:33Z

LCOV of commit `6e720d0` during Debug Builds #197

  lines......: 83.5% (15320 of 18343 lines)
  functions..: 74.7% (1544 of 2068 functions)
  branches...: no data found

Files changed coverage rate: n/a

Full coverage report

Copilot

Pull request overview

Adds an HTTP(S) probing mode to the existing netprobe input by introducing a libcurl curl_multi-driven transport integrated into the existing uvw/libuv event loop, and extends the netprobe handler to emit status-aware HTTP metrics (including optional per-phase timings).

Changes:

Introduces VisorHttpClient (libcurl multi + uvw poll/timer integration) and HttpProbe to issue periodic HTTP requests per target URL.
Extends netprobe input/handler signaling + metrics to record HTTP status outcomes, top status codes, response-time distributions, and optional per-phase quantiles.
Adds unit and end-to-end tests plus handler documentation for the new HTTP probe mode.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
src/inputs/netprobe/test_netprobe.cpp	Adds HTTP config validation + end-to-end HTTP probe tests using an in-process httplib server.
src/inputs/netprobe/NetProbeInputStream.h	Adds HTTP-related members and a new `probe_http_result` signal path.
src/inputs/netprobe/NetProbeInputStream.cpp	Parses HTTP targets/method, creates `HttpClient` + `HttpProbe` instances, and closes HTTP transport during loop teardown.
src/inputs/netprobe/HttpProbe.h	Declares `HttpProbe` netprobe implementation for HTTP requests via `HttpClient`.
src/inputs/netprobe/HttpProbe.cpp	Implements periodic request scheduling and result/error mapping to existing netprobe error types.
src/inputs/netprobe/CMakeLists.txt	Builds `HttpProbe` and links netprobe to the new `VisorHttpClient` library; updates test link deps.
src/handlers/netprobe/test_net_probe.cpp	Adds unit tests for HTTP status-aware counters, TopN status codes, phase group gating, and merge behavior.
src/handlers/netprobe/README.md	Documents HTTP probe configuration, success semantics, and new metrics/groups.
src/handlers/netprobe/NetProbeStreamHandler.h	Adds HTTP metrics group + per-target HTTP metrics fields and new manager APIs.
src/handlers/netprobe/NetProbeStreamHandler.cpp	Wires new HTTP result signal, routes HTTP failures separately, and serializes/merges new metrics.
libs/visor_http_client/test_http_client.cpp	Adds `HttpClient` unit tests against an in-process httplib server (200/404/refused/timeout/close-in-flight).
libs/visor_http_client/HttpTypes.h	Defines `HttpRequest`, `HttpResult`, and `HttpTimings` structs shared between probe/handler.
libs/visor_http_client/HttpClient.h	Declares `HttpClient` and its uvw/libcurl integration contracts.
libs/visor_http_client/HttpClient.cpp	Implements the libcurl multi + uvw socket/timer glue and timing extraction.
libs/visor_http_client/CMakeLists.txt	Adds the new static library and its unit test target.
libs/CMakeLists.txt	Includes the new `visor_http_client` subdirectory in the build.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

… races, README accuracy (Refs #697)

…ss_failure (Refs #697)

…re (Refs #697)

…nt (Refs #697)

…odes (Refs #697)

… (Refs #697)

Copilot

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 8 comments.

…validation, metric help-text, e2e server-ready/port checks (Refs #697)

leoparente · 2026-06-27T20:41:48Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c8053be1f7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…type/question validation, curl-error logging, request() reentrancy guard, POLL_NONE stop, DoH top_status_codes (Refs #697)

leoparente · 2026-06-27T22:40:29Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c9031e48f4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copilot

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 2 comments.

…d curl_slist_append OOM (Refs #697)

leoparente · 2026-06-28T00:49:25Z

@codex review

Copilot

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 2 comments.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5a7f4c393c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…o uppercase (Refs #697)

leoparente · 2026-06-28T14:30:36Z

@codex review

Copilot

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 3 comments.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 90029a8e2d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

… DoH response (TC bit + parsed-vs-declared records); accurate timeout comments (Refs #697)

leoparente · 2026-06-29T12:47:08Z

@codex review

Copilot

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 2 comments.

chatgpt-codex-connector · 2026-06-29T12:57:28Z

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: 3605341e69

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…EADME with actual semantics (Refs #697)

leoparente added 7 commits June 26, 2026 22:12

feat(http): add libcurl HttpClient transport (curl_multi on uvw loop)…

56a1bf0

… (Refs #697)

fix(http): guard HttpClient request/timer/socket callbacks against cl…

03f56ef

…osed state (Refs #697)

feat(netprobe): HttpProbe issuing requests via the libcurl HttpClient…

fae87be

… (Refs #697)

fix(netprobe): only allocate HttpClient for http streams; clarify tim…

8c50aa4

…er capture (Refs #697)

feat(netprobe): status-aware HTTP metrics, top_status_codes, http_res…

6d94a77

…ponse_phases group (Refs #697)

test(netprobe): end-to-end HTTP probe test + docs (Refs #697)

165c2e5

fix(http): bound redirects to 10, document timeout_msec:0, assert per…

a62451c

…-phase timings (Refs #697)

leoparente requested a review from Copilot June 27, 2026 12:39

Copilot started reviewing on behalf of leoparente June 27, 2026 12:40 View session

Copilot AI reviewed Jun 27, 2026

View reviewed changes

leoparente added 7 commits June 27, 2026 11:04

fix(http): handle curl alloc failures, fix watchdog/server-ready test…

4839975

… races, README accuracy (Refs #697)

fix(netprobe): lock _targets_metrics insert in process_attempts/proce…

4eb1fa9

…ss_failure (Refs #697)

build(deps): enable HTTP/2 in libcurl via libnghttp2 (Refs #697)

41ed764

feat(http): HttpClient request body, headers, and response-body captu…

3c618e4

…re (Refs #697)

feat(netprobe): DohProbe issuing DoH queries via the libcurl HttpClie…

03a40fc

…nt (Refs #697)

feat(netprobe): DNS-aware DoH metrics — dns_response_failures, top_rc…

bfa94a5

…odes (Refs #697)

test(netprobe): end-to-end DoH probe test + docs (Refs #697)

ea2876b

leoparente changed the title ~~feat(netprobe): HTTP(S) endpoint probe via libcurl (Refs #697)~~ feat(netprobe): HTTP(S) + DoH endpoint probes via libcurl (Refs #697) Jun 27, 2026

fix(http): include <stdexcept> for std::runtime_error (gcc/libstdc++)…

e9efa44

… (Refs #697)

leoparente self-assigned this Jun 27, 2026

leoparente requested a review from Copilot June 27, 2026 20:10

Copilot started reviewing on behalf of leoparente June 27, 2026 20:10 View session

Copilot AI reviewed Jun 27, 2026

View reviewed changes

fix(netprobe): address PR review — curl add_handle check, DoH method …

c8053be

…validation, metric help-text, e2e server-ready/port checks (Refs #697)

chatgpt-codex-connector Bot reviewed Jun 27, 2026

View reviewed changes

Comment thread libs/visor_http_client/HttpClient.cpp Outdated

Comment thread src/handlers/netprobe/NetProbeStreamHandler.cpp

feat(netprobe): v1 hardening — URL validation, NOSIGNAL, DoH content-…

c9031e4

…type/question validation, curl-error logging, request() reentrancy guard, POLL_NONE stop, DoH top_status_codes (Refs #697)

leoparente requested a review from Copilot June 27, 2026 22:40

Copilot started reviewing on behalf of leoparente June 27, 2026 22:40 View session

chatgpt-codex-connector Bot reviewed Jun 27, 2026

View reviewed changes

Comment thread src/inputs/netprobe/CMakeLists.txt

Copilot AI reviewed Jun 27, 2026

View reviewed changes

Comment thread libs/visor_http_client/HttpClient.cpp

Comment thread src/inputs/netprobe/NetProbeInputStream.h

refactor(http): encapsulate URL validation in visor_http_client; guar…

5a7f4c3

…d curl_slist_append OOM (Refs #697)

leoparente requested a review from Copilot June 28, 2026 00:49

Copilot started reviewing on behalf of leoparente June 28, 2026 00:49 View session

Copilot AI reviewed Jun 28, 2026

View reviewed changes

Comment thread libs/visor_http_client/HttpClient.cpp

Comment thread src/inputs/netprobe/NetProbeInputStream.cpp

chatgpt-codex-connector Bot reviewed Jun 28, 2026

View reviewed changes

Comment thread src/inputs/netprobe/CMakeLists.txt

Comment thread libs/visor_http_client/HttpClient.cpp

fix(netprobe): init curl before URL validation; normalize DoH qtype t…

90029a8

…o uppercase (Refs #697)

leoparente requested a review from Copilot June 28, 2026 14:30

Copilot started reviewing on behalf of leoparente June 28, 2026 14:31 View session

Copilot AI reviewed Jun 28, 2026

View reviewed changes

Comment thread src/inputs/netprobe/test_netprobe.cpp Outdated

Comment thread src/inputs/netprobe/test_netprobe.cpp Outdated

Comment thread src/inputs/netprobe/test_netprobe.cpp Outdated

chatgpt-codex-connector Bot reviewed Jun 28, 2026

View reviewed changes

Comment thread libs/visor_http_client/HttpClient.cpp

Comment thread src/inputs/netprobe/DohProbe.cpp

fix(netprobe): restrict redirects to http(s); require fully-parseable…

3605341

… DoH response (TC bit + parsed-vs-declared records); accurate timeout comments (Refs #697)

leoparente requested a review from Copilot June 29, 2026 12:46

Copilot started reviewing on behalf of leoparente June 29, 2026 12:47 View session

Copilot AI reviewed Jun 29, 2026

View reviewed changes

Comment thread src/handlers/netprobe/README.md Outdated

Comment thread libs/visor_http_client/CMakeLists.txt

chore(http): add Visor::Lib::Http alias; align http_status_failures R…

6e720d0

…EADME with actual semantics (Refs #697)

Uh oh!

Conversation

leoparente commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Architecture

Metrics

Config

Out of scope (v1)

Testing

Reviewer notes

Uh oh!

github-actions Bot commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

LCOV of commit 6e720d0 during Debug Builds #197

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

leoparente commented Jun 27, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

leoparente commented Jun 27, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

leoparente commented Jun 28, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

leoparente commented Jun 28, 2026

Uh oh!

Copilot AI left a comment

leoparente commented Jun 27, 2026 •

edited

Loading

github-actions Bot commented Jun 27, 2026 •

edited

Loading

LCOV of commit `6e720d0` during Debug Builds #197