feat(persistent): in-place solver updates (Solver framework + HiGHS/Gurobi/Xpress/Mosek) by FabianHofmann · Pull Request #718 · PyPSA/linopy

FabianHofmann · 2026-05-21T15:05:02Z

Changes proposed in this Pull Request

Adds an opt-in persistent-update framework so a built solver can be re-solved against a mutated Model without a full rebuild.

Core

linopy.persistent: ModelSnapshot, ModelDiff, StructuralKey, VarKind, RebuildReason. ModelDiff stores changes in flat-native arrays (bounds / var-types / RHS / signs / COO coefs / linear objective / sense) plus per-container VarSlice / ConSlice views.
ModelDiff.from_snapshot(snap, model) and ModelDiff.from_models(m1, m2) — snapshot-based and snapshot-free diffs.
_coef_dirty flag on constraints with RHS-setter short-circuit so RHS-only edits skip the coefficient re-walk.

Solver orchestration

Solver gains track_updates, lazy-build (first solve(model, …) builds), apply_update, update, disallow_rebuild, and structured rebuild reasons. Backends without persistent-update support short-circuit to rebuild.
Per-backend apply_update:
- HiGHS — changeColsBounds / changeColsIntegrality / changeRowBounds / changeCoeff / changeColsCost / changeObjectiveSense. Sign change → rebuild.
- Gurobi — setAttr(LB/UB/VType/RHS/Sense/Obj), chgCoeff, ModelSense. In-place sign change.
- Xpress — chgbounds / chgcoltype / chgrhs / chgrowtype / chgmcoef / chgobj / chgobjsense. In-place sign change.
- Mosek — chgvarbound / chgconbound / putvartypelist / putaijlist / putclist / putobjsense. Sign change → rebuild.

Tests

New test_persistent_snapshot_diff.py covering all ModelDiff semantics.
New parametrized test_persistent_apply_update.py running 9 cases × 4 backends (skipped per backend when license/installation is unavailable).
Cross-instance, pickle, and threading coverage in test_persistent_solver_extras.py.

Follow-ups (tracked)

Findings from a symmetric rolling-horizon benchmark on a PyPSA-Eur network (426k vars / 1.28M cons, 168h window, 24h step, HiGHS):

Basis policy after apply_update: the in-place path reuses the stale basis unconditionally. With HiGHS a valid basis skips presolve, so a 24h-shifted window hot-starts dual simplex on the full unpresolved LP — 126–219s vs 9.4s after clearSolver() (13–23x regression). Expose a reset_solver-style option on Solver.solve and/or auto-clear the basis when the diff touches a large fraction of rows. To be done by the user, add to docs
Sanitization on the Solver.solve(model=...) path: Model.solve runs sanitize_zeros / sanitize_infinities before building (drops ~11% of rows on the benchmark network); the bare Solver.solve path never sanitizes, so persistent users submit a larger problem. Can't be bolted on unconditionally — sanitize-induced mask changes between solves can themselves trigger rebuilds (STRUCTURAL_LABELS/SPARSITY).

Checklist

Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
Unit tests for new features were added (if applicable).
A note for the release notes doc/release_notes.rst of the upcoming release is included.
I consent to the release of this PR's code under the MIT license.

Tracks per-Constraint coefficient mutation via a single boolean slot, flipped in coeffs/vars/lhs setters. Pure-constant rhs writes now short-circuit and leave coeffs/vars buffers untouched (by identity), so rhs-only updates don't trigger expensive coefficient recompare on the persistent-solver fast path.

Pure-Python snapshot primitives for the persistent-solver Phase 1. Deep-copies value-side fields (var_lb/ub, con_rhs/sign, obj_linear), holds vlabels/clabels by reference, stores canonical CSR (indptr, indices) per constraint container. No Solver import.

Pure-function diff for the persistent-solver Phase 1. Detects structural, coord, sparsity, quadratic-objective, value-only var/con, and objective-linear/sense changes. Supports same_model fast path via _coef_dirty and cross-model full re-scan. Includes a focused test suite covering capture, mutation paths, deep-copy invariant, and the same_model toggle.

- supports_persistent_update class flag (default False) - snapshot/_rebuilds/_in_place_updates/_last_rebuild_reason fields - snapshot capture at end of direct _build, _clear_coef_dirty helper - apply_update stub raising UnsupportedUpdate - solve(model, assign) dispatcher with diff-or-rebuild path - update(model, apply=True) primitive returning ModelDiff - threading.Lock around diff+apply+resnapshot - __getstate__/__setstate__ drop native handle and snapshot

…date support Skip diff computation entirely when supports_persistent_update is False on apply, per plan: 'dispatcher checks flag before calling — if False, skips diffing entirely and goes to rebuild.'

Replace xarray-based snapshot and CSR pattern compare with per-row canonicalised numpy buffers; new ContainerVarUpdate / ContainerRowUpdate payloads. Gurobi/HiGHS apply_update rewritten around batched setAttr / changeColsBounds / changeColsCost / changeColsIntegrality; coefficient writes touch only changed cells. Cross-model diff now ~matches same-model cost for bound/rhs/coef-value sweeps.

compute_diff/Solver.solve/Solver.update grow an ignore_dims kwarg. None (default) keeps the current no-coord-check behaviour; any iterable opts into per-container coord-equality on every dim not in the set, supporting rolling-horizon workflows where e.g. the snapshot dim is expected to drift.

…_rebuild - Solver.from_name now accepts model=None; the first solve(m, ...) builds. - compute_diff folded into ModelDiff.from_snapshot classmethod; new ModelDiff.from_models diffs two linopy models directly. - Solver.solve grows disallow_rebuild=True, which raises RebuildRequiredError instead of falling back to a rebuild.

…m_models - Add `track_updates` flag (default False) to Solver; skip ModelSnapshot capture when disabled. Raise UpdatesDisabledError on solve(model)/update() if a built solver was constructed without tracking. - Rewrite ModelDiff.from_models to build directly from two models without capturing snapshots; share helpers with from_snapshot. - Update persistent tests to opt into track_updates=True; add coverage for the disabled path.

Cross-instance resolves now diff via from_models against the previously built model, with no snapshot. Same-instance mutation still raises UpdatesDisabledError. Snapshot recapture is skipped in this mode. Add cross-instance solve/update tests for the no-snapshot path.

Collapse _diff_objective QUAD_OBJ branches; cache n_coef_updates; short-circuit _canonicalize_rows when rows already sorted; tighten buffer extraction. Introduce VarKind enum used across snapshot/diff and HiGHS/Gurobi apply_update; reuse linopy.constants sign tokens. Move _clear_coef_dirty into ModelSnapshot.capture.

Source con buffers from Constraint.to_matrix_with_rhs, replacing the dense (n_rows, max_n_term) arrays with CSR (indptr, indices, data). Sign dtype adopts 'U1' across the persistent layer and apply_update in HiGHS/Gurobi consumes CSR-slice payloads instead of -1 masks. Deletes _canonicalize_rows and the _INT64_MAX sentinel.

Replace per-container ContainerVarUpdate/ContainerRowUpdate dicts with flat arrays (var_bounds_*, var_type_*, con_coef_* COO, con_rhs_*, con_sign_*) plus VarSlice/ConSlice per-container offsets for diagnostics. Add con_rhs_as_bounds() for ranged-row solvers. Backend apply_update bodies collapse to flat-array calls; remove duplicated label->position resolution.

Implement in-place model updates for Xpress (chgbounds/chgrhs/chgmcoef/ chgrowtype/chgobj/chgobjsense/chgcoltype) and Mosek (chgvarbound/ chgconbound/putaijlist/putclist/putvartypelist/putobjsense). Mosek rejects constraint sign change to trigger rebuild. Consolidate gurobi/highs apply_update tests into a single parametrized file that also covers xpress and mosek.

for more information, see https://pre-commit.ci

FabianHofmann · 2026-05-21T15:22:08Z

@FBumann here we go, if you want to take a first look. docs to come

FBumann · 2026-05-22T06:28:10Z

@FBumann here we go, if you want to take a first look. docs to come

Ill have a look ltoday

* hold solver lock through _run_direct so two threads calling solve(model) on the same Solver no longer race on the native handle (HiGHS returned 0.0 from the second concurrent solve). * narrow Optional ndarrays in persistent.diff.push_var / push_con and in HiGHS/Gurobi/Xpress/Mosek apply_update objective paths. * widen Constraint.rhs setter to ExpressionLike | VariableLike | ConstantLike to match the as_expression call in the body. * widen Constraints.__getitem__(str) return type to Constraint (the dominant case) so tests can set .rhs/.coeffs/.sign without ignores. * add docs for in-place solver updates.

FBumann · 2026-05-22T08:58:18Z

@FabianHofmann Sorry, I wont be able to review this today.

FabianHofmann · 2026-05-22T09:24:55Z

@FabianHofmann Sorry, I wont be able to review this today.

take your time, there is no hurry. I'll do some integration tests anyway

FBumann · 2026-05-22T14:54:21Z

Are there any conflicts with #717? Are we sure we want to merge and publish this before we go v1?

Just making sure...

MaykThewessen · 2026-05-23T16:43:37Z

Benched PR #718 ModelDiff vs hand-rolled changeRowsBounds path on a synthetic 69K-var / 160K-row DC-OPF, 8 weekly-style chunks (RHS + objective mutations only, identical structure across chunks). Per-chunk averages, chunk 0 excluded:

route	t_build	t_diff	t_apply	t_solve	t_total	in_place
blueprint + `changeRowsBounds` (hand-rolled)	0.000	0.001	0.006	0.759	0.766	0
ModelDiff sweep (fresh `Model` per chunk)	0.222	0.012	0.039	12.365	12.652	7
ModelDiff in-place (mutate `model.constraints[name].rhs.values[...]`)	0.003	0.010	0.041	4.919	4.982	7
ModelDiff in-place + manual `solver.solver_model.clearSolver()`	0.003	0.010	0.041	0.572	0.637	7

Diff compute + apply_update sum to ~50 ms — those are not the bottleneck. The regression is entirely in t_solve: after in-place RHS mutation the persistent path keeps HiGHS's prior basis, and HiGHS then skips presolve ("Solving LP with useful basis so presolve not used"). For LPs with strong presolve reduction (89 % row reduction on our production model, similar on this synthetic) the skipped presolve costs more than the warm basis saves.

Manually calling solver.solver_model.clearSolver() between solver.update() and solver.solve() recovers presolve and brings the persistent path slightly under the hand-rolled baseline. But that defeats the encapsulation — the user reaches past the Solver API to touch the native HiGHS handle.

Suggestion: expose an opt-in on the persistent API so users with presolve-heavy LPs can drop the warm basis without bypassing the encapsulation. Either:

a keep_basis: bool = True kwarg on Solver.solve() / Solver.update(), or
a Solver.clear_basis() method on the base class with backend-specific implementations (Highs.clearSolver(), Gurobi reset(), etc.).

Zero structural rebuilds observed across all 7 re-solves in both ModelDiff routes (RebuildReason.NONE). The diff correctly classifies these as RHS + coefficient-only changes — the persistent API is doing its job, the missing piece is just the basis-clear hook.

Caveats:

Synthetic DC-OPF surface (no SoC handoff, no NTC, no PST). Production LP is ~9× larger; presolve reduction and basis-warmstart behavior may differ at scale. Plan to repeat at production scale once the basis-clear hook lands (or we wire clearSolver() through manually).
Linopy pinned to PR feat(persistent): in-place solver updates (Solver framework + HiGHS/Gurobi/Xpress/Mosek) #718 head 089cf2e09. Bench harness is internal; happy to share details on request.

FBumann · 2026-05-27T08:25:01Z

@MaykThewessen Im not sure if the complexity and instability of manipulating .values directly is worth it. Does anyone have the time to create some sort of benchmkark to check about what kind of performance gain (absolute) we are talking here? The new update api already improves performance by miles, so id rather focus on stability instead of the last percentages of performance.
Is it actually a meaningful performance gain?

* feat: Variable.update() / Constraint.update() as canonical mutation API Introduces typed ``.update()`` methods on Variable and Constraint as the single, validated, multi-attribute mutation entry point. - ``Variable.update(lower=, upper=)``: validates non-constant inputs are rejected, new dims are rejected, and the resulting ``lower <= upper`` invariant holds across all coords. Returns ``self`` for chaining. - ``Constraint.update(rhs=, sign=)``: constant RHS only. The legacy ``c.rhs = variable`` rearrange-to-lhs path stays on the setter (different semantic, deserves its own explicit method). The existing ``.lower`` / ``.upper`` / ``.sign`` setters become thin shims that forward to ``.update()``, so single-attribute writes (``z.lower = 2``) stay ergonomic and the canonical validation runs in one place. The ``.rhs`` setter forwards constants through ``.update()`` and keeps the expression-rhs rearrange behaviour. This is the on-top experiment for the design discussion on #718. ``.lhs`` / ``.coeffs`` / ``.vars`` setters are untouched for now. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(update): Constraint.update accepts Variable/Expression rhs Mirrors the existing ``c.rhs = expr`` setter and ``add_constraints`` which both accept mixed-side input and rearrange the residual onto lhs. ``c.update(rhs=x + 5)`` now subtracts ``x`` from lhs and stores ``5`` on rhs. ``.rhs`` setter collapses to a one-line shim. Variable bound rejection of Variable/Expression is kept (bounds are numeric, not symbolic); docstring clarified to spell out that pandas / xarray / numpy arrays are first-class (time-varying bounds). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(update): extend Constraint.update to lhs/coeffs/vars; shim all setters Adds lhs / coeffs / vars to the canonical mutation API. All .lhs / .coeffs / .vars setters now forward to .update() — every Constraint mutation goes through one method with one validation path, one place that flips _coef_dirty. Composition rules: - lhs= replaces the whole expression first; subsequent rhs= rearrangement (Variable/Expression in rhs) sees the new lhs. - lhs= and coeffs= / vars= are mutually exclusive (whole replacement vs partial array update). - sign= is applied last so it composes cleanly. Internal Constraint.sanitize_zeros migrated to update(vars=, coeffs=) — no more internal setter calls in linopy/. 389 tests pass across mutation + persistent-solver suite. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(update): rename Constraint.update kwarg vars= -> variables= Avoids shadowing Python's vars() builtin. The .vars attribute on Constraint stays (it parallels the .data.vars internal name); only the kwarg gets the unambiguous spelling. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(update): accept positional ConstraintLike in Constraint.update Mirrors add_constraints' dispatch: c.update(x + 5 <= 3) is now shorthand for c.update(lhs=x, sign='<=', rhs=-2), extracted from the AnonymousConstraint / ConstraintBase the comparison produces. Mutually exclusive with the per-attribute kwargs; clear error when mixed. Also reverts the internal sanitize_zeros migration. The setters are pure shims forwarding to update(), so the migration didn't change behaviour or cost — just spelling. The original setter syntax reads more naturally there. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(update): note kwarg form is the targeted, cheap path The positional ConstraintLike form (c.update(x + 5 <= 3)) always rewrites lhs / sign / rhs and flips _coef_dirty. For hot loops that only touch one part, kwarg form (c.update(rhs=...)) skips the unchanged attributes and is materially cheaper. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(persistent): default ModelDiff.from_snapshot(same_model=False) Closes the A1 residual from the #718 review. The flag-trust path (`skip_coef_compare = same_model and not coef_dirty`) is correct through Constraint.update() (set in one place, shims forward), but `c.coeffs.values[...] = ...` still bypasses _coef_dirty. With same_model=True as the default, that bypass silently produces wrong diffs. Flip the default to False. Cross-model paths (the only production caller, Solver._update_locked, passes explicitly) are unaffected. Same-model warm-update paths now value-diff the CSR data — small perf hit (50-200ms at Mayk-scale per Mayk's bench), correct by default. Solver-aware callers who own the mutation contract can opt back into the optimization with `same_model=True`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: teach .update() in tutorials; mark setters as syntactic sugar - examples/manipulating-models.ipynb: rewrite mutation cells to use Variable.update / Constraint.update; setter form is mentioned in notes as syntactic sugar for the same call. - examples/creating-constraints.ipynb: reframe the CSRConstraint vs Constraint API table around .update() as the mutation API; setters are sugar. - Setter docstrings now say 'syntactic sugar for Constraint/Variable .update; do not add logic here so the contract stays single-sourced' — a directive to future contributors as much as to readers. No deprecation, no breaking change. .update() is the documented canonical mutation API; the seven setters continue to exist as one-line shims. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * deprecate(update): warn on mutation setters; promote .update() in docs Adds DeprecationWarning to all seven mutation setters (Variable.lower, Variable.upper, Constraint.coeffs, Constraint.vars, Constraint.sign, Constraint.rhs, Constraint.lhs). Each setter still forwards to .update() so existing code keeps working; the warning points at the canonical API. Internal sanitize_zeros migrated off setters (the last linopy/ caller). api.rst gains Modification sections listing .update() for both Variable and Constraint; tutorial notes rewritten to teach .update() and flag setters as deprecated. Release note added. dual.setter / solution.setter untouched — result assignment, not mutation, different deprecation track. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(update): edge-case coverage; document rhs-rearrangement invariant Constraint.update tests: lhs-only, coeffs-only (vars preserved), compound lhs+sign, mutually-exclusive lhs+coeffs and lhs+variables. Variable.update tests: upper-only, valid array bound. Migrate test_constraint_coef_dirty.py from the now-deprecated setters to .update(), exercising the canonical path; add positional-form and no-op cases. Net effect: same dirty-flag invariants, 7 fewer warnings per pytest run. Docs: Constraint.update rhs= gains a worked example showing the two forms (constant vs variable/expression). add_constraints rhs gets a matching note pointing at the linopy invariant so the rearrangement rule is documented at the creation site too. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * review(update): address inline feedback on #727 - Constraint._assign_lhs_expr → _assign_lhs (drop redundant suffix; the method already takes a LinearExpression, so the type was in the signature, not the name). - Add Constraint._assign_data(**fields) helper. Wraps the four ``self._data = assign_multiindex_safe(self.data, **kw)`` callsites inside update() (rhs / coeffs / vars / sign). Untouched: the same pattern in dual.setter, sanitize_missings, sanitize_infinities — those aren't update() and stay out of scope here. - Add types.CONSTANT_TYPES tuple, derived from ConstantLike via get_args so the two cannot drift. Variable.update bound validation flipped from negative (isinstance against a hand-rolled non_constant tuple) to positive (isinstance against CONSTANT_TYPES); drops a redundant in-function ``from linopy import expressions`` (the module-level import already covered it). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * deprecate(update): FutureWarning on DataArray as variables=; extract _validate_update Constraint - sanitize_zeros now writes _data via _assign_data directly (no longer round-trips through update(variables=DataArray), which would self-trigger the new deprecation warning). - Constraint.update(variables=...) emits FutureWarning when passed a raw DataArray of integer labels; Variable is the supported input. The path stays accepted for back-compat and will be removed alongside the v1 cleanup. Variable - Extract Variable._validate_update(*, lower, upper) — validates, broadcasts, and runs the cross-bound (lb<=ub) check, returning a dict ready for assignment. update() body shrinks to ~3 lines. Coord validation (parity with add_variables) deferred to #726 land. Tests - test_constraint_coef_dirty's variables= test now passes a Variable instead of a DataArray (matches the supported input). - test_constraint_vars_setter_with_array wrapped in pytest.warns(FutureWarning) — locks in the deprecation contract. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(constraints): _assign_data → _update_data; manage _coef_dirty inside The helper now writes fields AND flips _coef_dirty when the written set includes coeffs or vars. Callsites in update() (coeffs / vars branches) and sanitize_zeros drop their explicit `self._coef_dirty = True` lines — the rule lives in one place, can't be forgotten by future field additions. rhs / sign writes still don't dirty (correctly). _assign_lhs is untouched: it uses a different write mechanic (drop_vars + assign) and manages its own flag. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

FBumann · 2026-05-27T11:06:57Z

~~Blocked by #732~~

MaykThewessen · 2026-05-27T11:37:55Z

Benchmarked it. Isolated the mutation step (no solve) — typed constraint setter (con.rhs = arr / con.coeffs = arr, the same path .update() funnels through) vs raw in-place con.values[...]. linopy master, M1, median of 7 warm updates, chunk 0 excluded, 15% of rows mutated per chunk, 6 nnz/row.

n_rows	nnz	RHS typed	RHS raw	coef typed	coef raw
200 k	1.2 M	7.7 ms	0.03 ms	0.97 ms	0.14 ms
800 k	4.8 M	19.9 ms	0.09 ms	4.16 ms	0.55 ms
2.0 M	12 M	43.8 ms	0.23 ms	10.7 ms	1.43 ms

The 2 M row / 12 M nnz line matches our production LP (1.14 M cols × 1.97 M rows). Worst case — RHS + coeffs both updated — is ~52 ms typed vs ~1.7 ms raw, so ~51 ms/chunk saved by the raw path. Against a ~12 s solve that's ~0.4%; across a 52-chunk full-year run, ~2.7 s total.

Decomposed the typed cost to confirm it's the setter, not a benchmark artifact: assigning a prebuilt DataArray is 6.95 ms @ 200 k (the alignment inside the setter), the array .copy is only 0.15 ms. So the number is real setter work, and it's O(rows) — bounded and predictable, not a tail risk.

So I agree with @FBumann: the raw-.values escape hatch isn't worth it. ~0.4% of solve doesn't justify the freeze / editing() context manager, the internal-path audit, the dask-buffer inconsistency, and the lifecycle thaw/re-freeze plumbing. Withdrawing my freeze-CM suggestion — typed .update() / setters as the single mutation path is the right scope for this PR. Stability over the last fraction of a percent.

Bench script: benchmark/bench_linopy_mutation_path_typed_vs_raw_values.py (isolates mutation only, no solve), happy to upstream into linopy's benchmark/ if useful.

FabianHofmann · 2026-05-27T12:13:19Z

Benchmarked it. Isolated the mutation step (no solve) — typed constraint setter (con.rhs = arr / con.coeffs = arr, the same path .update() funnels through) vs raw in-place con.values[...]. linopy master, M1, median of 7 warm updates, chunk 0 excluded, 15% of rows mutated per chunk, 6 nnz/row.

n_rows nnz RHS typed RHS raw coef typed coef raw
200 k 1.2 M 7.7 ms 0.03 ms 0.97 ms 0.14 ms
800 k 4.8 M 19.9 ms 0.09 ms 4.16 ms 0.55 ms
2.0 M 12 M 43.8 ms 0.23 ms 10.7 ms 1.43 ms
The 2 M row / 12 M nnz line matches our production LP (1.14 M cols × 1.97 M rows). Worst case — RHS + coeffs both updated — is ~52 ms typed vs ~1.7 ms raw, so ~51 ms/chunk saved by the raw path. Against a ~12 s solve that's ~0.4%; across a 52-chunk full-year run, ~2.7 s total.

Decomposed the typed cost to confirm it's the setter, not a benchmark artifact: assigning a prebuilt DataArray is 6.95 ms @ 200 k (the alignment inside the setter), the array .copy is only 0.15 ms. So the number is real setter work, and it's O(rows) — bounded and predictable, not a tail risk.

So I agree with @FBumann: the raw-.values escape hatch isn't worth it. ~0.4% of solve doesn't justify the freeze / editing() context manager, the internal-path audit, the dask-buffer inconsistency, and the lifecycle thaw/re-freeze plumbing. Withdrawing my freeze-CM suggestion — typed .update() / setters as the single mutation path is the right scope for this PR. Stability over the last fraction of a percent.

Bench script: benchmark/bench_linopy_mutation_path_typed_vs_raw_values.py (isolates mutation only, no solve), happy to upstream into linopy's benchmark/ if useful.

great @MaykThewessen! very helpful. let's follow the easier route then and add non-supported value setting logics in the documentation

Restores symmetry with Constraint.vars (property) and data.vars (underlying xarray Dataset key) — the original rename traded one small asymmetry (read vs. mutate kwarg) for a worse one (Python property name vs. Dataset key name). The `vars()` builtin shadowing inside the kwarg position is benign: we never call `vars()` here, and dropping the rename also lets the top-level `linopy.variables` module be used directly inside the function body instead of importing `Variable as _Variable` to dodge the kwarg shadow. Closes #730. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Reverts cf845e1. Under the A/B framing for the .vars naming question, update(variables=...) is Category A — it accepts a linopy.Variable. The previous "restore symmetry with .vars / data.vars" argument conflated two layers: - Public Python API speaks about linopy variables → variables= - Internal xarray Dataset key stays as "vars" (xarray collision on Dataset.variables blocks renaming the key) The asymmetry between property/kwarg name and Dataset key name is principled (API layer vs. storage layer), not arbitrary — same pattern ORMs / serializers use. Keeping variables= here lines up with the broader .vars → .variables direction now being considered for properties. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

MaykThewessen · 2026-06-05T07:39:12Z

Took a first cut at A6 (incremental snapshot advance) as a follow-up branch off this PR's head, so it's ready to rebase + open once #718 lands.

Branch: MaykThewessen:feat/snapshot-incremental-advance (commit f64b265).

What

ModelSnapshot.advance(diff, model) replaces the full capture call in Solver._update_locked. It re-extracts only the containers in diff.changed_variables / diff.changed_constraints, patches obj_c at diff.obj_c_indices, updates obj_sense, and clears _coef_dirty on touched constraints. Unchanged containers' buffers are physically untouched (identity-preserved). First-time builds still go through capture.

Bench (M1, median of 7, isolated snapshot-refresh step)

n_rows	n_containers	recapture	advance	speedup
4 000	4	0.36 ms	0.07 ms	5.4x
100 000	10	3.43 ms	0.24 ms	14.3x
1 000 000	20	30.24 ms	0.98 ms	31x

Speedup scales as (n_containers - 1) / n_containers. Worst case (all updates in one container) is no regression: advance reduces to re-capturing that one container.

Scope notes

Not true O(|diff|). Still O(nnz in changed containers). True O(|diff|) needs an inverse map from global positions (used in ModelDiff) to local positions (used in ContainerConBuffers); fiddly enough that I'd defer it until a workload actually needs the last factor. Practical warm-update orchestrators already see >5x gain without it.
A2 overlap. advance owns the _coef_dirty clear for the containers it touched. If A2 lands (move dirty-clear out of capture), the matching clear in advance stays in the same place under the same condition (post-successful-apply).
Reject path. advance raises ValueError if called with a rebuild_required diff. Solver code already takes the _rebuild branch; the error is defense-in-depth.

Tests

test/test_persistent_snapshot_advance.py (8 tests) — equivalence (capture(post) == capture(pre).advance(diff, post) for RHS, bounds, objective-linear, combined), identity of unchanged buffers, empty-diff no-op, rebuild-required raises, _coef_dirty cleared. Full persistent suite green: 66/66.

Happy to open this as a real PR against main the moment #718 merges, or rebase onto a different base if that's easier. Or fold the change into #718 directly if @FabianHofmann prefers — small enough surface.

MaykThewessen · 2026-06-05T08:26:53Z

@FabianHofmann happy to fold the A6 commit (f64b265 on MaykThewessen:feat/snapshot-incremental-advance) directly into this PR — keeps the snapshot-refresh path consistent at landing time and avoids a stacked PR waiting on this one.

Two ways:

Cherry-pick f64b265 onto your feat/solver-update branch yourself — single commit, no conflicts against the current PR head.
Enable "Allow edits from maintainers" on the PR (or grant me push to your branch) and I'll push it.

Either works. Let me know which you prefer.

coroa

Ok, first set of comments. I still need to go over the solvers implementation.

Everything looks sane in general, i'd still try to make a couple of improvements:

persistent/diff.py can be cleaned up a bit. Nothing major, mostly moving stuff around. Maybe a preference to not use a COO constraint coeff diff.
persistent/snapshot.py and constraints.py: I'd like to make ModelSnapshots (or more explicitly their con and maybe var buffers) share the explicit underlying numpy objects with the Constraint objects in the case of CSRConstraints to save memory. coef_dirty is then not necessary anymore (for the CSR ones), because you can directly test whether it is still the same object. Arguments are memory and cpu efficiency.

coroa · 2026-06-10T09:57:37Z

+def _cat(parts: list[np.ndarray], dtype: type) -> np.ndarray:
+    if not parts:
+        return np.empty(0, dtype=dtype)
+    return np.concatenate(parts).astype(dtype, copy=False)
+
+
+def _cat_obj(parts: list[np.ndarray]) -> np.ndarray:
+    if not parts:
+        return _EMPTY_KIND
+    return np.concatenate(parts)
+
+
+def _cat_str(parts: list[np.ndarray]) -> np.ndarray:
+    if not parts:
+        return _EMPTY_U1
+    return np.concatenate(parts)


The _cat variants are all the same duplicated code.

ModelDiff.from_snapshot/from_models return RebuildReason on rebuild (NONE dropped); diff walkers moved onto _DiffBuilder with context in __init__, single _cat helper. Snapshot buffers share constraint arrays (identity fast path); CSRConstraint.sanitize_zeros copy-on-write. Use isinstance(val, ConstantLike) in Variable._validate_update.

FabianHofmann · 2026-06-11T11:44:19Z

@coroa thanks, addressed in 11b02e6:

from_snapshot / from_models now return ModelDiff | RebuildReason — RebuildReason.NONE and ModelDiff.rebuild_reason are gone, all ModelDiff fields are required (built only by _DiffBuilder.finalize()).
_diff_var/con_container and _diff_objective are now _DiffBuilder methods with push_* inlined; var_l2p/con_l2p/ignored live in __init__; the _cat variants are one helper; the zero-filled row_value_changed arrays are None now. Also dropped the always-True check_coords knob.
Memory sharing: _extract_con_buffers no longer copies. CSRConstraint snapshots share the stored arrays and the diff short-circuits on object identity (a is b) before comparing — O(1) for untouched frozen containers, and it replaces the unconditional _coef_dirty skip for CSR with a verified one. To make identity sound, CSRConstraint.sanitize_zeros() is now copy-on-write (it was the only in-place mutation path; everything else already rebinds).
isinstance(val, ConstantLike) — done.

Two pushbacks:

COO coef diff (your diff.py:651 comment): kept. All four backends consume per-entry triplets (changeCoeff / chgCoeff / chgmcoef / putaijlist), so storing CSR + mask would just move the same expansion into four apply_update bodies. The COO arrays only cover changed rows, so memory is bounded by changed-row nnz.
Var buffers stay masked copies (snapshot.py:64): they're small (2 floats/var) and they're the array-based safety net from the earlier .values[...] discussion — several tests mutate lower.values[...] raw and rely on the copied baseline to detect it. Sharing would compare an array against itself and silently miss exactly those edits. For constraints the trade-off differs: mutable extraction copies inherently, and CSRConstraint is frozen by contract — hence sharing is safe there.

Solvers part of your review still pending from your side — no rush.

FabianHofmann · 2026-06-11T11:49:12Z

@coroa thanks, I addressed most of the points. The refactoring was really needed (already pointed out by @FBumann). COO still looks like the right choice.

Note

AI written

from_snapshot / from_models now return ModelDiff | RebuildReason — RebuildReason.NONE and ModelDiff.rebuild_reason are gone, all ModelDiff fields are required (built only by _DiffBuilder.finalize()).
_diff_var/con_container and _diff_objective are now _DiffBuilder methods with push_* inlined; var_l2p/con_l2p/ignored live in __init__; the _cat variants are one helper; the zero-filled row_value_changed arrays are None now. Also dropped the always-True check_coords knob.
Memory sharing: _extract_con_buffers no longer copies. CSRConstraint snapshots share the stored arrays and the diff short-circuits on object identity (a is b) before comparing — O(1) for untouched frozen containers, and it replaces the unconditional _coef_dirty skip for CSR with a verified one. To make identity sound, CSRConstraint.sanitize_zeros() is now copy-on-write (it was the only in-place mutation path; everything else already rebinds).
isinstance(val, ConstantLike) — done.

Two pushbacks:

COO coef diff (your diff.py:651 comment): kept. All four backends consume per-entry triplets (changeCoeff / chgCoeff / chgmcoef / putaijlist), so storing CSR + mask would just move the same expansion into four apply_update bodies. The COO arrays only cover changed rows, so memory is bounded by changed-row nnz.
Var buffers stay masked copies (snapshot.py:64): they're small (2 floats/var) and they're the array-based safety net from the earlier .values[...] discussion — several tests mutate lower.values[...] raw and rely on the copied baseline to detect it. Sharing would compare an array against itself and silently miss exactly those edits. For constraints the trade-off differs: mutable extraction copies inherently, and CSRConstraint is frozen by contract — hence sharing is safe there.

# Conflicts: # doc/release_notes.rst

coroa · 2026-06-11T14:52:14Z

COO coef diff (your diff.py:651 comment): kept. All four backends consume per-entry triplets (changeCoeff / chgCoeff / chgmcoef / putaijlist), so storing CSR + mask would just move the same expansion into four apply_update bodies. The COO arrays only cover changed rows, so memory is bounded by changed-row nnz.

could move into a property on the model diff

maybe instead of changing the coefficient. rewriting the constraint would be faster. unsure whether this would lead to having to change the constraint mapping.

Variable.fix()/unfix() set both bounds atomically via update() instead of sequential deprecated setters (tripped new lower<=upper cross-validation). Fail fast on quadratic lhs in Constraint.update; type-narrowing fixes for mypy.

ModelDiff stores per-container _CoefDelta (changed rows referencing the CSR buffers); con_coef_rows/cols/vals materialize on first access via cached property. Expansion is now vectorized; backends guard on n_coef_updates. Follows coroa's suggestion on PR 718.

…(A2+A6) _DiffBuilder records target buffers/coords; ModelDiff.snapshot replaces the O(nnz) re-capture after in-place updates. ModelSnapshot.capture no longer mutates the model: the _coef_dirty clear moves to the solver, coupled to snapshot adoption (build + successful apply, never on apply=False).

Base Solver orchestrates the diff sections and validates up front (sign support; Mosek semi-continuous now fails before any native mutation); backends implement _apply_* hooks. Binary [0,1] re-clamp lifted to base with Gurobi no-op (VType 'B' implies bounds natively). self.sense now set uniformly; HiGHS vtype map cached; Xpress/Mosek list-conversion helpers.

…y (A5) update(model, apply=False) computes the diff without the solver lock (immutable snapshot buffers, same_model=False since _coef_dirty cannot be trusted concurrently). solve keeps the coarse lock: apply->run must be atomic and native handles are not thread-safe. Tests pin the non-blocking preview and the preview/apply asymmetry for raw .values mutations.

coroa

Ok, i also went over solvers. Not many comments.

I am only thinking about whether we can easily test whether removing and re-creating changed constraints might be faster for individual solvers? At least after some threshold of what was changed.

coroa · 2026-06-12T08:50:07Z

+    def _diff_unlocked(
+        self, model: Model, ignore_dims: Iterable[str]
+    ) -> ModelDiff | RebuildReason:
+        """
+        Compute a diff without the solver lock.
+
+        Snapshot and baseline refs are read once; snapshot buffers are
+        immutable after capture, so the walk is consistent even while a
+        concurrent apply swaps ``self.snapshot``. The fallback baseline
+        (``from_models``) is only consistent if no thread concurrently
+        mutates either Model.
+        """
+        snapshot = self.snapshot
+        if snapshot is not None:
+            return ModelDiff.from_snapshot(
+                snapshot, model, same_model=False, ignore_dims=ignore_dims
+            )
+        baseline = self.model
+        assert baseline is not None
+        return ModelDiff.from_models(baseline, model, ignore_dims=ignore_dims)


I would remove _unlocked from the name and use it from _update_locked

coroa · 2026-06-12T08:58:19Z

    model: Model | None = None
    io_api: str | None = None
    options: dict[str, Any] = field(default_factory=dict)
+    track_updates: bool = False


What's the memory impact of of the tracking? Is it small enough that a default of True would make sense?

coroa · 2026-06-12T10:07:34Z

+        if diff.con_rhs_indices.size:
+            self._apply_con_rhs(ctx, diff)
+        if diff.con_sign_indices.size:
+            self._apply_con_signs(ctx, diff.con_sign_indices, diff.con_sign_values)
+        if diff.n_coef_updates:
+            self._apply_con_coefs(
+                ctx, diff.con_coef_rows, diff.con_coef_cols, diff.con_coef_vals
+            )


i know from discussions with @brynpickering that he did experiments (with Gurobi, i think) where deleting and re-creating constraint rows was often faster than updating coefs. Can we test this?

Worth measuring, and the code is set up well to test it. The coef path is a scalar loop today: HiGHS changeCoeff and Gurobi chgCoeff are called per element in _apply_con_coefs, so this is exactly the case where the delete-and-re-add-row approach would batch better. Xpress (chgmcoef) and Mosek (putaijlist) already submit the whole COO list in one call, so there's less to win there.

Concrete shape: the _coef_dirty mask already identifies the changed rows and the COO arrays are grouped by row, so we know the exact row set to swap. Add a threshold: if the share of dirty coefs in a row (or the dirty-row count) exceeds a cutoff, delConstrs/deleteRows those rows and re-add them from the new CSR instead of per-element coefficient changes; below the cutoff keep the current path.

One caveat to fold into the measurement: deleting rows invalidates the saved basis for them. For Gurobi that's mostly free, but for HiGHS row deletion is one of the things that forces presolve, so it interacts with the basis-reuse regression in follow-up #1. We'd want to compare net solve time, not just the modify-call cost.

I'd suggest wiring both paths plus a threshold sweep into the same rolling-horizon PyPSA-Eur benchmark that produced the follow-up findings, and tracking it as a follow-up rather than blocking this PR: the scalar loop is correct, just potentially suboptimal past some change fraction.

_{🤖 AI-edited (Claude) reply, reviewed by @MaykThewessen.}

coroa · 2026-06-12T10:13:37Z

+        ignore_dims: Iterable[str] = (),
+        disallow_rebuild: bool = False,
+    ) -> ModelDiff | RebuildReason:
+        if apply and not type(self).supports_persistent_update:


classvars are also resolved on the instance.

Suggested change

if apply and not type(self).supports_persistent_update:

if apply and not self.supports_persistent_update:

coroa · 2026-06-12T10:16:00Z

+    def _update_locked(
+        self,
+        model: Model,
+        apply: bool,
+        ignore_dims: Iterable[str] = (),
+        disallow_rebuild: bool = False,
+    ) -> ModelDiff | RebuildReason:


Why does _update_locked even have to be called when apply=False. The code flow in this function looks a bit convoluted.

Maybe the we need to pull out the diff computation.

coroa · 2026-06-12T10:18:55Z

+    @classmethod
+    @functools.cache
+    def _vtype_map(cls) -> dict[VarKind, Any]:
+        return {
+            VarKind.CONTINUOUS: highspy.HighsVarType.kContinuous,
+            VarKind.BINARY: highspy.HighsVarType.kInteger,
+            VarKind.INTEGER: highspy.HighsVarType.kInteger,
+            VarKind.SEMI_CONTINUOUS: highspy.HighsVarType.kSemiContinuous,
+        }


Shouldn't this be a ClassVar constant?

coroa · 2026-06-12T10:20:06Z

+        if diff.var_type_positions.size:
+            self._apply_var_types(ctx, diff.var_type_positions, diff.var_type_kinds)
+            self._reclamp_binary_bounds(
+                ctx, diff.var_type_positions, diff.var_type_kinds
+            )


Is it possible that some solvers need to know about the vartypes before they accept bounds?

coroa · 2026-06-12T10:22:31Z

    def _build_solver_model(
        model: Model,
        explicit_coordinate_names: bool = False,
        set_names: bool = True,
    ) -> highspy.Highs:
        """Build a highspy.Highs instance that mirrors the linopy `model`."""
        if model.variables.sos:
            raise NotImplementedError(
                "SOS constraints are not supported by the HiGHS direct API. "
                "Use io_api='lp' instead."
            )

        M = model.matrices
        h = highspy.Highs()
        h.addVars(len(M.vlabels), M.lb, M.ub)
        if (
            len(model.binaries)
            + len(model.integers)
            + len(list(model.variables.semi_continuous))
        ):
            vtypes = M.vtypes
            integrality_map = {"C": 0, "B": 1, "I": 1, "S": 2}
            int_mask = (vtypes == "B") | (vtypes == "I") | (vtypes == "S")
            labels = np.arange(len(vtypes))[int_mask]
            integrality = np.array(
                [integrality_map[v] for v in vtypes[int_mask]], dtype=np.int32
            )
            h.changeColsIntegrality(len(labels), labels, integrality)

        c = M.c
        h.changeColsCost(len(c), np.arange(len(c), dtype=np.int32), c)


Should we try to re-use the _apply functions in the build_model or is this over kill?

coroa · 2026-06-12T10:23:11Z

+            raise UnsupportedUpdate("gurobi var count mismatch")
+        if len(gurobi_cons) != con_label_index.n_active_cons:
+            raise UnsupportedUpdate("gurobi con count mismatch")
+        return (gm, gurobi_vars, gurobi_cons)


namedtuple as context?

FBumann · 2026-06-12T15:10:20Z

I would say we should get this working and release, and try to work on performance tweaks from there, with the soon to be added benchmark tools

FabianHofmann and others added 19 commits May 19, 2026 15:19

test(persistent): smoke test Solver orchestrator with fake backend

61b3647

feat(solvers): short-circuit rebuild when backend lacks persistent-up…

ff1ae15

…date support Skip diff computation entirely when supports_persistent_update is False on apply, per plan: 'dispatcher checks flag before calling — if False, skips diffing entirely and goes to rebuild.'

feat(solvers): Gurobi apply_update for persistent solves

f67cec6

feat(solvers): HiGHS apply_update for persistent solves

8026293

test(persistent): cross-model, pickle, threading, failure-path coverage

b6601e3

[pre-commit.ci] auto fixes from pre-commit.com hooks

5c0a369

for more information, see https://pre-commit.ci

FabianHofmann requested review from FBumann and coroa May 21, 2026 15:21

Merge branch 'master' into feat/solver-update

b38f4f3

harden coords comparison

089cf2e

FBumann and others added 2 commits May 27, 2026 14:41

FabianHofmann mentioned this pull request Jun 1, 2026

perf(matrices): cache MatrixAccessor properties with cached_property #716

Merged

FBumann mentioned this pull request Jun 4, 2026

docs: cut release notes for v0.8.0 #759

Merged

fix: ensure dims in solution assignment

fbc5b7c

FBumann mentioned this pull request Jun 8, 2026

Fix variables via bound collapse instead of equality constraint #773

Merged

Merge branch 'master' into feat/solver-update

a4cf870

coroa reviewed Jun 10, 2026

View reviewed changes

Merge remote-tracking branch 'origin/master' into feat/solver-update

fb8d4a1

# Conflicts: # doc/release_notes.rst

FabianHofmann added 5 commits June 11, 2026 17:28

fix: CI failures after master merge

3a02029

Variable.fix()/unfix() set both bounds atomically via update() instead of sequential deprecated setters (tripped new lower<=upper cross-validation). Fail fast on quadratic lhs in Constraint.update; type-narrowing fixes for mypy.

coroa reviewed Jun 12, 2026

View reviewed changes

	if apply and not type(self).supports_persistent_update:
	if apply and not self.supports_persistent_update:

Conversation

FabianHofmann commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes proposed in this Pull Request

Follow-ups (tracked)

Checklist

Uh oh!

FabianHofmann commented May 21, 2026

Uh oh!

FBumann commented May 22, 2026

Uh oh!

FBumann commented May 22, 2026

Uh oh!

FabianHofmann commented May 22, 2026

Uh oh!

FBumann commented May 22, 2026

Uh oh!

MaykThewessen commented May 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

FBumann commented May 27, 2026

Uh oh!

FBumann commented May 27, 2026 • edited by FabianHofmann Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MaykThewessen commented May 27, 2026

Uh oh!

FabianHofmann commented May 27, 2026

Uh oh!

MaykThewessen commented Jun 5, 2026

What

Bench (M1, median of 7, isolated snapshot-refresh step)

Scope notes

Tests

Uh oh!

MaykThewessen commented Jun 5, 2026

Uh oh!

coroa left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

FabianHofmann commented Jun 11, 2026

Uh oh!

FabianHofmann commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coroa commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coroa left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FabianHofmann commented May 21, 2026 •

edited

Loading

MaykThewessen commented May 23, 2026 •

edited

Loading

FBumann commented May 27, 2026 •

edited by FabianHofmann

Loading

FabianHofmann commented Jun 11, 2026 •

edited

Loading

coroa commented Jun 11, 2026 •

edited

Loading