Core thesis. ReDevOps owns the logical intelligence plan and the governed mission lifecycle. Apache Spark owns distributed physical execution. Databricks is supported through a smaller adapter layer that composes with its managed AI, governance, search, observability and workflow services rather than duplicating them. One canonical logical model spans local JVM, open Apache Spark and Databricks; only the physical adapters and deployment envelope change.
Goal / state change │ Discovery Runtime — does this deserve attention? ▼ │ Execution Planner — which representations, capabilities and topology? ▼ │ Mission Runtime — how does intent become governed execution? ▼ │ Context Runtime + ReDevOps RAG — what should each node know? ▼ Local JVM │ Apache Spark │ Databricks │ external systems │ humans
ReDevOps keeps one specification and three implementations. Python (contextos) is the executable specification — the difficult part of engineering is defining behaviour, contracts and tests, and Python expresses those with the least syntactic overhead for humans and models alike (see Python is the specification). The Go port is the systems target — a low-overhead, single-static-binary runtime that also exposes the platform as a standing gRPC/HTTP service with a canonical contract IDL (one .proto for typed .NET/C# and any-language clients), remote .NET-hosted capability operators, Entra ID (Azure AD) authentication, and OpenTelemetry export (→ Azure Monitor). The service and gRPC dependencies are isolated in a separate module, so the Go runtime core stays zero-dependency — the natural fit for Microsoft/Azure and Windows deployments. This Kotlin port is the JVM / Spark target: Kotlin is a natural fit for Spark's Java libraries and the wider JVM data ecosystem, where Go is not. All three share the same canonical contracts and are held together by cross-language parity fixtures; they differ only in physical execution and deployment surface, never in the meaning of a plan, an evidence artifact, a policy or an outcome.
Enterprise AI increasingly combines retrieval, structured data, models, tools, streaming signals, human approvals and side effects. Most platforms expose these as individual services, leaving applications to hard-code when each is used, how evidence is assembled, whether a change deserves action, and how a goal becomes a safe, replayable execution. This paper specifies a Kotlin/JVM implementation of five ReDevOps components — Context Runtime, Mission Runtime, Discovery Runtime, the Execution Planner and ReDevOps RAG — as a portable library that runs in ordinary JVM applications and can use non-Databricks Apache Spark as an optional distributed backend.
A separate Databricks-only subset does not recreate Databricks AI Search, Unity AI Gateway, MLflow, Lakeflow, Unity Catalog, Lakebase, Genie or managed agents. It supplies the missing intent-to-execution layer: discovery of meaningful change, representation and capability planning, compilation of governed missions, cross-service evidence assembly and portable replay semantics.
TransformWithState: Java support, timers, TTL, composite state, schema evolution) for stateful detection, significance scoring and mission proposals.The same lifecycle applies in a local JVM process, an open Spark cluster and Databricks: WorldState/Goal → StateDelta → DiscoveryProposal → ExecutionPlan → CompiledMissionGraph → MissionRun + MissionEvents → EvidenceArtifacts → VerificationResult → Outcome → LearningObservation → updated WorldState. These are the platform's canonical public contracts, unchanged from v8 — v9 ports them to Kotlin, it does not redefine them.
| Contract | Purpose |
|---|---|
ProjectManifest | Goals, environments, capabilities, policies, budgets, data authorities, deployment profiles, retention. |
WorldState / StateDelta | Durable beliefs about the environment and meaningful changes since the prior observation. |
DiscoveryProposal | Why a change may deserve action — confidence, novelty, urgency, expected value, missing evidence, proposed goal. |
ExecutionPlan | The complete logical intelligence plan: representations, retrieval, models, topology, capabilities, constraints, verification, governance, cost, fallbacks. |
CompiledMissionGraph | Executable nodes and edges with contracts, dependencies, budgets, permissions, evidence requirements, recovery policies. |
MissionEvent | Append-only record of planning, execution, approvals, retries, artifacts, verification, state transitions. |
EvidenceArtifact | Typed, attributable evidence with source, lineage, time, authority, confidence, sensitivity, permitted purpose. |
VerificationResult / Outcome | Whether the goal — not merely the task — was satisfied, with explicit defects and residual uncertainty. |
LearningObservation | Structured feedback that updates cost/quality/routing/retrieval/discovery models without silently changing policy. |
Every core module compiles and runs without Spark or Databricks; Spark and Databricks artifacts are optional adapters. Public APIs stay Java-friendly (no Kotlin-only coroutine or inline/value types across compatibility boundaries without Java facades).
io.redevops.runtime ├── runtime-api canonical contracts + public SPI ├── execution-planner candidate generation, feasibility, scoring ├── context-runtime context and representation planning ├── mission-runtime compilation, execution, replay, approvals ├── discovery-runtime detectors, state deltas, significance scoring ├── redevops-rag retrieval planning, fusion, provenance ├── capability-sdk model · SQL · graph · API · tool · human adapters ├── runtime-local coroutine-based local backend ├── runtime-spark open Apache Spark backend ├── runtime-observability OTel events, metrics, trace schema └── runtime-testkit fixtures, simulators, replay + parity tests io.redevops.databricks ├── databricks-capabilities · databricks-governance · databricks-workflows ├── databricks-state · databricks-observability · databricks-bundle
The Execution Planner is the architectural center. It receives a goal, state, policy and available capabilities; generates feasible candidate plans; estimates cost, latency, quality and risk; selects a logical topology; and emits an explainable ExecutionPlan before a model or distributed job is invoked. Its planning dimensions are representation, capability, execution backend, topology, governance, verification and objective.
Logical vs. physical optimization. The Execution Planner decides whether to use SQL, retrieval, graph, model, agent, stream, workflow or human — the logical topology, and the evidence and verification plan. Spark Catalyst/AQE own relational and distributed physical execution; retrieval backends own index mechanics; model gateways own endpoint routing; workflow engines own scheduling. Databricks' own research on LLM-guided join-order exploration illustrates the boundary: an agent can explore candidate plans offline, validate and measure them, and retain improvements — but the model must not sit in the optimizer's millisecond hot path. ReDevOps applies the same rule to intelligence plans: model-assisted exploration and learning outside the deterministic path; validated plans inside it.
Mission Runtime compiles an ExecutionPlan into a CompiledMissionGraph, supervises its lifecycle, records events and evidence, invokes approvals, verifies outcomes and supports replay and recovery. It does not replace Spark scheduling or workflow infrastructure; it coordinates those systems as capabilities.
Mandatory rule. Spark transformations may generate evidence and action proposals. Governed side effects — deployments, writes to external transactional systems, messages, orders, approvals, irreversible operations — must execute through Mission Runtime outside retryable Spark transformations unless the sink provides explicit transactional and idempotent semantics. Agents and mission authority live on the driver / control plane; executors run deterministic partition work.
The lifecycle above is expressed as a small, fixed set of governed operators over the canonical contracts — the platform's syscall table. Every semantic operation is one of them: plan · discover · retrieve · act · merge · verify · promote · rollback · learn · explain. Each consumes and produces canonical contracts (the ABI), is metered, authorized and audited identically (the kernel), and emits provenance so explain() can reconstruct it (deterministic replay). This is what lets the runtime be described as an operating system for semantic execution rather than an orchestration engine: the operators are the primitives, the Execution Planner decides which to issue, and Mission Runtime schedules them under governance.
| Operator | Role | Home |
|---|---|---|
plan | Goal + state → logical ExecutionPlan over the decision axes | Execution Planner |
discover | Streams / changing state → significant DiscoveryProposals | Discovery Runtime |
retrieve | Query → evidence bundle with provenance | Context Runtime · ReDevOps RAG |
act | Execute a governed capability — the platform's side-effecting exec() | Mission Runtime · capability-sdk |
merge | Integrate sibling artifacts into one verified artifact + evidence + conflicts | Mission Runtime |
verify | Check an artifact against a claim → VerificationResult | Mission Runtime |
promote | Advance a verified artifact to a higher scope / environment | Mission Runtime |
rollback | Restore a checkpoint — the inverse of promote (the saga path) | Mission Runtime |
learn | Fold a LearningObservation back into cost / quality / routing models | Execution Planner |
explain | Reconstruct the provenance of any subject for audit | runtime-observability |
The set is deliberately small and mostly names operations the runtime already performs; merge is the one first-class addition. Its laws are what make the algebra more than a list: promote requires a passing verify; rollback ∘ promote returns state to its prior revision; every operator emits provenance; and the Execution Planner — not the caller — selects which operators run and with what strategy.
The original design assumed a frontier manager could integrate worker outputs directly. That holds until the integration exceeds the manager's effective context window — at which point direct merge is not merely lower-quality, it is infeasible. merge() promotes integration to a governed operator: it constructs a bounded working set (changed artifacts, dependent interfaces, architecture rules, affected tests, prior decisions, unresolved risks), retrieves evidence, detects conflicts, integrates, verifies and records provenance — producing a structured MergeResult with evidence, not merely source code.
Large merges compile into bounded hierarchical integration missions. Each level operates on a limited working set, and verified contracts become compression boundaries — a child mission inherits approved contracts instead of replaying historical reasoning:
worker outputs ▼ module merge · bounded working set · conflict detection · verify ▼ subsystem merge · only VERIFIED, conflict-free artifacts propagate up ▼ architecture merge · inherits approved contracts, not full history ▼ release verification · reviewer sees verified subsystems, not the whole repo
The planner selects the merge topology — direct, contract-first, hierarchical, evidence-first, sequential-verified or human-gated — on coupling, context size, worker capability, verification coverage, failure cost and budget. Because integration is now an operator with deterministic replay, verification and provenance, decentralized merges are safe: independent subsystem managers merge locally and propagate only verified artifacts upward — the same guarantee the Spark driver relies on when it fans partition work across executors.
Why this does not contradict our own benchmark. Our orchestration benchmark found that blanket formal orchestration merely ties a strong supervisor at small scale — so merge() is gated behind the planner and runs only when selected, never as a cost on every task. On the single-node profile the operator suite is therefore opt-in: with none configured the runtime uses direct manager integration — the benchmark-validated baseline — and pays nothing. On the Spark profile it is required by design. Spark exists for datasets too large to fit one machine's memory; by the same logic the integration context routinely exceeds a single model's window, so a bounded, evidence-first merge() is the design center of distributed execution, not an edge case. In short: Spark : RAM :: formal merge : the manager's context window.
Discovery Runtime turns streams and changing world state into selective attention — novelty, significance, urgency, expected value, policy relevance and missing evidence — then emits a DiscoveryProposal rather than launching an uncontrolled agent. Spark Structured Streaming TransformWithState is the preferred distributed backend (Java-accessible stateful processors, timers, TTL, initial state, schema evolution); Kotlin implements the Java interfaces behind Kotlin-native builders.
event streams / CDC / telemetry / documents
▼ deterministic parsing, filters, feature extraction
▼ keyed TransformWithState processor
· prior world state · novelty & drift metrics
· temporal windows/timers · suppression / cooldown / TTL
▼ DiscoveryProposal dataset → Execution Planner → optional governed missionQuality controls: deterministic reduction before model reasoning; explicit suppression/cooldown/duplicate policies; confidence calibration and false-positive cost; evidence sufficiency before mission initiation; human approval for high-impact or low-confidence proposals.
The open Spark profile is the complete distributed implementation — deployable on Kubernetes, YARN, standalone, managed Spark or on-prem, with no Databricks services.
| Driver / control plane | Executors / data plane |
|---|---|
| Goal interpretation and plan selection | Partition-local retrieval, filtering, scoring |
| Mission graph compilation and durable state | Feature extraction, candidate generation, deterministic transforms |
| Policy feasibility and approval orchestration | Local top-k, partial aggregation, graph partition ops |
| Model-mediated decisions and final synthesis | Batch-safe remote calls only via explicit idempotent adapters |
| Global verification, action execution, learning | Emit evidence and partial results — never autonomous mission state |
Backend selection: small request → local JVM; large bounded corpus → Spark batch; continuous keyed state → Structured Streaming; transactional side effect → Mission capability outside Spark; large evidence synthesis → Spark reduction, then a supervised model call.
Intentionally smaller, because Databricks already supplies managed search, model governance, observability, jobs, catalogs, state and agent experiences. ReDevOps adds planning and governed cross-service execution, and delegates AI Search (retrieval/index sync), Unity AI Gateway (provider access, quotas, telemetry), MLflow (tracing, evaluation, judges), Unity Catalog (governed assets, lineage), Lakeflow (managed execution), Lakebase/Delta (native state) and Genie/managed agents. The Kotlin artifact publishes a JDK/Scala/Spark-Connect compatibility matrix, compiles to the required JVM target, exposes Java entry points, keeps a classic-compute profile for capabilities needing broader Spark APIs, and ships Declarative Automation Bundle templates rather than asking adopters to reproduce version-sensitive builds.
| Area | Platform strength | ReDevOps differentiation |
|---|---|---|
| Agents | Genie, managed/custom agents, tool orchestration | The pre-agent decision: whether an agent is appropriate; portable mission semantics; outcome verification and replay. |
| Retrieval | Managed AI Search, enterprise grounding | Backend-neutral evidence planning, federation, authority/freshness priors, contradiction analysis, cross-store provenance. |
| Models | Managed endpoints, gateways, controls | Capability-level planning across models and non-model alternatives before invocation. |
| Governance | Unity Catalog permissions, lineage | Policy as a planning constraint, governed evidence objects, mission-level approvals, purpose-aware execution. |
| Evaluation | MLflow traces, datasets, judges | Goal-level verification plan, evidence sufficiency, learning tied back to planner decisions. |
| Discovery | Platform monitoring, product-surface agents | General-purpose, portable significance-to-mission runtime with explicit expected-value and policy gates. |
| Phase | Deliverable | Exit criterion |
|---|---|---|
| 0 · Ratify contracts | Canonical schemas, glossary, execution IR, policy + evidence model | An independent implementation reproduces the semantics from the spec. |
| 1 · Portable core | runtime-api, planner, context, mission, discovery + RAG interfaces; local backend | End-to-end local golden path passes deterministic tests. |
| 2 · Retrieval + knowledge map | Federated retrieval, fusion, provenance, asset/capability map | Multiple backends produce comparable evidence bundles and explain plans. |
| 3 · Spark batch | Dataset/DataFrame adapters, partition operators, distributed top-k, metrics | Partition-invariant parity and bounded-shuffle benchmarks pass. |
| 4 · Spark streaming discovery | TransformWithState detectors, timers, TTL, checkpoints, replay | Stateful restart and failure tests pass at target scale. |
| 5 · Mission-to-Spark compilation | Spark job nodes, evidence collection, verification, safe external-action boundary | A governed mission spans local and Spark nodes without duplicate effects. |
| 6 · Databricks subset | AI Search, SQL, MLflow, Unity Catalog, Lakeflow + state adapters; bundles | Databricks smoke suite passes on supported classic and/or serverless profiles. |
| 7 · Planner learning | Offline plan experiments, cost/quality estimates, shadow eval, promotion controls | Planner improves the measured objective without policy or quality regression. |
| 8 · Production hardening | Compatibility policy, security review, observability, docs, examples | A third party can deploy, operate, upgrade and audit the library. |
Contract tests (stable serialization, schema evolution, Java/Kotlin compatibility, backward-compat fixtures); semantic parity and partition invariance across local and Spark backends (subject to documented approximate-retrieval tolerances); retry/failure safety (no duplicate side effects); streaming-state tests (timers/TTL/checkpoint/restart/schema-evolution); architecture gates (no RAG special-wiring outside Capability interfaces; no autonomous executor agents; no Databricks dependency in core modules); Databricks compatibility (JDK/Scala/Spark-Connect matrix, JAR smoke, Unity Catalog permissions, MLflow export, bundle deploy); and full replay from manifest, plans, events, artifacts and capability versions.
For open Apache Spark: a portable intent-to-execution runtime that turns goals and state changes into optimized, governed plans across Spark, SQL, retrieval, models, tools and humans. For Databricks: an intent-planning and mission-governance layer that composes Databricks services into explainable, replayable business execution — it does not replace AI Search, Unity AI Gateway, MLflow, Unity Catalog, Lakeflow, Lakebase, Genie or managed agents. The distinction is simple: the platforms provide powerful execution and governance primitives; ReDevOps decides what should run, why, with what evidence, under which constraints, in which topology, and how success will be verified. Framed by its operator algebra, this is less an orchestration engine than an operating system for semantic execution: a small set of governed primitives — plan · discover · retrieve · act · merge · verify · promote · rollback · learn · explain — over canonical contracts, with the Execution Planner choosing which to issue.
The Execution Planner module is built to be cleanly separable and permissively licensed. If it proves valuable to the Apache Spark community, we intend to propose donating it to the Apache Software Foundation — a portable, engine-neutral intelligence planner above Catalyst, not a competitor to it.
The Kotlin implementation should be neither a mechanical port nor a Spark-specific rewrite. It establishes one portable logical runtime and two execution profiles: the full open Apache Spark profile (distributed retrieval, evidence processing and streaming discovery, no Databricks dependency) and a smaller Databricks subset that maps the same contracts to native services and omits what the platform already supplies. As managed stacks add better agents, search, gateways and observability, the value of another isolated tool falls and the value of a portable decision-and-mission layer rises: enterprises still need to determine whether action is warranted, choose representations and capabilities, enforce policy before computation, assemble sufficient evidence, govern side effects, verify outcomes and learn across heterogeneous systems. That is the layer v9 makes portable to the JVM and to Spark.
v9 owns the runtime-contract specifications the v10 harness depends on.
Every model-mediated capability declares an acceptance chain — candidate → schema validation → boundary validation → deterministic checks → reproducibility check → semantic verifier → policy evaluation → accepted artifact. Model self-assessment is never the gate; this is the universal form of the operator-algebra verify().
A typed object store holds immutable artifacts, live external handles, Mission-local and shared read-only objects, with ownership, visibility, leases, snapshots, content/compatibility hashes, audit, and GC. Handle classes: IMMUTABLE_ARTIFACT · LIVE_EXTERNAL_STATE · TEMPORARY_WORK_OBJECT · PRIVATE_USER_OBJECT · PUBLIC_REFERENCE. It also materializes, owns, and leases ContextViews. On Spark, references pass across executors; materialization runs on executors, while governed writes and promote() stay on the driver, exactly once.
All subsystems emit a versioned RuntimeEvent (NDJSON + JSON-Schema: runtime-event/v1 · dereference-event/v1 · capability-event/v1 · verification-event/v1 · policy-decision/v1 · trajectory-finding/v1 · session-summary/v1) — one forensic timeline, normalize-first. A DecisionGate returns ALLOW · DENY · NO_OVERRIDE · REQUIRE_REVIEW · UNABLE_TO_DECIDE, denying only when parsing, evaluation, evidence/audit persistence, and correlation state all succeed cleanly — an internal failure is never rendered as an affirmative safety decision (fail-open vs fail-closed declared per surface). Portable Mission Case Bundles (ContextViews, events, capability calls, decisions, evidence, SHA-256 manifest; build · verify · timeline · replay · diff) support audit, incident review, and benchmark reproduction — a manifest proves internal consistency, not completeness or source authenticity.
Code-as-action runs only inside a declared SandboxProfile, and enforcement uses a stricter static subset than detection (typed tool calls, statically-parsed commands, known targets, bounded parameters; ambiguous cases route to review). The harness capability-test suite asserts typed-I/O, pass-by-reference, harness-API, loop-program, object-state, and verification invariants independent of any end application.