← redevops.io
Draft · implementation · redevops.io

Portable AI Runtimes for Apache Spark

A Kotlin implementation of the three runtimes, the Execution Planner and ReDevOps RAG — with a smaller Databricks-native subset
v9 is an implementation and portability paper, not a new conceptual layer. It carries the exact canonical contracts of v1–v8 onto the JVM, adds Apache Spark as a distributed physical execution backend, and defines a deliberately smaller Databricks adapter that composes with — never duplicates — the managed platform. For architecture ratification and frontier-model implementation.

Core thesis. ReDevOps owns the logical intelligence plan and the governed mission lifecycle. Apache Spark owns distributed physical execution. Databricks is supported through a smaller adapter layer that composes with its managed AI, governance, search, observability and workflow services rather than duplicating them. One canonical logical model spans local JVM, open Apache Spark and Databricks; only the physical adapters and deployment envelope change.

Goal / state change
   │  Discovery Runtime   — does this deserve attention?
   ▼
   │  Execution Planner   — which representations, capabilities and topology?
   ▼
   │  Mission Runtime     — how does intent become governed execution?
   ▼
   │  Context Runtime + ReDevOps RAG — what should each node know?
   ▼
 Local JVM  │  Apache Spark  │  Databricks  │  external systems  │  humans

The three implementations

ReDevOps keeps one specification and three implementations. Python (contextos) is the executable specification — the difficult part of engineering is defining behaviour, contracts and tests, and Python expresses those with the least syntactic overhead for humans and models alike (see Python is the specification). The Go port is the systems target — a low-overhead, single-static-binary runtime that also exposes the platform as a standing gRPC/HTTP service with a canonical contract IDL (one .proto for typed .NET/C# and any-language clients), remote .NET-hosted capability operators, Entra ID (Azure AD) authentication, and OpenTelemetry export (→ Azure Monitor). The service and gRPC dependencies are isolated in a separate module, so the Go runtime core stays zero-dependency — the natural fit for Microsoft/Azure and Windows deployments. This Kotlin port is the JVM / Spark target: Kotlin is a natural fit for Spark's Java libraries and the wider JVM data ecosystem, where Go is not. All three share the same canonical contracts and are held together by cross-language parity fixtures; they differ only in physical execution and deployment surface, never in the meaning of a plan, an evidence artifact, a policy or an outcome.

Abstract

Enterprise AI increasingly combines retrieval, structured data, models, tools, streaming signals, human approvals and side effects. Most platforms expose these as individual services, leaving applications to hard-code when each is used, how evidence is assembled, whether a change deserves action, and how a goal becomes a safe, replayable execution. This paper specifies a Kotlin/JVM implementation of five ReDevOps components — Context Runtime, Mission Runtime, Discovery Runtime, the Execution Planner and ReDevOps RAG — as a portable library that runs in ordinary JVM applications and can use non-Databricks Apache Spark as an optional distributed backend.

A separate Databricks-only subset does not recreate Databricks AI Search, Unity AI Gateway, MLflow, Lakeflow, Unity Catalog, Lakebase, Genie or managed agents. It supplies the missing intent-to-execution layer: discovery of meaningful change, representation and capability planning, compilation of governed missions, cross-service evidence assembly and portable replay semantics.

Executive summary

Canonical contracts

The same lifecycle applies in a local JVM process, an open Spark cluster and Databricks: WorldState/Goal → StateDelta → DiscoveryProposal → ExecutionPlan → CompiledMissionGraph → MissionRun + MissionEvents → EvidenceArtifacts → VerificationResult → Outcome → LearningObservation → updated WorldState. These are the platform's canonical public contracts, unchanged from v8 — v9 ports them to Kotlin, it does not redefine them.

ContractPurpose
ProjectManifestGoals, environments, capabilities, policies, budgets, data authorities, deployment profiles, retention.
WorldState / StateDeltaDurable beliefs about the environment and meaningful changes since the prior observation.
DiscoveryProposalWhy a change may deserve action — confidence, novelty, urgency, expected value, missing evidence, proposed goal.
ExecutionPlanThe complete logical intelligence plan: representations, retrieval, models, topology, capabilities, constraints, verification, governance, cost, fallbacks.
CompiledMissionGraphExecutable nodes and edges with contracts, dependencies, budgets, permissions, evidence requirements, recovery policies.
MissionEventAppend-only record of planning, execution, approvals, retries, artifacts, verification, state transitions.
EvidenceArtifactTyped, attributable evidence with source, lineage, time, authority, confidence, sensitivity, permitted purpose.
VerificationResult / OutcomeWhether the goal — not merely the task — was satisfied, with explicit defects and residual uncertainty.
LearningObservationStructured feedback that updates cost/quality/routing/retrieval/discovery models without silently changing policy.

Kotlin module architecture

Every core module compiles and runs without Spark or Databricks; Spark and Databricks artifacts are optional adapters. Public APIs stay Java-friendly (no Kotlin-only coroutine or inline/value types across compatibility boundaries without Java facades).

io.redevops.runtime
├── runtime-api                 canonical contracts + public SPI
├── execution-planner           candidate generation, feasibility, scoring
├── context-runtime             context and representation planning
├── mission-runtime             compilation, execution, replay, approvals
├── discovery-runtime           detectors, state deltas, significance scoring
├── redevops-rag                retrieval planning, fusion, provenance
├── capability-sdk              model · SQL · graph · API · tool · human adapters
├── runtime-local               coroutine-based local backend
├── runtime-spark               open Apache Spark backend
├── runtime-observability       OTel events, metrics, trace schema
└── runtime-testkit             fixtures, simulators, replay + parity tests

io.redevops.databricks
├── databricks-capabilities · databricks-governance · databricks-workflows
├── databricks-state · databricks-observability · databricks-bundle

The Execution Planner

The Execution Planner is the architectural center. It receives a goal, state, policy and available capabilities; generates feasible candidate plans; estimates cost, latency, quality and risk; selects a logical topology; and emits an explainable ExecutionPlan before a model or distributed job is invoked. Its planning dimensions are representation, capability, execution backend, topology, governance, verification and objective.

Logical vs. physical optimization. The Execution Planner decides whether to use SQL, retrieval, graph, model, agent, stream, workflow or human — the logical topology, and the evidence and verification plan. Spark Catalyst/AQE own relational and distributed physical execution; retrieval backends own index mechanics; model gateways own endpoint routing; workflow engines own scheduling. Databricks' own research on LLM-guided join-order exploration illustrates the boundary: an agent can explore candidate plans offline, validate and measure them, and retain improvements — but the model must not sit in the optimizer's millisecond hot path. ReDevOps applies the same rule to intelligence plans: model-assisted exploration and learning outside the deterministic path; validated plans inside it.

Mission Runtime and the Spark boundary

Mission Runtime compiles an ExecutionPlan into a CompiledMissionGraph, supervises its lifecycle, records events and evidence, invokes approvals, verifies outcomes and supports replay and recovery. It does not replace Spark scheduling or workflow infrastructure; it coordinates those systems as capabilities.

Mandatory rule. Spark transformations may generate evidence and action proposals. Governed side effects — deployments, writes to external transactional systems, messages, orders, approvals, irreversible operations — must execute through Mission Runtime outside retryable Spark transformations unless the sink provides explicit transactional and idempotent semantics. Agents and mission authority live on the driver / control plane; executors run deterministic partition work.

A semantic operator algebra

The lifecycle above is expressed as a small, fixed set of governed operators over the canonical contracts — the platform's syscall table. Every semantic operation is one of them: plan · discover · retrieve · act · merge · verify · promote · rollback · learn · explain. Each consumes and produces canonical contracts (the ABI), is metered, authorized and audited identically (the kernel), and emits provenance so explain() can reconstruct it (deterministic replay). This is what lets the runtime be described as an operating system for semantic execution rather than an orchestration engine: the operators are the primitives, the Execution Planner decides which to issue, and Mission Runtime schedules them under governance.

OperatorRoleHome
planGoal + state → logical ExecutionPlan over the decision axesExecution Planner
discoverStreams / changing state → significant DiscoveryProposalsDiscovery Runtime
retrieveQuery → evidence bundle with provenanceContext Runtime · ReDevOps RAG
actExecute a governed capability — the platform's side-effecting exec()Mission Runtime · capability-sdk
mergeIntegrate sibling artifacts into one verified artifact + evidence + conflictsMission Runtime
verifyCheck an artifact against a claim → VerificationResultMission Runtime
promoteAdvance a verified artifact to a higher scope / environmentMission Runtime
rollbackRestore a checkpoint — the inverse of promote (the saga path)Mission Runtime
learnFold a LearningObservation back into cost / quality / routing modelsExecution Planner
explainReconstruct the provenance of any subject for auditruntime-observability

The set is deliberately small and mostly names operations the runtime already performs; merge is the one first-class addition. Its laws are what make the algebra more than a list: promote requires a passing verify; rollback ∘ promote returns state to its prior revision; every operator emits provenance; and the Execution Planner — not the caller — selects which operators run and with what strategy.

merge() as a first-class operator

The original design assumed a frontier manager could integrate worker outputs directly. That holds until the integration exceeds the manager's effective context window — at which point direct merge is not merely lower-quality, it is infeasible. merge() promotes integration to a governed operator: it constructs a bounded working set (changed artifacts, dependent interfaces, architecture rules, affected tests, prior decisions, unresolved risks), retrieves evidence, detects conflicts, integrates, verifies and records provenance — producing a structured MergeResult with evidence, not merely source code.

Large merges compile into bounded hierarchical integration missions. Each level operates on a limited working set, and verified contracts become compression boundaries — a child mission inherits approved contracts instead of replaying historical reasoning:

worker outputs
   ▼  module merge         · bounded working set · conflict detection · verify
   ▼  subsystem merge      · only VERIFIED, conflict-free artifacts propagate up
   ▼  architecture merge   · inherits approved contracts, not full history
   ▼  release verification · reviewer sees verified subsystems, not the whole repo

The planner selects the merge topology — direct, contract-first, hierarchical, evidence-first, sequential-verified or human-gated — on coupling, context size, worker capability, verification coverage, failure cost and budget. Because integration is now an operator with deterministic replay, verification and provenance, decentralized merges are safe: independent subsystem managers merge locally and propagate only verified artifacts upward — the same guarantee the Spark driver relies on when it fans partition work across executors.

Why this does not contradict our own benchmark. Our orchestration benchmark found that blanket formal orchestration merely ties a strong supervisor at small scale — so merge() is gated behind the planner and runs only when selected, never as a cost on every task. On the single-node profile the operator suite is therefore opt-in: with none configured the runtime uses direct manager integration — the benchmark-validated baseline — and pays nothing. On the Spark profile it is required by design. Spark exists for datasets too large to fit one machine's memory; by the same logic the integration context routinely exceeds a single model's window, so a bounded, evidence-first merge() is the design center of distributed execution, not an edge case. In short: Spark : RAM :: formal merge : the manager's context window.

Discovery Runtime on Spark

Discovery Runtime turns streams and changing world state into selective attention — novelty, significance, urgency, expected value, policy relevance and missing evidence — then emits a DiscoveryProposal rather than launching an uncontrolled agent. Spark Structured Streaming TransformWithState is the preferred distributed backend (Java-accessible stateful processors, timers, TTL, initial state, schema evolution); Kotlin implements the Java interfaces behind Kotlin-native builders.

event streams / CDC / telemetry / documents
   ▼  deterministic parsing, filters, feature extraction
   ▼  keyed TransformWithState processor
        · prior world state   · novelty & drift metrics
        · temporal windows/timers   · suppression / cooldown / TTL
   ▼  DiscoveryProposal dataset  →  Execution Planner  →  optional governed mission

Quality controls: deterministic reduction before model reasoning; explicit suppression/cooldown/duplicate policies; confidence calibration and false-positive cost; evidence sufficiency before mission initiation; human approval for high-impact or low-confidence proposals.

Open Apache Spark profile

The open Spark profile is the complete distributed implementation — deployable on Kubernetes, YARN, standalone, managed Spark or on-prem, with no Databricks services.

Driver / control planeExecutors / data plane
Goal interpretation and plan selectionPartition-local retrieval, filtering, scoring
Mission graph compilation and durable stateFeature extraction, candidate generation, deterministic transforms
Policy feasibility and approval orchestrationLocal top-k, partial aggregation, graph partition ops
Model-mediated decisions and final synthesisBatch-safe remote calls only via explicit idempotent adapters
Global verification, action execution, learningEmit evidence and partial results — never autonomous mission state

Backend selection: small request → local JVM; large bounded corpus → Spark batch; continuous keyed state → Structured Streaming; transactional side effect → Mission capability outside Spark; large evidence synthesis → Spark reduction, then a supervised model call.

Databricks subset

Intentionally smaller, because Databricks already supplies managed search, model governance, observability, jobs, catalogs, state and agent experiences. ReDevOps adds planning and governed cross-service execution, and delegates AI Search (retrieval/index sync), Unity AI Gateway (provider access, quotas, telemetry), MLflow (tracing, evaluation, judges), Unity Catalog (governed assets, lineage), Lakeflow (managed execution), Lakebase/Delta (native state) and Genie/managed agents. The Kotlin artifact publishes a JDK/Scala/Spark-Connect compatibility matrix, compiles to the required JVM target, exposes Java entry points, keeps a classic-compute profile for capabilities needing broader Spark APIs, and ships Declarative Automation Bundle templates rather than asking adopters to reproduce version-sensitive builds.

Where ReDevOps differs from the platforms

AreaPlatform strengthReDevOps differentiation
AgentsGenie, managed/custom agents, tool orchestrationThe pre-agent decision: whether an agent is appropriate; portable mission semantics; outcome verification and replay.
RetrievalManaged AI Search, enterprise groundingBackend-neutral evidence planning, federation, authority/freshness priors, contradiction analysis, cross-store provenance.
ModelsManaged endpoints, gateways, controlsCapability-level planning across models and non-model alternatives before invocation.
GovernanceUnity Catalog permissions, lineagePolicy as a planning constraint, governed evidence objects, mission-level approvals, purpose-aware execution.
EvaluationMLflow traces, datasets, judgesGoal-level verification plan, evidence sufficiency, learning tied back to planner decisions.
DiscoveryPlatform monitoring, product-surface agentsGeneral-purpose, portable significance-to-mission runtime with explicit expected-value and policy gates.

Portable lessons carried into all deployments

Implementation roadmap

PhaseDeliverableExit criterion
0 · Ratify contractsCanonical schemas, glossary, execution IR, policy + evidence modelAn independent implementation reproduces the semantics from the spec.
1 · Portable coreruntime-api, planner, context, mission, discovery + RAG interfaces; local backendEnd-to-end local golden path passes deterministic tests.
2 · Retrieval + knowledge mapFederated retrieval, fusion, provenance, asset/capability mapMultiple backends produce comparable evidence bundles and explain plans.
3 · Spark batchDataset/DataFrame adapters, partition operators, distributed top-k, metricsPartition-invariant parity and bounded-shuffle benchmarks pass.
4 · Spark streaming discoveryTransformWithState detectors, timers, TTL, checkpoints, replayStateful restart and failure tests pass at target scale.
5 · Mission-to-Spark compilationSpark job nodes, evidence collection, verification, safe external-action boundaryA governed mission spans local and Spark nodes without duplicate effects.
6 · Databricks subsetAI Search, SQL, MLflow, Unity Catalog, Lakeflow + state adapters; bundlesDatabricks smoke suite passes on supported classic and/or serverless profiles.
7 · Planner learningOffline plan experiments, cost/quality estimates, shadow eval, promotion controlsPlanner improves the measured objective without policy or quality regression.
8 · Production hardeningCompatibility policy, security review, observability, docs, examplesA third party can deploy, operate, upgrade and audit the library.

Testing and acceptance

Contract tests (stable serialization, schema evolution, Java/Kotlin compatibility, backward-compat fixtures); semantic parity and partition invariance across local and Spark backends (subject to documented approximate-retrieval tolerances); retry/failure safety (no duplicate side effects); streaming-state tests (timers/TTL/checkpoint/restart/schema-evolution); architecture gates (no RAG special-wiring outside Capability interfaces; no autonomous executor agents; no Databricks dependency in core modules); Databricks compatibility (JDK/Scala/Spark-Connect matrix, JAR smoke, Unity Catalog permissions, MLflow export, bundle deploy); and full replay from manifest, plans, events, artifacts and capability versions.

Positioning

For open Apache Spark: a portable intent-to-execution runtime that turns goals and state changes into optimized, governed plans across Spark, SQL, retrieval, models, tools and humans. For Databricks: an intent-planning and mission-governance layer that composes Databricks services into explainable, replayable business execution — it does not replace AI Search, Unity AI Gateway, MLflow, Unity Catalog, Lakeflow, Lakebase, Genie or managed agents. The distinction is simple: the platforms provide powerful execution and governance primitives; ReDevOps decides what should run, why, with what evidence, under which constraints, in which topology, and how success will be verified. Framed by its operator algebra, this is less an orchestration engine than an operating system for semantic execution: a small set of governed primitives — plan · discover · retrieve · act · merge · verify · promote · rollback · learn · explain — over canonical contracts, with the Execution Planner choosing which to issue.

The Execution Planner module is built to be cleanly separable and permissively licensed. If it proves valuable to the Apache Spark community, we intend to propose donating it to the Apache Software Foundation — a portable, engine-neutral intelligence planner above Catalyst, not a competitor to it.

Conclusion

The Kotlin implementation should be neither a mechanical port nor a Spark-specific rewrite. It establishes one portable logical runtime and two execution profiles: the full open Apache Spark profile (distributed retrieval, evidence processing and streaming discovery, no Databricks dependency) and a smaller Databricks subset that maps the same contracts to native services and omits what the platform already supplies. As managed stacks add better agents, search, gateways and observability, the value of another isolated tool falls and the value of a portable decision-and-mission layer rises: enterprises still need to determine whether action is warranted, choose representations and capabilities, enforce policy before computation, assemble sufficient evidence, govern side effects, verify outcomes and learn across heterogeneous systems. That is the layer v9 makes portable to the JVM and to Spark.

Amendment (v10-era) — acceptance chains, object store, normalized events, decision gates

v9 owns the runtime-contract specifications the v10 harness depends on.

Deterministic acceptance chains

Every model-mediated capability declares an acceptance chain — candidate → schema validation → boundary validation → deterministic checks → reproducibility check → semantic verifier → policy evaluation → accepted artifact. Model self-assessment is never the gate; this is the universal form of the operator-algebra verify().

Shared ArtifactObjectStore

A typed object store holds immutable artifacts, live external handles, Mission-local and shared read-only objects, with ownership, visibility, leases, snapshots, content/compatibility hashes, audit, and GC. Handle classes: IMMUTABLE_ARTIFACT · LIVE_EXTERNAL_STATE · TEMPORARY_WORK_OBJECT · PRIVATE_USER_OBJECT · PUBLIC_REFERENCE. It also materializes, owns, and leases ContextViews. On Spark, references pass across executors; materialization runs on executors, while governed writes and promote() stay on the driver, exactly once.

Normalized events, versioned schemas, decision gates, case bundles

All subsystems emit a versioned RuntimeEvent (NDJSON + JSON-Schema: runtime-event/v1 · dereference-event/v1 · capability-event/v1 · verification-event/v1 · policy-decision/v1 · trajectory-finding/v1 · session-summary/v1) — one forensic timeline, normalize-first. A DecisionGate returns ALLOW · DENY · NO_OVERRIDE · REQUIRE_REVIEW · UNABLE_TO_DECIDE, denying only when parsing, evaluation, evidence/audit persistence, and correlation state all succeed cleanly — an internal failure is never rendered as an affirmative safety decision (fail-open vs fail-closed declared per surface). Portable Mission Case Bundles (ContextViews, events, capability calls, decisions, evidence, SHA-256 manifest; build · verify · timeline · replay · diff) support audit, incident review, and benchmark reproduction — a manifest proves internal consistency, not completeness or source authenticity.

Code-as-action, harness tests

Code-as-action runs only inside a declared SandboxProfile, and enforcement uses a stricter static subset than detection (typed tool calls, statically-parsed commands, known targets, bounded parameters; ambiguous cases route to review). The harness capability-test suite asserts typed-I/O, pass-by-reference, harness-API, loop-program, object-state, and verification invariants independent of any end application.

ReDevOps Whitepaper v9 (draft, for architecture ratification and frontier-model implementation). One specification (Python), three implementations (Python · Go · Kotlin) held at contract parity. Kotlin targets the JVM / Apache Spark profile. redevops.io is a member of the NVIDIA Inception program.