Reference architecture · AI platform engineering

One organization.
Many frontier models.

A provider-neutral logical reference architecture for assigning specialist model workers on demand - with Temporal-backed workflow durability, explicit policy controls, and evidence-based completion criteria.

Portable rolesDynamic assignmentDurable workflow stateSeparated workspaces

System containers and execution flow

This logical component and runtime-flow view separates organization policy, durable orchestration, model access, governed tools, and evidence. It is not a deployment diagram or security proof. Arrowheads show the initiating direction; responses return on the same integration path unless separately drawn.

Provider-neutral synthetic organization Logical component + runtime flow · v1.2 · reviewed 5 September 2026
Provider-neutral frontier model organization reference architecture Human objectives and automated events start a Temporal workflow. A LangGraph application graph registered on a Temporal Worker can dynamically assign specialist worker nodes. Model calls pass through LiteLLM and tool calls through an MCP host. Temporal owns workflow history; application state, artifacts, and telemetry use separate stores and pipelines. A custom completion gate can return an evidence package, await an approval signal, or route work back for revision. REFERENCE ARCHITECTURE SCOPE · LOGICAL VIEW EXTERNAL MODEL PROVIDERS EXTERNAL TOOL SYSTEMS EXTERNAL ACTORS Human sponsor API request Schedule Repository event EVENT SOURCES Complete Evidence package decision + artifacts PORTABLE DEFINITION Organization spec roles · authority · assignment CONTROL Secrets + policy identity · permissions · risk HUMAN CONTROL Approval authority risk review · escalation ROUTING POLICY Capability registry model strengths · limits · cost TEMPORAL WORKER · LANGGRAPH PLUGIN DURABLE CONTROL PLANE Temporal Service history · timers · queues · signals APPLICATION GRAPH LangGraph route · fan out · merge · gate official integration: Public Preview DYNAMIC GRAPH WORKERS Specialist nodes Architect Research Builder Critic COMPLETION GATE tests · policy · evidence · confidence MODEL GATEWAY LiteLLM route · fallback · budget OAI OpenAI A Anthropic G Google LOCAL Other / local TOOL GATEWAY MCP host consent · permissions · tool log GITWorktreesworkspace separation Data + SaaS Browser Shell Workflow historyowned by Temporal PostgreSQL Artifact store OTel pipelinebackend not shown 1OBJECTIVE 2TRIGGER 3TASK 4ASSIGN 5INVOKE 6MCP CALL 7EVALUATE FAIL · REVISE 8RETURN RESULT ROLE CONTRACTS APPROVAL SIGNAL ROUTING POLICY DEFINITION + GOVERNANCE PLANE DURABLE ORCHESTRATION PLANE MODEL PLANE TOOL PLANE STATE + EVIDENCE PLANE
Runtime request or transitionPolicy or control dependencyState, evidence, or telemetry exportExternal integration requestApproval signalLogical scopeExternal service boundary

Elastic deployment and capability lifecycle

This deployment and dynamic view applies the components from View 01 to a hybrid execution topology. The organization and durable control plane persist; specialist Activity Workers, regional gateways, inference endpoints, and tool capacity can scale independently where policy and cold-start requirements permit.

Provider-neutral synthetic organization Deployment + dynamic view · v1.0 · reviewed 5 September 2026
Elastic deployment of the provider-neutral frontier organization The durable organization control plane stays available while LangGraph schedules specialist Activities onto Temporal Task Queues. A custom elasticity and placement controller observes backlog, latency, cost, quota, residency, and cold-start policy. Eligible Activity Worker pools and their model or tool capacity can scale across on-premises, AWS, Azure, and Google Cloud execution cells. After completion, workers drain and return to zero or a configured warm minimum. ELASTIC FRONTIER ORGANIZATION · HYBRID DEPLOYMENT VIEW PERSISTENT ORGANIZATION CONTROL PLANE PORTABLE GOVERNANCE Organization + policy roles · authority · budgets DURABLE CONTROL Temporal Service history · queues · timers MINIMUM WARM POOL LangGraph control Worker route · assign · merge · gate PLACEMENT INPUT Capability registry skill · region · cost · quota RELEASE CONTROL Approval + completion risk · evidence · cooldown EXTERNAL OBJECTIVE ROLE + AUTHORITY APPROVAL POLICY DURABLE DEMAND + PLACEMENT PLANE SCHEDULE ACTIVITIES ARCHITECT TASKS architect-tasks Activity Task Queue RESEARCH TASKS research-tasks Activity Task Queue BUILDER TASKS builder-tasks Activity Task Queue CRITIC TASKS critic-tasks Activity Task Queue IMPLEMENTATION COMPONENT Elasticity + placement controller backlog · schedule-to-start latency · utilization · budget residency · provider quota · cold start · configured warm minimum QUEUE METRICS PLACEMENT POLICY WORKER POLL / TASK DISPATCH SCALE · PLACE · DRAIN ON-PREMISES EXECUTION CELL sensitive · data-local · privileged access ACTIVITY WORKERSSpecialist pooleligible role queues 0-NREPLICAS LiteLLMmodel access MCP hosttool access Local models · private systems · sandboxed tools AWS EXECUTION CELL EKS · ECS · EC2 ACTIVITY WORKERSSpecialist pooleligible role queues 0-NREPLICAS LiteLLMregional instance MCP hostregional instance Bedrock · SageMaker · GPU endpoints · tools AZURE EXECUTION CELL AKS · VM scale sets ACTIVITY WORKERSSpecialist pooleligible role queues 0-NREPLICAS LiteLLMregional instance MCP hostregional instance Azure OpenAI · Azure AI · GPU endpoints · tools GOOGLE CLOUD EXECUTION CELL GKE · managed instance groups ACTIVITY WORKERSSpecialist pooleligible role queues 0-NREPLICAS LiteLLMregional instance MCP hostregional instance Vertex AI · GPU endpoints · tools UTILIZATION · COST · LATENCY CAPABILITY LIFECYCLE 1 · IDLEcontrol warm · pools 0/min 2 · DEMANDrole queues accumulate work 3 · SCALE OUTeligible cells start 0-N capacity 4 · EXECUTEmodel + tool Activities return evidence 5 · DRAINcooldown · return to 0/min
Task scheduling or dispatchPolicy dependencyScaling metricPlacement or scale commandExecution cell

Persistent definitions. Ephemeral teams.

The organization definition is a versioned application artifact. Each objective can create a temporary set of specialist worker nodes operating under explicit policy; this behavior is implemented by the application graph rather than supplied automatically by any one platform.

01

Portable organization

A versioned YAML, TOML, or equivalent application schema can define roles, authority, model policy, assignment rules, budgets, and typed work contracts. This schema is an architectural design choice - not a LangGraph or Temporal standard.

02

Durable workflow

Temporal Service persists Workflow Event History and coordinates timers, Task Queues, retries, and Signals. External calls execute as Activities; write Activities must be designed for idempotent retry or explicit compensation.

03

Graph orchestration

A LangGraph application graph can implement planning, conditional routing, dynamic fan-out through the Send API, merging, and completion gates. Those behaviors come from application nodes and edges, not from an autonomous built-in organization.

04

Independent evaluation

Specialists are selected through application policy. Prefer deterministic checks first; where an LLM judges another model, an independently chosen model family can reduce - but cannot eliminate - self- and family-preference risk.

05

Model gateway

LiteLLM can provide a common provider interface plus gateway authentication, authorization, routing, fallback, rate limits, spend tracking, and budgets. A custom capability registry preserves model-specific constraints rather than pretending all models are equivalent.

06

Tools and isolation

An MCP host creates clients that communicate with focused servers and must enforce consent and security policy. Git worktrees separate working directories and branches; they are not security sandboxes. Use containers or equivalent process isolation for untrusted execution.

07

State and evidence

Temporal owns durable Workflow Event History. PostgreSQL and artifact storage are optional application choices for cross-run state and retained evidence; they do not replace Temporal durability or a LangGraph checkpointer within the official integration.

08

Evaluation and telemetry

Completion policy can combine deterministic tests, evidence requirements, model-based review, and human approval. OpenTelemetry instrumentation emits and propagates telemetry; an external observability backend stores, queries, and visualizes it.

09

Capability Task Queues

Temporal Task Queues are durable demand buffers polled by one or more Worker processes. Separating Architect, Research, Builder, and Critic Activity work into capability-oriented queues is an application design choice that enables independent routing, throttling, and fleet scaling.

10

Elastic Worker pools

Activity Worker replicas can scale independently from the Temporal Service and the small control-Worker pool. Replica count, task slots, and poller count are separate controls; scale from backlog and schedule-to-start latency while respecting downstream provider quotas.

11

Hybrid placement

A custom placement controller can combine capability, residency, security, latency, cost, quota, and cold-start policy to select eligible on-premises or cloud execution cells. Temporal and LangGraph do not provide this multi-cloud placement policy automatically.

12

Scale-in safety

Workers should stop accepting new work, finish or safely abandon in-flight Activities, and drain before infrastructure termination. Scale-to-zero is appropriate only when startup latency is acceptable; slow model loading and privileged workloads may require a configured warm minimum.

Validated against current sources

Platform capabilities, deployment boundaries, and cloud scaling mechanisms were checked against current first-party documentation on 5 September 2026. The organization schema, capability queues, placement controller, completion policy, agent-role taxonomy, and combined operating model are explicitly identified as architectural design choices. Research evidence is labeled separately and does not constitute a platform guarantee.

Platform fact

LangGraph

StateGraph provides explicit state and controlled graph transitions; its Send API supports dynamically created worker nodes for orchestrator-worker flows.

Worker pattern ↗
Preview constraint

Temporal + LangGraph

The official integration is Python and Public Preview. Each graph node or task must declare Workflow or Activity execution; LLM, network, database, and file I/O run as Activities.

Integration guide ↗
Platform fact

Temporal Service

Temporal persists application state and supplies Workflow Event History, Task Queues, timers, Signals, and retry coordination so executions can resume after infrastructure failure.

Platform docs ↗
Platform fact

Temporal Activities

Activities contain failure-prone or nondeterministic operations. Temporal recommends idempotency because an Activity can execute more than once before completion is recorded.

Activity semantics ↗
Platform fact

LiteLLM

The gateway supports a common multi-provider interface, authentication and authorization, routing, fallback, rate limits, spend tracking, and budgets. Its current provider catalog includes Bedrock, SageMaker, Azure OpenAI, Azure AI, and Vertex AI.

Provider catalog ↗
Current spec

MCP 2026-07-28

MCP retains a host-client-server architecture but now has a stateless protocol core: each request carries its version and client capabilities.

Architecture spec ↗
Security duty

MCP tools

Servers validate inputs and enforce access controls; clients confirm sensitive operations, validate results, apply timeouts, and log tool use. These are implementation duties, not automatic protocol guarantees.

Tool security ↗
Platform fact

Git worktrees

Worktrees provide multiple working trees attached to one shared repository. The documented mechanism separates Git working state; process and security isolation require another control.

Git manual ↗
Platform fact

OpenTelemetry

OpenTelemetry generates, collects, and exports traces, metrics, and logs. It is not an observability backend; storage and visualization belong to other tools.

OTel overview ↗
Research evidence

Evaluator independence

Current research reports self- and same-family preference in LLM judges. Independent judges and deterministic checks mitigate risk but do not prove correctness.

Bias study ↗
Diagram standard

Scope boundaries

C4 distinguishes the software system in scope from external systems and reserves deployment details for deployment diagrams. This page therefore labels its view and external dependencies explicitly.

C4 guidance ↗
Platform fact

Temporal Task Queues

Task Queues are lightweight and created on demand. Workers poll when they have capacity; Workflow and Activity Tasks persist when Workers are unavailable, and queues load-balance work across Worker processes.

Task Queue semantics ↗
Implementation guidance

Temporal Worker capacity

Temporal distinguishes Worker replicas, task slots, and pollers. Current guidance covers backlog metrics, resource-based slot suppliers, and poller autoscaling for managing Worker fleet throughput.

Worker performance ↗
Deployment boundary

Temporal Cloud

Temporal Cloud manages Temporal Server capabilities but does not run customer applications or Workers. Worker programs execute in customer-controlled containers, Kubernetes clusters, virtual machines, or other hosting environments.

Cloud security model ↗
Cloud mapping

AWS elastic execution

AWS documents event-driven scaling on EKS with custom metrics and KEDA, including GPU inference scaling from queue depth and latency. Node capacity can scale through EKS Auto Mode, Karpenter, or Cluster Autoscaler.

EKS AI autoscaling ↗
Cloud mapping

Azure elastic execution

AKS supports pod, node, and event-driven scaling through HPA, cluster autoscaling, node autoprovisioning, and KEDA. KEDA is the relevant mechanism when an external backlog should activate workloads or permit scale-to-zero.

AKS scaling overview ↗
Cloud mapping

Google Cloud elastic execution

GKE can scale workloads from custom, Prometheus, or external metrics. Google documents queue backlog as an example signal for increasing replicas while work accumulates and reducing them as demand falls.

GKE metric autoscaling ↗
Back to top