Architecture

Rex is implemented as a small set of focused crates that form a pipeline:

1. Parsing (rex-parser): converts source text into a rex_ast::CompilationUnit { decls, body }.
2. Typing (rex-typesystem): Hindley–Milner inference + ADTs + type classes; produces a rex_typesystem::TypedExpr.
3. Execution (rex-engine): builds the host environment, prepares typed code into a CompiledProgram, and evaluates it to an owned rex_engine::Value.

The crates are designed so you can use them independently (e.g. parser-only tooling, typechecking-only checks, or embedding the full evaluator).

Crates

• rex-ast: shared AST types (Expr, Pattern, Decl, TypeExpr, CompilationUnit, symbols, spans).
• rex-parser: source parser. Entry point: rex_parser::parse.
  • Parsing enforces a fixed cap on AST nesting.
• rex-typesystem: type system. Entry points:
  • TypeSystem::new() to create an explicit typing environment.
  • infer_typed(&mut ts, expr) / infer(&mut ts, expr) for type inference.
  • The inference implementation itself lives in rex-typesystem/src/inference.rs; typesystem.rs now holds the shared core types, environments, and registration logic.
  • For untrusted code, set rex_typesystem::TypeSystemLimits::safe_defaults() before inference.
  • RegisteredValue carries a scheme, Rex-visible parameter names, and optional Markdown docs. TypeBundle is the JSON-facing persistence format for those registrations and their referenced, documented ADTs.
• rex-engine: host environment builder, compiler, and runtime evaluator. Entry points:
  • Builder::with_prelude(state)? to inject runtime constructors and builtin implementations (state can be ()).
  • standard_type_system()? to create a typing environment with the rex-engine standard prelude.
  • Builder::build_compiler() to consume the prepared builder into a compilation view.
  • Compiler::compile_program to consume the compiler and prepare a parsed program entry point into (CompiledProgram, Evaluator). Compiler::infer_* consumes the compiler for type-only checks.
  • Evaluator::run(compiled, inputs).await to execute one prepared program. inputs is a BTreeMap<String, Value> for the program’s external main interface; run consumes the evaluator, compiled program, and input map and returns a Value.
  • Builder carries host state as Builder<State> (State: Clone + Send + Sync + 'static); typed export callbacks receive an owned clone of State and return Result<T, EngineError>, typed export_async callbacks receive an owned clone of State and return Future<Output = Result<T, EngineError>>, while dynamic native APIs (export_native*) receive Context<State>, an instantiated type, and owned Values.
  • compile and evaluation APIs return EngineError; convenience entry points that cross phases return ExecutionError.
  • Host module injection API: Module + Export + Builder::inject_module for eager registration, or Importer<State> returning ResolvedModuleContent::module(...) for lazy Rust-backed modules. Named modules accept optional Markdown docs in Module::new.
• rex-proc-macro: #[derive(Rex)], #[rex::export], and #[rex::module] bridge documented Rust types, functions, and modules into Rex registrations.
• rex: top-level facade for embedding the pipeline in Rust applications.
• rex-cli: rex_cli command-line front-end around the pipeline.
• rex-lsp / rex-vscode: editor tooling.

rex-engine is organized internally around the same phases:

• builder/ owns builder-facing host/module registration.
• compiler/ owns program preparation, import rewriting, typechecking, module loading state, and CompiledProgram construction.
• evaluator/ owns execution, scheduling, native dispatch, Context, and the runtime core.
• modules/, memory/, and config.rs hold shared module identities, heap values/GC roots, and runtime options.

Runtime Ownership and GC Boundaries

rex-engine uses a moving copying collector. Heap has one owner and moves exactly once through Builder -> Compiler -> Evaluator; there is no heap mutex or public heap capability. Private heap cells contain moving edges, while evaluator frames, environments, scheduler work, and compiler state contain stable runtime-root tokens. The evaluator traverses and relocates those roots only at actual collection safepoints, not around every work item.

The public boundary is an owned Value tree. Before a host call starts, the evaluator performs a type-directed copy from internal roots to Value; the host future contains only those values and host state. On completion, the evaluator validates and imports the result while it again has exclusive heap access. Internal prelude intrinsics are deliberately separate: they operate synchronously through the private runtime/root scope and never use the host Value ABI.

This separation also leaves the internal representation free for a future LLVM backend and custom binary heap. Generated code can eventually use private tagged values and participate in the same GC safepoint/root protocol without changing host handlers or treating Value as a JIT ABI.

See Memory Management for the reference categories, allowed conversions, and collector invariants.

Design Notes

• Typed preparation: rex-engine prepares code into a typed form before execution. The current CompiledProgram stores a typed AST plus the environment snapshot needed to run it.
• Single-shot execution: evaluation is intentionally one-shot. CompiledProgram is moved into Evaluator::run with its runtime input map, consuming the evaluator as well. Prepare all required declarations/modules before constructing or consuming the evaluator.
• Single-use preparation: Builder::build_compiler(), Compiler::compile_program, and the Compiler::infer_* APIs consume their receivers. Each program run should create a fresh builder/compiler/evaluator lineage.
• Same-lineage runtime model: a CompiledProgram is intended to run on the evaluator produced from the same compiler. Rex programs are supplied as source and compiled per run, so Rex does not expose a portable compiled-artifact or cross-runtime linking model.
• Phase ownership: Builder owns embedder configuration only until build_compiler(). Compiler then owns preparation state: the type environment, runtime declaration environment, runtime registries, heap, module loader caches, and execution policy snapshot. Evaluator is built only by consuming that compiler, so runtime code cannot mutate modules or compile new declarations.
• Prelude ownership: rex-engine owns the standard prelude source, standard typing environment, and runtime contract. The split is:
  • typeclass and instance declarations written in Rex at rex-engine/src/prelude/typeclasses.rex
  • rex-engine/src/prelude/type_system.rs builds the prelude-enabled TypeSystem, including ADTs, parsed declarations, and primop schemes
  • rex-engine/src/prelude/mod.rs parses the Rex source and injects runtime method bodies/native implementations for Builder::with_prelude(state)?
  • rex-typesystem exposes generic registration/inference APIs and does not own the standard prelude
• Depth bounding: Some parts of the pipeline are naturally recursive (parsing deeply nested parentheses, matching deeply nested terms). The parser enforces a fixed AST-depth cap, and the typechecker-limit API provides bounded recursion for production/untrusted workloads.
• Import-use rewrite/validation: module processing resolves import aliases across expression vars, constructor patterns, type references, and class references; unresolved qualified alias members are rejected as module errors before runtime.
• Abstract module namespace: a ModuleId is a validated qualified Rex name such as std.prelude or ffmpeg.formats.av1. It does not encode a filesystem path, origin kind, URL, or content hash. Importers decide whether a module name maps to a file, database row, in-memory string, open editor buffer, generated AST, or Rust host module.
• Importer payloads and caching: importers are generic over host state and can return Rex source, a prebuilt CompilationPackage, or a Rust-backed Module<State>. Source imports are parsed into CompilationUnit values, then converted into CompilationPackage for compiler module processing. Source-derived and prebuilt packages are loaded through the SCC module graph path. Rust-backed modules are installed lazily through the same internal module installer used by eager inject_module, and are self-contained host modules rather than Rex source modules with nested import loading. Importer results are cached for one compile so the same request is not resolved repeatedly.

Intentional String Boundaries

Rex now prefers structured internal representations (for example NameRef, BuiltinTypeId, CanonicalSymbol, and module/type/class maps) across parser, type system, evaluator, and LSP rewrite paths. Remaining string usage is intentional in these boundary layers:

• Source text and parsing: the parser accepts source strings by definition.
• Human-facing diagnostics and display: error messages, hover text, CLI rendering, and debug output stringify symbols/types for readability.
• Protocol/serialization boundaries: JSON/LSP payloads are string-based and convert structured internal symbols/types at the edge.
• Module specifiers: parsed import names are textual before being resolved into validated ModuleId values and handed to importer policy.

Non-goal for this pass:

• Eliminating all .to_string() calls globally. The design target is to avoid stringly-typed core semantics, not to remove string conversion at UI/protocol boundaries.