The catalog descriptor

The catalog descriptor is a small binary file, one per unit, that lists what a runtime needs to route and check the traffic of that unit's interfaces: the interface numbers, the members and their ordinals, the maximum encoded size of each payload, the timing bounds, and the retired interfaces. A program reads it without the compiler and without decoding the IR. A unit is one ridl.toml with a [package] table and every source package in its directory tree; the unit's name is the manifest's name, which is also the name of its root source package (the design record).

What it is for

A process that carries interactions between other processes — a gateway, a broker, a bridge between two transports, or the engine of a runtime — does not link the generated crate of every unit it carries. It still has to answer questions such as:

Which interface does the number in this frame name, and is that interface retired?
Is this member a signal, an event, a command or a query, and which payload does it carry?
What is the largest payload this member can send, so that a buffer can be sized in advance, and so that a larger payload can be refused? The size counts the payload alone, without the envelope and the framing a transport adds.
What timing bound does this member declare?
Do both peers speak the same version of the unit?

The catalog descriptor answers each of these from one file per unit, read at startup.

What is built. The compiler writes the descriptor, the ridl-descriptor crate reads and verifies it, and ridl describe prints it. No gateway, broker or engine that reads it exists in this repository yet; those are the consumers it is designed for. The generated Rust face does not read the file either: it carries the catalog name and hash it needs as constants (see The catalog and the face). Codegen plugins do not read it; the codegen model carries the same facts (see Writing a codegen plugin).

Producing it

--emit catalog writes one <unit>.catalog.binfb file per unit that declares at least one interface, or a service with an inline body, in any of its source packages:

ridl build examples/cabin --emit catalog --out-dir out
# writes out/veh.cabin.catalog.binfb

A unit with no interface gets no file. The flag combines with the others, so one build can write the Rust crate and the descriptor together. The other emit targets stay one file per source package; only the catalog is per unit.

Reading it

ridl describe verifies a descriptor and prints it as JSON:

ridl describe out/veh.cabin.catalog.binfb

The top of the output for examples/cabin, with the 32 bytes of the hash left out. toolchain is the version of ridl that wrote the file, so it changes with each release; the excerpt shows it as <version>:

{
  "compatible": [],
  "hash": [ ... ],
  "interfaces": [
    {
      "members": [ ... ],
      "name": "Cabin",
      "number": 1,
      "provisional": true,
      "reserved_ordinals": []
    },
    ...
  ],
  "name": "veh.cabin",
  "retired": [],
  "toolchain": "<version>",
  "version": 1
}

The member average, a query, as it appears in members:

{
  "kind": "Query",
  "limit": null,
  "name": "average",
  "ordinal": 4,
  "payloads": [
    {
      "max_sizes": [
        { "bytes": 46, "cause": "Unspecified", "encoding": "FlatBuffers", "state": "Bounded" }
      ],
      "role": "request",
      "type_name": "Window"
    },
    {
      "max_sizes": [
        { "bytes": 44, "cause": "Unspecified", "encoding": "FlatBuffers", "state": "Bounded" }
      ],
      "role": "response",
      "type_name": "Average"
    }
  ],
  "timing": { "max_us": "200000", "min_us": null, "mode": "Range" }
}

The keys are in alphabetical order, and an absent timing bound or limit is null. A file that is not a valid descriptor, or that has a schema version this toolchain does not read, makes ridl describe exit with code 2. The CLI reference has the details.

What it contains

The catalog: the unit name; the hash, which identifies this version of the unit (see below); toolchain, the version of ridl that wrote the file; the schema version; the interfaces; the retired interfaces, each with its name and number, so a reader can refuse a peer that still uses one; and compatible, the hashes of earlier catalogs of the unit that the toolchain judged compatible or carried over unchanged with this one (see The compatible catalogs).
Each interface: its name, qualified relative to the unit (Cabin for an interface of the root source package, cluster.Speed for one of the source package cluster below it); its number, which is one space per unit; and provisional, which is true while the number is not yet recorded in the unit's interfaces.lock, the one lock file beside the unit's ridl.toml. Cabin above is provisional because examples/cabin has no lock file; a number becomes permanent when ridl lock records it. reserved_ordinals lists the ordinals that reserved tombstones hold. The ridl reference, section 11, is the rule for the names and the numbers.
Each member: its name, its ordinal (its position in the interface body), its kind, its payloads, its timing when the member declares one, and its limit, the payload limit in bytes that a [limits] entry of the unit's ridl.toml gives it, absent when no entry applies.
Each payload: its role (value for a signal or a fixed, occurrence for an event, request for a command or a query, response for a query's reply), its type_name, and one max_sizes row per encoding. A row is Bounded, with bytes the maximum encoded size of the payload, envelope and framing excluded; or Unbounded, with bytes 0 and a cause that says why the compiler cannot bound it in that encoding. For example, a FlatBuffers row is Unbounded when the payload's largest encoding would exceed 4 GiB, the most that FlatBuffers' 32-bit offsets can address. An encoding with no row has no size computed by this toolchain. A stream payload, a request of zero or of several parameters, and an inline T | E reply have no row in any encoding. A podbuf row is present for a payload with a podbuf root, sized by ridl_ir::projection::podbuf::max_size, and a proto3 row is present only for a payload that is a struct, a union, a named scalar, an enum or an enum set, the kinds the proto3 schema gives a message of their own. A named scalar, an enum and an enum set are sized as their root message <Name>Box.

The descriptor does not contain type layouts, field lists, constraints other than the ones that bound a payload's size, contract clauses, or initial values. A reader that needs a payload's shape reads the schema that the wire backend emits.

The compatible catalogs

The hash identifies one version of the unit exactly. compatible lists, as 32-byte hashes, the earlier versions of the unit that ridl diff's classifier, scoped to the unit, judged compatible with this one, newest first. When the current hash is the newest published one, the catalog did not change, and the published list is carried whatever the classifier's verdict. The current hash is never in it. A reader built for one of those versions can be accepted by a provider that serves this one, which is the use the frame specification makes of the list (section 6.1). No runtime in this repository accepts an attach on the strength of the list.

The list comes from the published baseline. ridl baseline records, for each unit with an interface, a <unit>.catalogs file beside the snapshots, and ridl build reads it. The list is empty when there is no baseline, when the unit has no file, and after a breaking change that changes the catalog, which restarts the chain. A file is used only when its newest hash is the hash recomputed from the snapshots published beside it; a file that does not describe those snapshots, such as one left behind by a hand-edited baseline, restarts the chain instead. A breaking change that leaves the catalog hash as it was, such as the removal of a declaration no interface reaches, keeps the chain. A change in another unit counts for this unit only when an interface of this unit reaches the changed declaration. ridlc build reads no baseline and writes an empty list. The CLI reference states the file format and the rules.

The catalog and the face

The catalog hash is a SHA-256 hash over the unit's interfaces, from every source package of the unit, their numbers, and every declaration they reach, in the unit or in another one. Doc comments, labels and deprecated do not change it; any other change to an interface or to a type it uses does, so a change in any source package of the unit that an interface reaches changes the one hash. A [limits] entry in the unit's ridl.toml that gives an interaction its payload limit changes the hash too, so adding, removing or editing such an entry changes the hash of the interaction's unit. The design record's hash section has the exact input.

The generated Rust face carries the same unit name and hash as its CATALOG. When a face binds to a port, it compares its CATALOG with the catalog of that port, and panics when they differ. So a face generated from a different version of the unit than the one its runtime serves fails at bind time, instead of misreading payloads. The face compares itself with its own port only; it does not compare catalogs with the party at the other end. The failures section of "Using the generated Rust code" shows what that check does.

Stability

The file is a FlatBuffers buffer with the file identifier RDLC. Its schema evolves by appending only: a field is added at the end of its table, and no field is removed, reordered or retyped. A reader built against an older schema ignores the fields it does not know. The version field is the schema version, now 1; a reader refuses a version other than the one it was built for.

A Rust program reads the file with the ridl-descriptor crate, published on crates.io. Its verify function checks the whole buffer before it returns a view, so a later read cannot fail on a malformed offset. With its default features off the crate is no_std with alloc and keeps verify, the accessors and finish; the parts that build a descriptor from the IR need the standard library and come with the std feature. The default dedup feature makes finish write each repeated vtable, string and byte vector once. With it off as well, a reader links no hashing crate, and finish writes a larger but valid file.

Not built yet

The design also defines a system descriptor, one per deployment. It is not built; the design record lists it with the other parts that are not built.

RIDL