output

output #

CategoryScope
Output transportbatch (queue target) · record (lane target)

Unified output step: writes the processed batch or record to exactly one target — a durable queue or an in-memory lane. Config unpacking rejects zero or multiple targets. queue_output is a registered alias for the queue form.

Configuration #

FieldTypeDefaultDescription
queuestring or objectTarget queue: bare name, or { name, label } object. Queue and lane are mutually exclusive.
lanestringTarget lane name. Record scope only — the lane carries live event pointers. Auto-created (block-on-full, default capacity) when no lane: stanza defines it.
ackboolLane target only: attach an acknowledgment receipt to the context’s AckSet, so the emitting consumer’s commit gate waits for settlement (cross-lane at-least-once).
codecstringQueue target: override the codec resolution.
message_fieldstring“messages”Batch scope (queue target): context key holding the batch.

Scope rules #

TargetScopeNotes
queuebatch (top level)Encodes the batch through the 3-level codec resolution.
lanerecord (inside for_each / lane_consumer)Placing it at the consumer top level errors per batch with “requires record scope”.

Example #

Lane hand-off — the main pipeline writes processed records into an in-memory lane instead of a second disk queue (staging stays the only disk write):

processor:
  - consumer:
      queue_selector:
        keys: ["staging-<streamID>"]
      message_field: messages
      processor:
        - for_each:
            message_field: messages
            parallel: 4
            processor:
              - grok: { field: message, patterns: ["..."] }
              - output:
                  lane: ingest-lane-<streamID>
Edit Edit this page