consumer #

Description #

The consumer processor is used to consume messages recorded in the queue without processing them. Its purpose is to provide an entry point for data consumption pipeline, which will be further processed by subsequent processors.

Configuration Example #

Here is a simple configuration example:

pipeline:
  - name: consume_queue_messages
    auto_start: true
    keep_running: true
    retry_delay_in_ms: 5000
    processor:
      - consumer:
          consumer:
            fetch_max_messages: 1
          max_worker_size: 200
          num_of_slices: 1
          idle_timeout_in_seconds: 30
          queue_selector:
            keys:
              - email_messages
          processor:
            - xxx1:
            - xxx2:

In the above example, it subscribes to and consumes the email_messages queue. The queue messages are stored in the context of the current pipeline. The consumer provides a processor parameter, which contains a series of processors that will be executed sequentially. If any processor encounters an error during execution, the consumer will exit without committing the batch of data.

Parameter Description #