https://docs.infinilabs.com/gateway/v1.31.2/Recent content in INFINI Gateway on INFINI Gateway | The Best Open Source Golang Search-focused Gateway for Easysearch/Elasticsearch/OpenSearchHugo -- gohugo.iohttps://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/echo/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/echo/echo # Description # The echo filter is used to output specified characters in the returned result. It is often used for debugging. Function Demonstration # Configuration Example # A simple example is as follows: flow: - name: hello_world filter: - echo: message: "hello infini\n" The echo filter allows you to set the number of times that same characters can be output repeatedly. See the following example.https://docs.infinilabs.com/gateway/v1.31.2/docs/tutorial/log4j2_filtering/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/tutorial/log4j2_filtering/Protect Elasticsearch from Apache Log4j Vulnerability # CVE Address https://github.com/advisories/GHSA-jfh8-c2jp-5v3q Vulnerability Description Apache Log4j is a very popular open source logging toolkit used for the Java runtime environment. Many Java frameworks including Elasticsearch of the latest version, use this component. Therefore, the scope of impact is huge. The latest vulnerability existing in the execution of Apache Log4j’s remote code was revealed recently. Attackers can construct malicious requests and utilize this vulnerability to execute arbitrary code on a target server.https://docs.infinilabs.com/gateway/v1.31.2/docs/tutorial/online_query_rewrite/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/tutorial/online_query_rewrite/Rewrite Your Elasticsearch Requests OnTheFly # In some cases, you may find that the QueryDSL generated by the service code is unreasonable. The general practice is to modify the service code and publish it online. If the launch of a new version takes a long time (for example, the put-into-production window is not reached, major network operation closure is in progress, or additional code needs to be submitted to go live), a large number of tests need to be performed.https://docs.infinilabs.com/gateway/v1.31.2/docs/overview/-comparison/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/overview/-comparison/https://docs.infinilabs.com/gateway/v1.31.2/docs/tutorial/request-logging/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/tutorial/request-logging/Elasticsearch Search/Request Log Analysis/Audit # INFINI Gateway can track and record all requests that pass through the gateway and analyze requests sent to Elasticsearch, to figure out request performance and service running status. Gateway configuration modification # After extracting the Extreme Gateway installation package, there will be a default configuration file called ‘gateway.yml’. With a simple modification, traffic analysis can be achieved. Typically, only this section needs to be modified.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/modules/floating_ip/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/modules/floating_ip/Floating IP # The embedded floating IP feature of INFINI Gateway can implement dual-node hot standby and failover. INFINI Gateway innately provides high availability for L4 network traffic, and no extra software and devices are required to prevent proxy service interruption caused by downtime or network failures. Note: This feature supports only Mac OS and Linux OS. The gateway must run as the user root. This feature relies on the ping and ifconfig commands of the target system.https://docs.infinilabs.com/gateway/v1.31.2/docs/getting-started/install/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/getting-started/install/Installing the Gateway # INFINI Gateway supports mainstream operating systems and platforms. The program package is small, with no extra external dependency. So, the gateway can be installed very rapidly. Installation Demo # Downloading # Automatic install curl -sSL http://get.infini.cloud | bash -s -- -p gateway The above script can automatically download the latest version of the corresponding platform’s gateway and extract it to /opt/gatewayhttps://docs.infinilabs.com/gateway/v1.31.2/docs/references/entry/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/entry/Service Entry # Defining an Entry # Each gateway must expose at least one service entrance to receive operation requests of services. In INFINI Gateway, the service entrance is called an entry, which can be defined using the following parameters: entry: - name: es_gateway enabled: true router: default network: binding: 0.0.0.0:8000 reuse_port: true tls: enabled: false The network.binding parameter can be used to specify the IP address and port to be bound and listened to after the service is started.https://docs.infinilabs.com/gateway/v1.31.2/docs/getting-started/configuration/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/getting-started/configuration/Configuration # The configuration of INFINI Gateway can be modified in multiple ways. CLI Parameters # INFINI Gateway provides the following CLI parameters: ✗ ./bin/gateway --help Usage of ./bin/gateway: -config string the location of config file, default: gateway.yml (default "gateway.yml") -debug run in debug mode, gateway will quit with panic error -log string the log level,options:trace,debug,info,warn,error (default "info") -v version The parameters are described as follows: config: Specifies the name of a configuration file.https://docs.infinilabs.com/gateway/v1.31.2/docs/tutorial/index_diff/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/tutorial/index_diff/Document-Level Index Diff Between Two Elasticsearch Clusters # INFINI Gateway is able to compare differences between two different indexes in the same or different clusters. In scenarios in which application dual writes, CCR, or other data replication solutions are used, differences can be periodically compared to ensure data consistency. Function Demonstration # How Is This Feature Configured? # Setting a Target Cluster # Modify the gateway.https://docs.infinilabs.com/gateway/v1.31.2/docs/user-cases/stories/indexing_speedup_for_big_index_rebuild/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/user-cases/stories/indexing_speedup_for_big_index_rebuild/How an Insurance Group Improved the Indexing Speed by 200x Times # Challenges # A large insurance group places common database fields in Elasticsearch to improve the query performance for its policy query service. The cluster is deployed on 14 physical machines, with 4 Elasticsearch instances deployed on each physical machine. The whole cluster has more than 9 billion pieces of data. The storage size of index primary shards is close to 5 TB, and about 600 million pieces of incremental data are updated every day.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/modules/force_merge/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/modules/force_merge/Active Merging of Index Segments # INFINI Gateway has an index segment merging service, which can actively merge index segment files to improve query speed. The index segment merging service supports sequential processing of multiple indexes and tracks the status of the merging task, thereby preventing cluster slowdown caused by concurrent operations of massive index segment merging tasks. Enabling the Service # Modify the gateway.yml configuration file by adding the following configuration:https://docs.infinilabs.com/gateway/v1.31.2/docs/user-cases/stories/a_cross_region_cluster_access_locality/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/user-cases/stories/a_cross_region_cluster_access_locality/Nearest Cluster Access Across Two Cloud Providers # Service Requirements # To ensure the high availability of the Elasticsearch service, Zuoyebang deploys a single Elasticsearch cluster on both Baidu Cloud and Huawei Cloud and requires that service requests be sent to the nearest cloud first. Deployment of a Single Elasticsearch Cluster on Dual Clouds # The Elasticsearch cluster uses an architecture with master nodes separated from data nodes.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/router/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/router/Service Router # INFINI Gateway judges the flow direction based on routers. A typical example of router configuration is as follows: router: - name: my_router default_flow: default_flow tracing_flow: request_logging rules: - method: - PUT - POST pattern: - "/_bulk" - "/{index_name}/_bulk" flow: - bulk_process_flow Router involves several important terms: Flow: Handling flow of a request. Flows can be defined in three places in a router. default_flow: Default handling flow, which is the main flow of service handling.https://docs.infinilabs.com/gateway/v1.31.2/docs/getting-started/docker/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/getting-started/docker/Container Deployment # INFINI Gateway supports container deployment. Installation Demo # Downloading an Image # The images of INFINI Gateway are published at the official repository of Docker. The URL is as follows: https://hub.docker.com/r/infinilabs/gateway Use the following command to obtain the latest container image: docker pull infinilabs/gateway:1.30.2-2396 Verifying the Image # After downloading the image locally, you will notice that the container image of INFINI Gateway is very small, with a size less than 25 MB.https://docs.infinilabs.com/gateway/v1.31.2/docs/overview/hardware/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/overview/hardware/https://docs.infinilabs.com/gateway/v1.31.2/docs/getting-started/helm/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/getting-started/helm/Helm Charts # INFINI Gateway supports deployment on K8s by using helm chart . The Chart Repository # Chart repository: https://helm.infinilabs.com. Use the follow command add the repository: helm repo add infinilabs https://helm.infinilabs.com Prerequisites # K8S StorageClass The default StorageClass of the Chart package is local-path, you can install it through here. If you want use other StorageClass(installed), you can create a YAML file (eg. vaules.https://docs.infinilabs.com/gateway/v1.31.2/docs/getting-started/optimization/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/getting-started/optimization/System Optimization # The operating system of the server where INFINI Gateway is installed needs to be optimized to ensure that INFINI Gateway runs in the best possible state. The following uses Linux as an example. System Parameters # sudo tee /etc/security/limits.d/21-infini.conf <<-'EOF' * soft nofile 1048576 * hard nofile 1048576 * soft memlock unlimited * hard memlock unlimited root soft nofile 1048576 root hard nofile 1048576 root soft memlock unlimited root hard memlock unlimited EOF Kernel Optimization # cat << SETTINGS | sudo tee /etc/sysctl.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/flow/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/flow/Handling Flow # Flow Definition # Requests received by each gateway are handled through a series of processes and then results are returned to the client. A process is called a flow in INFINI Gateway. See the following example. flow: - name: hello_world filter: - echo: message: "hello gateway\n" repeat: 1 - name: not_found filter: - echo: message: '404 not found\n' repeat: 1 The above example defines two flows: hello_world and not_found.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/elasticsearch/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/elasticsearch/Elasticsearch # Defining a Resource # INFINI Gateway supports multi-cluster access and different versions. Each cluster serves as one Elasticsearch back-end resource and can be subsequently used by INFINI Gateway in multiple locations. See the following example. elasticsearch: - name: local enabled: true endpoint: https://127.0.0.1:9200 - name: dev enabled: true endpoint: https://192.168.3.98:9200 basic_auth: username: elastic password: pass - name: prod enabled: true endpoint: http://192.168.3.201:9200 discovery: enabled: true refresh: enabled: true interval: 10s basic_auth: username: elastic password: pass The above example defines a local development test cluster named local and a development cluster named dev.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/context/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/context/Request Context # What Is Context # Context is the entry for INFINI Gateway to access relevant information in the current running environment, such as the request source and configuration. You can use the _ctx keyword to access relevant fields, for example, _ctx.request.uri, which indicates the requested URL. Embedded Request Context # The embedded _ctx context objects of an HTTP request mainly include the following: Name Type Description id uint64 Unique ID of the request tls bool Whether the request is a TLS request remote_ip string Source IP of the client remote_addr string Source IP address of the client, including port local_ip string Gateway local IP address local_addr string Gateway local IP address, including port elapsed int64 Time that the request has been executed (ms) request.https://docs.infinilabs.com/gateway/v1.31.2/docs/getting-started/benchmark/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/getting-started/benchmark/Benchmark Testing # You are advised to use the Elasticsearch-dedicated benchmark tool Loadgen to test the gateway performance. Highlights of Loadgen: Robust performance Lightweight and dependency-free Random selection of template-based parameters High concurrency Balanced traffic control at the benchmark end Validate server responses. Download URL: http://release.infinilabs.com/loadgen/ Loadgen # Loadgen is easy to use. After the tool is downloaded and decompressed, two files are obtained: one executable program and one configuration file loadgen.https://docs.infinilabs.com/gateway/v1.31.2/docs/tutorial/proxy_kibana/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/tutorial/proxy_kibana/Adding a TLS and Basic Security for Kibana # If you have multiple Kibana versions or your Kibana version is out of date, or if you do not set TLS or identity, then anyone can directly access Kibana. You can use the INFINI Gateway to quickly fix this issue. Using the HTTP Filter to Forward Requests # - http: schema: "http" #https or http host: "192.168.3.188:5602" Adding Authentication # - basic_auth: valid_users: medcl: passwd Replacing Static Resources in the Router # - method: - GET pattern: - "/plugins/kibanaReact/assets/illustration_integrations_lightmode.https://docs.infinilabs.com/gateway/v1.31.2/docs/tutorial/proxy_elasticsearch/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/tutorial/proxy_elasticsearch/Enable HTTPS/TLS + Basic Auth for Elasticsearch easily # If you have multiple Elasticsearch versions or your version is out of date, or if you do not set TLS or identity, then anyone can directly access Elasticsearch. You can use INFINI Gateway to quickly fix this issue. Define an Elasticsearch resource # Let’s define the Elasticsearch resources, config as bellow： elasticsearch: - name: prod enabled: true endpoint: http://192.168.3.201:9200 The prod refer to http://192.https://docs.infinilabs.com/gateway/v1.31.2/docs/tutorial/fix_count_in_search_response/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/tutorial/fix_count_in_search_response/Handle Count Structure of Different Elasticsearch Versions # To optimize performance in Elasticsearch 7.0 and later versions, search result matches are not accurately counted and the search result response body is adjusted. This will inevitably cause incompatibility with existing code. How can the problem be fixed quickly? Structure Diff # The search structure difference is as follows: The search structure used by Elasticsearch before version 7.0 is as follows.https://docs.infinilabs.com/gateway/v1.31.2/docs/tutorial/es-hadoop_integration/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/tutorial/es-hadoop_integration/Integrate with Elasticsearch-Hadoop # Elasticsearch-Hadoop utilizes a seed node to access all back-end Elasticsearch nodes by default. The hotspots and requests may be improperly allocated. To improve the resource utilization of back-end Elasticsearch nodes, you can implement precision routing for the access to Elasticsearch nodes through INFINI Gateway. Write Acceleration # If you import data by using Elasticsearch-Hadoop, you can modify the following parameters of Elasticsearch-Hadoop to access INFINI Gateway, so as to improve the write throughput:https://docs.infinilabs.com/gateway/v1.31.2/docs/tutorial/prometheus_integration/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/tutorial/prometheus_integration/Integration with Prometheus # Infini Gateway supports outputting metrics in Prometheus format, which is convenient for integration with Prometheus. Stats API # Access Gateway’s API endpoint, with URL parameter as below: http://localhost:2900/stats?format=prometheus ➜ ~ curl http://localhost:2900/stats\?format\=prometheus buffer_fasthttp_resbody_buffer_acquired{type="gateway", ip="192.168.3.23", name="Orchid", id="cbvjphrq50kcnsu2a8v0"} 1 buffer_stats_acquired{type="gateway", ip="192.168.3.23", name="Orchid", id="cbvjphrq50kcnsu2a8v0"} 7 buffer_stats_max_count{type="gateway", ip="192.168.3.23", name="Orchid", id="cbvjphrq50kcnsu2a8v0"} 0 system_cpu{type="gateway", ip="192.168.3.23", name="Orchid", id="cbvjphrq50kcnsu2a8v0"} 0 buffer_bulk_request_docs_acquired{type="gateway", ip="192.168.3.23", name="Orchid", id="cbvjphrq50kcnsu2a8v0"} 1 buffer_fasthttp_resbody_buffer_inuse{type="gateway", ip="192.168.3.23", name="Orchid", id="cbvjphrq50kcnsu2a8v0"} 0 stats_gateway_request_bytes{type="gateway", ip="192.168.3.23", name="Orchid", id="cbvjphrq50kcnsu2a8v0"} 0 system_mem{type="gateway", ip="192.https://docs.infinilabs.com/gateway/v1.31.2/docs/tutorial/routing_to_cluser_by_index/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/tutorial/routing_to_cluser_by_index/Unified access indices from different clusters in Kibana # Now there is such a demand, customers need to divide the data according to the business dimension, the index is split into three different clusters, to split the large cluster into multiple small clusters have many benefits, such as reduced coupling, bringing benefits to cluster availability and stability, but also to avoid the impact of a single business hotspot to affect other services, although splitting the cluster is a very common way to play, but the management is not so convenient, especially when querying data, it may be need to access the three sets of clusters separately APIs, even to switch between three different sets of Kibana to access the cluster’s data, is there a way to seamlessly unite them together?https://docs.infinilabs.com/gateway/v1.31.2/docs/tutorial/path_rewrite_by_javascript/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/tutorial/path_rewrite_by_javascript/Use JavaScript for complex query rewriting # Here is a use case： How does the gateway support cross-cluster search? I want to achieve: the input search request is lp:9200/index1/_search these indices are on three clusters, so need search across these clusters, how to use the gateways to switch to lp:9200/cluster01:index1,cluster02,index1,cluster03:index1/_search? we don’t want to change the application side, there are more than 100 indices, the index name not strictly named as index1, may be multiple indices together。https://docs.infinilabs.com/gateway/v1.31.2/docs/references/config/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/config/Other Configurations # Advanced Usage # Templates # Example: configs.template: - name: "es_gw1" path: ./sample-configs/config_template.tpl variable: name: "es_gw1" binding_host: "0.0.0.0:8000" tls_on_entry: true elasticsearch_endpoint: "http://localhost:9200" Name Type Description configs.template array Configuration templates, can specify multiple templates with corresponding parameters configs.template[].name string Name of the configuration configs.template[].path string Template configuration path configs.template[].variable map Template parameter settings, variables in the template are used as $[[variable_name]] Environment Variables # The Gateway supports the use of environment variables for flexible parameter control within the configuration.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/auto_generate_doc_id/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/auto_generate_doc_id/auto_generate_doc_id # Description # The auto_generate_doc_id filter is used to add a UUID (Universally Unique Identifier) to a document when creating a document without specifying the UUID explicitly. This is typically used when you don’t want the backend system to generate the ID automatically. For example, if you want to replicate the document between clusters, it’s better to assign a known ID to the document instead of letting each cluster generate its own ID.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/basic_auth/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/basic_auth/basic_auth # Description # The basic_auth filter is used to verify authentication information of requests. It is applicable to simple authentication. Configuration Example # A simple example is as follows: flow: - name: basic_auth filter: - basic_auth: valid_users: medcl: passwd medcl1: abc ... Parameter Description # Name Type Description valid_users map Username and passwordhttps://docs.infinilabs.com/gateway/v1.31.2/docs/references/processors/bulk_indexing/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/processors/bulk_indexing/bulk_indexing # Description # The bulk_indexing processor is used to asynchronously consume bulk requests in queues. Configuration Example # A simple example is as follows: pipeline: - name: bulk_request_ingest auto_start: true keep_running: true processor: - bulk_indexing: queue_selector.labels: type: bulk_reshuffle level: cluster Parameter Description # Name Type Description elasticsearch string The default Elasticsearch cluster ID, which will be used if elasticsearch is not specified in the queue Labels idle_timeout_in_seconds int Timeout duration of the consumption queue, which is set to 1 by default.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/bulk_request_mutate/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/bulk_request_mutate/bulk_request_mutate # Description # The bulk_request_mutate filter is used to mutate bulk requests of Elasticsearch. Configuration Example # A simple example is as follows: flow: - name: bulk_request_mutate filter: - bulk_request_mutate: fix_null_id: true generate_enhanced_id: true # fix_null_type: true # default_type: m-type # default_index: m-index # index_rename: # "*": index-new # index1: index-new # index2: index-new # index3: index3-new # index4: index3-new # medcl-dr3: index3-new # type_rename: # "*": type-new # type1: type-new # type2: type-new # doc: type-new # doc1: type-new .https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/bulk_request_throttle/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/bulk_request_throttle/bulk_request_throttle # Description # The bulk_request_throttle filter is used to limit the speed of Bulk requests to Elasticsearch. Configuration Example # A simple example is as follows: flow: - name: bulk_request_mutate filter: - bulk_request_throttle: indices: test: max_requests: 5 action: drop message: "test writing too fast。" log_warn_message: true filebeat-*: max_bytes: 512 action: drop message: "filebeat indices writing too fast。" log_warn_message: true Parameter Description # Name Type Description indices map The indices which wanted to throttle indices.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/bulk_reshuffle/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/bulk_reshuffle/bulk_reshuffle # Description # The bulk_reshuffle filter is used to parse batch requests of Elasticsearch based on document, sort out documents as needed, and archive and store them in queues. After documents are stored, the filter can rapidly return service requests, thereby decoupling front-end writing from back-end Elasticsearch clusters. The bulk_reshuffle filter needs to be used in combination with offline pipeline consumption tasks. When passing through queues generated by the bulk_reshuffle filter, metadata carries "type": "bulk_reshuffle" and Elasticsearch cluster information such as "elasticsearch": "dev", by default.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/bulk_response_process/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/bulk_response_process/bulk_response_process # Description # The bulk_response_process filter is used to process bulk requests of Elasticsearch. Configuration Example # A simple example is as follows: flow: - name: bulk_response_process filter: - bulk_response_process: success_queue: "success_queue" tag_on_success: ["commit_message_allowed"] Parameter Description # Name Type Description invalid_queue string Name of the queue that saves an invalid request. It is mandatory. failure_queue string Name of the queue that saves a failed request.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/cache/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/cache/cache # Description # The cache filter is composed of the get_cache and set_cache filters, which need to be used in combination. The cache filter is used to cache accelerated queries, prevent repeated requests, and reduce the query pressure of back-end clusters. get_cache Filter # The get_cache filter is used to acquire previous messages from the cache and return them to the client, without needing to access the back-end Elasticsearch.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/clone/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/clone/clone # Description # The clone filter is used to clone and forward traffic to another handling flow. It can implement dual-write, multi-write, multi-DC synchronization, cluster upgrade, version switching, and other requirements. Configuration Example # A simple example is as follows: flow: - name: double_write filter: - clone: flows: - write_to_region_a - write_to_region_b #last one's response will be output to client - name: write_to_region_a filter: - elasticsearch: elasticsearch: es1 - name: write_to_region_b filter: - elasticsearch: elasticsearch: es2 The above example copies Elasticsearch requests to two different remote clusters.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/processors/consumer/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/processors/consumer/consumer # Description # The consumer processor is used to consume messages recorded in the queue without processing them. Its purpose is to provide an entry point for data consumption pipeline, which will be further processed by subsequent processors. Configuration Example # Here is a simple configuration example: pipeline: - name: consume_queue_messages auto_start: true keep_running: true retry_delay_in_ms: 5000 processor: - consumer: consumer: fetch_max_messages: 1 max_worker_size: 200 num_of_slices: 1 idle_timeout_in_seconds: 30 queue_selector: keys: - email_messages processor: - xxx1: - xxx2: In the above example, it subscribes to and consumes the email_messages queue.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/context_filter/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/context_filter/context_filter # Description # The context_filter is used to filter traffic by request context. Configuration Example # A simple example is as follows: flow: - name: test filter: - context_filter: context: _ctx.request.path message: "request not allowed." status: 403 must: #must match all rules to continue prefix: - /medcl contain: - _search suffix: - _search wildcard: - /*/_search regex: - ^/m[\w]+dcl must_not: # any match will be filtered prefix: - /.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/context_limiter/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/context_limiter/context_limiter # Description # The context_limiter filter is used to control the traffic based on request context. Configuration Example # A configuration example is as follows: flow: - name: default_flow filter: - context_limiter: max_requests: 1 action: drop context: - _ctx.request.path - _ctx.request.header.Host - _ctx.request.header.Env The above configuration combines three context variables (_ctx.request.path, _ctx.request.header.Host, and _ctx.request.header.Env) into a bucket for traffic control. The allowable maximum queries per second (QPS) is 1 per second.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/context_parse/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/context_parse/context_parse # Description # context_parse filter is used to extract fields from context variables and store them in the context。 Configuration Example # A simple example is as follows: flow: - name: context_parse filter: - context_parse: context: _ctx.request.path pattern: ^\/.*?\d{4}\.(?P<month>\d{2})\.(?P<day>\d{2}).*? group: "parsed_index" In above flow, the context_parse can extract fields from request：/abd-2023.02.06-abc/_search，get two new fields: parsed_index.month and parsed_index.day。 Parameter Description # Name Type Description context string Context variable pattern string The regular expression used to extract the field skip_error bool Whether to ignore the error and returned directly, such like the context variable does not exist group string Set the group name, which the extracted fields can be placed under a separate grouphttps://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/context_regex_replace/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/context_regex_replace/context_regex_replace # Description # The context_regex_replace filter is used to replace and modify relevant information in the request context by using regular expressions. Configuration Example # A simple example is as follows: flow: - name: test filter: - context_regex_replace: context: "_ctx.request.path" pattern: "^/" to: "/cluster:" when: contains: _ctx.request.path: /_search - dump: request: true This example replaces curl localhost:8000/abc/_search in requests with curl localhost:8000/cluster:abc/_search. Parameter Description # Name Type Description context string Request context and corresponding key pattern string Regular expression used for matching and replacement to string Target string used for replacement A list of context variables that can be modified is provided below:https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/context_switch/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/context_switch/context_switch # Description # context_switch filter can be used to use context variables for conditional judgment and achieve flexible jumps. Configuration Example # A simple example is as follows: flow: - name: context_switch filter: - context_switch: context: logging.month default_flow: echo_message_not_found switch: - case: ["02","01"] action: redirect_flow flow: echo_message_01_02 - case: ["03"] action: redirect_flow flow: echo_message_03 Parameter Description # Name Type Description context string The name of context skip_error bool Whether to ignore the error and returned directly, such like the context variable does not exist default_action string Set the default action，could be redirect_flow or drop，default redirect_flow default_flow string Set the default flow stringify_value bool Whether to stringify the value，default true。 continue bool Whether to continue the flow after hit.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/processors/dag/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/processors/dag/dag # Description # The dag processor is used to manage the concurrent scheduling of tasks. Configuration Example # The following example defines a service named racing_example and auto_start is set to true. Processing units to be executed in sequence are set in processor, the dag processor supports concurrent execution of multiple tasks and the wait_all and first_win aggregation modes. pipeline: - name: racing_example auto_start: true processor: - echo: #ready, set, go message: read,set,go - dag: mode: wait_all #first_win, wait_all parallel: - echo: #player1 message: player1 - echo: #player2 message: player2 - echo: #player3 message: player3 end: - echo: #checking score message: checking score - echo: #announce champion message: 'announce champion' - echo: #done message: racing finished The echo processor above is very simple and is used to output a specified message.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/date_range_precision_tuning/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/date_range_precision_tuning/date_range_precision_tuning # Description # The date_range_precision_tuning filter is used to reset the time precision for time range query. After the precision is adjusted, adjacent repeated requests initiated within a short period of time can be easily cached. For scenarios with low time precision but a large amount of data, for example, if Kibana is used for report analysis, you can reduce the precision to cache repeated query requests to reduce the pressure of the back-end server and accelerate the front-end report presentation.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/drop/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/drop/drop # Description # The drop filter is used to discard a message and end the processing of a request in advance. Configuration Example # A simple example is as follows: flow: - name: drop filter: - drop:https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/dump/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/dump/dump # Description # The dump filter is used to dump relevant request information on terminals. It is mainly used for debugging. Configuration Example # A simple example is as follows: flow: - name: hello_world filter: - dump: request: true response: true Parameter Description # The dump filter is relatively simple. After the dump filter is inserted into a required flow handling phase, the terminal can output request information about the phase, facilitating debugging.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/processors/dump_hash/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/processors/dump_hash/dump_hash # Description # The dump_hash processor is used to export index documents of a cluster and calculate the hash value. Configuration Example # A simple example is as follows: pipeline: - name: bulk_request_ingest auto_start: true keep_running: true processor: - dump_hash: #dump es1's doc indices: "medcl-dr3" scroll_time: "10m" elasticsearch: "source" query: "field1:elastic" fields: "doc_hash" output_queue: "source_docs" batch_size: 10000 slice_size: 5 Parameter Description # Name Type Description elasticsearch string Name of a target cluster scroll_time string Scroll session timeout duration batch_size int Scroll batch size, which is set to 5000 by default slice_size int Slice size, which is set to 1 by default sort_type string Document sorting type, which is set to asc by default sort_field string Document sorting field indices string Index level string Request processing level, which can be set to cluster, indicating that node- and shard-level splitting are not performed on requests.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/elasticsearch/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/elasticsearch/elasticsearch # Description # The elasticsearch filter is used to forward requests to back-end Elasticsearch clusters. Configuration Example # Before using the elasticsearch filter, define one Elasticsearch cluster configuration node as follows: elasticsearch: - name: prod enabled: true endpoint: http://192.168.3.201:9200 The following shows a flow configuration example. flow: - name: cache_first filter: - elasticsearch: elasticsearch: prod The preceding example forwards requests to the prod cluster. Automatic Update # For a large cluster that contains many nodes, it is almost impossible to configure all back-end nodes individually.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/elasticsearch_health_check/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/elasticsearch_health_check/elasticsearch_health_check # Description # The elasticsearch_health_check filter is used to detect the health status of Elasticsearch in traffic control mode. When a back-end fault occurs, the filter triggers an active cluster health check without waiting for the results of the default polling check of Elasticsearch. Traffic control can be configured to enable the filter to send check requests to the back-end Elasticsearch at a maximum of once per second.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/flow/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/flow/flow # Description # The flow filter is used to redirect to or execute one or a series of other flows. Configuration Example # A simple example is as follows: flow: - name: flow filter: - flow: flows: - request_logging Context mapped flow: flow: - name: dns-flow filter: - flow: ignore_undefined_flow: true context_flow: context: _ctx.request.host context_parse_pattern: (?P<uuid>^[0-9a-z_\-]+)\. flow_id_template: flow_$[[uuid]] - set_response: status: 503 content_type: application/json body: '{"message":"invalid HOST"}' More information about context, please refer to Context .https://docs.infinilabs.com/gateway/v1.31.2/docs/references/processors/flow_replay/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/processors/flow_replay/flow_replay # Description # The flow_replay processor is used to asynchronously consume requests in the queue and use the asynchronous processing process for online requests to perform consumption processing. Configuration Example # A simple example is as follows: pipeline: - name: backup-flow-request-reshuffle auto_start: true keep_running: true singleton: true retry_delay_in_ms: 10 processor: - consumer: max_worker_size: 100 queue_selector: labels: type: "primary_write_ahead_log" consumer: group: request-reshuffle fetch_max_messages: 10000 fetch_max_bytes: 20485760 fetch_max_wait_ms: 10000 processor: - flow_replay: flow: backup-flow-request-reshuffle commit_on_tag: "commit_message_allowed" Parameter Description # Name Type Description message_field string The context field name that store the message obtained from the queue, default messages.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/processors/flow_runner/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/processors/flow_runner/flow_runner # Description # The flow_runner processor is used to asynchronously consume requests in a queue by using the processing flow used for online requests. Configuration Example # A simple example is as follows: pipeline: - name: bulk_request_ingest auto_start: true keep_running: true processor: - flow_runner: input_queue: "primary_deadletter_requests" flow: primary-flow-post-processing when: cluster_available: [ "primary" ] Parameter Description # Name Type Description input_queue string Name of a subscribed queue flow string Flow used to consume requests in consumption queues commit_on_tag string A message is committed only when a specified tag exists in the context of the current request.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/hash_mod/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/hash_mod/hash_mod # Description # The hash_mod filter is used to obtain a unique partition number using the hash modulo of the request’s context. It is generally used for subsequent request forwarding. Configuration Example # A simple example is as follows: flow: - name: default_flow filter: - hash_mod: # Hash requests to different queues source: "$[[_ctx.remote_ip]]_$[[_ctx.request.username]]_$[[_ctx.request.path]]" target_context_name: "partition_id" mod: 10 # Hash to 10 partitions add_to_header: true - set_context: context: _ctx.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/http/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/http/http # Description # The http filter is used to forward requests to a specified HTTP server as a proxy. Configuration Example # A simple example is as follows: flow: - name: default_flow filter: - basic_auth: valid_users: medcl: passwd - http: schema: "http" #https or http #host: "192.168.3.98:5601" hosts: - "192.168.3.98:5601" - "192.168.3.98:5602" Parameter Description # Name Type Description schema string http or https host string Target host address containing the port ID, for example, localhost:9200 hosts array Host address list.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/processors/index_diff/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/processors/index_diff/index_diff # Description # The index_diff processor is used to compare differences between two result sets. Configuration Example # A simple example is as follows: pipeline: - name: bulk_request_ingest auto_start: true keep_running: true processor: - index_diff: diff_queue: "diff_result" buffer_size: 1 text_report: true #If data needs to be saved to Elasticsearch, disable the function and start the diff_result_ingest task of the pipeline. source_queue: 'source_docs' target_queue: 'target_docs' Parameter Description # Name Type Description source_queue string Name of source data target_queue string Name of target data diff_queue string Queue that stores difference results buffer_size int Memory buffer size keep_source bool Whether difference results contain document source information text_report bool Whether to output results in text formhttps://docs.infinilabs.com/gateway/v1.31.2/docs/references/processors/indexing_merge/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/processors/indexing_merge/indexing_merge # Description # The indexing_merge processor is used to consume pure JSON documents in the queue, and merge them into bulk requests and save them in the specified queue. It needs to be consumed with the bulk_indexing processor, and batch writes are used instead of single requests to improve write throughput. Configuration Example # A simple example is as follows: pipeline: - name: indexing_merge auto_start: true keep_running: true processor: - indexing_merge: input_queue: "request_logging" elasticsearch: "logging-server" index_name: "infini_gateway_requests" output_queue: name: "gateway_requests" label: tag: "request_logging" worker_size: 1 bulk_size_in_mb: 10 - name: logging_requests auto_start: true keep_running: true processor: - bulk_indexing: bulk: compress: true batch_size_in_mb: 10 batch_size_in_docs: 5000 consumer: fetch_max_messages: 100 queues: type: indexing_merge when: cluster_available: [ "logging-server" ] Parameter Description # Name Type Description input_queue string Name of a subscribed queue worker_size int Number of threads that concurrently execute consumption tasks, which is set to 1 by default.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/javascript/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/javascript/javascript # Description # The javascript filter can be used to execute your own processing flow by crafting the scripts in javascript, which provide the ultimate flexibility. Configuration Example # A simple example is as follows: flow: - name: test filter: - javascript: source: > function process(ctx) { var console = require('console'); console.log("hello from javascript"); } The process in this script is a built-in function that handles incoming context and allows to write your custom business logic.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/processors/json_indexing/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/processors/json_indexing/json_indexing # Description # The json_indexing processor is used to consume pure JSON documents in queues and store them to a specified Elasticsearch server. Configuration Example # A simple example is as follows: pipeline: - name: request_logging_index auto_start: true keep_running: true processor: - json_indexing: index_name: "gateway_requests" elasticsearch: "dev" input_queue: "request_logging" idle_timeout_in_seconds: 1 worker_size: 1 bulk_size_in_mb: 10 Parameter Description # Name Type Description input_queue string Name of a subscribed queue worker_size int Number of threads that concurrently execute consumption tasks, which is set to 1 by default.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/ldap_auth/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/ldap_auth/ldap_auth # Description # The ldap_auth filter is used to set authentication based on the Lightweight Directory Access Protocol (LDAP). Configuration Example # A simple example is as follows: flow: - name: ldap_auth filter: - ldap_auth: host: "ldap.forumsys.com" port: 389 bind_dn: "cn=read-only-admin,dc=example,dc=com" bind_password: "password" base_dn: "dc=example,dc=com" user_filter: "(uid=%s)" The above configuration uses an online free LDAP test server, the test user is tesla, and the password is password.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/logging/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/logging/logging # Description # The logging filter is used to asynchronously record requests to the local disk to minimize the delay of requests. In scenarios with heavy traffic, you are advised to use other request filters jointly to reduce the total number of logs. Configuration Example # A simple example is as follows: flow: - name: test filter: - logging: queue_name: request_logging An example of a recorded request log is as follows:https://docs.infinilabs.com/gateway/v1.31.2/docs/references/processors/merge_to_bulk/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/processors/merge_to_bulk/merge_to_bulk # Description # The merge_to_bulk processor is used to consume pure JSON documents in the queue, and merge them into bulk requests and save them in the specified queue. It needs to be consumed with the consumer processor, and batch writes are used instead of single requests to improve write throughput. Configuration Example # A simple example is as follows: pipeline: - name: messages_merge_async_bulk_results auto_start: true keep_running: true singleton: true processor: - consumer: queue_selector: keys: - bulk_result_messages consumer: group: merge_to_bulk processor: - merge_to_bulk: elasticsearch: "logging" index_name: ".https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/queue/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/queue/queue # Description # The queue filter is used to save requests to a message queue. Configuration Example # Here is a simple example: flow: - name: queue filter: - queue: # Handle dirty_writes, second-commit queue_name: "primary_final_commit_log##$[[partition_id]]" labels: type: "primary_final_commit_log" partition_id: "$[[partition_id]]" message: "$[[_ctx.request.header.X-Replicated-ID]]#$[[_ctx.request.header.LAST_PRODUCED_MESSAGE_OFFSET]]#$[[_sys.unix_timestamp_of_now]]" when: equals: _ctx.request.header.X-Replicated: "true" Parameter Description # Name Type Description depth_threshold int Must be greater than the specified depth to be stored in the queue, default is 0 type string Specify the type of message queue, supports kafka and disk queue_name string Message queue name labels map Add custom labels to the newly created message queue topic message string Custom message content, supports variables save_last_produced_message_offset bool Whether to retain the Offset of the last successfully written message in the context for later use as a variable last_produced_message_offset_key string Custom variable name for storing the Offset of the last successfully written message in the context, default is LAST_PRODUCED_MESSAGE_OFFSEThttps://docs.infinilabs.com/gateway/v1.31.2/docs/references/processors/queue_consumer/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/processors/queue_consumer/queue_consumer # Description # The queue_consumer processor is used to asynchronously consume requests in a queue and send the requests to Elasticsearch. Configuration Example # A simple example is as follows: pipeline: - name: bulk_request_ingest auto_start: true keep_running: true processor: - queue_consumer: input_queue: "backup" elasticsearch: "backup" waiting_after: [ "backup_failure_requests"] worker_size: 20 when: cluster_available: [ "backup" ] Parameter Description # Name Type Description input_queue string Name of a subscribed queue worker_size int Number of threads that concurrently execute consumption tasks, which is set to 1 by default.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/ratio/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/ratio/ratio # Description # The ratio filter is used to forward normal traffic to another flow proportionally. It can implement canary release, traffic migration and export, or switch some traffic to clusters of different versions for testing. Configuration Example # A simple example is as follows: flow: - name: ratio_traffic_forward filter: - ratio: ratio: 0.1 flow: hello_world continue: true Parameter Description # Name Type Description ratio float Proportion of traffic to be migrated action string The action when hit, can be drop or redirect_flow, default is redirect_flow flow string New traffic processing flow continue bool Whether to continue flow after hit.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/record/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/record/record # Description # The record filter is used to record requests. Output requests can be copied to the console of Kibana for debugging. Configuration Example # A simple example is as follows: flow: - name: request_logging filter: - record: stdout: true filename: requests.txt Examples of the format of request logs output by the record filter are as follows: GET /_cluster/state/version,master_node,routing_table,metadata/* GET /_alias GET /_cluster/health GET /_cluster/stats GET /_nodes/0NSvaoOGRs2VIeLv3lLpmA/stats Parameter Description # Name Type Description filename string Filename of request logs stored in the data directory stdout bool Whether the terminal also outputs the characters.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/redirect/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/redirect/redirect # Description # redirect filter used to redirect request to specify URL address。 Configuration Example # A simple example is as follows: flow: - name: redirect filter: - redirect: uri: https://infinilabs.com Parameter Description # Name Type Description uri string The target URI code int Status code，default 302https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/redis_pubsub/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/redis_pubsub/redis_pubsub # Description # The redis filter is used to store received requests and response results to Redis message queues. Configuration Example # A simple example is as follows: flow: - name: redis_pubsub filter: - redis_pubsub: host: 127.0.0.1 port: 6379 channel: gateway response: true Parameter Description # Name Type Description host string Redis host name, which is localhost by default.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/processors/replay/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/processors/replay/replay # Description # The replay processor is used to replay requests recorded by the record filter. Configuration Example # A simple example is as follows: pipeline: - name: play_requests auto_start: true keep_running: false processor: - replay: filename: requests.txt schema: "http" host: "localhost:8000" Parameter Description # Name Type Description filename string Name of a file that contains replayed messages schema string Request protocol type: http or https host string Target server that receives requests, in the format of host:porthttps://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_api_key_filter/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_api_key_filter/request_api_key_filter # Description # When Elasticsearch conducts authentication through API keys, the request_api_key_filter is used to filter requests based on request API ID. Configuration Example # A simple example is as follows: flow: - name: test filter: - request_api_key_filter: message: "Request filtered!" exclude: - VuaCfGcBCdbkQm-e5aOx The above example shows that requests from VuaCfGcBCdbkQm-e5aOx will be rejected. See the following information. ➜ ~ curl localhost:8000 -H "Authorization: ApiKey VnVhQ2ZHY0JDZGJrUW0tZTVhT3g6dWkybHAyYXhUTm1zeWFrdzl0dk5udw==" -v * Rebuilt URL to: localhost:8000/ * Trying 127.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_api_key_limiter/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_api_key_limiter/request_api_key_limiter # Description # The request_api_key_limiter filter is used to control traffic by API key. Configuration Example # A configuration example is as follows: flow: - name: rate_limit_flow filter: - request_api_key_limiter: id: - VuaCfGcBCdbkQm-e5aOx max_requests: 1 action: drop # retry or drop message: "your api_key reached our limit" The above configuration controls the traffic with the API ID of VuaCfGcBCdbkQm-e5aOx and the allowable maximum QPS is 1 per second.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_body_json_del/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_body_json_del/request_body_json_del # Description # The request_body_json_del filter is used to delete some fields from a request body of the JSON format. Configuration Example # A simple example is as follows: flow: - name: test filter: - request_body_json_del: path: - query.bool.should.[0] - query.bool.must Parameter Description # Name Type Description path array JSON path key value to be deleted ignore_missing bool Whether to ignore processing if the JSON path does not exist.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_body_json_set/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_body_json_set/request_body_json_set # Description # The request_body_json_set filter is used to modify a request body of the JSON format. Configuration Example # A simple example is as follows: flow: - name: test filter: - request_body_json_set: path: - aggs.total_num.terms.field -> "name" - aggs.total_num.terms.size -> 3 - size -> 0 Parameter Description # Name Type Description path map It uses -> to identify the key value pair: JSON path and the value used for replacement.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_body_regex_replace/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_body_regex_replace/request_body_regex_replace # Description # The request_body_regex_replace filter is used to replace string content in a request body by using a regular expression. Configuration Example # A simple example is as follows: flow: - name: test filter: - request_body_regex_replace: pattern: '"size": 10000' to: '"size": 100' - elasticsearch: elasticsearch: prod - dump: request: true The above example changes the size from 10000 to 100 in the request body sent to Elasticsearch.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_client_ip_filter/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_client_ip_filter/request_client_ip_filter # Description # The request_client_ip_filter is used to filter traffic based on source user IP addresses of requests. Configuration Example # A simple example is as follows: flow: - name: test filter: - request_client_ip_filter: exclude: - 192.168.3.67 The above example shows that requests from 192.168.3.67 are not allowed to pass through. The following is an example of route redirection. flow: - name: echo filter: - echo: message: hello stanger - name: default_flow filter: - request_client_ip_filter: action: redirect_flow flow: echo exclude: - 192.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_client_ip_limiter/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_client_ip_limiter/request_client_ip_limiter # Description # The request_client_ip_limiter filter is used to control traffic based on the request client IP address. Configuration Example # A configuration example is as follows: flow: - name: rate_limit_flow filter: - request_client_ip_limiter: ip: #only limit for specify ips - 127.0.0.1 max_requests: 256 # max_bytes: 102400 #100k action: retry # retry or drop # max_retry_times: 1000 # retry_interval: 500 #100ms message: "your ip reached our limit" The above configuration controls the traffic with the IP address of 127.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_header_filter/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_header_filter/request_header_filter # Description # The request_header_filter is used to filter traffic based on request header information. Configuration Example # A simple example is as follows: flow: - name: test filter: - request_header_filter: include: - TRACE: true The above example shows that requests are allowed to pass through only when the headers of the requests contain TRACE: true. curl 192.168.3.4:8000 -v -H 'TRACE: true' Parameter Description # Name Type Description exclude array Header information used to refuse to allow requests to pass through include array Header information used to allow requests to pass through action string Processing action after filtering conditions are met.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_host_filter/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_host_filter/request_host_filter # Description # The request_host_filter is used to filter requests based on a specified domain name or host name. It is suitable for scenarios in which there is only one IP address but access control is required for multiple domain names. Configuration Example # A simple example is as follows: flow: - name: test filter: - request_host_filter: include: - domain-test2.com:8000 The above example shows that only requests that are used to access the domain name domain-test2.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_host_limiter/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_host_limiter/request_host_limiter # Description # The request_host_limiter filter is used to control traffic based on the request host (domain name). Configuration Example # A configuration example is as follows: flow: - name: rate_limit_flow filter: - request_host_limiter: host: - api.elasticsearch.cn:8000 - logging.elasticsearch.cn:8000 max_requests: 256 # max_bytes: 102400 #100k action: retry # retry or drop # max_retry_times: 1000 # retry_interval: 500 #100ms message: "you reached our limit" The above configuration controls the traffic used for accessing domain names api.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_method_filter/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_method_filter/request_method_filter # Description # The request_method_filter is used to filter traffic based on request method. Configuration Example # A simple example is as follows: flow: - name: test filter: - request_method_filter: exclude: - PUT - POST include: - GET - HEAD - DELETE Parameter Description # Name Type Description exclude array Methods of requests that are refused to pass through include array Methods of requests that are allowed to pass through action string Processing action after filtering conditions are met.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_path_filter/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_path_filter/request_path_filter # Description # The request_path_filter is used to filter traffic based on request path. Configuration Example # A simple example is as follows: flow: - name: test filter: - request_path_filter: must: #must match all rules to continue prefix: - /medcl contain: - _search suffix: - _count - _refresh wildcard: - /*/_refresh regex: - ^/m[\w]+dcl must_not: # any match will be filtered prefix: - /.kibana - /_security - /_security - /gateway_requests* - /.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_path_limiter/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_path_limiter/request_path_limiter # Description # The request_path_limiter filter is used to define traffic control rules for requests. It can implement index-level traffic control. Configuration Example # A configuration example is as follows: flow: - name: rate_limit_flow filter: - request_path_limiter: message: "Hey, You just reached our request limit!" rules: - pattern: "/(?P<index_name>medcl)/_search" max_qps: 3 group: index_name - pattern: "/(?P<index_name>.*?)/_search" max_qps: 100 group: index_name In the above configuration, the query is performed against the medcl query, the allowable maximum QPS is 3, and the QPS is 100 for queries performed against other indexes.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_reshuffle/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_reshuffle/request_reshuffle # Description # request_reshuffle can analyze non-bulk requests to Elasticsearch, archive them in a queue, and store them on disk first. This allows business-side requests to return quickly, decoupling the front-end writes from the back-end Elasticsearch cluster. request_reshuffle requires offline pipeline consumption tasks to work in conjunction. Configuration Example # Here is a simple example: flow: - name: backup-flow-request-reshuffle filter: - flow: flows: - set-auth-for-backup-flow - request_reshuffle: # Reshuffle none-bulk requests elasticsearch: "backup" queue_name_prefix: "request_reshuffle" partition_size: $[[env.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_user_filter/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_user_filter/request_user_filter # Description # When Elasticsearch conducts authentication in Basic Auth mode, the request_user_filter is used to filter requests by request username. Configuration Example # A simple example is as follows: flow: - name: test filter: - request_user_filter: include: - "elastic" The above example shows that only requests from elastic are allowed to pass through. Parameter Description # Name Type Description exclude array List of usernames, from which requests are refused to pass through include array List of usernames, from which requests are allowed to pass through action string Processing action after filtering conditions are met.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_user_limiter/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/request_user_limiter/request_user_limiter # Description # The request_user_limiter filter is used to control traffic by username. Configuration Example # A configuration example is as follows: flow: - name: rate_limit_flow filter: - request_user_limiter: user: - elastic - medcl max_requests: 256 # max_bytes: 102400 #100k action: retry # retry or drop # max_retry_times: 1000 # retry_interval: 500 #100ms message: "you reached our limit" The above configuration controls the traffic of users medcl and elastic and the allowable maximum QPS is 256 per second.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/response_body_regex_replace/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/response_body_regex_replace/response_body_regex_replace # Description # The response_body_regex_replace filter is used to replace string content in a response by using a regular expression. Configuration Example # A simple example is as follows: flow: - name: test filter: - echo: message: "hello infini\n" - response_body_regex_replace: pattern: infini to: world The result output of the preceding example is hello world. Parameter Description # Name Type Description pattern string Regular expression used for matching and replacement to string Target string used for replacementhttps://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/response_header_filter/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/response_header_filter/response_header_filter # Description # The response_header_filter is used to filter traffic based on response header information. Configuration Example # A simple example is as follows: flow: - name: test filter: ... - response_header_filter: exclude: - INFINI-CACHE: CACHED The above example shows that a request is not allowed to pass through when the header information of the response contains INFINI-CACHE: CACHED. Parameter Description # Name Type Description exclude array Response header information for refusing to allow traffic to pass through include array Response header information for allowing traffic to pass through action string Processing action after filtering conditions are met.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/response_header_format/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/response_header_format/response_header_format # Description # The response_header_format filter is used to convert keys in response header information into lowercase letters. Configuration Example # A simple example is as follows: flow: - name: test filter: - response_header_format:https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/response_status_filter/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/response_status_filter/response_status_filter # Description # The response_status_filter is used to filter traffic based on the status code responded by the back-end service. Configuration Example # A simple example is as follows: flow: - name: test filter: - response_status_filter: message: "Request filtered!" exclude: - 404 include: - 200 - 201 - 500 Parameter Description # Name Type Description exclude array Response code for refusing to allow traffic to pass through include array Response code for allowing traffic to pass through action string Processing action after filtering conditions are met.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/retry_limiter/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/retry_limiter/retry_limiter # Description # The retry_limiter filter is used to judge whether the maximum retry count is reached for a request, to avert unlimited retries of a request. Configuration Example # A simple example is as follows: flow: - name: retry_limiter filter: - retry_limiter: queue_name: "deadlock_messages" max_retry_times: 3 Parameter Description # Name Type Description max_retry_times int Maximum retry count. The default value is 3.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/rewrite_to_bulk/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/rewrite_to_bulk/rewrite_to_bulk # Description # rewrite_to_bulk can analyze ordinary document creation and modification operations in Elasticsearch and rewrite them as Bulk batch requests. Configuration Example # Here is a simple example: flow: - name: replicate-primary-writes-to-backup-queue filter: - flow: flows: - set-auth-for-backup-flow - rewrite_to_bulk: # Rewrite docs create/update/delete operation to bulk request - bulk_reshuffle: # Handle bulk requests when: contains: _ctx.request.path: /_bulk elasticsearch: "backup" queue_name_prefix: "async_bulk" level: cluster # Cluster, node, index, shard partition_size: 10 fix_null_id: true - queue: # Handle non-bulk requests <1.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/sample/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/sample/sample # Description # The sample filter is used to sample normal traffic proportionally. In a massive query scenario, collecting logs of all traffic consumes considerable resources. Therefore, you are advised to perform sampling statistics and sample and analyze query logs. Configuration Example # A simple example is as follows: flow: - name: sample filter: - sample: ratio: 0.2 Parameter Description # Name Type Description ratio float Sampling ratiohttps://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/set_basic_auth/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/set_basic_auth/set_basic_auth # Description # The set_basic_auth filter is used to configure the authentication information used for requests. You can use the filter to reset the authentication information used for requests. Configuration Example # A simple example is as follows: flow: - name: set_basic_auth filter: - set_basic_auth: username: admin password: password Parameter Description # Name Type Description username string Username password string Passwordhttps://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/set_context/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/set_context/set_context # Description # The set_context filter is used to set relevant information for the request context. Configuration Example # A simple example is as follows: flow: - name: test filter: - set_response: body: '{"message":"hello world"}' - set_context: context: # _ctx.request.uri: http://baidu.com # _ctx.request.path: new_request_path # _ctx.request.host: api.infinilabs.com # _ctx.request.method: DELETE # _ctx.request.body: "hello world" # _ctx.request.body_json.explain: true # _ctx.request.query_args.from: 100 # _ctx.request.header.ENV: dev # _ctx.response.content_type: "application/json" # _ctx.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/set_hostname/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/set_hostname/set_hostname # Description # The set_hostname filter is used to set the host or domain name to be accessed in the request header. Configuration Example # A simple example is as follows: flow: - name: set_hostname filter: - set_hostname: hostname: api.infini.cloud Parameter Description # Name Type Description hostname string Host informationhttps://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/set_request_header/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/set_request_header/set_request_header # Description # The set_request_header filter is used to set header information for requests. Configuration Example # A simple example is as follows: flow: - name: set_request_header filter: - set_request_header: headers: - Trial -> true - Department -> Engineering Parameter Description # Name Type Description headers map It uses -> to identify a key value pair and set header information.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/set_request_query_args/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/set_request_query_args/set_request_query_args # Description # The set_request_query_args filter is used to set the QueryString parameter information used for requests. Configuration Example # A simple example is as follows: flow: - name: set_request_query_args filter: - set_request_query_args: args: - size -> 10 Parameter Description # Name Type Description args map It uses -> to identify a key value pair and set QueryString parameter information.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/set_response/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/set_response/set_response # Description # The set_response filter is used to set response information to be returned for requests. Configuration Example # A simple example is as follows: flow: - name: set_response filter: - set_response: status: 200 content_type: application/json body: '{"message":"hello world"}' Parameter Description # Name Type Description status int Request status code, which is 200 by default. content_type string Type of returned content body string Returned structural bodyhttps://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/set_response_header/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/set_response_header/set_response_header # Description # The set_response_header filter is used to set the header information used in responses. Configuration Example # A simple example is as follows: flow: - name: set_response_header filter: - set_response_header: headers: - Trial -> true - Department -> Engineering Parameter Description # Name Type Description headers map It uses -> to identify a key value pair and set header information.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/sleep/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/sleep/sleep # Description # The sleep filter is used to add a fixed delay to requests to reduce the speed. Configuration Example # A simple example is as follows: flow: - name: slow_query_logging_test filter: - sleep: sleep_in_million_seconds: 1024 Parameter Description # Name Type Description sleep_in_million_seconds int64 Delay to be added, in millisecondshttps://docs.infinilabs.com/gateway/v1.31.2/docs/references/processors/smtp/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/processors/smtp/smtp # Description # The SMTP processor is used to send emails, supporting both plain text and HTML emails. It supports template variables and allows attachments to be embedded in the email body. The email message is comes from the pipeline context. Configuration Example # A simple example is as follows: pipeline: - name: send_email auto_start: true keep_running: true retry_delay_in_ms: 5000 processor: - consumer: consumer: fetch_max_messages: 1 max_worker_size: 200 num_of_slices: 1 idle_timeout_in_seconds: 30 queue_selector: keys: - email_messages processor: - smtp: idle_timeout_in_seconds: 1 server: host: "smtp.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/switch/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/switch/switch # Description # The switch filter is used to forward traffic to another flow along the requested path, to facilitate cross-cluster operations. No alternation is required for Elasticsearch clusters, and all APIs in each cluster can be accessed, including APIs used for index read/write and cluster operations. Configuration Example # A simple example is as follows: flow: - name: es1-flow filter: - elasticsearch: elasticsearch: es1 - name: es2-flow filter: - elasticsearch: elasticsearch: es2 - name: cross_cluste_search filter: - switch: path_rules: - prefix: "es1:" flow: es1-flow - prefix: "es2:" flow: es2-flow - elasticsearch: elasticsearch: dev #elasticsearch configure reference name In the above example, the index beginning with es1: is forwarded to the es1 cluster, the index beginning with es2: is forwarded to the es2 cluster, and unmatched indexes are forwarded to the dev cluster.https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/translog/Mon, 01 Jan 0001 00:00:00 +0000https://docs.infinilabs.com/gateway/v1.31.2/docs/references/filters/translog/translog # Description # The translog filter is used to save received requests to local files and compress them. It can record some or complete request logs for archiving and request replay. Configuration Example # A simple example is as follows: flow: - name: translog filter: - translog: max_file_age: 7 max_file_count: 10 Parameter Description # Name Type Description path string Root directory for log storage, which is the translog subdirectory in the gateway data directory by default category string Level-2 subdirectory for differentiating different logs, which is default by default.