Release Notes #

Information about release notes of INFINI Gateway is provided here.

Latest (In development) #

Breaking changes #

• chore: update default branch for vendor

Features #

• feat: auto issue certificates for domain
• feat: support filter config file (#620)
• feat: add configs param allow_generated_metrics_tasks
• feat: add param context for es api ClusterHealth and ClusterStats
• fix: getting cluster version with timeout
• feat: support passing query param level to cluster health api
• feat: crontab task support timezone
• feat: support tz draft
• feat: support customize event queue

Bug fix #

• fix: rollback for client register
• fix: reload when file is in dirty read
• fix: prevent consumer from advancing beyond writer’s segment
• fix: wait group usage in bulk_indexing processor
• fix: query_string query was ignored (#588)

Improvements #

• refactor: refactoring system config
• chore: add uuid to websocket session
• chore: add util to parse parameter, panic on missing
• chore: adjust logging format
• fix: refactoring inflight check
• chore: update license header
• chore: add util to register schema
• chore: log with ip and agent register when restart agent
• chore: update cli naming style (#589)

1.26.1 (2024-08-13) #

Improvements #

• Keep the same version number as INFINI Console.
• Synchronize updates for known issues fixed in the Framework.

1.26.0 (2024-06-07) #

Improvements #

• feat: add wildcard_domain filter
• chore: remove security filter and translog_viewer

1.25.0 (2024-04-30) #

Improvements #

• add push err and log pop timeout err

Bug fix #

• Fix date_range_precision_tuning work with filter query

1.24.0 (2024-04-15) #

Improvements #

• Refactoring http client tls config
• Write field routing to bulk metadata when field _routing exists in scrolled doc

Bug fix #

• Fix(reshuffle filter): make sure queue config always have label type
• Fix rotate config usage

1.23.0 (2024-03-01) #

Bug fix #

• Fix consumer offsets were not reset after deleting instance queues.

1.22.0 (2024-01-26) #

Bug fix #

• Fix update default EOF max retry times

Improvements #

• Unified version number with INFINI Console
• Optimize bytes pool
• Limit inflight request size
• Add limit to entry

1.21.0 (2023-12-29) #

Bug fix #

• Fix log error when temp file was missing

1.20.0 (2023-12-01) #

Bug fix #

• Fix the number of connections not being released and abnormal memory growth caused by Framework Bug

Improvements #

• Add configuration to allow setting fasthttp client parameters

1.19.0 (2023-11-16) #

Features #

• Add http processor
• Add basic-auth based security to API module
• Allow to register self to config manager
• Allow to panic on config error or not

Bug fix #

• Fix rewrite_to_bulk the issue with _type was missing in newer version
• Fix rewrite_to_bulk, support none-index doc operations

Improvements #

• Update test, assert parse result

1.18.0 (2023-09-21) #

Breaking changes #

• Finally removed request_body_truncate and response_body_truncate filter

Features #

• Support kafka based replication
• Add _util.generate_uuid to request context
• Add _util.increment_id.BUCKET_NAME to request context
• Add singleton to pipeline config, prevent multiple pipelines running at the same time
• Add a pluggable distributed locker implementation
• Add a general preference config for application
• Generalize abstraction for queue, refactoring disk_queue, complete kafka implementation
• Add merge_to_bulk processor, retired indexing_merge processor
• Add flow_replay processor, retired flow_runner processor
• Add replication_correlation for replication use case
• Add hash_mod filter
• Add new parameters to bulk_response_process filter
• Add request_reshuffle filter
• Add resource limit, allow to set max number of cpus or binding affinity
• Support nested variables in template
• Add rewrite_to_bulk filter

Bug fix #

• Fix retry delay was not working in pipeline
• Fix number was not supported in template
• Fix queue selector by labels, if more than one labels specified, they should all match together neither any match should be found

Improvements #

• Lowercase all module names
• Prefetch elasticsearch metadata during start
• Adding a shutdown signal to application scope
• Refactoring queue api, support kafka management
• Add enabled to badger module
• Allow to register module/plugin with priority
• Unified queue usage and initialization
• Optimize bulk_reshuffle filter’s performance, add header X-Bulk-Reshuffled to response
• Support to use variables in queue filter, allow to output last produced message offset

1.17.0 (2023-07-21) #

Features #

• Add filter consumer to subscribe message from queue.
• Add filter stmp to send email messages.

Improvements #

• Auto skip corrupted disk_queue file

1.16.0 (2023-06-30) #

Features #

• Add filter security to support unified identity management through the Console, and supports LDAP authentication login.

1.15.0 (2023-06-08) #

Breaking changes #

Features #

• Add filter auto_generate_doc_id to fill UUID during document creation

Bug fix #

• Fix floating_ip freqently switched between active and standby
• Fix elasticsearch overwriting upstream x-forwarded-for header.
• Fix queue_consumer high CPU usage when queue is empty.

Improvements #

1.14.0 (2023-05-25) #

Breaking changes #

Features #

• Support customize service name
• metrics added user_in_ms and sys_in_ms to track CPU utilization.
• elasticsearch added dial_timeout option.

Bug fix #

• Fix missing stdout outputs after real-time logging push turned on.
• logging fixed broken min_elapsed_time_in_ms option.
• Fix a issue that consumes idle queues causing high CPU usage. ​

1.13.0 (2023-05-11) #

Breaking changes #

Features #

• Allow to toggle each rules in router, disabled by default
• Added support for loong64 architecture.
• Added support for riscv64 architecture.
• elasticsearch added dial_timeout option.

Bug fix #

• Fix http response header from upstream was not well returned to client
• Fix duplicated running pipelines when hot-reloading pipeline configruations.
• bulk_indexing fix leaking goroutines when pipeline stopped or released.

Improvements #

• Prefer to set headers instead of add headers, avoid generate duplicated headers
• Improve the responsiveness of stopping pipeline.
• pipeline added enabled option to toggle the pipelines quickly.

1.12.1 (2023-04-20) #

Bug fix #

• elasticsearch fix potential http connection close.
• elasticsearch return proper error info when upstream timed out.

1.12.0 (2023-04-06) #

Breaking changes #

• bulk_indexing rename bulk.response_handle.retry_exception to bulk.response_handle.retry_rules

Improvements #

• bulk_indexing improve logging and default retry times.

Bug fix #

• bulk_indexing fix potential data loss when 429 status encountered.
• bulk_indeixng fix disk queue file retention rules not working properly.
• metrics fix prometheus metrics exporting
• pipeline fix auto-reloading
• badger fix potential file corruption.
• bulk_reshuffle fix potential memory leaking.

1.11.0 #

Breaking changes #

• Rename condition has_fields to exists

Features #

• echo added more options: status, continue, response, logging, logging_level.
• switch added continue and unescape options.
• Add max memory setting to system config
• Add flags to set max memory soft limit

Bug fix #

• Fix path init when install service

Improvements #

• unescape path in switch filter by default
• Update badger to v4
• Add more memory info to stats
• Escape metric name for prometheus

1.10.0 #

Breaking changes #

• bulk_indexing move bulk response handling controls from bulk to bulk.response_handle.

Features #

• Support environment variables in config
• Add option action to ratio filter, allow drop request immediately
• Add context_parser filter
• Add context_switch filter
• Add _sys.* to request context

Bug fix #

Improvements #

1.9.0 #

Breaking changes #

• Refactoring config for ip access control
• Disable elasticsearch metadata refresh by default
• Update default config path from configs to config
• Remove sample-configs, moved to dedicated integrated-testing project
• Remove field conntime, update field @timestamp to timestamp in logging filter
• Rename disorder_ to fast_

Features #

• Support listen on IPv6 address
• Add general health api
• Add request_ip to context
• Add badger filter plugin
• Allow to split produce and consume messages from s3
• Add bulk_request_throttle filter
• Support access request context and more output options in echo filter
• Add body_json to response context
• Add cert config to API module, support mTLS
• Add api to clear scroll context
• Floating_ip support stick by priority
• Add keystore util
• Allow to save success bulk results in bulk_indexing processor
• Enable watch and reload the major config file
• Support run background job in one goroutine
• Allow to handle async_bulk request logging
• Add config to control cluster health check while cluster not available, set default to false
• Allow to follow redirects in http filter, set default read and write timeout to 30s
• Support collect instance metrics to monitoring gateway
• Add json log format

Bug fix #

• Fix user was removed in logging filter
• Fix incorrect message size issue, reload when files changed in disk_queue
• Fix issue that index_diff could not finished automatically
• Fix hostname was not well updated in filter set_request_header or set_hostname
• Fix to check consumer’s lag instead of queue’s lag in flow_runner processor
• Fix file not found error for disk_queue
• Fix the delete requests was not proper handled in filter bulk_reshuffle, bulk_request_mutate and bulk_indexing processor
• Fix memory leak caused by misuse of bytes buffer
• Fix to handle the last request in replay processor
• Fix url args was not updated after change
• Fix memory leak when serving high-concurrent requests
• Fix nil id caused error when using sliced workers in bulk_indexing processor
• Fix index name with dot
• Refactoring time fields for orm, skip empty time
• Refactoring stats, allow to register extended stats
• Fix to restart gateway entrypoint on flow change
• Update ratio filter, fix random number, add header to ratio filter
• Fix query parameter no_cache was not well respected in get_cache filter
• Fix single delete request was ignored in bulk requests
• Fix request mutate filter

Improvements #

• Remove newline in indexing_merge and json_indexing processor
• Improve instance check, add config to disable
• Add option skip_insecure_verify to s3 module
• Improve instance check, enable config to disable
• Update the way to get ctx process info, optimize memory usage
• Improve indexing performance for bulk_indexing processor
• Refactoring disk_queue, speedup message consumption
• Enable segment compress for disk_queue by default
• Skip download s3 files when s3 was not enabled
• Add option to log warning messages for throttle filters
• Optimize hash performance for getting primary shardID and partitionID
• Add cache for get index routing table
• Optimize performance for bulk response processing
• Refactoring bulk_processor, pass meta info to payload func
• Don’t call payload func for delete action
• Improve queue consumer’s lag check
• Enable prepare flat files ahead for read by default, skip unnecessary file
• Add object pool for xxhash
• Refactoring disk_queue, handle consumer in-flight segments in memory
• Add config to remove duplicated newline for bulk_processor
• Add metric timestamp in stats api
• Improve error on routing table missing
• Refactoring bytes buffer and object pool, expose metrics via API
• Refactoring tasks pooling, support throttle and unified control
• Optimize badger file size and memory usage
• Refactoring time fields for orm, skip empty time
• Refactoring stats, allow to register extended stats
• Refactoring to handle bulk response results
• Add client_session_cache_size to tls setting
• Safety add newline to each bytes when handle bulk requests

1.8.1 #

Bug fix #

• Remove newline in document for processor es_scroll and dump_hash

1.8.0 #

Breaking changes #

• Remove config compress_on_message_payload from disk_queue
• Rename parameter consumer to name in consumer_has_lag condition
• Remove redundancy prefix name of the disk_queue files

Features #

• Add segment level based disk_queue file compression

Bug fix #

• Fix nil host in bulk_indexing processor
• Fix nil body in bulk_response_process filter
• Fix sliced consume in bulk_indexing processor

Improvements #

• Handle bulk stats to bulk_response_process and used in logging filter

1.7.0 #

Breaking changes #

Features #

• Add prometheus format to stats API
• Add redirect filter
• Add context_flow to flow filter
• Add permitted_client_ip_list to router
• Add Centos based docker image

Bug fix #

• Fix date_range_precision filter failed to parse on specify field
• Fix disk usage status in windows platform

Improvements #

• Merge events during config change, prevent unnecessary reload
• Handle templates when loading config
• Add cache to ldap_auth filter

1.6.0 #

Breaking changes #

• Update disk_queue folder structure, use UUID as folder name instead of the queue name
• Parameter mode was removed from bulk_reshuffle filter, only async was supported
• Rename filter bulk_response_validate to bulk_response_process

Features #

• Add metadata to queue
• Support subscribe queue by specify labels
• Support concurrent worker control for bulk_indexing processor
• Auto detect new queues for bulk_indexing processor
• Allow to consume queue messages over disk queue
• Auto sync disk_queue files to remote s3 in background
• Add api to operate gateway entry
• Support plugin auto discovery
• Add API to operate gateway entities
• Filter bulk_request_mutate support remove _type in bulk requests for es v8.0+
• Add elasticsearch adapter for version 8.0+
• Add http filter for general reverse proxy usage, like proxy Kibana
• Add consumer_has_lag condition to check queue status
• Add record filter to play requests easier
• Add zstd compress to disk_queue, disabled by default
• Add disorder_bulk_indexing processor
• Add javascript filter
• Add prefix and suffix to when conditions
• Add indexing_merge processor

Bug fix #

• Fix date_range_precision_tuning filter for complex range query
• Fix node availability initially check
• Fix basic_auth filter not asking user to input auth info in browser
• Fix null id not fixed in filter bulk_request_mutate and bulk_reshuffle
• Fix switch filter not forwarding when remove_prefix was disabled
• Fix buffer was not proper reset in flow_runner processor
• Fix entry not loading the pre-defined TLS certificates
• Fix set_basic_auth not proper reset previous user information
• Fix elapsed in request logging not correct
• Fix switch filter, use strings.TrimPrefix instead of strings.TrimLeft
• Fix the last query_string args can’t be deleted, parameter no_cache in get_cache filter fixed
• Fix s3 downloaded file corrupted

Improvements #

• Handle http public address, remove prefix if that exists
• Refactor bulk_reshuffle filter and bulk_indexing processor
• Should not fetch nodes info when elasticsearch discovery disabled
• Seamless consume queue message across files
• Persist consumer offset to local store
• Add API to reset consumer offset
• Refactoring ORM framework
• Expose error of mapping put
• Refactoring pipeline framework
• Improve multi-instance check, multi-instance disabled by default
• Add CPU and memory metrics to stats api
• Seamless fetch queue files from s3 server
• Proper handle 409 version conflicts in bulk requests
• Allow memory queue to retry 1s when it is full
• Proper handle the cluster available check
• Proper handle the cluster partial failure check
• Exit bulk worker while no new messages returned

1.5.0 #

Breaking changes #

Features #

• Add API to scroll messages from disk queue
• Prevent out of space, disk usage reserved for disk_queue
• Add context_filter and context_limiter for general purpose
• Add bulk_request_mutate filter
• Add basic_auth filter
• Add set_context filter
• Add context_regex_replace filter
• Add to_string property to request and response context

Bug fix #

• Fix bulk response validate incorrectly caused by jsonParser
• Fix nil exception in request_path_limiter caused by refactoring
• Fix big size document out of order caused by bulk buffer

Improvements #

• Fix TCP not keepalived in some case
• Add closing progress bar to pipeline module
• Add retry_delay_in_ms config to pipeline module
• Handle partial failure in bulk requests
• Optimize scroll performance of dump_hash processor
• Improve API directory

1.4.0 #

Breaking changes #

• Rename flow config filter_v2 to filter, only support new syntax
• Rename pipeline config pipelines_v2 to pipeline, processors to processor, only support new syntax
• Rename filter request_logging to logging
• Merge dump filters to dump filter
• Response headers renamed, dashboard may broken
• Remove filter request_body_truncate and response_body_truncate

Features #

• Add option to disable file logging output
• Add option compress to queue_consumer processor

Bug fix #

• Fix invalid host header setting in elasticsearch reverse proxy
• Fix cluster available health check
• Fix gzip encoding issue for requests forwarding

Improvements #

• Support string type in in condition

1.3.0 #

Breaking changes #

• Switch to use pipelines_v2 syntax only
• Rename filter disk_enqueue to queue
• Rename processor disk_queue_consumer to queue_consumer
• Rename filter redis to redis_pubsub

Features #

• Refactoring pipeline framework, support DAG based task schedule
• Add dump_hash and index_diffs processor
• Add redis output and redis queue adapter
• Add set_request_query_args filter
• Add ldap_auth filter
• Add retry_limiter filter
• Add request_body_json_set and request_body_json_del filter
• Add stats filter
• Add health_check config to elastic module
• Add API to pipeline framework, support _start and _stop pipelines

Bug fix #

• Fix data race issue in bulk_reshuffle
• Fix fix_null_id always executed in bulk_reshuffle
• Auto handle big sized documents in bulk requests

Improvements #

• Refactoring flow runner to service pipeline
• Optimize CPU and Memory usage
• Optimize index diff service, speedup and cross version compatibility
• Set the default max file size of queue files to 1 GB
• Proper handle elasticsearch failure during startup
• Support custom depth check to queue_has_lag condition
• Support multi hosts for elasticsearch configuration
• Add parameter auto_start to prevent pipeline running on start
• Add keep_running parameter to pipeline config
• Safety shutdown pipeline and entry service
• Support more complex routing pattern rules

1.2.0 #

Features #

• Support alias in bulk_reshuffle filter.
• Support truncate in request_logging filter.
• Handle 429 retry in json_indexing service.
• Add forcemerge service.
• Add response_body_regex_replace filter.
• Add request_body_regex_replace filter.
• Add sleep filter.
• Add option to log slow requests only.
• Add cluster and bulk status to request logging.
• Add filter_v2 and support _ctx to access request context.
• Add dump_context filter.
• Add translog filter, support rotation and compression.
• Add set_response filter.
• Add set_request_header filter.
• Add set_hostname filter.
• Add set_basic_auth filter.
• Add set_response_header filter.
• Add elasticsearch_health_check filter.
• Add drop filter.

Bug fix #

• Fix truncate body filter, correctly resize the body bytes.
• Fix cache filter.
• Fix floating_ip module.
• Fix dirty write in diskqueue.
• Fix compression enabled requests.
• Fix date_range_precision_tuning filter.
• Fix invalid indices status on closed indices #23.
• Fix document hash for elasticsearch 6.x.
• Fix floating_ip feature run with daemon mode.
• Fix async bulk to work with beats.

Improvements #

• Optimize memory usage, fix memory leak.

Acknowledgement #

Thanks to the following enterprises and teams #

• China Everbright Bank, China Citic Bank, BSG, Yogoo

Thanks to the following individual contributors #

• MaQianghua, YangFan, Tanzi, FangLi

1.1.0 #

• Request Logging and Dashboard.
• Support ARM Platform [armv5\v6\v7\v8(arm64)].
• Fix Elasticsearch Nodes Auto Discovery.
• Add Request Header Filter.
• Add Request Method Filter.
• Add Sample Filter.
• Request Logging Performance Optimized (100x speedup).
• Add Request Path Filter.
• Add Debug Filter.
• Add User Info to Logging Message.
• Support Routing Partial Traffic to Specify Processing Flow (by Ratio).
• Support Traffic Clone, Support Dual-Write or 1:N Write.
• Elasticsearch topology auto discovery, support filter by nodes,tags,roles.
• Backend failure auto detection, auto retry and select another available endpoint.
• Floating IP feature ready to use.
• Add bulk_reshuffle filter.

1.0.0 #

• Rewritten for performance
• Index level request throttle
• Request caching
• Kibana MAGIC speedup
• Upstream auto discovery
• Weighted upstream selections
• Max connection limit per upstream