ElastiFlow 6.3: Updated output default values for Elasticsearch and support for Elastic TSDS

6.3.0 - Elasticsearch output default values updated, new features including support for Elastic TSDS and OpenSearch AWS Sig v4 support.​

Breaking Changes​

• Elasticsearch Output: default option value changes

Beginning with ElastiFlow 6.3.0 the default values for the Elasticsearch output have been changed as follows.

1
end

1
collect

1
daily

1
rollover

• Kafka Output: default option value changes

Beginning with ElastiFlow 6.3.0 the default values for the Kafka output have been changed as follows. Performance testing has shown that this change can improve throughput.

1
0 

1
3 

1
1000

1
500

1
false

1
true

1
end

1
collect

• OpenSearch Output: default option value changes

Beginning with ElastiFlow 6.3.0 the default values for the OpenSearch output have been changed as follows.

1
end

1
collect

New Features​

• Elasticsearch Output: support for TSDS (TECHNOLOGY PREVIEW) - Support has been added to the Elasticsearch output for Time Series Data Streams (TSDS), introduced in Elasticsearch 8.7. Storing flow data using TSDS can result in a storage savings of 30-50% depending on the content of the flow records. TSDS also supports downsampling (initially for bytes and packets fields) which can result in even less storage capacity needed for historical data. Enabling TSDS does increase the ingest-related CPU load for Elasticsearch.

• OpenSearch Output: support for AWS Sig v4 - Support has been added for authentication via Sig v4. This is required when connecting to the AWS OpenSearch Serverless Service.

• Flow Processor: Juniper IFA - Support has been added for Juniper IFA records. The resulting IFA hop details are stored in the path index.

• YAML Configuration - The collector can now be configured via YAML files in addition to environment variables. The YAML file to be used can be specified using the -c or --config arguments. When both YAML and environment variables are set, environment variables will override the values from the YAML files.

Fixes​

• Flow Processor - Fixed a regression introduced in 6.2.2 which caused sample rates learned from option records to be ignored.

• Flow Processor - Fixed an issues which can cause a panic when a Netflow v9 packet contains excessive padding.

• Elasticsearch Output - Telemetry index templates are now created with the correct rollover alias.

• IPFIX IEs - Fixed Ixia AppID/Name values.

• HTTP-based Outputs - All HTTP-based outputs now set the Host header, as is required by some environments.

Updates​

• Flow UDP Input - Added 2055, 4739 and 6343 to default ports on which the input will listen.

• Flow Processor - Unsupported PEN-specific sFlow structures are now gracefully ignored, rather than rejecting the entire record.

• Flow Processor - Enrichment of network interface index values now supports SNMPv3.

• Flow Processor - Added ntop nDPI AppIDs to statically defined attribute values.

• Flow Processo - Added Viptela AppIDs to statically defined attribute values.

• IPFIX IEs - Added Versa Networks IEs

• IPFIX IEs - Added NetQuest SIP-related IEs

• IPFIX IEs - Added Ixia GTP-related IEs

Deprecations​

• While we have added support for configuration via YAML files in 6.3.0, the default method of configuration remains the use of environment variables set in the systemd unit file for the collector daemon. For example,  /etc/systemd/system/flowcoll.service.d/flowcoll.conf for the Unified Flow Collector binary flowcoll.

  In a future release, the default configuration method will be via YAML files, as described here.