Introducing ElastiFlow support for TSDS - Save up to 70% on Flow Data Storage Costs

“Complete Network Observability” is a top priority for most network teams. Network engineers need deep visibility and explorability into their networks to stay ahead of rising complexity and mounting security threats. This kind of observability usually comes with a trade-off between the desired depth of visibility and the cost to ingest and store vast amounts of flow data.

Many network observability solutions “solve” this cost problem by sampling flow data. This means they discard 70-99% of what could be valuable flow information to reduce storage costs. At ElastiFlow we don’t just talk about “Complete Network Observability”, we continuously work on ways to make this a reality for network engineers.

We are pleased to announce the release of ElastiFlow Unified Flow Collector 6.4, which includes the general availability release of the Time Series Data Streams (TSDS) output for Elasticsearch. With the release of TSDS support, we take a big step towards affordable complete network observability. Our customers using TSDS support during the technical preview saw a reduction in required storage capacity between 50 and 70%.

Why complete Network Observability requires unsampled flow data

Collecting unsampled flow data is crucial for gaining a comprehensive understanding of network traffic and ensuring effective network management and security. Unsampled flow data provides a representation of every packet that traverses a network, capturing information about the source and destination, protocol types, and other key attributes, unlike sampled flow data, which only captures a subset of packets at specific intervals.

One of the primary advantages of unsampled flow data is its ability to reveal subtle patterns and anomalies in network behavior. It gives network administrators the ability to detect and analyze irregularities, identify potential security threats, and troubleshoot performance issues precisely. Unsampled flow data is especially valuable in identifying low-frequency, high-impact events that may go unnoticed in sampled data. Moreover, unsampled flow data is essential for in-depth forensic analysis and compliance monitoring. This level of detail is crucial to meet the increasingly onerous regulatory compliance requirements - read this ElastiFlow blog post for more details. 

Collecting unsampled flow data is indispensable for complete Network Observability but potentially, it can add challenges in managing the volume and expense of the additional data - enter Elasticsearch TSDS.

How TSDS works

In version 8.7 (released Mar 30, 2023) ElastiFlow released a Time Series Database (TSDB) for Elasticsearch (Link to their release blog), which promises to reduce the storage requirements of time series metrics data by up to 70%. The ElastiFlow team developed 3 features specifically to streamline time series data:

1. Time series data streams (TSDS) leverage specific organization of time series data (e.g., by timestamp and dimension fields) to sort and store metric data more efficiently.

2. Synthetic source reduces storage space by not saving the original document data in _source, and instead reconstructing it from doc_values when needed. When combining the savings from TSDS and synthetic source, we reached storage savings of 55 to 70% depending on the kind of traffic observed.

3. Downsampling reduces the footprint of time series by storing them at lower granularity, giving you greater control over how much historical metric data you store. And since downsampling “just works” in Kibana with 8.7, it also speeds up dashboard visualizations. Note, downsampling needs to be configured.

How we adapted TSDS for Flow Records

After the release of Elasticsearch 8.7 we immediately started researching ways to use TSDS for flow data, which is not typically a time series data source. The primary challenge was defining the routing path, when the content of each flow record can vary between network device types and configurations. A poor choice could result in “hot spot” shards or nodes, limiting throughput and performance.

To create such a protocol-specific routing path value, we developed routing path generation algorithms for common network protocols. The collector uses these algorithms to generate a single hash value that is assigned as the routing path. This resulted in the ElastiFlow Collector generating massive storage savings without sacrificing the valuable information our users gain from being able to look at every flow.

How Well Does it Work?

To give an idea of the storage savings, let’s look into a typical customer scenario. This customer has an average flow rate of 10,000 flows per second. This results in roughly 864 million flows being stored per day. To calculate cost savings we’ll assume a conservative cost of storage of 185 $ per TB (your actual cost will likely be a bit higher).

We have customers who are saving 16,000$ / month in storage cost in their production environment since they have enabled TSDS during our tech preview of the feature.

Conclusion

If you are looking for a more comprehensive Network Observability solution, we’d love to talk. Getting started with ElastiFlow takes minutes and is free for up to 4,000 flow records per second. We also offer a 30-day free trial. More detailed pricing and support platforms are available on our website Subscription page. Thanks for reading!