Beyond Zeek: How ElastiFlow and NetQuest Deliver Zeek-Level Insight Through Enhanced Flow Data

Using NetQuest with ElasticFlow, companies can obtain data directly from high-speed network links, essential for generating feature-rich NetFlow data enriched with security insights. While typical routers, switches, and firewalls provide data useful for network operations and volumetric DDOS attacks, security teams often lack the depth needed for detailed analysis like a Zeek log. By capturing traffic in real-time, NetQuest and ElastiFlow deliver a comprehensive view of network behavior, enabling detailed analysis of communication patterns and detailed DNS, HTTP, and SSL information. This allows for identifying unusual or suspicious activity, such as unauthorized access attempts, malware attacks, lateral movement, or anomalous traffic patterns. The continuous data flow ensures real-time monitoring and quick responses to emerging security incidents. NetQuest further enhances flow data with fields vital to network security, similar to Zeek logs. Partnered with NetQuest, Elastiflow provides the visibility companies need while efficiently handling unsampled 1:1 flow data at large flows per second (FPS), equipping security teams with all the data necessary to investigate incidents and resolve them quickly. This blog explores the similarities between NetFlow data from NetQuest and Zeek logs, offering a deeper comparison of the two technologies. In many ways, the choice of which data format to deploy is driven by what the upstream analytics platform can consume, existing workflows, and existing processes for network traffic analysis.  

Why Flow is [As Good as, or Better] than Zeek for Network Security Monitoring with NetQuest and ElastiFlow enrichment

Background – What is Flow Data

Flow data, including NetFlow and IPFIX, provides a high-level abstraction of network traffic by capturing metadata about each packet traversing a network. Network flow data is a record of device communications that includes information about connections made over a network. This data is usually collected and sent by network devices and can include information such as Source and destination IP addresses, Ports, Protocols, Timestamps, Traffic volume, and Network interfaces. Unsampled Flow data is crucial for detailed network monitoring, offering a complete picture without sacrificing fidelity.

While flow data from routers, switches, and firewalls may not include the level of detail provided with Zeek, similar fields are included for flow using NetQuest. In addition, ElastiFlow enriches the flow records further to include valuable information for Geo-IP, BGP ASN, DNS, user-defined metadata, application name, and threat Intelligence, giving users even greater insight. 

Flow Data for Network Monitoring:

• Comprehensive Data: Captures every packet and byte, providing full visibility into network activity.

• Enrichment: To offer deeper insights, additional context, such as Threat-Intel, Geo IP, BGP ASN, application, and session-level data, can be enhanced.

• Efficiency: Flow data is a lightweight alternative to full packet capture, making it easier to collect, store, and analyze.

• Enhanced Flow: Probe vendors can enhance flow records with details similar to those of Zeek.

• Complete Visibility: Flow data can be generated from probes, routers, switches, and firewalls, making visibility north-south and east-west simple. 

Background – What is Zeek

Zeek, formerly known as Bro, is an open-source network analysis framework that focuses on deep packet inspection for detailed security monitoring. It generates rich logs that describe network activities, including HTTP sessions, DNS queries, and SSL handshakes.

Zeek for Network Monitoring:

• Detailed Logs: Provides extensive logs that can help identify complex security threats.

• Specialization: Primarily designed for security monitoring rather than broader network management.

• The Limitations of Zeek

• Cost to Generate: Requires significant computational resources, leading to higher operational costs.

• Availability: Needs specialized sensors, limiting its accessibility.

• Scalability: Deep packet inspection demands substantial processing power, challenging its scalability in large environments.

• Integration Complexity: Integrating Zeek data with upstream platforms can be resource-intensive.

• Deployability: Zeek sensors need to see packet data, making deployment challenging in large environments where north-south and east-west visibility is required. 

• Encrypted Traffic: Zeek rules perform matches on packet payload to identify threats. With encrypted traffic, these rules are no longer useful, limiting the product's usefulness. 

NetQuest Flow vs. Zeek: Comprehensive Comparison

Extensibility and Versatility

Flow data is ideal for security monitoring and valuable for other types of network monitoring. Its extensibility allows it to be leveraged across multiple platforms and use cases, offering a better return on investment.

Detecting Anomalous Traffic and Network Security Threats

Both Flow data and Zeek can detect anomalies. However, Flow's ability to scale efficiently and provide comprehensive coverage makes it a more versatile solution for detecting network security threats. 

Identifying Applications and User Conversations

Enriched Flow data offers application-level and session-level context, making it comparable to Zeek in identifying applications and user conversations. This enrichment allows for detailed traffic analysis without needing full packet inspection.

Extracting Session-Level Context

Both technologies excel in extracting session-level context, but Flow data's scalability and efficiency give it an edge in larger environments.

Handling Encrypted Traffic

While Zeek's deep packet inspection can provide insights into encrypted traffic, Flow data can offer valuable metadata for identifying encrypted sessions and monitoring network health.

The Value of Unsampled Flow Data

Sampling data might save resources but at the cost of missing critical details. Unsampled Flow data ensures no packet is overlooked, providing a complete and accurate picture of network activity.

Advantages of Unsampled Flow Data:

• Granularity: Captures every detail, making detecting and analyzing anomalies easier.

• Full Visibility: A comprehensive view of network traffic eliminates blind spots.

• Efficient Storage: Integration with advanced data compression technologies like Elasticsearch's TSDS ensures efficient storage, reducing the need for trade-offs.

Real-World Applications

ElastiFlow's Complete Network Insight Revolution:

• Financial Services: High-speed transactions demand precise, real-time monitoring.

• Healthcare: Critical data flows require comprehensive monitoring to ensure patient safety.

• Telecom: Managing petabytes of data necessitates detailed visibility to prevent service disruptions.

ElastiFlow provides the tools to monitor, analyze, and secure networks effectively. By capturing 1:1 unsampled Flow data, ElastiFlow eliminates the compromises of traditional sampling, offering unparalleled insight and control.

Fields comparison between NetQuest flow and Zeek logs:

To compare the flow field output from NetQuest to Zeek fields, we need to map the respective fields from each system and understand their formats and structures. Here's a basic comparison of typical flow field outputs from NetQuest and Zeek (formerly Bro).

Comparison

Notes

• NetQuest may combine some fields Zeek separates into originator and responder specifics, such as orig_pkts and resp_pkts, vs. a single Packets field.

• Zeek provides additional context, such as conn_state and history, that may not be present in NetQuest flow data.

• NetQuest's Bytes and Packets fields represent aggregate values, while Zeek breaks them down by originator and responder.

To analyze the advanced fields between NetQuest SNS1000 IPFIX templates and Zeek, we will focus on NetQuest's enriched Layer 7 protocol-specific fields and compare them to Zeek's detailed logging. 

A table compares the advanced DNS, HTTP, and TLS/SSL fields between NetQuest SNS1000 and Zeek.

Comparison Table: Advanced Fields

Notes

• Fields marked with "-" indicate no direct equivalent in the other system.

• NetQuest includes additional fields for HTTP and TLS/SSL not found in Zeek, such as http_proxied, ssl_session_id, ssl_certificate_subject_key_size, ssl_cert_hash, ssl_ja3, and ssl_ja3s.

• Zeek provides connection-specific information such as uid and detailed response information in DNS logs, which might not be directly included in NetQuest fields.

In conclusion, NetQuest, when used with Elastiflow, offers a powerful solution for obtaining and enriching high-speed flow data, addressing the needs of both network and security teams. The combination enables real-time traffic capture and detailed flow analysis, providing comprehensive visibility into network behavior. While traditional network devices like routers and firewalls generate useful flow data, they often lack the detail required for robust security monitoring. With NetQuest, unsampled flow data is enriched to include security-relevant insights into DNS, HTTP, and SSL transactions. In contrast, ElastiFlow further enriches the records to include Geo-IP, BGP ASN, Threat-Intel, User-defined metadata, and App-ID and App-Name, which can reveal anomalous or malicious activity patterns.

Moreover, the enriched flow data can rival the detailed analysis provided by Zeek. Flow data's efficiency, scalability, and ability to handle encrypted traffic make it a versatile and cost-effective alternative to Zeek in many large-scale network environments. A detailed comparison of flow and Zeek fields shows that the choice often depends on specific organizational needs, infrastructure capabilities, and analytics platforms. For many use cases, flow data enriched by NetQuest and Elastiflow proves to be as good as or better than Zeek, offering comprehensive network visibility and enriched security insights with greater efficiency.