Integrating AWS Data Firehose with NetObserv for Enhanced Collection of VPC Flow Logs

Introduction

As network engineers, ensuring seamless and efficient data flow is crucial for maintaining robust network observability. ElastiFlow’s NetObserv now supports the integration of Amazon Data Firehose (formerly Amazon Kinesis Data Firehose), offering a powerful method for importing VPC & Transit Gateway Flow Logs. This blog will guide you through the setup process and explain the benefits of using Amazon Data Firehose for sending VPC Flow Logs to NetObserv.

Why Use Amazon Data Firehose?

Amazon Data Firehose provides a reliable and scalable way to collect, process, and deliver streaming data. For ElastiFlow customers, using Firehose to import VPC Flow Logs into NetObserv presents several advantages:

1. Multi-Region and Multi-Account Support: Managing access rights for multiple AWS accounts or regions to a single S3 bucket can be complex. Data Firehose simplifies this by accepting flow logs from multiple sources across different regions.

2. Near Real-Time Data Delivery: Unlike the standard 10 to 15-minute intervals for S3, Data Firehose can deliver data with a 1-minute aggregation interval, enabling more timely insights.

3. Efficient Data Management: Data Firehose handles data transformation and batch processing, reducing the overhead on your systems and simplifying data ingestion workflows.

Setting Up Amazon Data Firehose for VPC Flow Logs

Step 1: Prepare an S3 Bucket

Ensure you have an S3 bucket available to receive the flow logs. This bucket will be the destination for the data streamed via Data Firehose.

Step 2: Create a Amazon Data Firehose Stream

1. Create a Stream: In the AWS Management Console, navigate to Data Firehose and create a new delivery stream with a Direct PUT source.

2. Set Destination: Configure the destination to your S3 bucket.

   

3. Configure Line Delimiter: While optional, disabling the new line delimiter setting is recommended to save space.

4. Set S3 Bucket Prefix: Use an appropriate prefix for easy identification and integration with NetObserv. This prefix should match the EF_AWS_VPC_FLOW_LOG_S3_PREFIX configuration setting in NetObserv.

   

Step 3: Configure VPC Flow Logs

1. Create Flow Log Output: In the VPC settings, create a new Flow Log and set the destination to the Data Firehose stream created in the previous step.

   

2. Copy Log Format: Ensure you use a consistent or the expected log format and copy this format for later use in NetObserv configuration.

   

Integrating with NetObserv

Step 1: Configure an S3 collector

You will need to ensure you have an S3 collector configured. If you are already using an S3 collector, you can skip this step. At a minimum, you will need to make sure that:

• EF_AWS_VPC_FLOW_LOG_S3_ENABLE is set to true, that

• EF_AWS_VPC_FLOW_LOG_S3_BUCKET is set to the S3 bucket you created, and that

• EF_AWS_VPC_FLOW_LOG_S3_PREFIX is set to the correct prefix. 

For additional information see our docs site. 

Step 2: Enable Data Firehose in the S3 Collector

Set the EF_AWS_VPC_FLOW_LOG_FIREHOSE_S3_ENABLE configuration to true. This informs NetObserv that the incoming data format is from Firehose rather than a direct VPC to S3 transfer.

Step 3: Set the log format for the VPC flow logs

EF_AWS_VPC_FLOW_LOG_FIREHOSE_S3_LOG_FORMAT: If using a custom log format, set this variable to match the log format used in your VPC Flow Logs. This ensures proper parsing and ingestion of flow log records.

Benefits of This Integration

Using Amazon Data Firehose to import VPC Flow Logs into NetObserv provides several key benefits for ElastiFlow customers:

1. Simplified Multi-Region Data Collection: Easily collect and centralize flow logs from multiple AWS regions and accounts without complex access management.

2. Enhanced Data Freshness: Receive and analyze near real-time data, improving the responsiveness and accuracy of network monitoring and threat detection.

3. Optimized Data Handling: Benefit from the efficient data transformation and batch processing capabilities of Data Firehose, reducing system overhead and simplifying data workflows.

Conclusion

Integrating AWS Data Firehose with ElastiFlow’s NetObserv enhances your network observability and security posture by providing a streamlined, efficient, and near real-time method for importing VPC Flow Logs. By following the setup guide outlined above, network engineers can leverage these capabilities to achieve superior insights and maintain robust network monitoring.

For more detailed documentation and support, visit the ElastiFlow website and start exploring the enhanced capabilities of NetObserv today.