Kubernetes DNS Traffic Analysis with Hubble: Custom Filters and Visualization Techniques

Hubble, the observability tool for Cilium, provides deep visibility into the network traffic within your Kubernetes clusters. By leveraging Hubble's capabilities, you can monitor DNS queries and responses, identify bottlenecks, and detect potential security threats. This article guides you through writing custom filters and visualizing DNS traffic data using Hubble.

Prerequisites

• A running Kubernetes cluster with Cilium and Hubble installed.
• kubectl configured to interact with your cluster.
• Basic understanding of Kubernetes networking and DNS.

Step 1: Understanding Hubble's Architecture

Hubble captures network traffic data as Flows, which are then processed and made available for querying and visualization. These Flows contain information about the source and destination of traffic, protocol, and other metadata. To effectively monitor DNS traffic, you need to understand how DNS queries and responses are represented as Flows within Hubble.

Step 2: Writing Custom Hubble Filters for DNS Traffic

Hubble filters allow you to select specific Flows based on criteria such as source and destination IP addresses, ports, and protocol. To monitor DNS traffic, you can create filters that target DNS port 53.

Using cilium CLI for Filtering

The cilium CLI provides a convenient way to interact with Hubble. You can use it to filter and display DNS-related Flows.

cilium hubble observe --protocol dns

This command filters and displays Flows with the DNS protocol. However, for more granular control, you can define custom filters using regular expressions.

Creating Custom Filters with Regular Expressions

Hubble supports filtering based on regular expressions applied to Flow properties. For DNS monitoring, you can filter based on the DNS query name.

1. Identify DNS Query Field: Determine which field in the Flow data contains the DNS query name. This typically involves inspecting the Flow data using cilium hubble observe without any filters to see the structure of the Flow.
2. Construct Regular Expression: Create a regular expression to match the DNS query names you are interested in. For example, to match queries for example.com, you can use the regex .*example\.com.*.
3. Apply the Filter: Use the cilium hubble observe command with the --fqdn option to apply the filter.

cilium hubble observe --fqdn '.*example\.com.*'

This command filters Flows where the FQDN (Fully Qualified Domain Name) matches the provided regular expression. Remember to escape special characters in the regex.

Example: Filtering DNS Queries for Specific Domains

Let's say you want to monitor DNS queries for api.example.com and db.example.com. You can combine these into a single regular expression:

cilium hubble observe --fqdn '.*(api|db)\.example\.com.*'

This command filters Flows that contain DNS queries for either api.example.com or db.example.com.

Step 3: Visualizing DNS Traffic Data

Hubble provides several options for visualizing network traffic data.

Hubble UI

The Hubble UI offers a graphical representation of Flows, allowing you to explore network traffic visually. You can filter Flows based on various criteria, including DNS queries and responses.

1. Access Hubble UI: Access the Hubble UI through the Cilium CLI or your Kubernetes dashboard.
2. Apply Filters: Use the UI's filtering options to select DNS traffic. You can filter by namespace, pod, and DNS query name.
3. Visualize Flows: Observe the graphical representation of Flows, highlighting DNS queries and responses. The UI allows you to see the latency and the direction of the traffic.

Grafana Integration

Hubble integrates with Grafana, a popular open-source data visualization tool. By setting up Grafana with Hubble, you can create custom dashboards to monitor DNS traffic metrics.

1. Install Grafana: Install Grafana in your Kubernetes cluster or use a managed Grafana service.
2. Configure Hubble as a Data Source: Add Hubble as a data source in Grafana, providing the necessary connection details.
3. Create Dashboards: Create Grafana dashboards to visualize DNS traffic metrics. You can use queries to extract data such as:
   • Number of DNS queries per domain.
   • DNS query latency.
   • DNS response codes.
   • Source and destination of DNS traffic.

Example Grafana Query

To visualize the number of DNS queries for a specific domain, you can use a PromQL query like this:

sum(rate(hubble_flows_dns_queries_total{fqdn="example.com"}[5m]))

This query calculates the rate of DNS queries for example.com over a 5-minute window.

Using jq for Custom Analysis

For advanced analysis and custom visualizations, you can use jq to process the JSON output from cilium hubble observe. This allows you to extract specific data points and create custom reports.

1. Capture Hubble Output: Capture the JSON output from cilium hubble observe.

cilium hubble observe --fqdn '.example.com.' -o json > dns_traffic.json
```

2. Process with jq: Use jq to extract relevant fields from the JSON data.

jq '.[] | {timestamp: .time, source: .source.labels, destination: .destination.labels, fqdn: .l7.dns.query}' dns_traffic.json
```

This command extracts the timestamp, source labels, destination labels, and FQDN from each Flow.

3. Create Custom Reports: Use the processed data to generate custom reports or visualizations using scripting languages like Python or tools like gnuplot.

Step 4: Identifying Bottlenecks and Security Threats

By visualizing DNS traffic data, you can identify potential bottlenecks and security threats.

Bottleneck Detection

• High DNS Latency: Identify domains with high DNS query latency, indicating potential DNS server issues or network congestion.
• Excessive DNS Queries: Detect excessive DNS queries from specific pods or namespaces, suggesting potential application misconfiguration or denial-of-service attacks.

Security Threat Detection

• Unauthorized DNS Queries: Identify DNS queries to domains that are not authorized by your security policies, indicating potential malware activity or data exfiltration.
• DNS Tunneling: Detect DNS tunneling attempts by analyzing the size and frequency of DNS queries.
• Suspicious Domains: Identify DNS queries to suspicious or known malicious domains.

Best Practices

• Use Regular Expressions Carefully: Ensure that your regular expressions are optimized for performance and do not cause excessive CPU usage.
• Monitor Resource Usage: Monitor the resource usage of Hubble and Grafana to ensure they are not impacting the performance of your Kubernetes cluster.
• Implement Alerting: Set up alerts based on DNS traffic metrics to notify you of potential issues or security threats.
• Secure Hubble: Secure access to Hubble and Grafana to prevent unauthorized access to network traffic data.

Conclusion

Monitoring DNS traffic with Hubble provides valuable insights into the network behavior of your Kubernetes clusters. By writing custom filters and visualizing DNS data, you can identify bottlenecks, detect security threats, and improve the overall performance and security of your applications. This guide provides a starting point for implementing DNS traffic analysis with Hubble, and you can customize these techniques to meet the specific needs of your environment.