How to trace network traffic of the OpenShift and Kubernetes in AWS

I will walk you through how to trace network traffic and how to capture the traffic in AWS for OpenShift and Kubernetes here.

The test request is sent by a client(JMeter) with “request-id” header, I will filter the tcp packets with the “request-id” header from tcpdump pcap files we captured on the node host. And I will compare it with ELB access logs and Flow logs in the VPC after that. I hope it’s helpful for understanding about your traffic analysis in Kubernetes platform on AWS. Look the above diagram, it’s helpful to figure out traffic flow in this test. This time CloudFront was not in the access path, but you can trace it with the same ways using CloudFront access logs and request ID.

Capture the traffic on the nodes running Ingress Router pods using tcpdump

Check the node running Ingress Router pods as follows.

Run the tcpdump on the node hosts through “oc debug node/<nodename>” or SSH. This time I tested it on OCPv4, so I will run tcpdump through oc CLI as follows.

Create a S3 bucket to store ELB access logs and Flow logs

Before enabling the AWS logging feature, we should create a S3 bucket first.

Put the bucket policy to be relaxed the permission for writing logs.

Enable the ELB access logs

We can enable the ELB access logs using annotations of the LoadBalancer Service simply. Refer ELB Access Logs on AWS for more details.

Enable the Flow logs on the VPC

We can specify the same S3 bucket ARN which is used in ELB access logs.

Run the JMeter client for a while enough to test !

Gather the tcpdump pcap files from the debug pods

We can download the ELB access logs and VPC Flow logs from our S3 bucket simply, but the tcpdump is more work than it. I’d like to show you the details.

Get the pcap files which is included a specific “request-id” traffic using “oc copy” after stopping tcpdump cmd in the debug pods.

Let’s analyze the traffic now

Filter the specific request session using “request-id: abcdefghijk” header in this test.

Comparison the traffic with the captured ELB access logs and VPC Flow logs

Here is all TCP packets in a session for “request-id: abcdefghijk”.

Filter the VPC Flow logs with the matched TCP port number. We can see additional information a view of AWS, such as the “action column”, we can check if the traffic was dropped by security groups.

Finally, check the ELB access logs matched with “request-id”, it show us the client source IP and the ELB internal IP when a specific request was received. And we can also know the ELB’s global IP from the log file name.

Conclusion

Throughout this test, we could verify the ELB access logs and the VPC Flow logs details. TCP packet provides more details of the traffic, but AWS logs provides informations related with AWS platform. If we can collect all related logs like this test for a specific request having and issue, we may gather more clue to troubleshoot in the OpenShift and Kubernetes on AWS environments.

Thank you for reading.