Go TLS tracing not working on Amazon Linux 2023

Describe the bug
Go TLS tracing does not work on Amazon Linux 2023.

To Reproduce
Provision EKS cluster (v1.30) with two node groups, one running AL2, one running AL2023. Deploy pixie. Run a Go (1.21 or 1.22) app (pod/deployment) that's making outbound HTTPS requests (HTTP 1.1). Go to px/http_data and try and observe said outbound traffic. Only traffic initiated by the pods running on AL2 node group would appear.

Expected behavior
Be able to see traffic initiated by pods scheduled on node group running AL2023.

Logs
Please attach the logs by running the following command:

./px collect-logs

App information (please complete the following information):

Pixie version: 0.14.10
K8s cluster version: EKS 1.30
Node Kernel version: 6.1.97-104.177.amzn2023.x86_64
Browser version: -

pem-AL2.log
pem-al2023.log

@SamuraiPrinciple

@SamuraiPrinciple thanks for reporting this. I'm aiming to try to reproduce the issue later this week and will report back when I have more information.

@ddelnano

Hi @ddelnano - many thanks for the quick reply, it really is appreciated. Do let me know if I could be of any help (more logs, screenshare, ...).

Take care and wish you a great day.

This is still on my radar. I'm expecting to have time to look into this at the end of the week.

That's great, thanks for the update. I've tried with the latest pixie version (0.14.11) and it's still not working.

@ddelnano

Hi @ddelnano hope all is well :) Wondering if you've managed to reproduce the issue? Happy to provide more details (EKS AMIs, pixie deployment config and sample go app) in case you didn't manage to get a repro...

Many thanks,
damjan

@SamuraiPrinciple

@SamuraiPrinciple sorry for missing to follow up on this! I'll be giving this some attention tomorrow or Thursday.

No, no, no problem at all - just let me know if you're struggling to reproduce it, very happy to make it as easy as possible for you.

Thanks for the reply and your hard work!

@SamuraiPrinciple

Since our existing Go tls tracing test hasn't been updated for Go 1.22 yet, I wanted to rule out any obvious incompatibility. We will eventually have that version under test, but until that is upstream I've created this branch which adds coverage for Go 1.22. The test passes on that branch, which is a good sign. My next step is to reproduce the issue on EKS w/ AL2023.

@SamuraiPrinciple while I haven't tried the EKS steps yet, can you able to share an example Go application that exhibits the problem and the compiler version?

Hi, thanks for picking this up! I will provide you with the sample app later, but just as quick note - if I rotate EKS nodes so that they use AL2 AMIs, the tracing then works with everything else being exactly the same.

No problem! Looking forward to seeing the example app as I didn't have luck with the reproduction on EKS.

https://github.com/samuraiprinciple/pixie-repro

The repo contains a simple go program that polls https://example.com as well as a Dockerfile that can be used to build an image. There is also a very simple k8s deployment manifest that runs two containers (one for HTTP 1.1 and the other for HTTP/2).

Thanks for the quick response. Does the HTTP/2 case work on AL2? Pixie's HTTP/2 tracing should only work for GRPC and when Go binaries have debug symbols (docs).

As for the HTTP 1.1 case, can you explain more on the success case (when running on AL2)? I crafted a sample application very similar to yours and didn't see any traffic for AL2 or AL2023. I didn't consider my case a reproduction since I thought it was more similar to #899, which was never fully understood.

I'll be able to continue debugging this on my end, but I'd still like to understand the AL2 case to help narrow down where the problem might be.

Hi, mega-thanks for the quick reply! To be honest, we're not using HTTP/2 (i.e. we're actively disabling it - long story, but it's problematic for various reasons, unrelated to pixie), so I've only included that 'just in case'. I can confirm that on AL2 I can indeed see payloads for outbound TLS requests (HTTP 1.1).

For your AL2 environment, can you run the PEM with --stirling_conn_trace_pid=${pid_of_working_app} and upload the PEM log to this issue?

You can add that arg to the DaemonSet even though the PID won't match on the other instances. Just make sure to grab the logs from the instance that has the application with the PID.

Hi! Many thanks for the investigation and the info. I can confirm that installing linux headers package indeed resolves the issue. The HTTP 1.1 tls connections can be traced (I have noticed that gzipped responses are not rendered in UI - "resp_body: <removed: non-text content-type>,"). Happy to close this issue.

No problem, sorry it took some time for me to dig into this and greatly appreciate your help throughout the process!

I'm hopeful that surfacing the lack of linux headers in Pixie's diagnostic tools will help uncover and fix these problems before they become month long bugs! That tooling should be making its way in a release in the next few weeks (across vizier, operator, cli, etc).

Glad I could be useful :) and many, many thanks for your help with this!

Use `px/agent_status_diagnostics` script within px cli to detect miss…

Navigation Menu

Repository navigation

Go TLS tracing not working on Amazon Linux 2023 #1986

6 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Go TLS tracing not working on Amazon Linux 2023 #1986

Description

Activity

ddelnano commented on Aug 20, 2024

SamuraiPrinciple commented on Aug 20, 2024

ddelnano commented on Aug 28, 2024

SamuraiPrinciple commented on Aug 28, 2024

SamuraiPrinciple commented on Dec 9, 2024

ddelnano commented on Dec 10, 2024

SamuraiPrinciple commented on Dec 10, 2024

ddelnano commented on Dec 12, 2024

SamuraiPrinciple commented on Dec 12, 2024

ddelnano commented on Dec 12, 2024

SamuraiPrinciple commented on Dec 12, 2024

SamuraiPrinciple commented on Dec 12, 2024

ddelnano commented on Dec 12, 2024

SamuraiPrinciple commented on Dec 12, 2024

ddelnano commented on Dec 12, 2024

6 remaining items

SamuraiPrinciple commented on Dec 14, 2024

ddelnano commented on Dec 17, 2024

SamuraiPrinciple commented on Dec 18, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions