-
Notifications
You must be signed in to change notification settings - Fork 61
Open
Labels
bugSomething isn't workingSomething isn't working
Description
What happened?
Latest kernel headers have been installed on multiple AlmaLinux 9.5 hosts.
Host A
$ uname -a
Linux hostA 5.14.0-503.23.2.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC x86_64 x86_64 x86_64 GNU/Linux
$ dnf list installed | egrep -i kernel
kernel-headers.x86_64 5.14.0-503.23.2.el9_5 @appstream
kernel-devel.x86_64 5.14.0-503.23.2.el9_5 @appstream
Host B
$ uname -a
Linux hostB 5.14.0-503.14.1.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC x86_64 x86_64 x86_64 GNU/Linux
$ dnf list installed | egrep -i kernel
kernel-headers.x86_64 5.14.0-503.23.1.el9_5 @appstream
kernel-devel.x86_64 5.14.0-503.14.1.el9_5 @appstream
Log from the kernel-collector pod running on Host A
resolving kernel headers...
cleaning up stale kprobes...
launching kernel collector...
+ exec /srv/kernel-collector --host-distro unknown --kernel-headers-source pre_installed --config-file=/etc/network-explorer/config.yaml --disable-nomad-metadata --log-console --info
2025-02-18 12:17:23.334641+00:00 info [p:1694674 t:1694674] Starting Kernel Collector version 0.10.2 (release)
2025-02-18 12:17:23.334645+00:00 info [p:1694674 t:1694674] Kernel Collector agent ID is FAIDLOGL7BWGAVS5Y3U3DE471BO6REDKDZ11
2025-02-18 12:17:23.334647+00:00 info [p:1694674 t:1694674] Running on:
sysname: Linux
nodename: fw01
release: 5.14.0-503.23.2.el9_5.x86_64
version: #1 SMP PREEMPT_DYNAMIC Wed Feb 12 05:52:18 EST 2025
machine: x86_64
2025-02-18 12:17:23.334661+00:00 info [p:1694674 t:1694674] HTTP Metrics: Enabled
2025-02-18 12:17:23.334662+00:00 info [p:1694674 t:1694674] Socket stats interval in seconds: 10
2025-02-18 12:17:23.334662+00:00 info [p:1694674 t:1694674] Userland TCP: Disabled
2025-02-18 12:17:24.344563+00:00 info [p:1694674 t:1694674] Kernel Collector version 0.10.2 (release) started on host fw01
2025-02-18 12:17:24.344632+00:00 info [p:1694674 t:1694674] Node label has been set in config: 'environment':'pluto'
2025-02-18 12:17:30.358637+00:00 info [p:1694674 t:1694674] connecting to opentelemetry-ebpf-reducer:7000 (binary)...
In file included from ../../../src/collector/kernel/bpf_src/render_bpf.c:39:
In file included from include/net/tcp.h:35:
In file included from include/net/sock_reuseport.h:5:
In file included from include/linux/filter.h:9:
include/linux/bpf.h:350:10: error: invalid application of 'sizeof' to an incomplete type 'struct bpf_rb_root'
return sizeof(struct bpf_rb_root);
^ ~~~~~~~~~~~~~~~~~~~~
include/linux/bpf.h:350:24: note: forward declaration of 'struct bpf_rb_root'
return sizeof(struct bpf_rb_root);
^
include/linux/bpf.h:352:10: error: invalid application of 'sizeof' to an incomplete type 'struct bpf_rb_node'
return sizeof(struct bpf_rb_node);
^ ~~~~~~~~~~~~~~~~~~~~
include/linux/bpf.h:352:24: note: forward declaration of 'struct bpf_rb_node'
return sizeof(struct bpf_rb_node);
^
include/linux/bpf.h:354:10: error: invalid application of 'sizeof' to an incomplete type 'struct bpf_refcount'
return sizeof(struct bpf_refcount);
^ ~~~~~~~~~~~~~~~~~~~~~
include/linux/bpf.h:354:24: note: forward declaration of 'struct bpf_refcount'
return sizeof(struct bpf_refcount);
^
include/linux/bpf.h:377:10: error: invalid application of '__alignof' to an incomplete type 'struct bpf_rb_root'
return __alignof__(struct bpf_rb_root);
^ ~~~~~~~~~~~~~~~~~~~~
include/linux/bpf.h:377:29: note: forward declaration of 'struct bpf_rb_root'
return __alignof__(struct bpf_rb_root);
^
include/linux/bpf.h:379:10: error: invalid application of '__alignof' to an incomplete type 'struct bpf_rb_node'
return __alignof__(struct bpf_rb_node);
^ ~~~~~~~~~~~~~~~~~~~~
include/linux/bpf.h:379:29: note: forward declaration of 'struct bpf_rb_node'
return __alignof__(struct bpf_rb_node);
^
include/linux/bpf.h:381:10: error: invalid application of '__alignof' to an incomplete type 'struct bpf_refcount'
return __alignof__(struct bpf_refcount);
^ ~~~~~~~~~~~~~~~~~~~~~
include/linux/bpf.h:381:29: note: forward declaration of 'struct bpf_refcount'
return __alignof__(struct bpf_refcount);
^
../../../src/collector/kernel/bpf_src/tcp-processor/bpf_tcp_send_recv.h:184:53: error: no member named 'iov' in 'struct iov_iter'
bpf_probe_read(&iov, sizeof(iov), &(msg->msg_iter.iov));
~~~~~~~~~~~~~ ^
../../../src/collector/kernel/bpf_src/tcp-processor/bpf_tcp_send_recv.h:393:53: error: no member named 'iov' in 'struct iov_iter'
bpf_probe_read(&iov, sizeof(iov), &(msg->msg_iter.iov));
~~~~~~~~~~~~~ ^
8 errors generated.
Failed to compile eBPF code for the Linux distro 'unknown' running kernel version 5.14.0-503.23.2.el9_5.x86_64.
troubleshoot item bpf_compilation_failed (os=Linux,flavor=unknown,headers_src=pre_installed,kernel=5.14.0-503.23.2.el9_5.x86_64): ProbeHandler couldn't load BPFModule: Success
This usually means that kernel headers weren't installed correctly.
Please reach out to support and include this log in its entirety so we can diagnose and fix
the problem.
In the meantime, please install kernel headers manually on each host before running
the Kernel Collector.
To manually install kernel headers, follow the instructions below:
- for Debian/Ubuntu based distros, run:
sudo apt-get install --yes "linux-headers-`uname -r`"
- for RedHat based distros like CentOS and Amazon Linux, run:
sudo yum install -y "kernel-devel-`uname -r`"
2025-02-18 12:17:34.163486+00:00 error [p:1694674 t:1694674] Cannot initialize BPF program, res=-1
2025-02-18 12:17:34.163578+00:00 error [p:1694674 t:1694674] Exception during BPFHandler initialization, closing connection: ProbeHandler couldn't load BPFModule: Success
2025-02-18 12:17:34.163588+00:00 error [p:1694674 t:1694674] troubleshoot item bpf_compilation_failed (os=Linux,flavor=unknown,headers_src=pre_installed,kernel=5.14.0-503.23.2.el9_5.x86_64): ProbeHandler couldn't load BPFModule: Success
Steps to Reproduce
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update open-telemetry
helm install opentelemetry-ebpf . -f ./values.yaml -n monitor
check logs of kernel collector pod
Expected Result
Metrics being sent from the kernel-collector to the reducer.
Actual Result
Errors in data collection.
eBPF Collector version
v0.10.2
Environment information
Environment
OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")
eBPF Collector configuration
k get cm opentelemetry-ebpf-config -n monitor -o yaml
apiVersion: v1
data:
config.yaml: |
labels:
environment: blue
kind: ConfigMap
metadata:
creationTimestamp: "2025-02-17T17:33:00Z"
labels:
app.kubernetes.io/instance: opentelemetry-ebpf
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: opentelemetry-ebpf
app.kubernetes.io/version: v0.10.2
argocd.argoproj.io/instance: opentelemetry-ebpf
helm.sh/chart: opentelemetry-ebpf-0.1.4
name: opentelemetry-ebpf-config
namespace: monitor
Log output
Additional context
No response
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working