Skip to content

Cannot initialize BPF program for Linux distro 'unknown' running kernel version 5.14.0-503.23.2.el9_5.x86_64 #293

@elaurensx

Description

@elaurensx

What happened?

Latest kernel headers have been installed on multiple AlmaLinux 9.5 hosts.

Host A

$ uname -a
Linux hostA 5.14.0-503.23.2.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC x86_64 x86_64 x86_64 GNU/Linux
$ dnf list installed | egrep -i kernel
kernel-headers.x86_64   5.14.0-503.23.2.el9_5   @appstream
kernel-devel.x86_64   5.14.0-503.23.2.el9_5   @appstream

Host B

$ uname -a
Linux hostB 5.14.0-503.14.1.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC x86_64 x86_64 x86_64 GNU/Linux
$ dnf list installed | egrep -i kernel
kernel-headers.x86_64   5.14.0-503.23.1.el9_5   @appstream
kernel-devel.x86_64   5.14.0-503.14.1.el9_5   @appstream

Log from the kernel-collector pod running on Host A

resolving kernel headers...
cleaning up stale kprobes...
launching kernel collector...
+ exec /srv/kernel-collector --host-distro unknown --kernel-headers-source pre_installed --config-file=/etc/network-explorer/config.yaml --disable-nomad-metadata --log-console --info
2025-02-18 12:17:23.334641+00:00 info [p:1694674 t:1694674] Starting Kernel Collector version 0.10.2 (release)
2025-02-18 12:17:23.334645+00:00 info [p:1694674 t:1694674] Kernel Collector agent ID is FAIDLOGL7BWGAVS5Y3U3DE471BO6REDKDZ11
2025-02-18 12:17:23.334647+00:00 info [p:1694674 t:1694674] Running on:
   sysname: Linux
  nodename: fw01
   release: 5.14.0-503.23.2.el9_5.x86_64
   version: #1 SMP PREEMPT_DYNAMIC Wed Feb 12 05:52:18 EST 2025
   machine: x86_64
2025-02-18 12:17:23.334661+00:00 info [p:1694674 t:1694674] HTTP Metrics: Enabled
2025-02-18 12:17:23.334662+00:00 info [p:1694674 t:1694674] Socket stats interval in seconds: 10
2025-02-18 12:17:23.334662+00:00 info [p:1694674 t:1694674] Userland TCP: Disabled
2025-02-18 12:17:24.344563+00:00 info [p:1694674 t:1694674] Kernel Collector version 0.10.2 (release) started on host fw01
2025-02-18 12:17:24.344632+00:00 info [p:1694674 t:1694674] Node label has been set in config: 'environment':'pluto'
2025-02-18 12:17:30.358637+00:00 info [p:1694674 t:1694674] connecting to opentelemetry-ebpf-reducer:7000 (binary)...
In file included from ../../../src/collector/kernel/bpf_src/render_bpf.c:39:
In file included from include/net/tcp.h:35:
In file included from include/net/sock_reuseport.h:5:
In file included from include/linux/filter.h:9:
include/linux/bpf.h:350:10: error: invalid application of 'sizeof' to an incomplete type 'struct bpf_rb_root'
                return sizeof(struct bpf_rb_root);
                       ^     ~~~~~~~~~~~~~~~~~~~~
include/linux/bpf.h:350:24: note: forward declaration of 'struct bpf_rb_root'
                return sizeof(struct bpf_rb_root);
                                     ^
include/linux/bpf.h:352:10: error: invalid application of 'sizeof' to an incomplete type 'struct bpf_rb_node'
                return sizeof(struct bpf_rb_node);
                       ^     ~~~~~~~~~~~~~~~~~~~~
include/linux/bpf.h:352:24: note: forward declaration of 'struct bpf_rb_node'
                return sizeof(struct bpf_rb_node);
                                     ^
include/linux/bpf.h:354:10: error: invalid application of 'sizeof' to an incomplete type 'struct bpf_refcount'
                return sizeof(struct bpf_refcount);
                       ^     ~~~~~~~~~~~~~~~~~~~~~
include/linux/bpf.h:354:24: note: forward declaration of 'struct bpf_refcount'
                return sizeof(struct bpf_refcount);
                                     ^
include/linux/bpf.h:377:10: error: invalid application of '__alignof' to an incomplete type 'struct bpf_rb_root'
                return __alignof__(struct bpf_rb_root);
                       ^          ~~~~~~~~~~~~~~~~~~~~
include/linux/bpf.h:377:29: note: forward declaration of 'struct bpf_rb_root'
                return __alignof__(struct bpf_rb_root);
                                          ^
include/linux/bpf.h:379:10: error: invalid application of '__alignof' to an incomplete type 'struct bpf_rb_node'
                return __alignof__(struct bpf_rb_node);
                       ^          ~~~~~~~~~~~~~~~~~~~~
include/linux/bpf.h:379:29: note: forward declaration of 'struct bpf_rb_node'
                return __alignof__(struct bpf_rb_node);
                                          ^
include/linux/bpf.h:381:10: error: invalid application of '__alignof' to an incomplete type 'struct bpf_refcount'
                return __alignof__(struct bpf_refcount);
                       ^          ~~~~~~~~~~~~~~~~~~~~~
include/linux/bpf.h:381:29: note: forward declaration of 'struct bpf_refcount'
                return __alignof__(struct bpf_refcount);
                                          ^
../../../src/collector/kernel/bpf_src/tcp-processor/bpf_tcp_send_recv.h:184:53: error: no member named 'iov' in 'struct iov_iter'
  bpf_probe_read(&iov, sizeof(iov), &(msg->msg_iter.iov));
                                      ~~~~~~~~~~~~~ ^
../../../src/collector/kernel/bpf_src/tcp-processor/bpf_tcp_send_recv.h:393:53: error: no member named 'iov' in 'struct iov_iter'
  bpf_probe_read(&iov, sizeof(iov), &(msg->msg_iter.iov));
                                      ~~~~~~~~~~~~~ ^
8 errors generated.

Failed to compile eBPF code for the Linux distro 'unknown' running kernel version 5.14.0-503.23.2.el9_5.x86_64.

troubleshoot item bpf_compilation_failed (os=Linux,flavor=unknown,headers_src=pre_installed,kernel=5.14.0-503.23.2.el9_5.x86_64): ProbeHandler couldn't load BPFModule: Success

This usually means that kernel headers weren't installed correctly.

Please reach out to support and include this log in its entirety so we can diagnose and fix
the problem.


In the meantime, please install kernel headers manually on each host before running
the Kernel Collector.

To manually install kernel headers, follow the instructions below:

  - for Debian/Ubuntu based distros, run:

      sudo apt-get install --yes "linux-headers-`uname -r`"

  - for RedHat based distros like CentOS and Amazon Linux, run:

      sudo yum install -y "kernel-devel-`uname -r`"

2025-02-18 12:17:34.163486+00:00 error [p:1694674 t:1694674] Cannot initialize BPF program, res=-1
2025-02-18 12:17:34.163578+00:00 error [p:1694674 t:1694674] Exception during BPFHandler initialization, closing connection: ProbeHandler couldn't load BPFModule: Success
2025-02-18 12:17:34.163588+00:00 error [p:1694674 t:1694674] troubleshoot item bpf_compilation_failed (os=Linux,flavor=unknown,headers_src=pre_installed,kernel=5.14.0-503.23.2.el9_5.x86_64): ProbeHandler couldn't load BPFModule: Success

Steps to Reproduce

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update open-telemetry
helm install opentelemetry-ebpf . -f ./values.yaml -n monitor
check logs of kernel collector pod

Expected Result

Metrics being sent from the kernel-collector to the reducer.

Actual Result

Errors in data collection.

eBPF Collector version

v0.10.2

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

eBPF Collector configuration

k get cm opentelemetry-ebpf-config -n monitor -o yaml
apiVersion: v1
data:
  config.yaml: |
    labels:
      environment: blue
kind: ConfigMap
metadata:
  creationTimestamp: "2025-02-17T17:33:00Z"
  labels:
    app.kubernetes.io/instance: opentelemetry-ebpf
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: opentelemetry-ebpf
    app.kubernetes.io/version: v0.10.2
    argocd.argoproj.io/instance: opentelemetry-ebpf
    helm.sh/chart: opentelemetry-ebpf-0.1.4
  name: opentelemetry-ebpf-config
  namespace: monitor

Log output

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions