Linkerd / Jaeger trace collection not working. received corrupt message of type InvalidContentType, no spans received

[mephmanx]

Im having issues with my linkerd / jaeger install. I am following the guides online and everything installs but the proxy sidecars keep throwing this error /warning.

[ 0.004040s] INFO ThreadId(01) linkerd2_proxy: release 2.308.0 (a6e47d7) by linkerd on 2025-07-15T17:58:06Z
[ 0.009288s] INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[ 0.011521s] INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[ 0.011557s] INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[ 0.011567s] INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[ 0.011573s] INFO ThreadId(01) linkerd2_proxy: Tap interface on 0.0.0.0:4190
[ 0.011580s] INFO ThreadId(01) linkerd2_proxy: SNI is default.examples.serviceaccount.identity.linkerd.cluster.local
[ 0.011585s] INFO ThreadId(01) linkerd2_proxy: Local identity is default.examples.serviceaccount.identity.linkerd.cluster.local
[ 0.011591s] INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[ 0.011596s] INFO ThreadId(01) linkerd2_proxy: Tracing collector at collector.linkerd-jaeger:4317 (collector.linkerd-jaeger.serviceaccount.identity.linkerd.cluster.local)
[ 0.014071s] INFO ThreadId(01) policy:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_pool_p2c: Adding endpoint addr=10.42.0.35:8090
[ 0.014713s] INFO ThreadId(01) dst:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_pool_p2c: Adding endpoint addr=10.42.0.35:8086
[ 0.014732s] INFO ThreadId(02) identity:identity{server.addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_pool_p2c: Adding endpoint addr=10.42.0.34:8080
[ 0.021230s] INFO ThreadId(01) tracing:controller{addr=collector.linkerd-jaeger:4317}: linkerd_pool_p2c: Adding endpoint addr=10.43.161.169:4317
[ 0.023854s] WARN ThreadId(01) tracing:controller{addr=collector.linkerd-jaeger:4317}:endpoint{addr=10.43.161.169:4317}: linkerd_reconnect: Failed to connect error=endpoint 10.43.161.169:4317: received corrupt message of type InvalidContentType error.sources=[received corrupt message of type InvalidContentType]
[ 0.024736s] INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity id=default.examples.serviceaccount.identity.linkerd.cluster.local
[ 0.130674s] WARN ThreadId(01) tracing:controller{addr=collector.linkerd-jaeger:4317}:endpoint{addr=10.43.161.169:4317}: linkerd_reconnect: Failed to connect error=endpoint 10.43.161.169:4317: received corrupt message of type InvalidContentType error.sources=[received corrupt message of type InvalidContentType]
[ 0.353973s] WARN ThreadId(01) tracing:controller{addr=collector.linkerd-jaeger:4317}:endpoint{addr=10.43.161.169:4317}: linkerd_reconnect: Failed to connect error=endpoint 10.43.161.169:4317: received corrupt message of type InvalidContentType error.sources=[received corrupt message of type InvalidContentType]
[ 0.761195s] WARN ThreadId(01) tracing:controller{addr=collector.linkerd-jaeger:4317}:endpoint{addr=10.43.161.169:4317}: linkerd_reconnect: Failed to connect error=endpoint 10.43.161.169:4317: received corrupt message of type InvalidContentType error.sources=[received corrupt message of type InvalidContentType]
Here is my install process / script

Confirm version matches

  sudo linkerd version --kubeconfig=/etc/rancher/rke2/rke2.yaml
  sudo kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml create namespace linkerd
  sudo kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml create namespace linkerd-viz
  sudo kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml create namespace linkerd-jaeger
  sudo kubectl annotate namespace linkerd [field.cattle.io/projectId=local:infrastructure](http://field.cattle.io/projectId=local:infrastructure) --overwrite --kubeconfig=/etc/rancher/rke2/rke2.yaml
  sudo kubectl annotate namespace linkerd-viz [field.cattle.io/projectId=local:infrastructure](http://field.cattle.io/projectId=local:infrastructure) --overwrite --kubeconfig=/etc/rancher/rke2/rke2.yaml
  sudo kubectl annotate namespace linkerd-jaeger [field.cattle.io/projectId=local:infrastructure](http://field.cattle.io/projectId=local:infrastructure) --overwrite --kubeconfig=/etc/rancher/rke2/rke2.yaml
  sudo kubectl annotate namespace linkerd [scheduler.alpha.kubernetes.io/node-selector=](http://scheduler.alpha.kubernetes.io/node-selector=)"dedicated=srvr-node" --overwrite --kubeconfig=/etc/rancher/rke2/rke2.yaml
  sudo kubectl annotate namespace linkerd-viz [scheduler.alpha.kubernetes.io/node-selector=](http://scheduler.alpha.kubernetes.io/node-selector=)"dedicated=srvr-node" --overwrite --kubeconfig=/etc/rancher/rke2/rke2.yaml
  sudo kubectl annotate namespace linkerd-jaeger [scheduler.alpha.kubernetes.io/node-selector=](http://scheduler.alpha.kubernetes.io/node-selector=)"dedicated=srvr-node" --overwrite --kubeconfig=/etc/rancher/rke2/rke2.yaml
  sudo kubectl annotate ns linkerd [linkerd.io/inject=disabled](http://linkerd.io/inject=disabled) --overwrite --kubeconfig=/etc/rancher/rke2/rke2.yaml
  sudo kubectl annotate ns linkerd-viz [linkerd.io/inject=disabled](http://linkerd.io/inject=disabled) --overwrite --kubeconfig=/etc/rancher/rke2/rke2.yaml
  sudo kubectl annotate ns linkerd-jaeger [linkerd.io/inject=disabled](http://linkerd.io/inject=disabled) --overwrite --kubeconfig=/etc/rancher/rke2/rke2.yaml
  echo ":inbox_tray: Installing Gateway API CRDs..."
  # sudo kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml
  sudo kubectl --kubeconfig=/etc/rancher/rke2/rke2.yaml apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/experimental-install.yaml
  # 2) Install the CRDs
  sudo linkerd install --crds \
    --kubeconfig /etc/rancher/rke2/rke2.yaml \
  | sudo kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml apply -f -
  sudo linkerd install \
    --kubeconfig /etc/rancher/rke2/rke2.yaml \
    --set clusterNetworks="10.42.0.0/16\,10.43.0.0/16" \
    --set proxyInit.ignoreOutboundPorts="4567\,4568\,4317\,2379" \
    --set-string global.tracing.enabled=true \
    --set-string proxy.tracing.enabled=true \
    --set-string proxy.tracing.otlp.endpoint="collector.linkerd-jaeger.svc.cluster.local:4317" \
    --set-string proxy.telemetry.spans.enabled=true \
    --set-string proxy.telemetry.spans.level=trace \
    | sudo kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml apply -f -
  # 4) Wait for Linkerd to come up
  sudo linkerd check --kubeconfig /etc/rancher/rke2/rke2.yaml
  # Install Jaeger extension with collector config
  cat <<EOF | sudo tee /tmp/linkerd-jaeger-values.yaml
  collector:
    enabled: true
    useOtel: true
    config:
      receivers:
        otlp:
          protocols:
            grpc:
              endpoint: 0.0.0.0:4317
      exporters:
        jaeger:
          endpoint: jaeger.linkerd-jaeger.svc.cluster.local:14250
          tls:
            insecure: true
      processors:
        batch: {}
        k8sattributes: {}
      service:
        pipelines:
          traces:
            receivers: [otlp]
            processors: [batch, k8sattributes]
            exporters: [jaeger]
  ui:
    tls:
      enabled: false
    auth:
      enabled: false
  EOF
  # After generating the Jaeger manifest
  sudo linkerd jaeger install -f /tmp/linkerd-jaeger-values.yaml \
    --kubeconfig /etc/rancher/rke2/rke2.yaml > /tmp/jaeger-install.yaml
  sudo yq -i '
    (. | select(.kind == "Deployment" and .metadata.name == "collector") |
      .spec.template.metadata.annotations["[linkerd.io/inject](http://linkerd.io/inject)"]) = "disabled" |
    (. | select(.kind == "Deployment" and .metadata.name == "jaeger") |
      .spec.template.metadata.annotations["[linkerd.io/inject](http://linkerd.io/inject)"]) = "disabled"
  ' /tmp/jaeger-install.yaml
  # Apply the patched manifest
  sudo kubectl apply -f /tmp/jaeger-install.yaml --kubeconfig=/etc/rancher/rke2/rke2.yaml
  # Wait for rollout to complete after patch
  sudo kubectl rollout status deploy/jaeger -n linkerd-jaeger --kubeconfig /etc/rancher/rke2/rke2.yaml
  sudo linkerd jaeger check --kubeconfig /etc/rancher/rke2/rke2.yaml
  # 6) Install the Viz extension (dashboard + embedded Jaeger UI),
  #    disabling *only* the dashboard’s TLS + auth
  sudo linkerd viz install \
    --set dashboard.tls.enabled=false \
    --set dashboard.auth.enabled=false \
    --set jaeger.enabled=true \
    --set dashboard.enforcedHostRegexp=".*" \
    --kubeconfig /etc/rancher/rke2/rke2.yaml \
  | sudo kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml apply -f -
  # 7) Tell Linkerd this namespace hosts the viz extension
  sudo kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml \
    label ns linkerd-viz [linkerd.io/extension=viz](http://linkerd.io/extension=viz) --overwrite
  # 8) Validate the viz extension
  sudo linkerd viz check --kubeconfig /etc/rancher/rke2/rke2.yaml
  sudo kubectl --kubeconfig=/etc/rancher/rke2/rke2.yaml -n linkerd-viz set env deployment/web LINKERD_DASHBOARD_ALLOW_ORIGIN="*"
  sudo kubectl rollout status deploy/web -n linkerd-viz --kubeconfig /etc/rancher/rke2/rke2.yaml

[7:46 PM] Chris
my linkerd jaeger check looks good
ubuntu@srvr-node:~$ sudo linkerd jaeger check --kubeconfig /etc/rancher/rke2/rke2.yaml
linkerd-jaeger

√ linkerd-jaeger extension Namespace exists
‼️ jaeger extension pods are injected
could not find proxy container for collector-75568c7c6f-b5t28 pod
see https://linkerd.io/2/checks/#l5d-jaeger-pods-injection for hints
‼️ jaeger injector pods are running
container "linkerd-proxy" in pod "collector-75568c7c6f-b5t28" is not ready
see https://linkerd.io/2/checks/#l5d-jaeger-pods-running for hints
√ jaeger extension proxies are healthy
√ jaeger extension proxies are up-to-date
√ jaeger extension proxies and cli versions match
Status check results are √
ubuntu@srvr-node:~$

[kflynn]

@mephmanx, so sorry I missed this! Let me ask the silly question first: which version of Linkerd is this? We've made a bunch of changes in this recently so I want to make sure I'm looking at the right stuff.

[mephmanx]

this is the version I am trying to use

  curl -sL https://run.linkerd.io/install | sh -s -- --version version-2.18
  
  here is the start of the install process that I am using

https://github.com/dotcomrow/tf-k8s-cluster-infra/blob/d2bc68c4ebfeb02dd3e93a204cb3727c36bb5eb8/terraform/config/cloud_init_srvr.tftpl#L1492

My entire platform build is open source here