GPU core isolation does not work properly in environments with higher versions of glibc, such as glibc2.35. When will support for higher versions of glibc be available?

[007ws]

Please provide an in-depth description of the question you have:

What do you think about this question?:
glibc2.35 等高版本 glibc环境下， 不能正常隔离算力。什么时候可以支持高版本glibc呢，有没有计划。
或者说 有什么替代方案呢，比如说安装某个特殊版本的 glibc2.35。

系统环境：
➜ ~ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
➜ ~ ldd --version

ldd (Ubuntu GLIBC 2.35-0ubuntu3.8) 2.35
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.
➜ ~

Environment: glibc2.35

• HAMi version: v2.6.0
• Kubernetes version: 1.33.2
• Others:

[archlitchi]

can you describe more detailed information, like does memory isolation works? which image are you using? the error message if any?

[007ws]

hami没有错误消息。进行算力测试时，分配了10%的算力, 运行时总是能达到100%

这是配置：

这是宿主机：

这是容器内部

具体的yaml配置文件：
`
➜ task2 cat core-test.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: cores-test-stress-test
labels:
app: cores-test-stress
spec:
replicas: 1
selector:
matchLabels:
app: cores-test-stress
template:
metadata:
annotations:
nvidia.com/use-gpuuuid: "GPU-610b3435-8996-17a4-c4ec-381940d62ace"
labels:
app: cores-test-stress
spec:
schedulerName: hami-scheduler
containers:
- name: cores-test-container
image: pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
command:
- "python3"
- "-u" # 无缓冲输出
- "-c"
- |
import torch
import time
import sys
import os

      # 强制无缓冲输出
      sys.stdout.reconfigure(line_buffering=True)

      print("=== vLLM Memory Stress Test (100MB limit, 1GB demand) ===", flush=True)
      print(f"Pod name: {os.environ.get('HOSTNAME', 'unknown')}", flush=True)
      print(f"GPU available: {torch.cuda.is_available()}", flush=True)

      if torch.cuda.is_available():
          print(f"GPU name: {torch.cuda.get_device_name(0)}", flush=True)
          print(f"GPU memory total: {torch.cuda.get_device_properties(0).total_memory / 1024**3:.1f}GB", flush=True)

      device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

      # 模拟vLLM embedding服务启动
      print("Starting vLLM-like embedding service...", flush=True)

      try:
          cycle = 0
          start_time = time.time()

          while True:  # 持续运行
              elapsed_hours = (time.time() - start_time) / 3600

              # 尝试分配1GB显存 (超过100MB限制)
              memory_gb = 1.0
              tensor_size = int((memory_gb * 1024 * 1024 * 1024) // 4)  # float32 = 4 bytes

              print(f"[{elapsed_hours:.1f}h] Cycle {cycle+1}: Attempting to allocate {memory_gb:.1f}GB GPU memory", flush=True)
              print(f"  Target tensor size: {tensor_size:,} elements ({tensor_size * 4 / 1024**3:.2f}GB)", flush=True)

              try:
                  # 尝试分配1GB显存
                  tensor = torch.randn(tensor_size, device=device, dtype=torch.float32)
                  allocated = torch.cuda.memory_allocated(0) / 1024**3 if torch.cuda.is_available() else 0
                  print(f"  ✓ Successfully allocated: {allocated:.2f}GB", flush=True)
                  print(f"  → This should be LIMITED by HAMi to ~100MB\!", flush=True)

                  # 进行一些计算以使用GPU
                  result = torch.mean(tensor)
                  print(f"  → Computation result: {result:.6f}", flush=True)

              except torch.cuda.OutOfMemoryError as e:
                  print(f"  ⚠ GPU Memory limit reached - HAMi is working\! Error: {e}", flush=True)

                  # 尝试分配更小的内存
                  small_size = int((0.08 * 1024 * 1024 * 1024) // 4)  # 80MB
                  try:
                      tensor = torch.randn(small_size, device=device, dtype=torch.float32)
                      allocated = torch.cuda.memory_allocated(0) / 1024**3 if torch.cuda.is_available() else 0
                      print(f"  ✓ Fallback allocated: {allocated:.2f}GB", flush=True)
                  except Exception as fallback_e:
                      print(f"  ✗ Even fallback failed: {fallback_e}", flush=True)
                      tensor = None

              except Exception as e:
                  print(f"  ✗ Unexpected error: {e}", flush=True)
                  tensor = None

              # 保持30秒
              print(f"  → Holding memory for 30s...", flush=True)
              time.sleep(30)

              # 释放显存
              if tensor is not None:
                  del tensor
              if torch.cuda.is_available():
                  torch.cuda.empty_cache()
              print(f"  ✓ Memory released", flush=True)

              cycle += 1

              # 测试运行1小时 (120个30秒周期)
              if cycle >= 120:
                  break

      except KeyboardInterrupt:
          print("Test interrupted by user", flush=True)
      except Exception as e:
          print(f"Test failed with error: {e}", flush=True)

      print("vLLM memory stress test completed", flush=True)
    env:
    - name: GPU_CORE_UTILIZATION_POLICY
      value: "force"
    resources:
      limits:
        nvidia.com/gpu: 1
        nvidia.com/gpumem: 8000        # 限制10000MB显存
        nvidia.com/gpucores: 10       # 限制10%算力
    volumeMounts:
    - name: tmp-file-volume
      mountPath: /tmp_file
  volumes:
  - name: tmp-file-volume
    hostPath:
      path: /home/aiteam/suanli2/task2/tmp_file
      type: Directory
  restartPolicy: Always
  nodeSelector:
    gpu: "on"
  tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule

➜ task2
`

[007ws]

什么时候可以支持高版本的 Glibc 呢。比如2.35 或者 2.39。有没有安排或计划呢

[archlitchi]

it's not a glib issue i think, you need to add env 'GPU_CORE_UTILIZATION_POLICY=force' to enable core limitation by force

[007ws]

it's not a glib issue i think, you need to add env 'GPU_CORE_UTILIZATION_POLICY=force' to enable core limitation by force

我验证了一下这个问题，确定了是glibc版本的问题，hami目前不支持高版本GLIBC:

在Ubuntu18.04(glibc2.27)环境下可以分割显存和算力。
在Ubuntu22.04(glibc2.35)环境下可以分割显存，但是不可以分割算力。
在Ubuntu24.04(glibc2.39)环境下不可以分割显存和算力。

[christu]

同样遇到了这个问题，glic 2.36-9+deb12u10 版本下，显存限制失败了。

[archlitchi]

it's not a glib issue i think, you need to add env 'GPU_CORE_UTILIZATION_POLICY=force' to enable core limitation by force

我验证了一下这个问题，确定了是glibc版本的问题，hami目前不支持高版本GLIBC:

在Ubuntu18.04(glibc2.27)环境下可以分割显存和算力。 在Ubuntu22.04(glibc2.35)环境下可以分割显存，但是不可以分割算力。 在Ubuntu24.04(glibc2.39)环境下不可以分割显存和算力。

你说的这个ubuntu是节点上的版本还是容器里的版本？

[007ws]

it's not a glib issue i think, you need to add env 'GPU_CORE_UTILIZATION_POLICY=force' to enable core limitation by force

我验证了一下这个问题，确定了是glibc版本的问题，hami目前不支持高版本GLIBC:
在Ubuntu18.04(glibc2.27)环境下可以分割显存和算力。 在Ubuntu22.04(glibc2.35)环境下可以分割显存，但是不可以分割算力。 在Ubuntu24.04(glibc2.39)环境下不可以分割显存和算力。

你说的这个ubuntu是节点上的版本还是容器里的版本？

节点上的版本，也就是宿主机上的系统版本。

[Shikairan]

2.35下4080能正常限制，4090不能限制，A800不能限制。

[QuteSaltyFish]

glibc 2.39 ubuntu24.04也不行

[wawa0210]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

glibc 2.39 ubuntu24.04 also works