utils.tqdm is not thread-safe #3285
Description
Describe the bug
Running snapshot_download() in a ThreadPoolExecutor is raising
File ".venv/lib/python3.12/site-packages/tqdm/contrib/concurrent.py", line 24, in ensure_lock
del tqdm_class._lock
^^^^^^^^^^^^^^^^
AttributeError: type object 'tqdm' has no attribute '_lock'
This seems the same issue reported in the datasets library: huggingface/datasets#7660
Reproduction
This is a simple code which is regularly getting into the error.
from concurrent.futures import ThreadPoolExecutor, as_completed
from huggingface_hub import snapshot_download
from huggingface_hub.utils import disable_progress_bars
def task(task_id: int):
print(f"[{task_id}] Stating model download")
disable_progress_bars()
download_path = snapshot_download(
repo_id="ds4sd/docling-layout-old",
)
print(f"[{task_id}] {download_path=}")
print(f"[{task_id}] Done model download")
def main():
NUM_THREADS = 10
with ThreadPoolExecutor(max_workers=NUM_THREADS) as pool:
futures = [pool.submit(task, task_id) for task_id in range(NUM_THREADS)]
datas = []
for future in as_completed(futures):
datas.append(future.result())
if __name__ == "__main__":
main()Logs
System info
- huggingface_hub version: 0.34.3
- Platform: macOS-15.5-arm64-arm-64bit
- Python version: 3.12.7
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Running in Google Colab Enterprise ?: No
- Token path ?: /Users/dol/.cache/huggingface/token
- Has saved token ?: False
- Configured git credential helpers: osxkeychain
- FastAI: N/A
- Tensorflow: N/A
- Torch: 2.7.1
- Jinja2: 3.1.6
- Graphviz: N/A
- keras: N/A
- Pydot: N/A
- Pillow: 11.3.0
- hf_transfer: N/A
- gradio: 5.40.0
- tensorboard: N/A
- numpy: 2.3.2
- pydantic: 2.11.7
- aiohttp: 3.12.15
- hf_xet: 1.1.5
- ENDPOINT: https://huggingface.co
- HF_HUB_CACHE: /Users/dol/.cache/huggingface/hub
- HF_ASSETS_CACHE: /Users/dol/.cache/huggingface/assets
- HF_TOKEN_PATH: /Users/dol/.cache/huggingface/token
- HF_STORED_TOKENS_PATH: /Users/dol/.cache/huggingface/stored_tokens
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_ENABLE_HF_TRANSFER: False
- HF_HUB_ETAG_TIMEOUT: 10
- HF_HUB_DOWNLOAD_TIMEOUT: 10