BGE-M3 LoRA Fine-Tune

Fine-tuned BAAI/bge-m3 with LoRA for domain-specific dense retrieval, on a single consumer GPU (12GB VRAM).

HF Hub: Lzvick/bge-m3-ir-research-lora-v1

Config

Setting	Value
Base model	BAAI/bge-m3
Hardware profile	rtx_5070_12gb (single GPU, no DeepSpeed)
Method	LoRA (r=16, alpha=32, dropout=0.1, targets: query,key,value)
Trainable params	2,359,296 / 570,114,048 (0.41%)
Epochs	3
Effective batch size	128 (4 per-device × 32 grad-accum)
Learning rate	1e-5
Query / passage max len	256 / 256
Train group size	8 (1 positive + 7 negatives per query)
Precision	bf16

Data

Split	File	Examples	Used for
Train	data/train/train_mixed.jsonl	2,072	LoRA fine-tuning
Validation	data/val/val.jsonl	109	Held-out sanity check (post-training)
Test	not yet built	—	data/eval/ is empty; run src/build_eval_dataset.py to produce a BEIR-format test set for a full benchmark (evaluation/run_comparison.py)

Results

Quick sanity check (evaluation/smoke_test.py): mean cosine-similarity margin between query→positive and query→negative on the 109 held-out validation examples, LoRA adapter merged in-memory (no disk merge).

Model	Mean pos-neg margin	Trainable params
Baseline (BAAI/bge-m3)	0.0849	—
Fine-tuned (LoRA)	0.0861	0.41%

The fine-tune widened the margin by +0.0012 — a weak positive signal, not a strong one. This is expected given the small dataset and short run (3 epochs over 2,072 examples ≈ 51 optimizer steps total).

This is a coarse sanity check only, not a retrieval benchmark — it confirms the LoRA weights learned something in the right direction, nothing more. A real nDCG/Recall/MRR comparison against baseline requires building a test set and running the full evaluation pipeline (see Data table above).

Known issues fixed along the way

• transformers 5.x removed Trainer.tokenizer (renamed to processing_class); FlagEmbedding's EncoderOnlyEmbedderM3Trainer._save() still read the old attribute, crashing on every checkpoint save. Patched with a compat shim in pipeline/train_m3_lora.py.
• LoRA training saves an adapter only (adapter_config.json, adapter_model.safetensors) plus colbert_linear.pt / sparse_linear.pt — not a standalone model. It must be merged with the base model before FlagModel/BGEM3FlagModel-based tools (evaluation/run_comparison.py, src/retriever.py) can load it.