fix(bulk_writer): preserve remote upload prefixes on Windows

[VectorPeak]

Normalize RemoteBulkWriter object-key construction so Windows path separators cannot drop the configured remote upload prefix.

What Problem This Solves

RemoteBulkWriter._upload() converts each generated local bulk file into the object key that is sent to S3/MinIO or Azure Blob Storage. That object key is later also stored in batch_files, so this path is both the upload target and the path Milvus uses for bulk import.

The current code derives the local suffix with string replacement and then feeds that suffix back into Path.joinpath():

relative_file_path = str(file_path).replace(str(super().data_path), "")
minio_file_path = str(Path.joinpath(self._remote_path, relative_file_path.lstrip("/"))).lstrip("/")

This mixes two different path domains:

• file_path and super().data_path are local filesystem paths, so on Windows they use \ separators.
• minio_file_path is an object-storage key, so it should use stable POSIX-style / separators regardless of the client OS.

On Windows, a generated file such as:

C:\tmp\bulk_writer\<uuid>\1.jsonl

can produce this suffix after string replacement:

\1.jsonl

The code only calls .lstrip("/"), so the leading Windows backslash remains. When that value is passed to Path.joinpath(self._remote_path, "\1.jsonl"), Windows treats "\1.jsonl" as a rooted path. A rooted path resets the join, so the configured remote prefix is discarded.

Local reproduction before the fix:

remote_base: \bulk\test\00000000-0000-0000-0000-000000000001
relative_file_path: '\\1.jsonl'
joined_object_key: \1.jsonl
expected_prefix_present: False

That means a Windows upload configured for:

bulk/test/<uuid>/1.jsonl

can instead send and record:

\1.jsonl

The practical impact is that the blob is uploaded outside the intended bulk/test/<uuid>/ object-key prefix, and batch_files records the same wrong path for later bulk import. The generated file itself is valid; the bug is in translating a local Windows file path into a remote object-storage key.

Change

Build remote object names as POSIX object-key strings instead of host filesystem paths.

The fix keeps the two path domains separate:

• use Path(...).resolve().relative_to(super().data_path) only to compute the local file path relative to the local bulk writer directory
• convert that relative local path with .as_posix() so nested bulk files become stable object-key suffixes such as 1.jsonl or 1/part.parquet
• combine the configured remote prefix with the relative suffix as a POSIX string, not with Path.joinpath()

After the change, Windows local paths are normalized before they become object-storage keys:

local file path:     C:\tmp\bulk_writer\<uuid>\1.jsonl
relative suffix:     1.jsonl
remote object key:   bulk/test/<uuid>/1.jsonl

This keeps bucket/container checks, overwrite behavior, supported file extensions, upload client selection, batch_files shape, and local cleanup unchanged. It only changes the object-key construction step inside RemoteBulkWriter._upload().

Evidence

The focused unit test now verifies the bug boundary directly at the upload call. It creates a RemoteBulkWriter with remote_path="bulk/test", writes a local .jsonl file under the writer's local data directory, and patches the storage calls so no real S3/Azure service is contacted.

The test asserts that _upload_object(object_name=...) receives a remote object key that:

• equals the path stored in uploaded_files
• retains the configured bulk/test/<uuid>/ prefix
• ends with /1.jsonl
• contains no Windows backslashes

Expected object key after the fix:

bulk/test/<uuid>/1.jsonl

This proves the path translation layer now treats the remote upload target as an object-storage key rather than a host filesystem path.

Possible call chain / impact

RemoteBulkWriter.commit(...)
  -> LocalBulkWriter.commit(call_back=self._upload)
    -> LocalBulkWriter._flush(...)
      -> persists generated bulk files under the local writer directory
    -> RemoteBulkWriter._upload(file_list)
      -> computes object_name from the local file path
      -> _object_exists(object_name)
      -> _upload_object(object_name=...)
      -> batch_files records the uploaded object key

The affected path is narrow: only RemoteBulkWriter's conversion from a local generated file path to the remote S3/Azure object key is changed. In practice, this matters for Windows clients because the local file separator can otherwise leak into object-key construction.

Everything around that path stays the same. This PR does not change how bulk files are generated, how schemas or file extensions are validated, how S3/MinIO or Azure clients are initialized, how bucket/container existence and overwrite checks work, or how local files are cleaned up after upload.

Validation

• mamba run -n prw-pymilvus-py3.11 python -m pytest tests/unit/test_remote_bulk_writer.py -q - passed, 4 tests
• mamba run -n prw-pymilvus-py3.11 python -m ruff check pymilvus/bulk_writer/remote_bulk_writer.py tests/unit/test_remote_bulk_writer.py - passed
• git diff --check - passed

[sre-ci-robot]

Welcome @VectorPeak! It looks like this is your first PR to milvus-io/pymilvus 🎉

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[VectorPeak]

/assign @tedxu

Hi @tedxu, this PR is ready for review. It keeps the change scoped to RemoteBulkWriter object-key construction on Windows and adds focused unit coverage for the path normalization behavior. DCO is passing, and the local validation details are included in the PR description.

[XuanYang-cn]

/assign
/unassign @tedxu

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.91%. Comparing base (fd9b79a) to head (11e8b30).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #3658   +/-   ##
=======================================
  Coverage   93.91%   93.91%           
=======================================
  Files          72       72           
  Lines       15487    15487           
=======================================
  Hits        14545    14545           
  Misses        942      942           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[XuanYang-cn]

Thanks for the focused fix. Please handle the relative local_path regression and run formatting so the lint checks can pass.

[VectorPeak]

Addressed the requested changes from review:

• Resolved super().data_path before relative_to() in RemoteBulkWriter._upload() so relative local_path values work correctly.
• Added test_upload_jsonl_file_with_relative_local_path as regression coverage.
• Ran local validation:
  • uv run --group dev pytest tests/unit/test_remote_bulk_writer.py -q -> 5 passed
  • uv run --group dev ruff check pymilvus tests -> passed
  • uv run --group dev black pymilvus tests --check --diff -> touched files are Black-formatted; on this Windows machine the full run hit an unrelated multiprocessing formatter error in tests/unit/prepare/test_search.py, so I also verified the touched files directly.

Latest pushed commit: 7a202403 with DCO sign-off. DCO is now passing. GitHub Actions for the latest signed commit have not reported Code checker/Test runs yet, so this may still need the repository-side workflow approval/rerun step.

@XuanYang-cn could you please re-review when the checks are available?

[VectorPeak]

Hi @XuanYang-cn, the requested changes have been addressed in the latest signed commit 7a202403:

• RemoteBulkWriter._upload() now resolves the local data path before relative_to().
• Added regression coverage for relative local_path values.
• Re-ran the focused local validation and DCO is passing.

Please feel free to continue the review when you have time.

[XuanYang-cn]

@VectorPeak PLZ push an empty commit to re-triggers github's actions. I tried but cannot workaround this github bug.

[VectorPeak]

Hi @XuanYang-cn, done. I pushed an empty signed commit 11e8b30 to re-trigger GitHub Actions for this PR.

I'll keep an eye on the checks as they come back.

[XuanYang-cn]

/lgtm
/approve

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: VectorPeak, XuanYang-cn

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [XuanYang-cn]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[pymilvus-bot]

❌ Backport Failed
Hi @VectorPeak, I could not cherry-pick this to 2.6 due to merge conflicts. Please backport manually.

(cc @XuanYang-cn)

[pymilvus-bot]

✅ Backport Created
Hi @VectorPeak, Backport PR for 3.0 has been created: #3665

(cc @XuanYang-cn)

[pymilvus-bot]

Backport Created: 2.6 -> #3669