Skip to content

enhance: Support streaming read and write of vector index files #43824

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

enhance: Support streaming read and write of vector index files #43824

wants to merge 1 commit into from
+480 −30

Conversation

foxspy
Copy link
Contributor

@foxspy foxspy commented Aug 11, 2025

issue: #42032

@sre-ci-robot sre-ci-robot added area/compilation size/L Denotes a PR that changes 100-499 lines. labels Aug 11, 2025
@sre-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: foxspy
To complete the pull request process, please assign jiaoew1991 after the PR has been reviewed.
You can assign the PR to them by writing /assign @jiaoew1991 in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mergify Mergify
Copy link
Contributor

mergify bot commented Aug 11, 2025

@foxspy

Invalid PR Title Format Detected

Your PR submission does not adhere to our required standards. To ensure clarity and consistency, please meet the following criteria:

  1. Title Format: The PR title must begin with one of these prefixes:
  • feat: for introducing a new feature.
  • fix: for bug fixes.
  • enhance: for improvements to existing functionality.
  • test: for add tests to existing functionality.
  • doc: for modifying documentation.
  • auto: for the pull request from bot.
  1. Description Requirement: The PR must include a non-empty description, detailing the changes and their impact.

Required Title Structure:

[Type]: [Description of the PR]

Where Type is one of feat, fix, enhance, test or doc.

Example:

enhance: improve search performance significantly

Please review and update your PR to comply with these guidelines.

@mergify Mergify
Copy link
Contributor

mergify bot commented Aug 11, 2025

@foxspy cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

@foxspy foxspy changed the title [Enhancement]: Support streaming read and write of vector index files [enhance]: Support streaming read and write of vector index files Aug 12, 2025
@mergify Mergify
Copy link
Contributor

mergify bot commented Aug 12, 2025

@foxspy cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Aug 12, 2025

@foxspy go-sdk check failed, comment rerun go-sdk can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Aug 12, 2025

@foxspy cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

@foxspy foxspy changed the title [enhance]: Support streaming read and write of vector index files enhance: Support streaming read and write of vector index files Aug 12, 2025
@mergify mergify bot added kind/enhancement Issues or changes related to enhancement and removed do-not-merge/invalid-pr-format labels Aug 12, 2025
@mergify Mergify
Copy link
Contributor

mergify bot commented Aug 12, 2025

@foxspy go-sdk check failed, comment rerun go-sdk can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Aug 12, 2025

@foxspy cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Aug 12, 2025

@foxspy cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

@codecov Codecov
Copy link

codecov bot commented Aug 13, 2025 •
edited
Loading

Codecov Report

❌ Patch coverage is 82.30088% with 20 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.38%. Comparing base (e6d7f34) to head (9c36315).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
internal/core/src/storage/RemoteInputStream.cpp 80.55% 7 Missing ⚠️
internal/core/src/storage/MemFileManagerImpl.cpp 0.00% 6 Missing ⚠️
internal/core/src/storage/DiskFileManagerImpl.cpp 83.87% 5 Missing ⚠️
internal/core/src/storage/FileManager.h 83.33% 1 Missing ⚠️
internal/core/src/storage/MinioChunkManager.h 0.00% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           master   #43824       +/-   ##
===========================================
+ Coverage   73.72%   79.38%    +5.65%     
===========================================
  Files         331     1560     +1229     
  Lines       32194   225064   +192870     
===========================================
+ Hits        23734   178659   +154925     
- Misses       8460    39955    +31495     
- Partials        0     6450     +6450
Components Coverage Δ
Client 79.18% <ø> (∅)
Core 73.74% <82.30%> (+0.02%) ⬆️
Go 80.48% <ø> (∅)
Files with missing lines Coverage Δ
internal/core/src/index/VectorDiskIndex.cpp 75.13% <100.00%> (+0.13%) ⬆️
internal/core/src/segcore/DeletedRecord.h 96.07% <ø> (ø)
internal/core/src/storage/DiskFileManagerImpl.h 60.00% <ø> (ø)
internal/core/src/storage/LocalChunkManager.h 77.77% <100.00%> (+6.34%) ⬆️
internal/core/src/storage/MemFileManagerImpl.h 50.00% <ø> (ø)
internal/core/src/storage/RemoteInputStream.h 100.00% <100.00%> (ø)
internal/core/src/storage/RemoteOutputStream.cpp 100.00% <100.00%> (ø)
internal/core/src/storage/RemoteOutputStream.h 100.00% <100.00%> (ø)
internal/core/src/storage/ThreadPool.h 97.22% <ø> (ø)
...nternal/core/src/storage/azure/AzureChunkManager.h 100.00% <100.00%> (ø)
... and 5 more

... and 1225 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@sre-ci-robot sre-ci-robot added size/XL Denotes a PR that changes 500-999 lines. and removed size/L Denotes a PR that changes 100-499 lines. labels Aug 14, 2025
@mergify Mergify
Copy link
Contributor

mergify bot commented Aug 14, 2025

@foxspy cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Aug 14, 2025

@foxspy go-sdk check failed, comment rerun go-sdk can trigger the job again.

@foxspy
Copy link
Contributor Author

foxspy commented Aug 14, 2025

/run-cpu-e2e

@foxspy
Copy link
Contributor Author

foxspy commented Aug 14, 2025

rerun go-sdk

@foxspy
Copy link
Contributor Author

foxspy commented Aug 14, 2025

/run-cpu-e2e

2 similar comments
@foxspy
Copy link
Contributor Author

foxspy commented Aug 14, 2025

/run-cpu-e2e

@foxspy
Copy link
Contributor Author

foxspy commented Aug 14, 2025

/run-cpu-e2e

@mergify mergify bot added the ci-passed label Aug 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bigsheeper bigsheeper Awaiting requested review from bigsheeper

@czs007 czs007 Awaiting requested review from czs007

Assignees
No one assigned
Labels
area/compilation ci-passed dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement size/XL Denotes a PR that changes 500-999 lines.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@foxspy @sre-ci-robot