Skip to content

[Bugfix] Preserve empty values in Mooncake Store REST GET#2587

Merged
ykwd merged 2 commits into
kvcache-ai:mainfrom
VectorPeak:fix
Jun 24, 2026
Merged

[Bugfix] Preserve empty values in Mooncake Store REST GET#2587
ykwd merged 2 commits into
kvcache-ai:mainfrom
VectorPeak:fix

Conversation

@VectorPeak

@VectorPeak VectorPeak commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Description

This PR fixes a value-preservation bug in the Mooncake Store REST API GET /api/get/{key} path. The handler previously treated an empty byte payload as a missing key, so a value that could be stored through the REST API could not be read back reliably.

The original REST read path used the value returned by self.store.get(key) to decide whether the key existed:

value = self.store.get(key)
if not value:
    return 404

That check conflates two different states: an existing empty byte payload (b"") and an absent object. Reviewer feedback also pointed out that the real MooncakeDistributedStore.get() binding may return zero-length bytes when the underlying get_buffer(key) result is null, so get() alone is not a reliable existence signal.

The updated handler now uses the store's explicit existence API before reading the value:

exists = self.store.is_exist(key)
if exists == 0:
    return 404
if exists < 0:
    return 500
value = self.store.get(key)

For empty byte values, the handler re-checks existence after get() returns b"". This keeps valid stored empty payloads readable while avoiding a false 200 response when the binding surfaces a missing/null buffer as zero-length bytes.

Changes Made

  • Switched REST GET missing-key detection from value truthiness/None checks to self.store.is_exist(key).
  • Preserved b"" as a valid 200 response body for existing keys.
  • Returned 500 when is_exist() reports a negative error or when get() unexpectedly returns None after a successful existence check.
  • Updated tests so missing keys match the real binding behavior: is_exist(key) == 0 while get(key) may still be b"".
  • Added coverage for empty byte payloads, existence-check failure, unexpected None from get(), and the empty-byte ambiguity after get().

Evidence

The original bug can be reproduced with an isolated handler-style case where the store returns an existing empty byte payload:

self.fake_store.is_exist = lambda key: True
self.fake_store.get = lambda key: b""
request.match_info = {"key": "empty_value_key"}
resp = await self.service.handle_get(request)

Before this fix, the handler evaluated b"" as false and returned the missing-key branch:

status: 404
body: {"error": "Key not found"}

After this fix, the key existence check distinguishes an existing empty payload from a missing key:

status: 200
body: b""

A real missing key is represented by is_exist(key) == 0 and still returns 404 Key not found even if get(key) would produce b"".

REST GET empty byte value reproduction

Possible Call Chain / Impact

The affected path is the Mooncake Store REST API read flow:

mooncake-wheel/mooncake/mooncake_store_service.py
  -> MooncakeStoreService.start_http_service(...)
  -> web.get('/api/get/{key}', self.handle_get)
  -> MooncakeStoreService.handle_get(...)
  -> self.store.is_exist(key)
  -> self.store.get(key)
  -> web.Response(body=value, content_type='application/octet-stream')

This PR only changes the REST GET response decision for values returned through the REST shim. It does not change writes, deletion, mounting, store setup, or the underlying Mooncake distributed store APIs.

Module

  • Transfer Engine (mooncake-transfer-engine)
  • Mooncake Store (mooncake-store)
  • Mooncake EP (mooncake-ep)
  • Mooncake PG (mooncake-pg)
  • Integration (mooncake-integration)
  • P2P Store (mooncake-p2p-store)
  • Python Wheel (mooncake-wheel)
  • Common (mooncake-common)
  • Mooncake RL (mooncake-rl)
  • CI/CD
  • Docs
  • Other

Type of Change

  • Bug fix
  • New feature
  • Refactor
  • Breaking change
  • Documentation update
  • Performance improvement
  • Other

How Has This Been Tested?

Test commands:

python -m unittest mooncake-wheel.tests.test_mooncake_store_service_api.StoreServiceApiTest.test_handle_get_success mooncake-wheel.tests.test_mooncake_store_service_api.StoreServiceApiTest.test_handle_get_empty_bytes mooncake-wheel.tests.test_mooncake_store_service_api.StoreServiceApiTest.test_handle_get_empty_bytes_rechecks_existence mooncake-wheel.tests.test_mooncake_store_service_api.StoreServiceApiTest.test_handle_get_not_found mooncake-wheel.tests.test_mooncake_store_service_api.StoreServiceApiTest.test_handle_get_none_value_is_store_failure mooncake-wheel.tests.test_mooncake_store_service_api.StoreServiceApiTest.test_handle_get_exist_check_failure mooncake-wheel.tests.test_mooncake_store_service_api.StoreServiceApiTest.test_handle_get_store_exception
git diff --check

Test results:

  • Unit tests pass
  • Integration tests pass (if applicable)
  • Manual testing done (describe below)

Targeted GET handler tests passed:

Ran 7 tests in 0.017s

OK

git diff --check passed with no patch errors. It emitted Windows line-ending warnings only.

I also ran the full mooncake-wheel.tests.test_mooncake_store_service_api file locally before this update. It had unrelated existing failures around handle_put_missing_value and unmount call expectations, so I did not include that as a passing validation claim.

Checklist

  • I have performed a self-review of my own code
  • I have formatted my code using ./scripts/code_format.sh
  • I have run pre-commit run --all-files and all hooks pass
  • I have updated the documentation (if applicable)
  • I have added tests to prove my changes are effective
  • For changes >500 LOC: I have filed an RFC issue (not applicable; this PR changes fewer than 100 lines)

AI Assistance Disclosure

  • No AI tools were used
  • AI tools were used (specify below)

AI assistance was used to identify the edge case, make the handler/test updates, respond to review feedback, and draft this PR description. The submitter is responsible for reviewing and validating the final changes.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the handle_get method in mooncake_store_service.py to check if the retrieved value is explicitly None instead of evaluating its truthiness. This allows empty values, such as empty bytes, to be successfully returned with a 200 OK status rather than incorrectly triggering a 404 Not Found error. The mock Response class and the test case test_handle_get_empty_bytes in the test suite have been updated to support and verify this new behavior. There are no review comments to address, and I have no feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@VectorPeak
VectorPeak marked this pull request as ready for review June 24, 2026 03:31
value = self.store.get(key)

if not value:
if value is None:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix assumes self.store.get(key) returns None for a missing key, but the real MooncakeDistributedStore.get() binding returns zero-length bytes when get_buffer(key) is null. With the new if value is None check, a real missing key falls through and returns 200 with an empty body instead of 404.

The updated unit test passes because the fake store models missing keys as None, which does not match the real binding behavior. Please either change the binding contract so missing keys return None, or make handle_get() distinguish existence separately, for example by checking is_exist(key) before returning b"" as a valid stored value.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this. I updated the REST GET path to use the binding's existence contract instead of using the value truthiness/None result to decide whether to return 404.

handle_get() now checks self.store.is_exist(key) first, returns 404 when it returns 0, returns 500 on a negative existence-check result, and only then calls get(). Empty bytes are preserved as a valid 200 response for existing keys. I also added a second existence check when get() returns b"" so the REST handler does not turn a missing/null buffer into a false 200 response.

The unit tests now model the real binding behavior: the missing-key case uses is_exist == 0 while get() may still be b"". I also added coverage for existence-check failure, unexpected None from get(), and the empty-byte ambiguity after get().

Validation:

Ran 7 focused GET handler tests in 0.017s

OK

git diff --check also passes locally, with only Windows LF/CRLF warnings.

@Lin-z-w Lin-z-w left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix. LGTM.

@ykwd
ykwd merged commit 6cc11d3 into kvcache-ai:main Jun 24, 2026
22 checks passed
he-yufeng added a commit to he-yufeng/Mooncake that referenced this pull request Jun 24, 2026
handle_exist used bool(is_exist(key)) to build the response body.
is_exist() returns 0 (not found), a positive integer (found), or a
negative integer (store error). bool(-1) is True, so a store error was
silently surfaced as HTTP 200 {"exists": true} instead of HTTP 500.

Fix: check exists < 0 and return 500 "Exist check failed", mirroring
the pattern introduced for handle_get in kvcache-ai#2587. Also replace bool(exists)
with exists > 0 for clarity.

Adds test_handle_exist_store_error to cover the previously untested
negative-return path.

Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
ZhijunLStudio pushed a commit to ZhijunLStudio/Mooncake that referenced this pull request Jul 2, 2026
…#2587)

Co-authored-by: VectorPeak <VectorPeak@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants