Skip to content

Hive lock-free commits can break commits using existing Hive locking mechanism #26360

@laithalzyoud

Description

@laithalzyoud

This PR #25445 introduced lock-free commits in iceberg HMS catalog, however if locking is used then there is no need to pass {"expected_parameter_key": "metadata_location", "expected_parameter_value", metadataLocation} to the environmentContext. This leads to unnecessary operation on the metastore side to verify that the metadata location is still the same before commiting while Hive locking is already being used to ensure the commit is atomic, example SQL query:

UPDATE "TABLE_PARAMS"
SET "PARAM_VALUE" = 'P2' -- The new path
WHERE "TBL_ID" = 123
  AND "PARAM_KEY" = 'metadata_location'
  AND "PARAM_VALUE" = 'P1'; -- The expected old path

I'm using HMS 3.1.2 and this was a breaking change where commits were not succeeding (probably due to a bug in HMS or underlying MySQL DB) that caused transactions to become stuck in the DB leading to timeouts and failed commits and eventually the MySQL DB used by the HMS wasn't able to process other transactions timely leading to timeout across all other HMS operations until the underlying DB was restarted to clear the stuck transactions and the Trino version was rolled back.

This change should only affect Trino deplyoments that want to use lock-free commits and have iceberg.hive-catalog.locking-enabled set as false

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions