core/state: introduce the TransitionState object (verkle transition part 1) #31634

gballet · 2025-04-14T08:04:39Z

This is the first part of #31532

It maintains a series of conversion maker which are to be updated by the conversion code (in a follow-up PR, this is a breakdown of a larger PR to make things easier to review). They can be used in two modes:

At genesis, by forcing the setting of a conversion marker as "ended". It replaces EnableVerkleAtGenesis.
During the conversion, by storing the conversion markers when the block has been processed. This is meant to be written in a function that isn't currently present, hence this TODO.

Part of #31583

core/state/database.go

Signed-off-by: Guillaume Ballet <[email protected]>

rjl493456442 · 2025-07-25T03:28:54Z

I have added some core changes into this PR, specifically:

remove the transition state pre-initialization around the Genesis, which i believe it's not the best solution
change the CurrentAccountAddress to CurrentAccountHash, getting rid of the pointer

Let's ship the transition state structure definition in this PR first and then pile up more stuff on top.

rjl493456442 · 2025-07-29T12:00:43Z

core/state/database.go

+// TransitionState is a structure that holds the progress markers of the
+// translation process.
+type TransitionState struct {
+	CurrentAccountAddress *common.Address // addresss of the last translated account


Using a pointer here is unnecessarily complex, we can replace it with the account address hash instead, reasons:

Verkle migration is traversed in the order of account address hashes

A zero hash can represent that nothing has been migrated yet

Using a hash allows the entire TransitionState to remain a pure value type, with no pointers involved

We have already discussed that multiple times. I could confirm that not using the pointer was breaking the changes owing to some side effect somewhere else that I don't remember just now, I'll have to debug when I get back to Berlin - but this PR has already been sitting for >7 months and it's time to be productive and merge it. Once I am able to perform transitions with master, we can redesign this since we'd have a reference.

Regarding the hash, this means that if you have a storage tree for which the conversion takes more than one block, you need to resolve the address from the hash to open the storage tree. that, in turns, means that you need something like the preimage store, which is very slow. What we use is some side file that has been generated to be in enumeration order, and we would have to backtrack that file for an unknown amount of bytes (well, we could store that offset as well, but then what's the difference with a pointer?)

Regarding the hash, this means that if you have a storage tree for which the conversion takes more than one block, you need to resolve the address from the hash to open the storage tree. that, in turns, means that you need something like the preimage store, which is very slow.

I don't agree with this. The entire conversion is hash-based, we traverse in trie order, which has nothing to do with the account address. I do agree it's quite annoying that the ordering is determined by the hash.

it's time to be productive and merge it

I totally agree. But the process of moving things forward involves integrating codes piece by piece, in the proper way. Unfortunately, some necessary changes are inevitable sometimes.

I could confirm that not using the pointer was breaking the changes owing to some side effect somewhere else that I don't remember just now

Yeah, maybe I miss something. I can live with the current workaround. But I will always keep my suggestions to simplify the code.

rjl493456442 · 2025-07-29T12:01:16Z

core/state/database.go

+}
+
+// Copy returns a deep copy of the TransitionState object.
+func (ts *TransitionState) Copy() *TransitionState {


This function can be removed if the AccountAddress is replaced with AccountAddressHash

rjl493456442 · 2025-07-29T12:02:20Z

core/state/database.go

@@ -184,6 +236,7 @@ func (db *CachingDB) Reader(stateRoot common.Hash) (Reader, error) {
 			readers = append(readers, newFlatReader(snap))
 		}
 	}
+	ts := db.LoadTransitionState(stateRoot)


It's very expensive before the Verkle activation;

The TransitionState won't be existent before the verkle migration, each lookup ends up with an non-existent database lookup, which traverses the entire LSM Tree layers.

that's a fair point. and I see that your (now deleted) changes serve as a guard. But it lacks the ability to distinguish between the transition and post-transition phases. That might still save some hits though.

As I suggested previously, we should expose IsVerkle information to the state reader, so that we can easily distinguish these scenarios respectively:

Before the verkle activation: IsVerkle == false

The first verkle block: IsVerkle == true && rawdb.ReadTransitionState(stateRoot) == null

The blocks within the conversion:

the conversion hasn't started: IsVerkle == true && rawdb.ReadTransitionState(stateRoot) != null && state.Start == false && state.End == false

the conversion has started: IsVerkle == true && rawdb.ReadTransitionState(stateRoot) != null && state.Start == true && state.End == false

The blocks after the conversion: IsVerkle == true && rawdb.ReadTransitionState(stateRoot) != null && state.End == true

#32084 this change simplifies the thing, exposing the IsVerkle without passing the flag around the entire codebase.

The last unsolved puzzle for me is: how to get rid of TransitionState checking
after the conversion. We can probably store a singleton entry, tracking the block
number and block hash for the last block of conversion.

If deep reorg occurs and rewinds the chain back to conversion, this flag should
be removed.

The benefit is: we can skip the entire TransitionState checking before the verkle, it's the main blocker from merging this change.

rjl493456442 · 2025-07-29T12:02:39Z

core/state/database.go

@@ -223,7 +276,11 @@ func (db *CachingDB) ReadersWithCacheStats(stateRoot common.Hash) (ReaderWithSta

 // OpenTrie opens the main account trie at a specific root hash.
 func (db *CachingDB) OpenTrie(root common.Hash) (Trie, error) {
-	if db.triedb.IsVerkle() {
+	ts := db.LoadTransitionState(root)


It's very expensive before the Verkle activation;

The TransitionState won't be existent before the verkle migration, each lookup ends up with an non-existent database lookup, which traverses the entire LSM Tree layers.

this is the one I don't think we can replace with db.triedb.IsVerkle() because we won't be able to tell the difference between a transition and a post-transition. This being said, it might be enough to guard a call to db.LoadTransitionState...

The only part I'm not clear on, is how you would check that the db is loaded? don't you have to do some kind of empty marker reading either way?

this is the one I don't think we can replace with db.triedb.IsVerkle() because we won't be able to tell the difference between a transition and a post-transition.

The idea for checking db.triedb.IsVerkle first is to skip the entire TransitionState checking before the verkle.
This state is essential to distinguish between a transition and a post-transition.

rjl493456442 · 2025-07-29T12:02:43Z

core/state/database.go

@@ -235,10 +292,11 @@ func (db *CachingDB) OpenTrie(root common.Hash) (Trie, error) {

 // OpenStorageTrie opens the storage trie of an account.
 func (db *CachingDB) OpenStorageTrie(stateRoot common.Hash, address common.Address, root common.Hash, self Trie) (Trie, error) {
+	ts := db.LoadTransitionState(stateRoot)


It's very expensive before the Verkle activation;

The TransitionState won't be existent before the verkle migration, each lookup ends up with an non-existent database lookup, which traverses the entire LSM Tree layers.

Signed-off-by: Guillaume Ballet <[email protected]> Co-authored-by: Gary Rong <[email protected]>

Signed-off-by: Guillaume Ballet <[email protected]>

…32366) This add some of the changes that were missing from #31634. It introduces the `TransitionTrie`, which is a façade pattern between the current MPT trie and the overlay tree. --------- Signed-off-by: Guillaume Ballet <[email protected]> Co-authored-by: rjl493456442 <[email protected]>

gballet marked this pull request as ready for review April 14, 2025 10:49

gballet requested review from fjl, karalabe, rjl493456442 and zsfelfoldi as code owners April 14, 2025 10:49

gballet added the verkle label Apr 14, 2025

gballet commented Apr 28, 2025

View reviewed changes

core/state/database.go Outdated Show resolved Hide resolved

gballet commented Apr 28, 2025

View reviewed changes

core/state/database.go Outdated Show resolved Hide resolved

gballet mentioned this pull request Apr 28, 2025

cmd/evm/internal/t8ntool: support for verkle-at-genesis in evm t8n (verkle transition part 3) #31730

Draft

gballet force-pushed the rebase-master-add-transition-state branch from bb038c6 to 7352993 Compare May 5, 2025 07:59

gballet force-pushed the rebase-master-add-transition-state branch from c38f272 to c0578c1 Compare July 11, 2025 15:16

gballet added 3 commits July 15, 2025 12:15

core/state: introduce the TransitionState object

d3a07a2

fix failing tests

7fd5179

fix the race condition by using lru.Cache

86183a8

Signed-off-by: Guillaume Ballet <[email protected]>

gballet force-pushed the rebase-master-add-transition-state branch from e3f7af8 to 86183a8 Compare July 15, 2025 10:27

rjl493456442 force-pushed the rebase-master-add-transition-state branch from 20731b8 to 86183a8 Compare July 28, 2025 07:55

rjl493456442 reviewed Jul 29, 2025

View reviewed changes

rjl493456442 added the status:triage label Jul 29, 2025

gballet and others added 5 commits August 4, 2025 12:38

don't store conversion state in genesis db in this PR

a483135

Signed-off-by: Guillaume Ballet <[email protected]> Co-authored-by: Gary Rong <[email protected]>

move transition state to its own file

f0f51d1

Signed-off-by: Guillaume Ballet <[email protected]> Co-authored-by: Gary Rong <[email protected]>

make LoadTransitionState cacheless and move it to the overlay pkg

0f37c99

Signed-off-by: Guillaume Ballet <[email protected]> Co-authored-by: Gary Rong <[email protected]>

revert more changes to simplify this PR

24f848d

Signed-off-by: Guillaume Ballet <[email protected]>

core, consensus: avoid transition state read in reader construction

320ea13

rjl493456442 approved these changes Aug 5, 2025

View reviewed changes

rjl493456442 added this to the 1.16.3 milestone Aug 5, 2025

rjl493456442 removed the status:triage label Aug 5, 2025

rjl493456442 merged commit cf50026 into ethereum:master Aug 5, 2025
2 of 3 checks passed

gballet mentioned this pull request Aug 7, 2025

trie, core/state: add the transition tree (verkle transition part 2) #32366

Merged

core/state: introduce the TransitionState object (verkle transition part 1) #31634

core/state: introduce the TransitionState object (verkle transition part 1) #31634

Uh oh!

Conversation

gballet commented Apr 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rjl493456442 commented Jul 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rjl493456442 Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rjl493456442 Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

gballet commented Apr 14, 2025 •

edited

Loading

rjl493456442 commented Jul 25, 2025 •

edited

Loading

rjl493456442 Jul 29, 2025 •

edited

Loading

rjl493456442 Jul 31, 2025 •

edited

Loading