On-demand calculation for Ghostdag for Higher Levels #494

coderofstuff · 2024-06-22T00:38:07Z

Introduction

Currently, for each block that we process header processor will calculate ghostdag for all levels. This preprocessing allows for the faster calculation of the pruning proof during pruning but imposes a significant overhead for the processing of each block. It is far better to defer this higher level GD calculation for when it's actually needed - during pruning proof building.

Main Changes

During header processing, calculate only the ghostdag for the header at level 0 (primary ghostdag). So instead of calculating for 255 levels, we only calculate for the one.
Calculate the ghostdag for when it's needed (when build_pruning_proof is called):
- during initial IBD
- during pruning point movement
Introduces 2 new prefixes:
- TempGhostdag
- TempGhostdagCompact
Includes upgrade logic from version 3 to 4 that doesn't require a DB reset
- Deletes all higher level GD records which are no longer necessary given that these will be calculated on-demand

Not included

minimization of proof size

someone235

It shouldn't be a blocker for merge, but it would be nice if each function will have a comment with a bit of explanation, because it can be hard to get the guiding logic for the whole process if you're not familiar with it

consensus/src/consensus/services.rs

consensus/src/pipeline/header_processor/processor.rs

consensus/src/pipeline/pruning_processor/processor.rs

consensus/src/processes/pruning_proof/mod.rs

consensus/src/processes/ghostdag/ordering.rs

Co-authored-by: Ori Newman <[email protected]>

apply_proof only inserts parent entries for a header from the proof into the relations store for a level if there was GD data in the old stores for that header. This adds a check to filter out parent records not in relations store

…cient root This happens when there's not enough headers in the pruning proof but it satisfies validation

relations.get_parents on GD gets extra parents that aren't in the current GD store. so get_blue_work throws an error next, ORIGIN was mising from the GD so add that

…nt_proof

The new prefixes added are compatible with the old version. We don't want to trigger a db delete with this change

It's already on there

This prevents the case where new root is an anticone of old root

When building pruning point proof ghostdag data, ignore blocks before root

consensus/src/processes/pruning_proof/mod.rs

michaelsutton

Initial pass -- minor details out of core changes

consensus/src/pipeline/header_processor/processor.rs

consensus/src/processes/ghostdag/protocol.rs

- use BlueWorkType - fix some comments

coderofstuff changed the base branch from master to dev June 22, 2024 01:04

someone235 requested changes Jun 22, 2024

View reviewed changes

coderofstuff and others added 24 commits June 25, 2024 18:56

Refactor pruning proof validation to many functions

5d3b10f

Co-authored-by: Ori Newman <[email protected]>

Use blue score as work for higher levels

40f1cc9

Co-authored-by: Ori Newman <[email protected]>

Remove pruning processor dependency on gd managers

4af6773

Co-authored-by: Ori Newman <[email protected]>

Consistency renaming

1df5a22

Co-authored-by: Ori Newman <[email protected]>

Update db version

d12592c

Co-authored-by: Ori Newman <[email protected]>

GD Optimizations

2bea765

Co-authored-by: Ori Newman <[email protected]>

Remove remnant of old impl. optimize db prefixes

902b217

Ensure parents are in relations; Add comments

7f1f412

apply_proof only inserts parent entries for a header from the proof into the relations store for a level if there was GD data in the old stores for that header. This adds a check to filter out parent records not in relations store

Match depth check to block_at_depth logic

f49478a

Use singular GD store for header processing

879c135

Relax the panic to warn when finished_headers and couldn't find suffi…

56c0b40

…cient root This happens when there's not enough headers in the pruning proof but it satisfies validation

Error handling for gd on higher levels

43e9f9e

relations.get_parents on GD gets extra parents that aren't in the current GD store. so get_blue_work throws an error next, ORIGIN was mising from the GD so add that

remove using deeper requirements in lower levels

34f20ab

Fix missed references to self.ghostdag_stores in validate_pruning_poi…

2654b25

…nt_proof

Refactoring for single GD header processing

ba04929

Add assertion to check root vs old_root

a45b571

Lint fix current_dag_level

edb5cd3

Keep DB Version at 3

e81394f

The new prefixes added are compatible with the old version. We don't want to trigger a db delete with this change

Cleanup apply_proof logic and handle more ghostdag_stores logic

0e8c788

remove simpa changes

56f9dab

Remove rewriting origin to primary GD

c5be8ad

It's already on there

More refactoring to use single GD store/manager

8d15e27

Lint fixes

1c6b585

warn to trace for common retry

273aa81

coderofstuff force-pushed the gd-optimization branch from d347f1b to 273aa81 Compare June 26, 2024 00:56

coderofstuff added 3 commits June 25, 2024 23:11

Address initial comments

f9b3fda

Remove "primary" in ghostdag store/manager references

ca8bb42

Add small safety margin to proof at level 0

61183fa

This prevents the case where new root is an anticone of old root

coderofstuff added 5 commits August 27, 2024 01:07

More merge fixes

4c9f997

Merge pull request #9 from someone235/gd-proof-ignore-root-past

42a3030

When building pruning point proof ghostdag data, ignore blocks before root

Merge branch 'dev' into gd-optimization

4f2aace

Merge branch 'dev' into gd-optimization

bc2e648

Refactor relations services into self

56b4392

someone235 requested changes Aug 29, 2024

View reviewed changes

consensus/src/processes/pruning_proof/mod.rs Outdated Show resolved Hide resolved

consensus/src/processes/pruning_proof/mod.rs Outdated Show resolved Hide resolved

consensus/src/processes/pruning_proof/mod.rs Outdated Show resolved Hide resolved

someone235 requested changes Aug 29, 2024

View reviewed changes

consensus/src/processes/pruning_proof/mod.rs Outdated Show resolved Hide resolved

coderofstuff added 2 commits August 29, 2024 20:00

Use blue_work for find_selected_parent_header_at_level

66d4ebe

Comment fixes and small refactor

18158e6

coderofstuff requested review from someone235 and michaelsutton August 30, 2024 04:17

coderofstuff added 5 commits September 1, 2024 11:03

Revert rename to old root

8d042b2

Merge branch 'dev' into gd-optimization

09f1443

Merge branch 'dev' into gd-optimization

2913bda

Merge branch 'dev' into gd-optimization

095a9f3

Lint fix from merged code

65948d3

coderofstuff force-pushed the gd-optimization branch from 6bd441a to 65948d3 Compare September 6, 2024 23:49

michaelsutton deleted the branch kaspanet:master September 11, 2024 18:34

michaelsutton closed this Sep 11, 2024

michaelsutton reopened this Sep 11, 2024

michaelsutton changed the base branch from dev to master September 11, 2024 18:42

coderofstuff added 6 commits September 18, 2024 12:16

Merge branch 'master' into gd-optimization

6021a1e

Merge branch 'master' into gd-optimization

009f9a8

Merge branch 'master' into gd-optimization

a60bd8c

Merge branch 'master' into gd-optimization

f1d6904

Merge branch 'master' into gd-optimization

362e564

Merge branch 'master' into gd-optimization

7b7d659

michaelsutton reviewed Oct 15, 2024

View reviewed changes

coderofstuff added 2 commits October 15, 2024 15:44

Some cleanup

2e8deab

- use BlueWorkType - fix some comments

remove last reference to ghostdag_primary_*

4168456

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

On-demand calculation for Ghostdag for Higher Levels #494

On-demand calculation for Ghostdag for Higher Levels #494

coderofstuff commented Jun 22, 2024 •

edited

Loading

someone235 left a comment

michaelsutton left a comment

On-demand calculation for Ghostdag for Higher Levels #494

Are you sure you want to change the base?

On-demand calculation for Ghostdag for Higher Levels #494

Conversation

coderofstuff commented Jun 22, 2024 • edited Loading

Introduction

Main Changes

Not included

someone235 left a comment

Choose a reason for hiding this comment

michaelsutton left a comment

Choose a reason for hiding this comment

coderofstuff commented Jun 22, 2024 •

edited

Loading