Klever-Go KVM: Throttler slot leak in trie account-data sync causes epoch bootstrap / state sync DoS

Missing slot release paths

After StartProcessing(), the following error paths return without EndProcessing():

1. trie.NewTrie(...) returns an error
2. trie.NewTrieSyncer(...) returns an error
3. trieSyncer.StartSyncing(...) returns an error

Why this matters

NumGoRoutinesThrottler is a strict bounded counter:

  func (ngrt *NumGoRoutinesThrottler) CanProcess() bool {
      valCounter := atomic.LoadInt32(&ngrt.counter)
      return valCounter < ngrt.max
  }

  func (ngrt *NumGoRoutinesThrottler) StartProcessing() {
      atomic.AddInt32(&ngrt.counter, 1)
  }

  func (ngrt *NumGoRoutinesThrottler) EndProcessing() {
      atomic.AddInt32(&ngrt.counter, -1)
  }

  Once leaked, a slot remains consumed for the lifetime of that throttler instance.

  The parent loops in both syncers wait for capacity before starting the next account-data trie sync:

  for !u.throttler.CanProcess() {
      select {
      case <-time.After(timeBetweenRetries):
          continue
      case <-ctx.Done():
          return common.ErrTimeIsOut
      }
  }

So after enough failures, further roots stop progressing and the sync operation eventually returns time is out.

Bootstrap impact

Epoch bootstrap uses these syncers directly and aborts on any error:

  err = e.syncUserAccountsState(e.epochStartMeta.Header.TrieRoot)
  if err != nil {
      return nil, nil, err
  }

  err = e.syncKappAccountsState(e.epochStartMeta.Header.KAppsTrieRoot)
  if err != nil {
      return nil, nil, err
  }

The throttlers for these paths are real bounded throttlers created from numConcurrentTrieSyncers.

Proof of Concept

I verified the bug with the real timeout path, not only with a canceled context.

The PoC below uses:

• a real NumGoRoutinesThrottler with capacity 2
• a real trieSyncer.StartSyncing()
• an empty trie-node cache and a request handler that never supplies nodes
• a short sync timeout (1s) so StartSyncing() returns trie.ErrTimeIsOut

After the first failed sync, one slot remains leaked. After the second failed sync, the throttler is exhausted.

PoC test

  package syncer

  import (
        "context"
        "testing"
        "time"

        commonmock "github.com/klever-io/klever-go/common/mock"
        corethrottler "github.com/klever-io/klever-go/core/throttler"
        "github.com/klever-io/klever-go/data"
        "github.com/klever-io/klever-go/data/trie"
        triestats "github.com/klever-io/klever-go/data/trie/statistics"
        "github.com/stretchr/testify/require"
  )

  func newBaseSyncerForTimeoutPOC(t *testing.T) *baseAccountsSyncer {
        t.Helper()

        storageManager, err := trie.NewTrieStorageManagerWithoutPruning(commonmock.NewMemDbMock())
        require.NoError(t, err)

        return &baseAccountsSyncer{
                hasher:                    commonmock.HasherMock{},
                marshalizer:               &commonmock.MarshalizerMock{},
                trieSyncers:               make(map[string]data.TrieSyncer),
                dataTries:                 make(map[string]data.Trie),
                trieStorageManager:        storageManager,
                requestHandler:            &commonmock.RequestHandlerStub{},
                timeout:                   time.Second,
                cacher:                    commonmock.NewCacherStub(),
                maxTrieLevelInMemory:      5,
                name:                      "timeout-poc",
                maxHardCapForMissingNodes: 1,
        }
  }

  func TestPOC_UserAccountsSyncer_LeaksThrottlerSlotOnTrieTimeout(t *testing.T) {
        thr, err := corethrottler.NewNumGoRoutinesThrottler(2)
        require.NoError(t, err)

        s := &userAccountsSyncer{
                baseAccountsSyncer: newBaseSyncerForTimeoutPOC(t),
                throttler:          thr,
        }

        err = s.syncDataTrie([]byte("missing-root-1"), triestats.NewTrieSyncStatistics(), context.Background())
        require.ErrorIs(t, err, trie.ErrTimeIsOut)
        require.True(t, thr.CanProcess())

        err = s.syncDataTrie([]byte("missing-root-2"), triestats.NewTrieSyncStatistics(), context.Background())
        require.ErrorIs(t, err, trie.ErrTimeIsOut)
        require.False(t, thr.CanProcess())
  }

  func TestPOC_KappAccountsSyncer_LeaksThrottlerSlotOnTrieTimeout(t *testing.T) {
        thr, err := corethrottler.NewNumGoRoutinesThrottler(2)
        require.NoError(t, err)

        s := &kappAccountsSyncer{
                baseAccountsSyncer: newBaseSyncerForTimeoutPOC(t),
                throttler:          thr,
        }

        err = s.syncDataTrie([]byte("missing-root-1"), triestats.NewTrieSyncStatistics(), context.Background())
        require.ErrorIs(t, err, trie.ErrTimeIsOut)
        require.True(t, thr.CanProcess())

        err = s.syncDataTrie([]byte("missing-root-2"), triestats.NewTrieSyncStatistics(), context.Background())
        require.ErrorIs(t, err, trie.ErrTimeIsOut)
        require.False(t, thr.CanProcess())
  }

Command used

  go test ./data/syncer -run 'TestPOC_(User|Kapp)AccountsSyncer_LeaksThrottlerSlotOnTrieTimeout' -count=1

Result

  ok    github.com/klever-io/klever-go/data/syncer      4.005s

This confirms the leak with the real timeout path from trieSyncer.StartSyncing().

Impact

An attacker who can repeatedly cause trie-node sync failures or timeouts during bootstrap can consume the bounded sync throttler until no capacity remains.

Once enough slots are leaked:

• additional account-data trie sync attempts stop making progress
• the parent loop waits until context timeout
• SyncAccounts() fails
• epoch bootstrap fails

This is a core node availability issue. It affects fresh/restarting nodes and validators that need to bootstrap or resync state.

This is not a theoretical issue:

• StartSyncing() performs network-dependent trie-node retrieval
• it already has explicit timeout / failure paths
• the leaked throttler slots are confirmed by runtime PoC

Recommended Fix

Release the slot with defer immediately after StartProcessing() and cancel the defer only if ownership is intentionally transferred, which is not the case here.

Example fix pattern:

  func (u *userAccountsSyncer) syncDataTrie(rootHash []byte, ssh data.SyncStatisticsHandler, ctx context.Context) error {
      u.throttler.StartProcessing()
      defer u.throttler.EndProcessing()

      u.syncerMutex.Lock()
      defer u.syncerMutex.Unlock()

      if _, ok := u.dataTries[string(rootHash)]; ok {
          return nil
      }

      dataTrie, err := trie.NewTrie(...)
      if err != nil {
          return err
      }

      trieSyncer, err := trie.NewTrieSyncer(arg)
      if err != nil {
          return err
      }

      u.trieSyncers[string(rootHash)] = trieSyncer
      return trieSyncer.StartSyncing(rootHash, ctx)
  }

The same pattern should be applied to:

• data/syncer/userAccountsSyncer.go
• data/syncer/kappAccountsSyncer.go

References

• data/syncer/userAccountsSyncer.go
• data/syncer/kappAccountsSyncer.go
• data/trie/sync.go
• core/throttler/numGoRoutinesThrottler.go
• core/bootstrap/process.go
• SECURITY.md