当我尝试在Redpanda（或Apache Kafka）上创建主题时，任务因为这个特定错误而失败：

{
    "name": "MY-table",
    "connector": {
        "state": "RUNNING",
        "worker_id": "172.28.0.30:8083"
    },
    "tasks": [
        {
            "id": 0,
            "state": "FAILED",
            "worker_id": "172.28.0.30:8083",
            "trace": "org.apache.kafka.connect.errors.ConnectException: Creation of database schema history topic failed, please create the topic manually\n\tat io.debezium.storage.kafka.history.KafkaSchemaHistory.initializeStorage(KafkaSchemaHistory.java:567)\n\tat io.debezium.relational.HistorizedRelationalDatabaseSchema.initializeStorage(HistorizedRelationalDatabaseSchema.java:90)\n\tat io.debezium.connector.sqlserver.SqlServerConnectorTask.start(SqlServerConnectorTask.java:88)\n\tat io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:141)\n\tat org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.initializeAndStart(AbstractWorkerSourceTask.java:280)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:202)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:259)\n\tat org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.run(AbstractWorkerSourceTask.java:77)\n\tat org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:236)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\nCaused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidPartitionsException: Can not increase partition count due to FD limit\n\tat java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)\n\tat java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2022)\n\tat org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:180)\n\tat io.debezium.storage.kafka.history.KafkaSchemaHistory.initializeStorage(KafkaSchemaHistory.java:554)\n\t... 13 more\nCaused by: org.apache.kafka.common.errors.InvalidPartitionsException: Can not increase partition count due to FD limit\n"
        }
    ],
    "type": "source"
}

解决方案

问题原因是Redpanda（或Kafka）用户允许的最大打开文件数（ulimit -n）太低。

Redpanda容器的默认FD限制为1024。

虽然我使用了docker-compose堆栈，包括Debezium、Redpanda和 ClickHouse，但我还是检查了Redpanda容器当前的FD使用情况。

在Redpanda容器上检查当前FD使用情况

#Check FD limit inside Redpanda container
docker exec 604259b9b636 sh -c "ulimit -n"

#Check how many FDs are currently open
ls /proc/$(docker inspect --format '{{.State.Pid}}' 604259b9b636)/fd | wc -l

#Check host-level limits for that process
cat /proc/$(docker inspect --format '{{.State.Pid}}' 604259b9b636)/limits | grep "open files"

我执行了这些命令来查明1024的值是从哪里配置的

echo "=== OS Default ===" && ulimit -n
echo "=== /etc/security/limits.conf ===" && grep nofile /etc/security/limits.conf
echo "=== Docker daemon.json ===" && cat /etc/docker/daemon.json 2>/dev/null || echo "Not set"
echo "=== systemd Docker limit ===" && systemctl show docker | grep LimitNOFILE
echo "=== OS version ===" && cat /etc/os-release | grep PRETTY

修复 — 生产服务器（零停机）

步骤1 — 创建daemon.json

sudo tee /etc/docker/daemon.json <<EOF
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "5"
  },
  "storage-driver": "overlay2",
  "default-ulimits": {
    "nofile": {
      "Name": "nofile",
      "Hard": 1048576,
      "Soft": 1048576
    }
  }
}
EOF

步骤2 — 重新加载Docker守护进程（不会停止任何容器）

sudo systemctl reload docker

# Verify daemon picked it up
docker info | grep -i "ulimit\|nofile" 2>/dev/null
cat /etc/docker/daemon.json

步骤3 — 立即应用到正在运行的Redpanda（无需重启）

PID=$(docker inspect --format '{{.State.Pid}}' 604259b9b636)
sudo prlimit --pid $PID --nofile=1048576:1048576

# Verify
sudo cat /proc/$PID/limits | grep "open files"
# Expected: Max open files  1048576  1048576

步骤4 — 重启失败的连接器

要实现永久解决方案，需要在docker-compose文件中添加以下块

yaml
services:
  redpanda:
    image: redpandadata/redpanda:...
    ulimits:
      nofile:
        soft: 1000000
        hard: 1000000
    ...