在PostgreSQL 18中,当 `pgoutput` 跳过空事务时,在没有心跳表的情况下推进 `confirmed_flush_lsn`
升级到PostgreSQL 18之后,pgoutput 不再为仅涉及出版物之外表的事务发送 BEGIN/COMMIT。这是设计使然(参见相关问题)。我们的复制槽的 confirmed_flush_lsn 在大量这类“空交易”负载下不会推进。
我们使用一个自定义的Spring Boot消费者,通过 PGReplicationStream(pgoutput,proto_version=4)从逻辑复制槽读取。只有在收到并处理来自流的消息时,确认的LSN才会推进。
我们已经尝试/排除了以下情况:
- 心跳表(出版物之外)
向不在出版物中的虚拟表写入数据,在PG17时会生成 BEGIN/COMMIT 对。PG18已经完全消除了这些对,因此这种做法已不起作用。
* 心跳表在出版物内
工作机制上可行,但我们把所有解码后的消息转发到Kafka,因此每次心跳写入都会产生一个虚假的变更事件,消费者必须过滤掉。我们希望避免污染Kafka主题。
* status-interval-sec > 0(周期性的待机状态更新)
在 pg_replication_slots 中推进 write_lsn/flush_lsn,但不会推进 confirmed_flush_lsn,这正是我们需要防止WAL累积的部分。
是否有办法在PostgreSQL 18的 pgoutput 逻辑复制槽上推进 confirmed_flush_lsn,在工作负载主要由出版物外的事务组成且不使用心跳表时?
具体来说,我在寻找:
- 一个WAL发送端 /
pgoutput选项(槽选项、proto_version、出版物参数等),使服务器发出某种周期性的位置标记或保活信号,携带当前的WAL位置——客户端可以据此有意义地调用stream.setFlushedLSN()/stream.forceUpdateStatus()。 - 或者一种服务器端机制(GUC、复制命令等),能够定期将槽刷新到当前的WAL位置,而不需要客户端可见的变更事件。
环境:
- PostgreSQL 18.2
- JDBC驱动42.7.x,
PGReplicationStream pgoutput,proto_version=4- Java 17 / Spring Boot 2.4.13
解决方案
该改动是在PostgreSQL v15中由 [PostgreSQL commit d5a9d86d8ffcadc52ff3729cd00fbd83bc38643c] 引入的。
从提交信息和代码注释,我们可以得到一些有趣的信息。
From the commit message:
When skipping empty transactions in synchronous replication mode, we send
a keepalive message to avoid delaying such transactions.
A code comment:
/*
* Maintain a per-transaction level variable to track whether the transaction
* has sent BEGIN. BEGIN is only sent when the first change in a transaction
* is processed. This makes it possible to skip sending a pair of BEGIN/COMMIT
* messages for empty transactions which saves network bandwidth.
*
* This optimization is not used for prepared transactions because if the
* WALSender restarts after prepare of a transaction and before commit prepared
* of the same transaction then we won't be able to figure out if we have
* skipped sending BEGIN/PREPARE of a transaction as it was empty. This is
* because we would have lost the in-memory txndata information that was
* present prior to the restart. This will result in sending a spurious
* COMMIT PREPARED without a corresponding prepared transaction at the
* downstream which would lead to an error when it tries to process it.
*
* XXX We could achieve this optimization by changing protocol to send
* additional information so that downstream can detect that the corresponding
* prepare has not been sent. However, adding such a check for every
* transaction in the downstream could be costly so we might want to do it
* optionally.
*
* We also don't have this optimization for streamed transactions because
* they can contain prepared transactions.
*/
这表明有两种可能的解决办法:
- 在你的订阅中使用流式模式。
对于逻辑复制,这将和以下情况一起工作
sql
ALTER SUBSCRIPTION sub_name SET (streaming = parallel);
在协议层面,你需要调用 START_REPLICATION 并使用 pgoutput 选项 streaming parallel。
2.使用同步复制。这大概不是你想要的做法。