Streaming replication slots are a pending feature in PostgreSQL 9.4, as part of the logical changeset extraction feature.

What are they for, what do you need to know, what changes?

What are replication slots?

Streaming replication slots are a new facility introduced in PostgreSQL 9.4. They are a persistent record of the state of a replica that is kept on the master server even when the replica is offline and disconnected.

They aren’t used for physical replication by default, so you’ll only be dealing with them if you enable their use. This post explains what they do, why they’re useful for physical replication, and why they’re necessary for logical replication.

Why slots?

As part of the ongoing work Andres has been doing on log-streaming logical replication, changeset extraction, and bidirectional replication, there’s been the need to better control WAL retention.

As streaming replication currently works, the master doesn’t make any effort to retain extra WAL for a standby that’s fallen behind. If the standby gets too far behind, the master will delete WAL segments the standby still needs to replay, and the standby cannot recover. You get an error like:

ERROR:  requested WAL segment 00000001000000010000002D has already been removed

To prevent this, you are currently expected to configure continuous archiving and provide a restore_command to give the replica access to the archive, or be willing to re-create a standby that falls behind.

A less reliable alternative is to set wal_keep_segments to a “high enough” value to ensure that the replica never falls too far behind – but there’s no real way to guess what is high enough, and the higher the value is set the greater the risk the master will be able to run out of space in pg_xlog. So generally archiving is preferred.

The master could retain WAL for a replica while it’s connected, but as soon as the replica disconnects for any reason the master forgets all about it. As transient disconnects are common, this means there’s not much point in trying to retain WAL just for connected replicas.

We’ve gotten away with this so far because the option of falling back to continuous archiving has been available for when the master discards WAL the replicas still need. When combined with or scripts likeWAL-E, a shared network file system, or a shared host to scp archives to/from, it’s done the job pretty well. It’s a bit confusing for new users, but it works well.

With log streaming logical replication we can’t fall back on continuous archiving anymore, because the master must read WAL and decode it to send the results to the replicas; it can’t just send raw WAL records. Replicas cannot just read raw WAL from the master and extract what they require themselves because only the master has a record of transaction ID statuses, the local oids of objects, etc.

We could just teach the master to run restore_command (something people seem to assume happens anyway) – but that could get very inefficient when many replicas want the same WAL archive. We’d need a caching system and other complexity. It also doesn’t get rid of the need for the admin to manage retention for archived WAL.

Unlike with physical replication, the option of just re-seeding a replica that has fallen too far behind is not acceptable for logical replication. There may be other databases on the replica host that contain their own original data. In the bidirectional replication case, the replica database may also contain newdata not yet replicated to other nodes. So we can’t just write the replica off and make a new pg_basebackupfrom the master or re-rsync it like we do with physcial replication.

The introduction of slots is intended to deal with this issue by giving the master persistent knowledge of the state of replicas, even when they are disconnected. While it was added for logical replication, the concept is useful in physical replication as well, as it lets the master retain as much WAL as is required by all its standbys and no more. There’s no guesswork like there is with wal_keep_segments. There’s no need to set up a separate WAL archiving system and manage retention in it. So long as you have enough space in pg_xlog for the extra WAL, it just works. If a replica disconnects the master can keep WAL for it until it reconnects and replays the WAL.

The downside of this approach is that it adds another way for unbounded WAL growth to fill pg_xlog – when a replica cannot receive or replay WAL fast enough or remains offline for a long time, the master will keep on retaining WAL until pg_xlog fills up and it starts reporting errors to clients. This can already happen on a master with long checkpoint intervals under heavy load, but it’s much easer to trigger with slots as a replica that goes ofline indefinitely will cause pg_xlog to fill up.

While monitoring and adjustment of pg_xlog capacity is already necessary for a busy PostgreSQL server, it is much more so for a server that is using slots to manage WAL retention for replication.

Operational impact

By default, physical replication will not use slots. Nothing changes. You must use WAL archiving on top of streaming to ensure reliable replication.

You can enable the use of replication slots for physical replication with the primary-slotname parameter in recovery.conf. The physical replication slot must already have been created on the master with the pg_create_physical_replication_slot(...) function; see the docs.

When using slots to manage physical replication, WAL archiving is no longer required for replication (though you may wish to retain it for PITR anyway). You don’t need to deal with the difficulties of managing WAL retention amongst multiple replicas that may or may not be online, with replay positions that are not known to the master when offline. You don’t need to manage the space in the archive store.

Instead, you need to keep an eye on space in pg_xlog. WAL retention is managed automatically for you, but you must ensure there’s enough space available.

If a replica is retired from service or unrecoverably lost due to failure of the host, you must manually remove the slot from the master because the master won’t delete any xlog until you do. If you have a system where the master produces xlog rapidly, this may require a lot of pg_xlog space or fairly quick action. A monitoring system like Icinga or Zabbix is strongly recommended.

A replica that is unable to keep up the master must be fixed or retired to prevent the master from running out of xlog space.

With slots you no longer need to monitor the replicas to determine their state, as accurate and up to date information is always available in the pg_replication_slots view. So all you need to do is monitor the master’s pg_xlog space and the status of all replication slots.

Any wal_keep_segments parameter will be respected as a minimum for WAL retention.

So: using slots for physical replication is a trade-off. You don’t need archiving anymore, but you have to monitor the state of the system more closely to avoid disrupting the master.

Future work

To reduce the need for a large pg_xlog space allocation on what could be expensive high-performance storage, in future slots may be extended to support moving WAL to a separate location once it’s no longer required for the master’s own checkpointing.

参考:

http://blog.2ndquadrant.com/postgresql-9-4-slots/

Streaming replication slots in PostgreSQL 9.4的更多相关文章

  1. 配置PostgreSQL Streaming Replication集群

    运行环境: Primary: 192.168.0.11 Standby: 192.168.0.21, 192.168.0.22 OS: CentOS 6.2 PostgreSQL: 9.1.2 版本以 ...

  2. PostgreSQL 9.3 Streaming Replication 状态监控

    postgresql是使用Streaming Replication来实现热备份的,热备份的作用如下: 灾难恢复 高可用性 负载均衡,当你使用Streaming Replication来实现热备份(h ...

  3. PostgreSQL Streaming Replication的FATAL ERROR

    磨砺技术珠矶,践行数据之道,追求卓越价值回到上一级页面: PostgreSQL集群方案相关索引页     回到顶级页面:PostgreSQL索引页[作者 高健@博客园  luckyjackgao@gm ...

  4. PostgreSQL的streaming replication

    磨砺技术珠矶,践行数据之道,追求卓越价值回到上一级页面: PostgreSQL集群方案相关索引页     回到顶级页面:PostgreSQL索引页[作者 高健@博客园  luckyjackgao@gm ...

  5. postgresql Streaming Replication监控与注意事项

    一监控Streaming Replication集群 1 pg_stat_replication视图(主库端执行) pid Wal sender process的进程ID usesysid 执行流复制 ...

  6. [笔记] postgresql 流复制(streaming replication)

    基本环境说明: os:FreeBSD 9.3 postgresql version: master:192.168.56.101 standby:192.168.56.102 安装过程略,基于pkg包 ...

  7. PostgreSQL 9.3发布

    9月9日,PostgreSQL全球开发组宣布了9.3版发布的消息.从2010年9.0版开始,PostgreSQL已经连续四个版本稳定地按时在每年9月中旬发布,从一个侧面也显示了开发团队的强大实力. 9 ...

  8. PostgreSQL Replication之第十四章 扩展与BDR

    在这一章中,将向您介绍一个全新的技术,成为BDR.双向复制(BDR),在PostgreSQL的世界里,它绝对是一颗冉冉升起的新星.在不久的将来,许多新的东西将会被看到,并且人们可以期待一个蓬勃发展的项 ...

  9. PostgreSQL Replication之扩展与BDR

    在这一章中,将向您介绍一个全新的技术,成为BDR.双向复制(BDR),在PostgreSQL的世界里,它绝对是一颗冉冉升起的新星.在不久的将来,许多新的东西将会被看到,并且人们可以期待一个蓬勃发展的项 ...

随机推荐

  1. (六)ICMP、ping

    ping命令是ICMP协议 127.0.0.1 ifconfig -all

  2. 协同过滤 CF & ALS 及在Spark上的实现

    使用Spark进行ALS编程的例子可以看:http://www.cnblogs.com/charlesblc/p/6165201.html ALS:alternating least squares ...

  3. 在centos7上作用mongodb

    安装服务端 yum install mongodb-server 安装客户端 yum install mongodb 版本 mongo --version 是否安装了mongodbwhich mong ...

  4. 修改内联CSS(点击按钮连续改变文字大小、位置,.animate()方法)

    $(document).ready(function(){            $('#swticher-large').click(function(){                var $ ...

  5. zoj 1789 The Suspects

    好高兴,又AC一道 ,不过是很类似的两道..还是好高兴呀思想跟2833是一样的,不过要重新设计输入和输出.老师上课又重新讲解了一下,因为嫌疑人已知是0,所以加入集中时应该默认让数值小的做树根,即最终让 ...

  6. Java里正则表达式

    java.util.regex是一个用正则表达式所订制的模式来对字符串进行匹配工作的类库包.它包括两个类:Pattern和Matcher Pattern 一个Pattern是一个正则表达式经编译后的表 ...

  7. 微信网页授权,微信登录,oauth2

    微信官方文档: http://mp.weixin.qq.com/wiki 微信公众平台OAuth2.0授权详细步骤如下: 1. 用户关注微信公众账号.2. 微信公众账号提供用户请求授权页面URL.3. ...

  8. 前台发送字符串给后台,格式为(a,b,c,d),后台将字符串转换成数组遍历后进而更新CheckId

    using Newtonsoft.Json; using System; using System.Collections.Generic; using System.Configuration; u ...

  9. 2、Linux系统root用户忘记密码的重置方式

    .界面按空格暂停,按E .找到UTF-,在后面空格后输入init=/bin/sh 然后CHRL+X启动 .进入到这个界面,输入mount -o remount,rw / .输入passwd..然后输入 ...

  10. Python SocketServer源码分析

    1      XXXServer 1.1      BaseSever 提供基础的循环等待请求的处理框架.使用serve_forever启动服务,使用shutdown停止.同时提供了一些可自行扩展的方 ...