configuring a warm standby, the easy way

PITR made easy

Joshua DrakeUnited States PostgreSQL

Software in the Public InterestCommand Prompt, Inc.

What is PITR

Log shipping/Warm Standby/Replication

What can you do with it?

Simple asynchornous failover scenariosLow impact backups

Simple archivals

What can't you do with it?

Read only slave

Why is PITR hard?

It was written by open source engineers.

Uses the many tools to one solution

rsync?nfs?

walmgr?scp?

Push or pull?Something else?

What makes PITR easy?

PITR toolsBSD Licensed

Written in Python(theoretically cross platform)

What is PITRTools

A simple wrapper around all the utilities you already have.

ssh,rsync,pg_standby,postgresql

Features?

Warm standbyCold Storage

Fail over (actionable)Arbitrary alerts (monitoring)

How does it work?

Queue

SlaveMastersuccess

else

The process on the archiver/master

On the archiver/Master:● Configure ssh key for postgres user to standby/slave

● Configure archiver.ini● cmd_archiver -C /path/to/archiver.ini -I● Edit the postgresql.conf change:

● archive_mode to on● archive_command to:

● cmd_archiver -C /path/to/archiver.ini -F %p● restart postgresql● apply cmd_standby.sql to database of pitr user (usually postgres)● psql -U postgres < /path/to/cmd_standby/sql

The archiver

The archiver calls cmd_archive.py which calls rsync to “push” the file to the slave or queue.

Usage: cmd_archiver [options] arg1 arg2

Options: -h, --help show this help message and exit -F FILE, --file=FILE Archive file -C FILE, --config=FILE the name of the archiver config file -f, --flush Flush all remaining archives to slave -I, --init Initialize master environment

The archiver config file

Useful but obvious parameters

[DEFAULT]state: onlinepgdata: /var/lib/postgresql/8.3/mainrsync_bin: /usr/bin/rsyncrsync_version = 2 # Because RH ships old softwareuser: postgrestimeout: 10debug: onssh_debug: off

The “other” archiver options

r_archivedir: /var/lib/postgresql/archivel_archivedir: /var/lib/postgresql/archiversync_version = 2slaves: 192.168.1.201,192.168.1.202notify_ok: /var/lib/postgresql/etc/archiver_ok.shnotify_warning: /var/lib/postgresql/etc/archiver_warning.shnotify_critical: /var/lib/postgresql/etc/archiver_critical.sh

cmd_standby.py

Works in conjunction with pg_standby Supports failover Supports recovery to a point in time Included with 8.3, available for 8.1 and 8.2 (8.1

is flaky)

The process on the slave

On the standby/slave:● Configure SSH key for postgres user to master● Configure cmd_standby.ini● Initialize environment

● cmd_standby -C /path/to/cmd_standby.ini -I● Start base backup

● cmd_standby -C /path/to/cmd_standby.ini -B● Start standby

● cmd_standby -C /path/to/cmd_standby.ini -S

The standby configuration

[DEFAULT]pgversion: 8.2numarchives: 10ssh: /usr/bin/sshrsync: /usr/bin/rsyncpg_standby: /usr/lib/postgresql/8.3/bin/pg_standbypg_ctl: /usr/lib/postgresql/8.3/bin/pg_ctlr_psql: /usr/lib/postgresql/8.3/bin/psqlport: 6000master_public_ip: 192.168.3.254master_local_ip: 127.0.0.1user: postgresdebug: off ssh_timeout: 30

Configuring the standby cont...

archivedir: /data2/pgsql/archive/pgdata: /data1/pgsql/data/postgresql_conf: /var/lib/postgresql/etc/postgresql.confpg_hba_conf: /var/lib/postgresql/etc/pg_hba.confnotify_critical: /var/lib/postgresql/etc/standby_critical.shnotify_warning: /var/lib/postgresql/etc/standby_warning.shnotify_ok: /var/lib/postgresql/etc/standby_ok.shaction_failover: /var/lib/postgresql/pitr_tools/failover.sh

Start the standby

Usage: cmd_standby [options] arg1 arg2

Options: -h, --help show this help message and exit -A start|stop, --action=start|stop|stop_basebackup Start or Stop PostgreSQL -B, --basebackup Start/Stop a base backup -C FILE, --config=FILE Name of the archiver config file -F VALUE, --failover=VALUE If you are serious, set -F999 -I, --dbinit Use before -B -P, --ping Is my master alive? -R TIMESTAMP, --recovertotime=TIMESTAMP To restore to a specific point in time -S, --standby Enter standby mode

Is the archiver working?

Is the standby is working?

How do I?

Use PITRTools to simplify backups? Take the archiver offline? Configure send_nsca (nagios) for alerts Donate to PostgreSQL?

Simplify backups

On the standby/slave: cmd_standby -C /var/lib/postgresql/etc/cmd_standby.ini -Astop

tar -czvf /backups/database.`date +%a`.tar.gz /var/lib/pgsql/data

cmd_standby -C /var/lib/postgresql/etc/cmd_standby.ini -S

You could also use rsync with a network volume to make the backup more efficient (after stopping the standby)

Take the archiver offline

Edit /var/lib/postgresql/etc/cmd_archiver.ini[DEFAULT]

; online or offline

state: offline

When you are ready to bring it back online, just change to online. No restarts on archiver or standby required.

Configure send_nsca

Example ok.sh for send_nsca

#!/bin/sh

echo "shoggoth;CMD_ARCHIVER;0;Archived to Yugg"| \

/usr/sbin/send_nsca -to 10 -H monitor -d ';' -c /etc/nagios/send_nsca.cfg

Donate to PostgreSQL

For United States PostgreSQL: https://www.postgresql.us/donate

For PostgreSQL Generally: http://www.postgresql.org/about/donate

Questions

Hit me