How to setup Canonical Identity Service (Candid) in the HA mode |My Technical Blog

Intro

Canonical Identity Service (Candid) does not require any HA-specific extensions / settings and can be deployed in the HA mode in various ways. Which one is the best one, however? It is hard to say, but the reference architecture is always the most obvious choice. In the following post we will setup Candid in the HA mode based on the following tool set:

PostgreSQL - reliable and highly available backend
Corosync & Pacemaker - messaging and service management
HAProxy - load balancing and SSL termination

This architecture is shown in the following diagram:

The following part contains step-by-step instructions to install and configure Candid in the HA mode. All services are set on LXD containers with Ubuntu Bionic. It is assumed that LDAP is used as the Identity Provider.

Prerequisites

Generate CA-signed certificates for the following services:

Candid - candid.crt + candid.key with FQDN of candid.example.com
PostgreSQL - server.crt + server.key with FQDN of postgres.example.com

Launch LXD containers:

candid-ha-0
candid-ha-1
candid-ha-2

Ensure the following FQDNs are resolvable (e.g. by editing /etc/hosts file on containers):

candid.example.com (pointing to Candid VIP)
ldap.example.com (pointing to Identity Provider IP)
postgres.example.com (pointing to PostgreSQL VIP)

Ensure CA certificate (/etc/ssl/certs/ca.pem) is installed on all containers

PostgreSQL

On all containers install and stop PostgreSQL:

# apt -y install postgresql

# systemctl stop postgresql

On candid-ha-0 container disable PostgreSQL:

# systemctl disable postgresql

On all containers change ownership and permissions of PostgreSQL certificate and key:

# chown postgres:postgres /var/lib/postgresql/10/main/server.crt

# chown postgres:postgres /var/lib/postgresql/10/main/server.key

# chmod 600 /var/lib/postgresql/10/main/server.key

On all containers configure PostgreSQL:

# sed -i "s^ssl_cert_file = '/etc/ssl/certs/ssl-cert-snakeoil.pem'^#ssl_cert_file = '/etc/ssl/certs/ssl-cert-snakeoil.pem'^" /etc/postgresql/10/main/postgresql.conf

# sed -i "s^ssl_key_file = '/etc/ssl/private/ssl-cert-snakeoil.key'^#ssl_key_file = '/etc/ssl/private/ssl-cert-snakeoil.key'^" /etc/postgresql/10/main/postgresql.conf

# cat <<EOF >> /etc/postgresql/10/main/postgresql.conf

listen_addresses = '*'

max_connections = 300

wal_level = hot_standby

synchronous_commit = on

archive_mode = off

max_wal_senders = 10

wal_keep_segments = 256

hot_standby = on

restart_after_crash = off

hot_standby_feedback = on

ssl_cert_file = '/var/lib/postgresql/10/main/server.crt'

ssl_key_file = '/var/lib/postgresql/10/main/server.crt'

EOF

# cat /etc/postgresql/10/main/pg_hba.conf | head -n -3 > /tmp/pg_hba.conf; mv /tmp/pg_hba.conf /etc/postgresql/10/main/pg_hba.conf

# cat <<EOF >> /etc/postgresql/10/main/pg_hba.conf

host replication postgres 10.130.194.10/32 trust

host replication postgres 10.130.194.11/32 trust

host replication postgres 10.130.194.12/32 trust

host replication postgres 10.130.194.253/32 trust

host replication postgres 10.130.194.254/32 trust

hostssl candid candid 10.130.194.10/32 md5

hostssl candid candid 10.130.194.11/32 md5

hostssl candid candid 10.130.194.12/32 md5

hostssl candid candid 10.130.194.253/32 md5

hostssl candid candid 10.130.194.254/32 md5

EOF

NOTE: Replace IP addresses with IP addresses of all containers and IP addresses reserved for Candid and PostgreSQL VIP.

On candid-ha-1 container start PostgreSQL:

# systemctl start postgresql

On candid-ha-2 container initiate the replication and start PostgreSQL:

# rm -rf /var/lib/postgresql/10/main/

# sudo -u postgres pg_basebackup -h 10.130.194.11 -D /var/lib/postgresql/10/main -v --wal-method=stream

# systemctl start postgresql

NOTE: Replace IP addresses with IP addresses of candid-ha-1 container.

HAProxy

On all containers install HAProxy:

# apt -y install haproxy

On all containers configure HAProxy:

cat <<EOF > /etc/haproxy/haproxy.cfg

defaults

timeout connect 10s

timeout client 1m

timeout server 1m

global

tune.ssl.default-dh-param 2048

frontend candid

bind *:443 ssl crt /etc/ssl/private/candid.pem

reqadd X-Forwarded-Proto:\ https

option http-server-close

default_backend candid

backend candid

balance source

hash-type consistent

server candid-ha-1 10.130.194.10:8081 check

server candid-ha-2 10.130.194.11:8081 check

server candid-ha-3 10.130.194.12:8081 check

EOF

NOTE: candid.pem is a concatenation of candid.crt and candid.key files (cat candid.crt candid.key > candid.pem)

NOTE: Replace IP addresses with IP addresses of containers.

On all containers restart HAProxy:

# systemctl restart haproxy

Corosync & Pacemaker

On all containers install Corosync and Pacemaker:

# apt -y install crmsh

On all containers configure Corosync:

# cat <<EOF > /etc/corosync/corosync.conf

totem {

version: 2

token: 3000

token_retransmits_before_loss_const: 10

join: 60

consensus: 3600

vsftype: none

max_messages: 20

clear_node_high_bit: yes

secauth: off

threads: 0

ip_version: ipv4

rrp_mode: none

transport: udpu

}

quorum {

provider: corosync_votequorum

}

nodelist {

node {

ring0_addr: 10.130.194.10

nodeid: 1000

}

node {

ring0_addr: 10.130.194.11

nodeid: 1001

}

node {

ring0_addr: 10.130.194.12

nodeid: 1002

}

logging {

fileline: off

to_stderr: yes

to_logfile: no

to_syslog: yes

syslog_facility: daemon

debug: off

logger_subsys {

subsys: QUORUM

debug: off

}

EOF

NOTE: Replace IP addresses with IP addresses of containers.

On all containers restart Corosync:

# systemctl restart corosync

On candid-ha-0 container configure Pacemaker (run the command and replace the data):

# crm configure edit

node 1000: candid-ha-0 \

attributes pgsql-data-status=DISCONNECT

node 1001: candid-ha-1 \

attributes pgsql-data-status=LATEST

node 1002: candid-ha-2 \

attributes pgsql-data-status=DISCONNECT

primitive haproxy lsb:haproxy \

op monitor interval=15s

primitive pgsql pgsql \

params rep_mode=sync pgctl="/usr/lib/postgresql/10/bin/pg_ctl" psql="/usr/bin/psql" pgdata="/var/lib/postgresql/10/main/" socketdir="/var/run/postgresql" config="/etc/postgresql/10/main/postgresql.conf" logfile="/var/log/postgresql/postgresql-10-ha.log" master_ip=10.130.194.253 node_list="candid-ha-0 candid-ha-1 candid-ha-2" primary_conninfo_opt="keepalives_idle=60 keepalives_interval=5 keepalives_count=5" restart_on_promote=true \

op start timeout=60s interval=0s on-fail=restart \

op monitor timeout=60s interval=4s on-fail=restart \

op monitor timeout=60s interval=3s on-fail=restart role=Master \

op promote timeout=60s interval=0s on-fail=restart \

op demote timeout=60s interval=0s on-fail=stop \

op stop timeout=60s interval=0s on-fail=block \

op notify timeout=60s interval=0s

primitive res_candid_vip IPaddr2 \

params ip=10.130.194.254 cidr_netmask=32 \

op monitor interval=10s \

meta migration-threshold=0

primitive res_pgsql_vip IPaddr2 \

params ip=10.130.194.253 cidr_netmask=32 \

op monitor interval=10s \

meta migration-threshold=0 target-role=Started

ms ms_pgsql pgsql \

meta master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true

clone haproxy-clone haproxy

location cli-prefer-res_pgsql_vip res_pgsql_vip role=Started inf: candid-ha-0

order ord_demote 0: ms_pgsql:demote res_pgsql_vip:stop symmetrical=false

order ord_promote inf: ms_pgsql:promote res_pgsql_vip:start symmetrical=false

location pgsql_on_two_nodes ms_pgsql -inf: candid-ha-0

colocation pgsql_vip inf: res_pgsql_vip ms_pgsql:Master

property cib-bootstrap-options: \

have-watchdog=false \

dc-version=1.1.18-2b07d5c5a9 \

cluster-infrastructure=corosync \

cluster-name=debian \

stonith-enabled=false \

last-lrm-refresh=1534598484

rsc_defaults rsc-options: \

resource-stickiness=INFINITY \

migration-threshold=10

NOTE: Replace IP addresses with IP addresses reserved for Candid and PostgreSQL VIP.

Candid

On all containers install Candid:

# apt -y install candid

On candid-ha-1 container create PostgreSQL user and database for Candid:

# su postgres -c "createuser candid -P"

# su postgres -c "createdb candid -O candid"

On all containers export CANDID_URL variable:

# export CANDID_URL="https://candid.example.com"

# echo "export CANDID_URL=\"https://candid.example.com\"" >> /root/.bashrc

On candid-ha-0 container create admin credentials for Candid API:

# candid put-agent --admin --agent-file admin.agent

# candid put-agent --admin --agent-file services.keys

On all containers configure Candid:

# cat <<EOF > /etc/candid/config.yaml

listen-address: 10.130.194.10:8081

location: 'https://candid.example.com'

private-addr: 10.130.194.10

storage:

type: postgres

connection-string: dbname=candid user=candid password=candid host=postgres.example.com

private-key: Xh/hbA92cqSAFunu3IgVK0VeZrZvtpR7E50OXR39S48=

public-key: Olf//8WpzSnIFm0HwJX4WCoXlTkw1ndAAvFGP1nj71U=

admin-agent-public-key: JsJOh7kXuONBGvgF2kunmbn+gcg8MpoBfMVMrB8RrTw=

resource-path: /usr/share/candid/

access-log: /var/log/identity/access.log

identity-providers:

- type: ldap

name: ldap

domain: example

url: ldap://ldap.example.com/dc=example,dc=com

ca-cert: |

-----BEGIN CERTIFICATE-----

MIIEGzCCAwOgAwIBAgIJAI67J2tCUZWMMA0GCSqGSIb3DQEBCwUAMIGjMQswCQYD

VQQGEwJQTDETMBEGA1UECAwKU29tZS1TdGF0ZTEPMA0GA1UEBwwGS3Jha293MRww

GgYDVQQKDBNJVHN0ZWVyIFR5dHVzIEt1cmVrMRMwEQYDVQQLDApDb25zdWx0aW5n

MRgwFgYDVQQDDA93d3cuaXRzdGVlci5jb20xITAfBgkqhkiG9w0BCQEWEm9mZmlj

ZUBpdHN0ZWVyLmNvbTAeFw0xODAyMDExMTQ1NTZaFw0yODAxMzAxMTQ1NTZaMIGj

MQswCQYDVQQGEwJQTDETMBEGA1UECAwKU29tZS1TdGF0ZTEPMA0GA1UEBwwGS3Jh

a293MRwwGgYDVQQKDBNJVHN0ZWVyIFR5dHVzIEt1cmVrMRMwEQYDVQQLDApDb25z

dWx0aW5nMRgwFgYDVQQDDA93d3cuaXRzdGVlci5jb20xITAfBgkqhkiG9w0BCQEW

Em9mZmljZUBpdHN0ZWVyLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC

ggEBAM3cE1zSJgQw3XNzOn0Z7pcwlHg6B2/ubOQ1L6UDmQNFqdz0Zmg5nSTPpeU6

VlxrUz8YogiISEl549v92TjBSw7SrDTexUNqKNeHdF6wdVQpEsU8hZbndP1sgYH8

2ONYTKG1sqs03JS8gdbb8ZBJYQGiqT2owOLU43QTlVl1KE5yq5b7PwgUlqCfSMbG

FUlE51YBcbv0DYDILJ5trbslAT3xXCk9Lbxyi7cW87fB9mfvkmd48jZb1yl2EY1V

qFiHTrLw0TK+JcI49psxccOy1aXzKJjVbjTt3l/d2mCUIh76S5AlBOiLmn9zOo2G

GA0LtTm7lgJVD1kahpf5NbNAVTUCAwEAAaNQME4wHQYDVR0OBBYEFN8zFHBCIM4T

BPEBeRonPyf7h7K4MB8GA1UdIwQYMBaAFN8zFHBCIM4TBPEBeRonPyf7h7K4MAwG

A1UdEwQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAAvpzYUcMT2Z7ohbUpV94ZOz

8bL919UozNY0V9lrcbnI5v/GlibnNDd/lE7/kBZAdpJMFpzYLxQdBdukXNsQ66fu

UCj7OVZbnhykN6aiAmB7NHHb4gp6Eu9Aan5Cfky2UE66FmZRMulNMH+l0B64AJ9h

crRUIGpsK0BrLl6KITE7OB9Qbjm8VSsRBxDy1MrdwGjDyeWCVIU5YRGcs/j5X45k

OeK8S1KwpuU8/wjkP5lYKUeNRDXIbduWsNAYbLLY8N1wWh+373IuZg3OkfSkIEV7

ApcRh/uwdJKsx0ebO8aHTDCiBi4AYGDcAumsmpY1CAaWBDzdja77bQocI3qDV60=

-----END CERTIFICATE-----

dn: cn=admin,dc=example,dc=com

password: admin

user-query-filter: (objectClass=posixAccount)

user-query-attrs:

id: uid

email: mail

display-name: displayName

group-query-filter: (&(objectClass=groupOfNames)(member={{.User}}))

EOF

NOTE: Replace the following entries:

listen-address - IP address of the container with port
private-addr - IP address of the container
password - password of PostgreSQL candid user
private-key - value of private from services.keys file
public-key - value of public from services.keys file
admin-agent-public-key - value of public from admin.keys file
domain - LDAP domain
url - LDAP URL
ca-cert - content of the ca.pem file
dn - LDAP bind credentials
password - LDAP bind password

On all containers restart Candid service:

# systemctl restart candid

At this point you should be able to access Candid at https://candid.example.com

My Technical Blog