How to setup Canonical Identity Service (Candid) in the HA mode

Intro


Canonical Identity Service (Candid) does not require any HA-specific extensions / settings and can be deployed in the HA mode in various ways. Which one is the best one, however? It is hard to say, but the reference architecture is always the most obvious choice. In the following post we will setup Candid in the HA mode based on the following tool set:
  • PostgreSQL - reliable and highly available backend
  • Corosync & Pacemaker - messaging and service management
  • HAProxy - load balancing and SSL termination
This architecture is shown in the following diagram:


The following part contains step-by-step instructions to install and configure Candid in the HA mode. All services are set on LXD containers with Ubuntu Bionic. It is assumed that LDAP is used as the Identity Provider.

Prerequisites



PostgreSQL


On all containers install and stop PostgreSQL:

# apt -y install postgresql

# systemctl stop postgresql

On candid-ha-0 container disable PostgreSQL:

# systemctl disable postgresql

On all containers change ownership and permissions of PostgreSQL certificate and key:

# chown postgres:postgres /var/lib/postgresql/10/main/server.crt

# chown postgres:postgres /var/lib/postgresql/10/main/server.key

# chmod 600 /var/lib/postgresql/10/main/server.key

On all containers configure PostgreSQL:

# sed -i "s^ssl_cert_file = '/etc/ssl/certs/ssl-cert-snakeoil.pem'^#ssl_cert_file = '/etc/ssl/certs/ssl-cert-snakeoil.pem'^" /etc/postgresql/10/main/postgresql.conf

# sed -i "s^ssl_key_file = '/etc/ssl/private/ssl-cert-snakeoil.key'^#ssl_key_file = '/etc/ssl/private/ssl-cert-snakeoil.key'^" /etc/postgresql/10/main/postgresql.conf

# cat <<EOF >> /etc/postgresql/10/main/postgresql.conf
listen_addresses = '*'
max_connections = 300
wal_level = hot_standby
synchronous_commit = on
archive_mode = off
max_wal_senders = 10
wal_keep_segments = 256
hot_standby = on
restart_after_crash = off
hot_standby_feedback = on
ssl_cert_file = '/var/lib/postgresql/10/main/server.crt'
ssl_key_file = '/var/lib/postgresql/10/main/server.crt'
EOF

# cat /etc/postgresql/10/main/pg_hba.conf | head -n -3 > /tmp/pg_hba.conf; mv /tmp/pg_hba.conf /etc/postgresql/10/main/pg_hba.conf

# cat <<EOF >> /etc/postgresql/10/main/pg_hba.conf
host    replication     postgres        10.130.194.10/32        trust
host    replication     postgres        10.130.194.11/32        trust
host    replication     postgres        10.130.194.12/32        trust
host    replication     postgres        10.130.194.253/32       trust
host    replication     postgres        10.130.194.254/32       trust
hostssl candid          candid          10.130.194.10/32        md5
hostssl candid          candid          10.130.194.11/32        md5
hostssl candid          candid          10.130.194.12/32        md5
hostssl candid          candid          10.130.194.253/32       md5
hostssl candid          candid          10.130.194.254/32       md5
EOF

NOTE: Replace IP addresses with IP addresses of all containers and IP addresses reserved for Candid and PostgreSQL VIP.

On candid-ha-1 container start PostgreSQL:

# systemctl start postgresql

On candid-ha-2 container initiate the replication and start PostgreSQL:

# rm -rf /var/lib/postgresql/10/main/

# sudo -u postgres pg_basebackup -h 10.130.194.11 -D /var/lib/postgresql/10/main -v --wal-method=stream

# systemctl start postgresql

NOTE: Replace IP addresses with IP addresses of candid-ha-1 container.

HAProxy


On all containers install HAProxy:

# apt -y install haproxy

On all containers configure HAProxy:

cat <<EOF > /etc/haproxy/haproxy.cfg
defaults
    timeout connect 10s
    timeout client 1m
    timeout server 1m

global
    tune.ssl.default-dh-param 2048

frontend candid
    bind *:443 ssl crt /etc/ssl/private/candid.pem
    reqadd X-Forwarded-Proto:\ https
    option http-server-close
    default_backend candid

backend candid
    balance source
    hash-type consistent
    server candid-ha-1 10.130.194.10:8081 check
    server candid-ha-2 10.130.194.11:8081 check
    server candid-ha-3 10.130.194.12:8081 check
EOF

NOTE: candid.pem is a concatenation of candid.crt and candid.key files (cat candid.crt candid.key > candid.pem)

NOTE: Replace IP addresses with IP addresses of containers.

On all containers restart HAProxy:

# systemctl restart haproxy

Corosync & Pacemaker


On all containers install Corosync and Pacemaker:

# apt -y install crmsh

On all containers configure Corosync:

# cat <<EOF > /etc/corosync/corosync.conf
totem {
  version: 2
  token: 3000
  token_retransmits_before_loss_const: 10
  join: 60
  consensus: 3600
  vsftype: none
  max_messages: 20
  clear_node_high_bit: yes
  secauth: off
  threads: 0
  ip_version: ipv4
  rrp_mode: none
  transport: udpu
}

quorum {
  provider: corosync_votequorum
  }

nodelist {
  node {
    ring0_addr: 10.130.194.10
    nodeid: 1000
  }
  node {
    ring0_addr: 10.130.194.11
    nodeid: 1001
  }
  node {
    ring0_addr: 10.130.194.12
    nodeid: 1002
  }
}

logging {
  fileline: off
  to_stderr: yes
  to_logfile: no
  to_syslog: yes
  syslog_facility: daemon
  debug: off
  logger_subsys {
    subsys: QUORUM
    debug: off
  }
}
EOF

NOTE: Replace IP addresses with IP addresses of containers.

On all containers restart Corosync:

# systemctl restart corosync

On candid-ha-0 container configure Pacemaker (run the command and replace the data):

# crm configure edit

node 1000: candid-ha-0 \
        attributes pgsql-data-status=DISCONNECT
node 1001: candid-ha-1 \
        attributes pgsql-data-status=LATEST
node 1002: candid-ha-2 \
        attributes pgsql-data-status=DISCONNECT
primitive haproxy lsb:haproxy \
        op monitor interval=15s
primitive pgsql pgsql \
        params rep_mode=sync pgctl="/usr/lib/postgresql/10/bin/pg_ctl" psql="/usr/bin/psql" pgdata="/var/lib/postgresql/10/main/" socketdir="/var/run/postgresql" config="/etc/postgresql/10/main/postgresql.conf" logfile="/var/log/postgresql/postgresql-10-ha.log" master_ip=10.130.194.253 node_list="candid-ha-0 candid-ha-1 candid-ha-2" primary_conninfo_opt="keepalives_idle=60 keepalives_interval=5 keepalives_count=5" restart_on_promote=true \
        op start timeout=60s interval=0s on-fail=restart \
        op monitor timeout=60s interval=4s on-fail=restart \
        op monitor timeout=60s interval=3s on-fail=restart role=Master \
        op promote timeout=60s interval=0s on-fail=restart \
        op demote timeout=60s interval=0s on-fail=stop \
        op stop timeout=60s interval=0s on-fail=block \
        op notify timeout=60s interval=0s
primitive res_candid_vip IPaddr2 \
        params ip=10.130.194.254 cidr_netmask=32 \
        op monitor interval=10s \
        meta migration-threshold=0
primitive res_pgsql_vip IPaddr2 \
        params ip=10.130.194.253 cidr_netmask=32 \
        op monitor interval=10s \
        meta migration-threshold=0 target-role=Started
ms ms_pgsql pgsql \
        meta master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
clone haproxy-clone haproxy
location cli-prefer-res_pgsql_vip res_pgsql_vip role=Started inf: candid-ha-0
order ord_demote 0: ms_pgsql:demote res_pgsql_vip:stop symmetrical=false
order ord_promote inf: ms_pgsql:promote res_pgsql_vip:start symmetrical=false
location pgsql_on_two_nodes ms_pgsql -inf: candid-ha-0
colocation pgsql_vip inf: res_pgsql_vip ms_pgsql:Master
property cib-bootstrap-options: \
        have-watchdog=false \
        dc-version=1.1.18-2b07d5c5a9 \
        cluster-infrastructure=corosync \
        cluster-name=debian \
        stonith-enabled=false \
        last-lrm-refresh=1534598484
rsc_defaults rsc-options: \
        resource-stickiness=INFINITY \
        migration-threshold=10

NOTE: Replace IP addresses with IP addresses reserved for Candid and PostgreSQL VIP.

Candid


On all containers install Candid:

# apt -y install candid

On candid-ha-1 container create PostgreSQL user and database for Candid:

# su postgres -c "createuser candid -P"

# su postgres -c "createdb candid -O candid"

On all containers export CANDID_URL variable:

# export CANDID_URL="https://candid.example.com"

# echo "export CANDID_URL=\"https://candid.example.com\"" >> /root/.bashrc

On candid-ha-0 container create admin credentials for Candid API:

# candid put-agent --admin --agent-file admin.agent

# candid put-agent --admin --agent-file services.keys

On all containers configure Candid:

# cat <<EOF > /etc/candid/config.yaml
listen-address: 10.130.194.10:8081
location: 'https://candid.example.com'
private-addr: 10.130.194.10
storage:
  type: postgres
  connection-string: dbname=candid user=candid password=candid host=postgres.example.com
private-key: Xh/hbA92cqSAFunu3IgVK0VeZrZvtpR7E50OXR39S48=
public-key: Olf//8WpzSnIFm0HwJX4WCoXlTkw1ndAAvFGP1nj71U=
admin-agent-public-key: JsJOh7kXuONBGvgF2kunmbn+gcg8MpoBfMVMrB8RrTw=
resource-path: /usr/share/candid/
access-log: /var/log/identity/access.log
identity-providers:
  - type: ldap
    name: ldap
    domain: example
    url: ldap://ldap.example.com/dc=example,dc=com
    ca-cert: |
      -----BEGIN CERTIFICATE-----
      MIIEGzCCAwOgAwIBAgIJAI67J2tCUZWMMA0GCSqGSIb3DQEBCwUAMIGjMQswCQYD
      VQQGEwJQTDETMBEGA1UECAwKU29tZS1TdGF0ZTEPMA0GA1UEBwwGS3Jha293MRww
      GgYDVQQKDBNJVHN0ZWVyIFR5dHVzIEt1cmVrMRMwEQYDVQQLDApDb25zdWx0aW5n
      MRgwFgYDVQQDDA93d3cuaXRzdGVlci5jb20xITAfBgkqhkiG9w0BCQEWEm9mZmlj
      ZUBpdHN0ZWVyLmNvbTAeFw0xODAyMDExMTQ1NTZaFw0yODAxMzAxMTQ1NTZaMIGj
      MQswCQYDVQQGEwJQTDETMBEGA1UECAwKU29tZS1TdGF0ZTEPMA0GA1UEBwwGS3Jh
      a293MRwwGgYDVQQKDBNJVHN0ZWVyIFR5dHVzIEt1cmVrMRMwEQYDVQQLDApDb25z
      dWx0aW5nMRgwFgYDVQQDDA93d3cuaXRzdGVlci5jb20xITAfBgkqhkiG9w0BCQEW
      Em9mZmljZUBpdHN0ZWVyLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
      ggEBAM3cE1zSJgQw3XNzOn0Z7pcwlHg6B2/ubOQ1L6UDmQNFqdz0Zmg5nSTPpeU6
      VlxrUz8YogiISEl549v92TjBSw7SrDTexUNqKNeHdF6wdVQpEsU8hZbndP1sgYH8
      2ONYTKG1sqs03JS8gdbb8ZBJYQGiqT2owOLU43QTlVl1KE5yq5b7PwgUlqCfSMbG
      FUlE51YBcbv0DYDILJ5trbslAT3xXCk9Lbxyi7cW87fB9mfvkmd48jZb1yl2EY1V
      qFiHTrLw0TK+JcI49psxccOy1aXzKJjVbjTt3l/d2mCUIh76S5AlBOiLmn9zOo2G
      GA0LtTm7lgJVD1kahpf5NbNAVTUCAwEAAaNQME4wHQYDVR0OBBYEFN8zFHBCIM4T
      BPEBeRonPyf7h7K4MB8GA1UdIwQYMBaAFN8zFHBCIM4TBPEBeRonPyf7h7K4MAwG
      A1UdEwQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAAvpzYUcMT2Z7ohbUpV94ZOz
      8bL919UozNY0V9lrcbnI5v/GlibnNDd/lE7/kBZAdpJMFpzYLxQdBdukXNsQ66fu
      UCj7OVZbnhykN6aiAmB7NHHb4gp6Eu9Aan5Cfky2UE66FmZRMulNMH+l0B64AJ9h
      crRUIGpsK0BrLl6KITE7OB9Qbjm8VSsRBxDy1MrdwGjDyeWCVIU5YRGcs/j5X45k
      OeK8S1KwpuU8/wjkP5lYKUeNRDXIbduWsNAYbLLY8N1wWh+373IuZg3OkfSkIEV7
      ApcRh/uwdJKsx0ebO8aHTDCiBi4AYGDcAumsmpY1CAaWBDzdja77bQocI3qDV60=
      -----END CERTIFICATE-----
    dn: cn=admin,dc=example,dc=com
    password: admin
    user-query-filter: (objectClass=posixAccount)
    user-query-attrs:
      id: uid
      email: mail
      display-name: displayName
    group-query-filter: (&(objectClass=groupOfNames)(member={{.User}}))
EOF

NOTE: Replace the following entries:
  • listen-address - IP address of the container with port
  • private-addr - IP address of the container
  • password - password of PostgreSQL candid user
  • private-key - value of private from services.keys file
  • public-key - value of public from services.keys file
  • admin-agent-public-key - value of public from admin.keys file
  • domain - LDAP domain
  • url - LDAP URL
  • ca-cert - content of the ca.pem file
  • dn - LDAP bind credentials
  • password - LDAP bind password
On all containers restart Candid service:

# systemctl restart candid

At this point you should be able to access Candid at https://candid.example.com

No comments:

Post a Comment