How to setup Let’s Encrypt SSL on Multiple Servers using Nginx + Lsyncd

Installing Let’s Encrypt on standalone servers with cPanel, or plain Linux servers without any control panel is an easy task that can be done within minutes.

However, days ago our tech team faced a new interesting challenge. One of our customers required to install let’s encrypt on a website that is running across many Nginx servers using round robin dns (rrDNS) balancing.

Just in case, I will assume you already have Nginx and Let’s Encrypt working on at least one single Nginx server, if you haven’t, I will recommend you to read this post: Install Let’s Encrypt on Nginx + CentOS

Let’s move to the fun part. This is the best way we’ve found to have Let’s Encrypt SSL running over mulitple servers.

setup let's encrypt with nginx on multiple servers

Update, September 13, 2017: after publishing this and running several tests, I’ve found this not to be the best solution, as ssl renewal may not work as expected due to the rrDNS setup and the Let’s Encrypt verification process. Let’s Encrypt over rrDNS + Nginx will only work perfectly if you are using a shared storage service across all your clustered / balanaced servers (like NFS for example on rrDNS setups) for your .well-known/ directory, otherwise it may fail to verify the .well-known/acme-challenge/RANDOM URL file if it hits another server who doesn’t contain that active URL structure, more information can be found on Let’s Encrypt forums.

Configure Let’s Encrypt on Multiple Servers

Add the .well-known location configuration to your Nginx vhost config file, as you se below:

server {
        access_log off;
        error_log  logs/nixcp.com-error_log warn;

        listen       80;
        server_name  nixcp.com www.nixcp.com;
        root   /var/www/nixcp.com;
        index  index.html index.php;

	location /.well-known {
                root /var/www/nixcp.com;
        }

Here is the full Nginx configuration for a Let’s Encrypt based website:

### nixcp.com

server {
        access_log off;
        error_log  logs/nixcp.com-error_log warn;

        listen       80;
        server_name  nixcp.com www.nixcp.com;
        root   /var/www/nixcp.com;
        index  index.html index.php;

        # SSL configuration
	listen 443 ssl;
	ssl on;
	ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
	ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
	ssl_prefer_server_ciphers On;
	ssl_dhparam /etc/nginx/conf/dhparams.pem;
        ssl_certificate /etc/letsencrypt/live/nixcp.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/nixcp.com/privkey.pem;
	ssl_session_cache shared:SSL:20m;
	ssl_session_timeout 10m;
	# End of SSL Configuration

	location /.well-known {
                root /var/www/nixcp.com;
        }

	# Parseo de php con php-fpm
        location ~ \.(php|shtml)$ {
            try_files $uri =404;
            fastcgi_pass unix:/tmp/php5-fpm.sock;
	    fastcgi_index  index.shtml;
            fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
            include        fastcgi_params;
            fastcgi_buffer_size 128k;
	    fastcgi_read_timeout 150;
            fastcgi_buffers 256 4k;
            fastcgi_busy_buffers_size 256k;
            fastcgi_temp_file_write_size 256k;
       }
}

Let’s Encrypt SSL replication across mulitple servers

Lsyncd is a great tool to synchronize file and directories over Linux servers. In this case I’m using lsyncd to sync the Nginx vhost configuration files located at /etc/nginx/conf.d/*, and also the /etc/letsencrypt directories, which contains the .pem, .key files and the rest of the configuration files required to have Let’s Encrypt SSL working.

Setting up lsyncd

yum install lsyncd -y

or using apt-get for Debian/Ubuntu distros:

apt-get install lsyncd

Generate log directory and files:

mkdir -p /var/log/lsyncd -p
touch /var/log/lsyncd/lsyncd.log
touch /var/log/lsyncd/lsyncd-status.log

Edit /etc/lsyncd.conf file

nano -w /etc/lsyncd.conf

Then try to setup your lsyncd.conf configuration as you see below.

Make sure you replace:

  • 192.168.1.105 with your ssh remote servers IP, this is an example for only one remote servers, but you can add as many servers as you like.
  • /etc/nginx/conf.d/ with your real nginx vhost configuration directory.
  • -p 22 with the real SSH port you are using on the remote servers.
  • nixcp.com with the real domin name you are using.
settings {
logfile = "/var/log/lsyncd/lsyncd.log",
statusFile = "/var/log/lsyncd/lsyncd-status.log",
statusInterval = 2
}

sync {
default.rsync,
source="/etc/nginx/conf.d/",
target="192.168.1.105:/etc/nginx/conf.d/",
rsync = {
compress = true,
acls = true,
verbose = true,
owner = true,
group = true,
perms = true,
rsh = "/usr/bin/ssh -p 22 -o StrictHostKeyChecking=no"
}
}

sync {
default.rsync,
source="/etc/letsencrypt/",
target="192.168.1.105:/etc/letsencrypt/",
rsync = {
compress = true,
acls = true,
verbose = true,
owner = true,
group = true,
perms = true,
rsh = "/usr/bin/ssh -p 22 -o StrictHostKeyChecking=no"
}
}

After that, restart lsyncd:

service lsyncd restart

or

systemctl restart lsyncd.service

Let’s Encrypt SSL renewal

The renewal let’s encrypt script must be run only on the “master” server, that is the one that syncs the let’s encrypt ssl certificate to the “slave” servers.

Important: this will only work if you are using a shared storage service across all your clustered / balanaced servers (like NFS for example on rrDNS setups) for your .well-known/ directory, otherwise it may fail to verify the .well-known/acme-challenge/RANDOM URL file if it hits another server who doesn’t contain that active URL structure, more information can be found on Let’s Encrypt forums.

Setup a weekly cron to make sure your ssls are always renewed:

crontab -e

The paste this at the end of the file:

01 1 * * 1 /usr/bin/certbot renew >> /var/log/le-ssl-renew.log
06 1 * * 1 /sbin/service nginx reload

That’s all, now you know how to setup let’s encrypt ssl over multiple Nginx servers.

Let me know if you face any issues during the setup process.

About the Author: Martin Keler

Leave a Reply

Your email address will not be published. Required fields are marked *