/ Tutorials / By

HAProxy with SSL and Let’s Encrypt

Secure HAProxy with SSL

Perhaps you’ve already tested a little with Let’s Encrypt or read my article on Nginx with Let’s Encrypt. That I am a big fan of HAProxy should have become clear 🙂

What I have not written yet: HAProxy with SSL Securing. Now I’m going to get this article. Let’s Encrypt is a new certification authority that provides simple and free SSL certificates. The CA is embedded in all relevant browsers, so you can use Let’s Encrypt to secure your web pages. And all at no cost.

Which is even more important to me personally:: So far, I have always set my Icinga on every SSL certificate. So I was able to make sure to always be informed in time if an SSL certificate threatened to expire. In the case of monitoring, the procedure then always proceeded:

  • Check the key length (whether it is still sufficient)
  • Possibly. Renewal of SSL certificate or create new SSL key and new SSL CRT
  • Then credit card through Thawte or Comodo pull me, meanwhile with bad interfaces around
  • Then wait, wait, wait …. and wait
  • Then, in a next step, switching the SSL certificates on the servers 
  • Restart the services and check that everything is running

If you have something more than a handful of SSL certificates for your servers and services, then that is really hard. So I started some time ago to exchange the SSL certificates by Let’s Encrypt certificates. First I started with my webserver, but in the meantime I have already converted a few mailserver. Thanks to the HAProxy software, I can use the same configuration for almost anything that can be layer-4 loadbalanced.

This article is based on my article Installation of HAProxy on Debian. If you are in a hurry and do not want to read my article, simply install the HAProxy setup.
[sc name=”Tutorials-Signup”]

Preparation

If I work with SSL on web servers and use a HAProxy (or even an F5 or Broacade Loadbalancer) in front of the web servers, I usually do not use the encryption on the webserver itself. I have SSL on the loadbalancer terminate and work in the backend with an unencrypted connection. Especially with hardware load balancers and pages with a large amount of HTTPS sessions, this reduces a lot of CPU load. Loadbalancers have mostly built specially for SSL-designed CPU cards, which are many times faster.

Since the connections between HAProxy and the web servers are made via a private network, the danger of a man-in-the-middle (MitM) attack is manageable. First, one would have to compromise my internal network in order to get to the unencrypted data packages. If this is possible, I would have a completely different problem anyway.

First, create a self-signed SSL certificate

For the initial tests, you first create a self-signed SSL certificate. In my example is so far not an OpenSSL installed, if necessary, do it quickly now:

root@haproxy:~# apt-get -y install openssl

Use the following command to create your self-signed SSL certificate and move it to /etc/ssl/private.

root@haproxy:~# openssl req -nodes -x509 -newkey rsa:2048 -keyout /etc/ssl/private/test.key -out /etc/ssl/private/test.crt -days 30
Generating a 2048 bit RSA private key
..........................................+++
......................................+++
writing new private key to '/etc/ssl/private/test.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:DE
State or Province Name (full name) [Some-State]:NRW
Locality Name (eg, city) []:Cologne
Organization Name (eg, company) [Internet Widgits Pty Ltd]:gridscale Cloud Computing
Organizational Unit Name (eg, section) []:gridscale Beta Labs
Common Name (e.g. server FQDN or YOUR name) []:*
Email Address []:

You can fill all the properties for the certificate with your personal data, or simply leave it blank. We will replace the certificate by a Let’s Encrypt certificate.

Now create a pem file by copying key and certificate to a file. That goes with:

root@haproxy:~# cat /etc/ssl/private/test.key /etc/ssl/private/test.crt > /etc/ssl/private/test.pem

Configuration of HAProxy with SSL listener for HTTPS

On the HAProxy, you can use the following configuration to create a listener on port 443 (HTTPS) and instruct HAProxy to both schedule SSL and work on Layer-7.

frontend ssl_443
 bind *:443 ssl crt /etc/ssl/private/test.pem
 mode http
 http-request set-header X-Forwarded-For %[src]
 reqadd X-Forwarded-Proto:\ https
 option http-server-close
 default_backend ssl_443

backend ssl_443
 mode http
 balance leastconn
 server web1 10.0.0.2:80 check
 server web2 10.0.0.3:80 check

Short Parameter-Check:

  • bind *:443 ssl crt /etc/ssl/private/test.pem creates a listener on port 443 with an SSL certificate that is located in the appropriate path.
  • Mode http as well as the whole header manipulations I had already explained in the previous article
  • option http-server-close is new. This parameter is unimportant for the first, but it allows the web server to keepalive. This makes the surfing experience on your websites a bit faster.

After restarting the HAProxy, your web servers should be accessible on port 443. If you call directly the info.php, you see also which additional headers are now available. Compare nevertheless times the info.php over Port 80 and over Port 443.

Installation of Let’s Encrypt on HAProxy

On the HAProxy system, the Let’s Encrypt Suite must be installed so that you can request SSL certificates. Since Let’s Encrypt issues domain validated certificates, you first need a DNS entry pointing to the IP address of your HAProxy. I use the following DNS ‘haproxy.kmsg.cc’.

First, install a few tools on the HAProxy system that are later required by Let’s Encrypt.

root@haproxy:~# apt-get -y install bc git

Then clone the repository of Let’s Encrypt:

root@haproxy:~# git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt

Now you have everything on the system that is needed for Let’s Encrypt.

Temporarily switch your HAProxy off, there are still a few configurations necessary:

root@haproxy:~# /etc/init.d/haproxy stop

How Let’s Encrypt Suite Works

Times quite roughly to the course of Let’s Encrypt Suite. The cloned suite works with so-called modules for various services and programs. These modules usually do all the work. You create the certificates, perform the validation, and finally configure your service.

There is no such module for implementation in HAProxy. Therefore, it is helpful to understand how Let’s Encrypt works. With this knowledge, you can then build a workaround that works with a HAProxy.

The Let’s Encrypt Auto module connects to the Let’s Encrypt servers and exchanges a token there. This token is placed in a web server directory and then called by the Let’s Encrypt servers. If the token matches, the Let’s Encrypt Server sign your certificate.

Adjust HAproxy for Let’s Encrypt

Actually quite simply, if there would not be a small hook. Your default HAProxy does not deliver any content, but redirects the queries to the backend. The backend, in turn, neither speak SSL (so do not create certificate requests), nor does it know anything about what you are doing on the HAProxy.

Say the token cannot be exchanged and thus you get no signed certificate.

I’d like to work around this problem by instructing HAProxy to handle all requests to the Let’s Encrypt token directory differently than any other request. To do this, you first need the names of the directories that Let’s Encrypt requests on your web server. Now adjust the HAProxy configuration accordingly.

frontend port_80
 bind *:80
 mode http
 acl lets_encrypt path_beg /.well-known/acme-challenge/
 use_backend lets_encrypt if lets_encrypt 
 default_backend port_80

backend port_80
 mode http
 http-request set-header X-Forwarded-For %[src]
 balance roundrobin
 option httpchk HEAD / HTTP/1.0
 server web1 10.0.0.2:80 check
 server web2 10.0.0.3:80 check

Pay attention to the inserted ACL. If a request is made to the HAProxy that begins with ‘.well-known’ in the URI, it will be redirected to the backend ‘lets_encrypt’.

A definition for the backend ‘lets_encrypt’ is missing. The probably simplest backend entry for HAProxy is enough for this:

backend lets_encrypt
 mode http
 server local localhost:60001

This makes it clear to HAProxy that all requests to the backend should be delivered via the localhost. What is missing is still a web server, which also responds to localhost port 60001. This webserver letsencrypt directly.

Now start your HAProxy and look in the log file /var/log/haproxy.log if there are any errors logged there. If you have the “admin stats” activated (more in an article about a few advanced configurations of HAProxy), you should now see the new configuration on the admin page of HAProxy.

Now test times whether the exception rule in the HAProxy for the URL .well-known works. Call the URL in the browser, for example: ‘http://haproxy.kmsg.cc/.well-known/acme-challenge’. If you have done everything right, you get an error (503 Service Unavailable from the HAProxy). Because currently is on the localhost port 60001 not a service started, so far HAProxy cannot forward a request.

Let’s Encrypt Certificate Request

Now comes the penultimate step, requesting the Let’s Encrypt certificate. First create the directory /etc/letsencrypt/ (if it does not already exist) and create a configuration file “cli.ini”).

rsa-key-size = 4096
email = email@domain.tld
authenticator = standalone
standalone-supported-challenges = http-01

You can also pass the parameters to the CLI tool, if you prefer to work without a configuration file. Now go to the directory / opt / letsencrypt (Reminder: Here you had the Let’s Encrypt Suite cloned) and execute the following command. Replace –domains with your DNS for which you want to request a certificate.

root@haproxy:/opt/letsencrypt# ./letsencrypt-auto certonly --domains haproxy.kmsg.cc --renew-by-default --http-01-port 60001 --agree-tos

Voilà!You will find in / etc / letsencrypt / live / a directory with your DNS name and in it are key, certificate, chain etc …

Now the new certificate only has to be entered into the haproxy.cnf. Unfortunately, you cannot directly use the files created by Let’s Encrypt, since HAProxy needs everything in one file.

I put all certificates for HAProxy to / etc / haproxy / cert. I think this is clear, but DevOps regularly breaks the right structure. So just think about how you want to do it 🙂

root@haproxy:~# mkdir -p /etc/haproxy/cert
root@haproxy:~# cat /etc/letsencrypt/live/haproxy.kmsg.cc/fullchain.pem /etc/letsencrypt/live/haproxy.kmsg.cc/privkey.pem > /etc/haproxy/cert/haproxy.kmsg.cc.pem

Now edit the listener for port 443 in your haproxy.cnf:

frontend ssl_443
# bind *:443 ssl crt /etc/ssl/private/test.pem
  bind *:443 ssl crt /etc/haproxy/cert/
  mode http
  http-request set-header X-Forwarded-For %[src]
  reqadd X-Forwarded-Proto:\ https
  option http-server-close
  default_backend ssl_443

And restart HAProxy. Then test if you can access your DNS via SSL.

Last step, automatic renewal of certificates

Let’s Encrypt only allow certificates for a period of 90 days. This means you must renew the certificate within this time.

The easiest way to do this is to enter the above command into the cronjob. If you want to create a lot of certificates for a domain, a warning: Let’s Encrypt has installed a so-called rate limit. This rate limit counts the certificate requests based on the domain (caution, not subdomain!). Each request for a subdomain counts into the counter of the main domain. Do not forget that you do not ask the servers of Let’s Encrypt too often after a renewal (otherwise you are unlucky), but not too rarely (otherwise your certificate is suddenly invalid).

I use a small shell script for automation. You can find it on our BitBucket repository and can Download the URL directly. Just download it to /opt.

root@haproxy:~# wget https://bitbucket.org/gridscale/letsencrypt/raw/HEAD/le-autorenew.sh -o /opt/le-renew-webroot

Edit the parameter “exp_limit” in the script and set it, e.g. To any value between 10 and 30. Brief overview of what I do in the script:

  • I read the existing directories in /etc/letsencrypt/live
  • For each directory I check the key and CA file
  • In the CA I look for how long the certificate is still valid. If it is longer than exp_limit, jump to the next certificate.
  • If the certificate is valid for less than exp_limit, then I read the certificate names (you can deposit more than one DNS per certificate)
  • Then I generate the Let’s Encrypt command and try to have the certificate extended
  • If this is successful, I exchange the certificate in HAProxy and reload the service. If the operation fails, I try to restore the previous certificate.

The script is only hacked together quickly and should serve as an example. It would certainly be good to build a little more error handling. I do not start the following scenarios:

  • If the DNS is no longer valid for a domain, Let’s Encrypt cannot perform domain validation. Let’s Encrypt then acknowledges with an error. It would be better if the deposited domain names were checked for the registry status (i.e. in the root zone of the respective NIC) and then the responsible nameservers would be asked for the entered DNS.
  • There is no error handler for HAProxy in the script.
  • No error and info reporting in the script

In brief: Use the script carefully or build it if necessary. If I have a quiet hour, I hack the bash script maybe once again in a Python script and improve the deficiencies. If you want to do the work, I am glad about your application which I will publish here gladly.

If you still want to trust my script, simply create a cronjob so it runs once per day.