it would be great if, as part of the SSL integration, ikiwiki-hosting would call up on the letsencrypt.org gods to install proper certs.

First implementation

so far I have done this by hand, using the main client (previously called letsencrypt, now called certbot, those instructions may need to be updated):

1. define some handy variables:

    domain=example.com
    email=nobody@$domain
    username=w-example
    webroot=/home/$username/public_html/
   

2. install the letsencrypt client software:

   • if you are on Debian sid, stretch or jessie (with backports):

        apt install certbot
     

   • otherwise, run the magic commands:

        git clone https://github.com/certbot/certbot
        cd certbot
        ./cerbot-auto --help
     

3. request a certificate:

    sudo -s
    [ -d .git ] && source venv/bin/activate # only use virtualenv if checking out from git
    certbot certonly --webroot \
      --agree-tos \
      --webroot-path="$webroot" \
      --domains $domain --email $email
   

   ... notice how you need to change the two last lines for your environment.

4. this will create a certificate in /etc/letsencrypt/live/example.com. install it where ikiwiki expects it:

    ln -s /etc/letsencrypt/live/$domain/cert.pem /etc/ikiwiki-hosting/config/$username/ssl.crt
    ln -s /etc/letsencrypt/live/$domain/fullchain.pem /etc/ikiwiki-hosting/config/$username/ssl.chain
    ln -s /etc/letsencrypt/live/$domain/privkey.pem /etc/ikiwiki-hosting/config/$username/ssl.key
   

5. regenerate the apache configs and restart apache:

    ikisite enable $domain
    service apache2 reload # or restart?
   

Ikiwiki hosting would need to perform steps 3 to 5, and probably have a Recommends on certbot. -- anarcat

Potential clients

Finally, note that there is now a fairly large set of Let's Encrypt clients, here's a short list (those have not been reviewed or tested for compatiblity with ikiwiki-hosting):

• dehydrated, bash-only client that should work all Debian releases back to Wheezy (formerly known as letsencrypt.sh, in Debian jessie, stretch and sid)
• simp_le, much simpler implementation in Python (not in Debian)
• acme, root-less, more minimal implementation than the official client, in Go (in Debian stretch and sid)
• lego, root-less, fairly featureful, also in Debian
• acme-tiny is a very small client (~200 lines of python code) that aims to be easily audited and to do only one thing well. for example, you need to generate your RSA key yourself, which may actually be a good thing considering how ikiwiki-hosting works (in Debian stretch and sid)

Note that I have also considered simply using the --apache plugin to just have LE manage all those certs itself. This could conflict with ikiwiki-hosting, but since it runs fairly frequently, it could just win in the end. It's not a great solution for sites managed by the control panel, but for my use case, it could just work. Unforunately, it parses configs in sites-available instead of just sites-enabled which breaks it for me. --anarcat

Renewals

Also, renewals need to be processed in some way. It seems that it's possible to just run:

letsencrypt -c /etc/letsenctypt/renewal/$domain.conf --renew-by-default

--renew-by-default renews every time it's run; --keep-until-expiring instead waits until 30 days before expiry and renews then. I think that's the simpler approach, and the same letsencrypt command line can then both obtain the original cert and renew it. --Joey

True, makes a lot more sense! --anarcat

Actually, the -c /etc/letsenctypt/renewal/$domain.conf doesn't work at all. The 0.4 client features a new renew command that does the right thing (including renewing only if necessary). I tested it here and it seems to work fairly well. My complete update run looks more something like this now: --anarcat

letsencrypt renew --non-interactive
git -C /etc add -A letsencrypt
git commit -m"renew all certs"

So the software name changed: it's certbot now. And yes, the renew command seems to work better. There is a --post-hook that can restart apache and do various things with certs, but unfortunately prior to 0.7, it runs all the time. Normally, the --post-hook is one of those variables that would be saved on renewal, but I haven't tested this because I don't want to have that hook run twice a day. So I'm waiting for this to hit backports before automating this completely, especially since I do not want to diverge from the package's /etc/cron.d/certbot cronjob. My test renewal line now looks like:

certbot --dry-run renew --renew-hook 'service apache2 reload; etckeeper commit "certbot renewed domains: $RENEWED_DOMAINS"'

I'll test this again in a few weeks, hopefully when the new package hits backports. --anarcat

The software has caught up. To avoid customizing the config file, I had to put the renew-hook in the /etc/letsencrypt/cli.ini config file, as such:

renew-hook = service apache2 reload & etckeeper commit "certbot renewed domains: $RENEWED_DOMAINS"

Notice the & instead of ; because the latter breaks the LE config file parser. With that, the configured domains are renewed automatically! To add new domains, I just need to repeat steps 3 to 5 above. Note that extra webroots can be specified with:

certbot certonly -w /home/foo -d foo.example.com -w /home/bar -d bar.example.com

I.e. the last webroot path is attached to the next domain specified. Next step is to code this in Perl, I guess? --anarcat

Rate limits

Regarding rate limits, they don't seem to be a problem unless you run a really huge site (500 registrations per 3 hours per IP!). The limits are:

• 100 Names/Certificate (how many domain names you can include in a single certificate)
• 20 Certificates per Domain per week
• 500 Registrations/IP address per 3 hours
• 300 Pending Authorizations/Account per week

That seems to be 5 certificates per domain, not subdomain. Eg, I asked for tmp.kitenet.net 5 times and now it's not allowing getting a cert for downloads.kitenet.net.

So, this seems a pretty big problem; ikiwiki-hosting sites often use subdomains, and if so, quite likely use more than 5. It could keep a site site http-only until it got a cert. But with 100 sites, that would be nearly half a year of delay! (Update: With the new 20 certs/domain/week, that drops to 5 weeks delay.)

And, what about renewal, wouldn't that be rate limited too? (Update: No, it seems that renewals are not rate limited in the 20 certs/domain/week bucket.)

Seems like getting a cert for 100 subdomains under branchable.com (eg), and then moving onto the next cert etc would be the best we could do. But that's a lot more complicated. In particular, renewal needs to use the same set of 100 domains, and if one of the domains is deleted in the meantime, renewal of those 100 probably won't work, and a new certificate would need to be generated for the 99 remaining. --Joey

Ouch, yes, that is a significant problem for branchable!!! Maybe this should be brought up upstream... I wonder if this couldn't be a way to finance letsencrypt, to have larger providers pay a monthly fee for the service, and bypass those limits. --anarcat

WP.com switched to LE, so there's probably a way around those. --anarcat

I asked around and the limits were bumped to 20 certs/domain/week recently. Also, they can tweak the limits upon request. --[[anarcat]

Discussion

Oh, and this could be a plugin, not sure if that would make sense, but it's the way it works on that side of the world. --anarcat

I would much rather use something small enough to be reviewable, particularly if we're going to have to run it unattended as root; the official Let's Encrypt client is huge. We don't have to worry about portability to non-Debian environments or non-Apache web servers, which takes away a lot of the reason to use something so big. --smcv

Kind of agreed, but I'm more interested in ameanability to automation than overall code size. https://github.com/hlandau/acme is one client that seems promising. At the moment only letsencrypt is seems packaged in debian yet and that's the first bar. --Joey

Agreed as well. Just scratching an itch here... The acme client looks interesting, but it's go, which brings a whole set of dependencies in. But I like its philosophy and it could integrate well. Also note that the above instructions fail to take into account the canonical and alias names of the site, which breaks openid logins... Something to fix in my future renewals... --anarcat

Here's an interesting puppet implementation that mentions another golang client (lego), which has the interesting property of supporting DNS challenges. The puppet code itself uses the shell script previously mentioned. --anarcat

Note that Let's Encrypt started supporting wildcard certificates recently, but only with DNS challenges.. --anarcat

---

how to only enable ssl once a certificate is available

Tip from Joey:

Letsencrypt has a bit of a chicken and egg problem, in that the http server has to be up to get the certificate, but the apache site config file has to contain both the http and the https vhost. And the https vhost needs to point to cert files not available until letsencrypt has run.

smcv was thinking about dealing with that by generating a temporary self-signed cert first. But, I don't like the possibility that a http site would come up with such a cert during normal operation.

I found a way around this when adding letsencrypt support to propellor: Make the apache site config file contain only the http vhost, and use IncludeOptional to include another file which contains the https vhost. Generate that file only once letsencrypt has generated the cert.

Note that IncludeOptional has to be used with a wildcard to avoid it being an error if the included file doesn't exist. So, something like:

IncludeOptional /etc/apache2/sites-available/$domain/*.conf

The way I did this here was to generate the cert with the HTTP-only vhost, and only after the cert was generated create the HTTPS vhost... Can't you have a modified HTTPS vhost generated only after the cert is generated? --anarcat

Yes, I think your approach at the top works; ikisite enable $domain notices the certs and regenerates an apache config to use them. --Joey

Worked design

Add a use_letsencrypt config setting in the ikiwiki-hosting plugin. (done)

Make /etc/ikiwiki-hosting/config/$username/domain.{crt,key,chain} be used for a domain when available, instead of ssl.{crt,key,chain}. This allows urlaliases to each get their own letsencrypt cert. (done)

Add ikisite letsencrypt site, which first sets use_letsencrypt and then runs certbot (or whatever, it can be made configurable if desired) to try to get a certificate for all of the site's domain(s). If certbot succeeds, it symlinks the cert into place, and calls ikisite enable site to update the web server config to use the cert. (done)

Add ikisite letsnotencrypt site which undoes that. (done)

Then add a daily cron job that looks for sites that have use_letsencrypt set, but don't have letsencrypt certs. (Check that the cert files are symlinks to tell). For each such site, it runs ikisite letsencrypt to try again to get the cert. (done in ikisite maintaincerts)

Another cron job can run certbot renew to handle all renewals. Seems this doesn't need to rejigger the certificate symlinks or the apache config for renewed sites, so it should be nice and simple. (done in ikisite maintaincerts)

ikisite domains will need to check use_letsencrypt. If it's set, and a new domain is being added to a site, or a domain removed, it needs to delete the flag file and run ikisite letsencrypt to (try to) get a new certificate. What if that fails? It might be possible to keep using the old letsencrypt certificate in some cases, eg when adding an alias. Certianly not in the case where the site's primary domain has changed. The safe option is certainly to remove the old letsencrypt certificate if it fails, and ikisite enable will then disable ssl support for that site, until the cron job later succeeds in getting a certificate. (done)

Once this is tested and working well, it can be made the default, by setting use_letsencrypt in the autosetup/ files. For that to work, every ikisite command that creates a site (including branching, restoring from backup, etc) should run ikisite letsencrypt at the end when the site has use_letsencrypt configured. It can take a minute or so for letsencrypt to run, so best to do that in the background so as to not slow down site creation etc.

Note: there is already a cron job that runs certbot renew in the Debian package, watch out for dupes... Otherwise: design sounds okay, although I'm not sure why the flag file is necessary. --anarcat

issues

• If a site's urlalias includes some domain that is not in the DNS, or no longer points to the host, this is normally not a problem. But, it will prevent certbot getting a cert for that site, because it cannot verify it.

  This affects both ikisite letsencrypt and ikisite domains.

  ikisite domains does not currently ensure that all urlaliases are set in the dns. This check could be added, it might cause some problem though.

  Note that this is only a problem because the urlaliases are passed to certbot as --host. That is needed because templates/apache-sitealias.tmpl includes a https configuration. That lets redirection from the https urlalias to the main url work.

  But also, if a cert is issued for domains A and B, and B later falls out of the DNS, renewal of the cert will fail. So this can lead to renewal failures later down the line.

  Partial fix would be re-running certbot without the urlaliases if it fails with them. Better to have the main url encrypted even if redirection to it cannot be. But, that would leave the apache-sitealias.tmpl generating a config that uses the wrong cert for https, which is bad if some of the urlaliases are accessible in the dns. So, this fix needs a way to prevent that template from using https in this case.

  Alternatively, could improve the UI so the site admin can see when letsencrypt failed, and why. (Worth doing in any case, but I'm not sure where to put it in the UI.)

  Or, could make apache-sitealias.tmpl use per-domain cert files. This avoids all this complexity. Only the Registration/IP rate limit might be a problem with doing this, but it's sufficiently high. (done)

• Sites with a apache.conf.tmpl password protecting the site will not be able to use letencrypt, unless it's relaxed to let .well-known be accessed w/o password. This can be fixed by adding this to the end of their apache.conf.tmpl file:

  /public_html/.well-known> AuthType None Require all granted