Skip to content

Renew Shibboleth SSL Certificate

Overview

This guide describes the process for renewing and updating the SSL certificate for shibboleth.statista.com on the Shibboleth instances.

ℹ️ Configuration Management: Certificate files and Shibboleth configuration are version-controlled in the shibboleth-server-configuration repository. Use the repository's deployment script to push certificate changes to both servers.

Prerequisites

  • Access to cheapsslshop account (credentials in LastPass)
  • Access to DFN AAI metadata management (credentials in LastPass)
  • SSH access to ec2-euc-shib-01 and ec2-euc-shib-02 instances
  • VPN connection established
  • SSH key: euc-shib.pem
  • Local clone of the shibboleth-server-configuration repository

Important Notes

⚠️ Warning: The systemctl restart shibd command doesn't work reliably. Always stop the service first with systemctl stop shibd and then start it with systemctl start shibd.

⚠️ Warning: Service start can take 90 minutes (not 15-20 as previously documented). Only stop/start one instance at a time to maintain service availability.

Steps

1. Renew Certificate

  1. Log in to cheapsslshop (credentials in LastPass)
  2. Renew the certificate for shibboleth.statista.com

2. Generate CSR

  1. SSH into one of the Shibboleth instances
  2. Use the existing private key to generate a Certificate Signing Request (CSR)
  3. Submit the CSR to complete the certificate renewal

3. Add New Certificate to Configuration Repository (Day 0)

  1. Clone or update the shibboleth-server-configuration repository:

    cd /path/to/shibboleth-server-configuration
git pull

  2. Add the new certificate file to etc/shibboleth/ (e.g., shibboleth.statista.com-2026.crt.pem)

  3. Edit etc/shibboleth/shibboleth2.xml to add the new certificate in the CredentialResolver section:

    <CredentialResolver type="Chaining">
   <!-- old certificate -->
   <CredentialResolver type="File" key="shibboleth.statista.com.key.pem" certificate="shibboleth.statista.com-2025.crt.pem"/>
   <!-- new certificate -->
   <CredentialResolver type="File" key="shibboleth.statista.com.key.pem" certificate="shibboleth.statista.com-2026.crt.pem"/>
</CredentialResolver>

  4. Commit and push changes:

    git add .
git commit -m "Add 2026 certificate for renewal"
git push

  5. Deploy to both servers using the deployment script:

    ./deploy.sh
# Select option 3 (Both servers)

  6. Restart shibd on ec2-euc-shib-02 first:

    ssh user@ec2-3-67-78-187.eu-central-1.compute.amazonaws.com 'sudo systemctl stop shibd && sudo systemctl start shibd'

  7. Wait for ec2-euc-shib-02 to complete startup (~90 minutes) and verify:

    curl https://shibboleth.statista.com/Shibboleth.sso/Status -k

  8. After ec2-euc-shib-02 is healthy, restart ec2-euc-shib-01:

    ssh user@ec2-52-57-79-135.eu-central-1.compute.amazonaws.com 'sudo systemctl stop shibd && sudo systemctl start shibd'

Reference: DFN AAI Documentation - Step 4 at SP

4. Update DFN AAI Metadata (Day 0)

  1. Log in to DFN AAI metadata management (credentials in LastPass)
  2. Add the new certificate to the metadata
  3. Wait 24 hours for propagation

5. Change Certificate Order (Day 1)

  1. In the shibboleth-server-configuration repository, edit etc/shibboleth/shibboleth2.xml and change the order of certificates (new certificate first):

    <CredentialResolver type="Chaining">
   <!-- new certificate -->
   <CredentialResolver type="File" key="shibboleth.statista.com.key.pem" certificate="shibboleth.statista.com-2026.crt.pem"/>
   <!-- old certificate -->
   <CredentialResolver type="File" key="shibboleth.statista.com.key.pem" certificate="shibboleth.statista.com-2025.crt.pem"/>
</CredentialResolver>

  2. Commit and push:

    git add etc/shibboleth/shibboleth2.xml
git commit -m "Change certificate order - prioritize 2026 cert"
git push

  3. Deploy and restart both servers (one at a time):

    ./deploy.sh
# Select option 3 (Both servers)

Reference: DFN AAI Documentation - Step 5 at SP

6. Remove Old Certificate from Metadata (Day 1)

  1. Log in to DFN AAI metadata management
  2. Remove the old certificate from the metadata
  3. Wait 24 hours for propagation across DFN AAI and UK Federation IdPs

7. Remove Old Certificate from Configuration (Day 2)

  1. In the shibboleth-server-configuration repository, edit etc/shibboleth/shibboleth2.xml and remove the old certificate:

    <CredentialResolver type="Chaining">
   <!-- new certificate -->
   <CredentialResolver type="File" key="shibboleth.statista.com.key.pem" certificate="shibboleth.statista.com-2026.crt.pem"/>
</CredentialResolver>

  2. Optionally remove or archive the old certificate file from etc/shibboleth/

  3. Commit and push:

    git add .
git commit -m "Remove old 2025 certificate"
git push

  4. Deploy and restart both servers (one at a time):

    ./deploy.sh

Verification

After each restart, always verify the service is running properly:

curl https://shibboleth.statista.com/Shibboleth.sso/Status -k

Check the logs for any issues:

ssh user@hostname 'sudo tail -f /var/log/shibboleth/shibd.log'

Troubleshooting

For major issues, escalate to the responsible team. See Statista Atlassian for related troubleshooting documentation.

A word on: Shibboleth Service Provider (shibd) Startup Time

Normal Behavior

Expected startup time: ~1.5 hours

The Shibboleth daemon takes approximately 90 minutes to start due to processing large federation metadata files totaling ~270 MB of XML.

During startup, shibd parses thousands of IdP definitions and validates cryptographic signatures on all metadata. The signature validation is the most time-consuming step, but ensures metadata authenticity.

Why This Is Acceptable

  • Restarts only occur during annual certificate updates
  • Service remains available on the other instance during restart
  • No production impact when properly planned

Emergency Workaround

If faster startup is critical during an incident, you can temporarily disable signature validation by commenting out all <MetadataFilter type="Signature".../> lines in /etc/shibboleth/shibboleth2.xml. This reduces startup time to 15-20 minutes.

⚠️ Warning: Disabling signature validation reduces security. Only use in time-critical situations and re-enable afterward. Always backup the configuration file first.

Last updated: March 24, 2026 at 10:39
By: Axel Tetzlaff