Page tree
Skip to end of metadata
Go to start of metadata

The FaaS Operations Team uses this document to adapt newly created instances of FaaS machines (cloned from the then current "template" machine) and make it specific for an NREN, using the information from FaaS instances configuration parameters.

Table of Contents

1. FaaS template instance

VM template.faas.geant.net is the FaaS template instance that holds the deployment of software stack and setup used for FaaS. Each customer instance should be cloned from the template instance.

The FaaS software stack consists of:

This guide describes configuration changes that need to be done for each instance cloned from the template instance and software update procedures.

Configuration steps that should be performed only on the production instances (and not on training or test) are labeled with the symbol (star).

2. Firewall configuration

All users should use HTTPS in order to communicate with the server, so the machine is visible from the public network only on HTTPS. Here is the list of allowed services:

  1. Outbound
    • TCP port 443 opened to everyone
    • TCP port 80 opened to everyone
    • TCP port 25 should be open to the provided smarthost or if there is no smarthost to everyone
    • TCP port 1792 opened to se-tug-hsm1.sunet.se
  2. Inbound
    • TCP port 443 opened to everyone
    • SSH access allowed only from PSNC management IP addresses

For outbound traffic TCP port 80 could be limited to http://mds.edugain.org/ and TCP port 443 could be removed for production use, but since yum and pip download packages via http/https we should leave these ports open.

For HSM access, we are currently using only se-tug-hsm1.sunet.se, but another HSM appliance in Denmark will be available at some point, allowing for geo-redundant access to the HSM service. So, when that happens we’ll need to update firewall configuration.

3. System changes

Change the root password on each new instance:

passwd root

4. Software stack configuration

4.1. Apache web server

If you are using this deployment guide for creation of training instances, then skip the sections 4.1.1 and 4.1.2. For production instances you will need to follow the procedure for getting a new certificate since these instances will not use template wildcard certificate.

4.1.1. (star)Creating TLS certificate

  1. Based on the certificate information from the FaaS instances configuration parameters page, create a new private key and CSR. (For the exact values of supported/desired key size and hashing algorithm check FaaS instances configuration parameters.)

    openssl req -nodes -newkey rsa:4096 -sha256 -keyout webserver-key.pem -out webserver.csr
  2. Replace the secret key in /etc/pki/tls/private/webserver-key.pem with the newly created one:

    mv -f webserver-key.pem /etc/pki/tls/private/
    chown root.root /etc/pki/tls/private/webserver-key.pem
    chmod 0400 /etc/pki/tls/private/webserver-key.pem
  3. Send a copy of the CSR to faas@lists.geant.org. FaaS team will then forward the CSR to the customer for requesting the certificate.
  4. After receiving the certificate from FaaS team, rename the certificate to webserver-cert.pem, replace the server certificate in /etc/pki/tls/certs/webserver-cert.pem with the new one and set appropriate permissions:

    chown root.root /etc/pki/tls/certs/webserver-cert.pem
    chmod 0644 /etc/pki/tls/certs/webserver-cert.pem
  5. Set up the certificate chain:
    • For NRENs that are using TCS, generate certificate chain by running the following command:

      cd /etc/pki/tls/
      make

      (info) After you run make follow the further instructions for the available targets (arguments).

    • NRENs using other CAs (i.e., not TCS) should supply us with the appropriate certificate chain (or documentation). Rename the certificate chain to chain.pem, replace the certificate chain file in /etc/pki/tls/chain.pem with the new one and set appropriate permissions:

      chown root.root /etc/pki/tls/chain.pem
      chmod 0644 /etc/pki/tls/chain.pem
      
    • In case of a CA without any intermedate CAs (e.g. MREN) simply comment out the SSLCertificateChainFile directive in /etc/httpd/conf.d/ssl.conf.

4.1.2. (star) Testing web server TLS

Restart web server and test with the following command, replacing HOSTNAME with the publicly visible server name also contained in the certiticate itself, and the referenced CAfile with a copy of the root CA certificate for the CA used (e.g. TCS in the example below):

openssl s_client -CAfile /etc/pki/tls/AddTrustExternalCARoot.pem -connect HOSTNAME:443 < /dev/null 2>&1 | fgrep Verify

This should produce the output:

Verify return code: 0 (ok)

4.1.3. Apache httpd configuration

Modify /etc/httpd/conf.d/ssl.conf file and set the ServerName in the eponymous directive.

#omitted for the purpose of this guide!

<VirtualHost *:443>
#omitted for the purpose of this guide!
ServerName CHANGE_SERVER_NAME
#omitted for the purpose of this guide!

4.2. MySQL database

  1. Change the password for the mysql root user
    • as root user run mysqladmin command to reset password. You can use pwgen(1) to generate new password, e.g.:

      pwgen -s 18 | head -1
      mysqladmin password <type_generated_password>
    • write new password to file /root/.my.cnf and make sure it’s still only readable by root (chmod 0600).
    • execute command:

      mysqladmin flush-privileges
  2. Change password for database user rr3user
    • Generate and note down another password, e.g. using pwgen again (as done for the MySQL root account)

    • login to mysql database as root user, no password required after following the above steps:

      mysql mysql
    • issue UPDATE statement to modify user table, pasting in the freshly created/generated password:

      UPDATE user SET Password = PASSWORD('<type_generated_password>') WHERE User = 'rr3user';
      FLUSH PRIVILEGES;
      QUIT

      This password also needs to be configured within Jagger later.

4.3. Postfix configuration

  1. If customer provides us with a smarthost for email notifications, change relayhost in /etc/postfix/main.cf so that it points to the smarthost:

    relayhost = CHANGE_SMART_HOST
  2. Make sure the smarthost accepts connections on port 25:

    echo QUIT | nc -w5 CHANGE_SMART_HOST 25
  3. Restart postfix and test email delivery by sending an email to yourself:

    date | mail -s "`hostname -f` email test" your@email.example.com

    Email headers on the recieved message should reveal that this email traversed the NREN's smarthost successfully.

  4. If email delivery is not working, check log files for error message and send it to FaaS development team.

4.4. Jagger

  1. In file /opt/rr3/application/config/config.php:
    • Change base_url variable:

      $config['base_url'] = 'https://CHANGE_SERVER_NAME/';
  2. In file /opt/rr3/application/config/config_rr.php:
    • set default registration authority (the unique identifier of the SAML metadata registrar, i.e, of the federation):

      $config['registrationAutority'] = 'CHANGE_UNIQUE_FEDERATION_URL';
    • set list of languages in two-letter ISO 639-1 codes with single quotes and separated by commas, for example array ('en', 'rs'). By default should be the official languages of the country/region served by the NREN, plus English:

      $config['langselectlimit'] = array(CHANGE_LANGUAGE_LIST);
    • set the default language in two-letter ISO 639-1 code. Should be one of the official languages of the country/region served by the NREN.

      $config['langselectdefault'] = CHANGE_DEFAULT_LANGUAGE;
    • set permission who can do translation of Jagger UI and into what language (optional):

      $config['translator_access']['CHANGE_LANGUAGE'] = 'CHANGE_USER';

      (warning) Depending on the language (e.g. Lithuanian) you need to change permissions for rr_lang.php file in the appropriate language directory (e.g. /opt/rr3/application/language/lt) to allow apache user to write to this file.

      (warning) You need to set max_input_vars to 2000 in /etc/php.ini file (the main reason to use PHP > 5.3 here).

    • change the site logo:
      • upload new logo into /opt/rr3/images directory;
      • refernce the file name of the new logo in the site_logo variable in the config_rr.php file:

        $config['site_logo'] = 'CHANGE_LOGO'; 
  3. In file /opt/rr3/application/config/database.php:
    • change password for rr3user (password is already set in step 2 of the MySQL database section):

      $db['default']['username'] = 'rr3user';
      $db['default']['password'] = '<type_rr3user_password>';
  4. In file /opt/rr3/application/config/email.php:
    • Change SMTP sender:

      $config ['mail_from'] = 'CHANGE_EMAIL_FROM';
    • Since JaggerMailer is running under control of supervisor process, restart supervisord to apply changes:

      service supervisord restart

      Test it's running successfully (pid and uptime will vary, of course):

      # supervisorctl status
      JaggerMailer   RUNNING    pid 11807, uptime 23 days, 1:35:31

4.5. (star) HSM access

For training instances skip this section, as training machines only use SoftHSM-protected keys.

  1. Create key pair to secure HSM connections:

    vtl createcert -n `hostname -f`
  2. Send the created certificate (/usr/safenet/lunaclient/cert/client/FILENAME.pem) to noc@nordu.net (Cc'ing faas-operations@geant.net) and wait for confirmation that the client certificate has been deployed.

  3. Change user and file system permissions for FILENAMEKey.pem:

    chown root.pyff /usr/safenet/lunaclient/cert/client/FILENAMEKey.pem
    chmod 0640 /usr/safenet/lunaclient/cert/client/FILENAMEKey.pem
  4. When you have confirmation from the NDN NOC, verify the connection to the HSM:

    vtl verify 

4.6. Pyff configuration

  1. Edit /opt/faas/pipelines/edugain-upstream.fd, replacing all CHANGE_* strings with provided values (cf. FaaS instances configuration parameters):

    - load:
        - http://127.0.0.1/metadata/federation/eduGAIN/metadata.xml
    - select
    - xslt:
        stylesheet: tidy.xsl
    - xslt:
        stylesheet: pubinfo.xsl
        publisher: CHANGE_UNIQUE_FEDERATION_URL
    - xslt:
        stylesheet: pp.xsl
    - finalize:
        Name: CHANGE_UNIQUE_FEDERATION_URL
        cacheDuration: PT6H
        validUntil: P8D
    - sign:
        key:pkcs11:///usr/safenet/lunaclient/lib/libCryptoki2_64.so:1/faas-prod-private
    - publish:
        output: /opt/faas/pyff-output/edugain-upstream.xml
  2. Edit /opt/faas/pipelines/federation-downstreams.fd, replacing all CHANGE_* strings with provided values:

    - load:
        - http://127.0.0.1/metadata/federation/production/metadata.xml as Local
        - http://mds.edugain.org/ as eduGAIN /opt/faas/credentials/edugain.crt
    - fork:
        - select:
            - "eduGAIN!//md:EntityDescriptor[md:Extensions/mdrpi:RegistrationInfo/@registrationAuthority and not(md:Extensions/mdrpi:RegistrationInfo/@registrationAuthority='CHANGE_UNIQUE_FEDERATION_URL')]"
        - "Local!//md:EntityDescriptor"
        - fork merge:
            - select: Local
        - xslt:
            stylesheet: /opt/faas/tooling/cleanup.xsl
        - xslt:
            stylesheet: pubinfo.xsl
            publisher: CHANGE_UNIQUE_FEDERATION_URL
        - xslt:
            stylesheet: /opt/faas/tooling/regauth.xsl
        - xslt:
            stylesheet: pp.xsl
        - finalize:
            Name: CHANGE_UNIQUE_FEDERATION_URL
            cacheDuration: PT6H
            validUntil: P8D
        - sign:
            key: pkcs11:///usr/safenet/lunaclient/lib/libCryptoki2_64.so:1/faas-prod-private
        - publish:
            output: /opt/faas/pyff-output/federation-downstream.xml
    - fork:
        - select: "Local!//md:EntityDescriptor"
        - xslt:
            stylesheet: pubinfo.xsl
            publisher: CHANGE_UNIQUE_FEDERATION_URL
        - xslt:
            stylesheet: pp.xsl
        - finalize:
            Name: CHANGE_UNIQUE_FEDERATION_URL
            cacheDuration: PT6H
            validUntil: P8D
        - sign:
            key: pkcs11:///usr/safenet/lunaclient/lib/libCryptoki2_64.so:1/faas-prod-private
        - publish:
            output: /opt/faas/pyff-output/local-entities-downstream.xml

4.7. Cron configuration

After configuring pyff pipelines you should do the following:

  1. Open cron file for editing (as user root):

    crontab -e
  2. Activate (uncomment) all cron jobs for user root after the line “# FaaS”:

    */10 * * * * su - pyff -c "/opt/faas/tooling/md-watch.py >/dev/null"
    10 1 * * * su - pyff -c "/opt/faas/tooling/pyff.py production >/dev/null 2>&1"
    15 1 * * * su - pyff -c "/opt/faas/tooling/pyff.py eduGAIN >/dev/null 2>&1"

    This will cause SAML metadata for the "production" and eduGAIN (upstream) "federations" to be refreshed and re-signed daily at the given hour (here: 1:10 and 1:15 a.m. local time – note that FaaS instances have UTC as local time!), even when no changes to the local metadata occured.
    The md-watch.py script will only re-sign metadata if local metadata changes within Jagger have occured. By default this script is run every 10 minutes via cron, but this can be adjusted to taste. (Running more often will cause changes to be picked up/published sooner.)

4.8. (star) Shibboleth SP configuration (optional)

Many Jagger deployments (and all training and test instances) won't federate their SAML federation registry right away (or ever). So initially there is no need to configure the Shibboleth SP, as its own SAML metadata would need to be registered into Jagger first, as well as any IDPs to log in to Jagger with.

  1. Regenerate key material for SAML SP:

    umask 0077
    cd /etc/shibboleth
    ./keygen.sh -f -u shibd -g shibd -h CHANGE_SERVER_NAME
    chmod 0644 sp-cert.pem
  2. Edit /etc/shibboleth/shibboleth2.xml, replacing all CHANGE_* strings with provided values:

    <ApplicationDefaults entityID="https://CHANGE_SERVER_NAME/shibboleth"
      REMOTE_USER="eppn persistent-id targeted-id">
    ...
    <SSO discoveryProtocol="SAMLDS" discoveryURL="https://CHANGE_SERVER_NAME/shibboleth-ds/">
      SAML2 SAML1
    </SSO>
    ...
    <Errors supportContact="CHANGE_EMAIL_FROM"
      helpLocation="/about.html"
      styleSheet="/shibboleth-sp/main.css"/>
  3. Test configuration (might complain due to missing/incorrect SAML metadata from Jagger, if nothing has been put into Jagger previously):

    LD_LIBRARY_PATH=/opt/shibboleth/lib64 shibd -t
  4. Restart shibd:

    /etc/init.d/shibd restart
  5. Ensure shibd starts after reboots:

    chkconfig shibd on

5. Log files

Maintain system and application logs for audit trails. Following logs should be kept:

  • Standard system logs: messages, cron, secure etc.
  • Logs of specific services run from FaaS: web server, web applicaiton, pyff, shibboleth, mysql, postfix.

All logs should be rotated weekly, where at least 4 weeks sets should be kept, or as otherwise mandated by the respective laws under which the infrastructure is operated.

Location of log files:

  • Standard system logs (messages, cron, secure etc.): /var/log/
  • Apache logs: /var/log/httpd/
  • Web application: /var/log/rr3/
  • Pyff logs: /var/log/faas
  • Shibboleth logs: /var/log/shibboleth/ and /var/log/shibboleth-www/
  • Mysql logs: /var/log/mysqld.log
  • Postfix logs: /var/log/maillog

6. Backup

Backup is only important for production instances; there is no need to back up training instances as those do not contain real data and are generally rather short-lived by nature.

For each VM:

  1. Create backup of /opt/etc and /var/local/mysqlbackup directories, recursively, on a daily basis. Backed up files and directories should be kept two weeks at minimum. Ideally at least the last 2 generations of backed-up files should be kept (i.e., the current one and the previous one).

  2. Create a [hot] copy of the VM on a monthly basis. The last two images should be kept for quick disaster recovery of the configured system.

 

 

 

 

  • No labels