Versions Compared

Old Version 43

changes.mady.by.user Maria Haider

Saved on

compared with

New Version Current

changes.mady.by.user Maria Haider

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Please follow below guide after the tech team has cut the new release. You need to be authorized to push the image to docker.sunet.se, see this.

Build thiss-js

  1. git clone https://github.com/TheIdentitySelector/thiss-js
  2. git checkout <version-to-be-released>
  3. make docker && make docker_push_sunet 

...

  1. sunet

Build thiss-mdq

  1.  git clone git@github.com:TheIdentitySelector/thiss-mdq.git
  2.   make

Beta deployment 

Frontend (use.thiss.io)

...

To deploy <version> to beta (use.thiss.io)

  1. Clone the git repository in your computer (git@github.com:TheIdentitySelector/thiss-ops.git)
  2. Open global/overlay/etc/hiera/data/common.yaml, check the 'max-age' under 'cache_control_beta'. If the current s-max age is 2 days or 172800 seconds, you need to change it to 10 hours, two days before the deployment. If it is one day, you need to change it to 10 hours exactly 1 day before. We need to do it so the old cache gets cleared from non-fastly cache nodes and new files are requested from our backend during the deployment.
  3. Wait as long as the old 's-max-age' and then proceed to the next step. You can even check in the 'developer tools' of your browser (e.g. Chrome, Firefox) that the response headers have the new s-maxage of 10 hours. You can try by using https://demo.beta.seamlessaccess.org/ 
  4. Update the ds_version_beta in global/overlay/etc/hiera/data/common.yaml.
Code Block
languagebash
ds_version_beta: '2.1.51'
cache_control_beta: 'public, max-age=36000, must-revalidate, s-maxage=172800, proxy-revalidate'
  1. Run git add global/overlay/etc/hiera/data/common.yaml and git commit. You should ofcourse have right to commit in the repository.
  2. Run the script thiss-ops/bump-tag afterwards.
  3. To verify that the new version is installed, log in to the servers static-1.thiss.io and static-2.thiss.io and enter 'run-cosmo -v'. 
  4. You can check the status by running the command service docker-thiss_js status.
  5. You can also enter 'docker ps' in order to see if the new version is present on docker image tag.
  6. We may have to do more than one version upgrade to make it backward compatible, check https://github.com/TheIdentitySelector/thiss-js/blob/staging/RELEASE.md
  7. Check that the login works using demo.beta.seamlessaccess.org. You can check the transaction check & page speed in Pingdom after each upgrade. Page speed shows the response headers and other information for each file loaded from https://use.thiss.io.
  8. It is also good to check the HIT ratio (observability > service overview) and origin latency under CDN services from Fastly GUI for the service 'use.thiss.io'. Chek that the HIT ratio is high enough, should be around 98 to 99 percent, latency is way below 5 seconds.
  9. After the verification step is done, everything looks good and open global/overlay/etc/hiera/data/common.yaml and change 'cache_control_beta to original value after 10 hours have passed.

Verification

...

  1. Code Block
    cache_control_beta: 'public, max-age=36000, must-revalidate, s-maxage=172800, proxy-revalidate'
  2. Log in to the servers static-1.thiss.io, static-2.thiss.io, static-1.aws2.thiss.io & static-2.aws2.sthiss.io and enter 'run-cosmo -v'. 
  3. Restart the service by running the command service sunet-thiss_js restart.
  4. One liner to check that service is working by showing expected version and s-maxage.
    Code Block
    run-cosmos && service sunet-thiss_js restart && sleep 5 && curl http://localhost/manifest.json && echo "" && docker exec thiss_js-thiss-js-1 env |grep "s-maxage"
  5. It is also good to check the HIT ratio & origin offload (observability > service overview) from Fastly GUI for the service 'use.thiss.io'. Chek that the HIT ratio & origin offload are high enough, should be around 98 to 99 percent. 
  6. Wait as long as the old 's-max-age' and then proceed to the next step. You can even check in the 'developer tools' of your browser (e.g. Chrome, Firefox) that the response headers have the new s-maxage of 10 hours. You can try by using https://demo.beta.seamlessaccess.org/ 
  7. Update the ds_version_beta in global/overlay/etc/hiera/data/common.yaml.


Code Block
languagebash
ds_version_beta: '2.1.51'


  1. Run git add global/overlay/etc/hiera/data/common.yaml and git commit. You should ofcourse have right to commit in the repository.
  2. Run the script thiss-ops/bump-tag afterwards.
  3. To verify that the new version is installed, log in to the servers static-1.thiss.io, static-2.thiss.io, static-1.aws2.thiss.io & static-2.aws2.sthiss.io and enter 'run-cosmo -v'. 
  4. Restart the service by running the command service sunet-thiss_js restart.
  5. You can check the status by running the command service sunet-thiss_js status.
  6. You can run  curl -k https://localhost/manifest.json to see that it shows current version and no errors.
  7. You can also enter 'docker ps' in order to see if the new version is present on docker image tag.
  8. One liner to check that service is working by showing expected version. (the seconds for sleeping may need to be adjusted as it will take time to download the new docker image)
    Code Block
    run-cosmos && service sunet-thiss_js restart && sleep 15 && curl http://localhost/manifest.json && echo "" && docker ps
  9. We may have to do more than one version upgrade to make it backward compatible, check https://github.com/TheIdentitySelector/thiss-js/blob/staging/RELEASE.md
  10. Check that the login works using demo.beta.seamlessaccess.org. You can check the transaction check & page speed in Pingdom after each upgrade. Page speed shows the response headers and other information for each file loaded from https://use.thiss.io

...

  1. .
  2. It is also good to check the HIT ratio & origin offload (observability > service overview) from Fastly GUI for the service 'use.thiss.io'. Chek that the HIT ratio & origin offload are high enough, should be around 98 to 99 percent. 
  3. After the verification step is done, everything looks good and open global/overlay/etc/hiera/data/common.yaml and change 'cache_control_beta to original value after 10 hours have passed.
  4. Restart the service by running the command service sunet-thiss_js restart. One liner to check that service is working by showing expected version and s-maxage.
    Code Block
    run-cosmos && service sunet-thiss_js restart && sleep 5 && curl http://localhost/manifest.json && echo "" && docker exec thiss_js-thiss-js-1 env |grep "s-maxage"

Verification

Verify that the changes have taken effect - this may take a while depending on how quickly the CDN picks up the changes. Find out which changes should be tested, check with the developer team or technical lead Leif Johansson. In addition to that, it should be checked that the discovery service works by visiting https://use.thiss.io. It is good to do it in a private window of your browser in case your browser has cached the old version. Click on the 'Login' button and see that it is possible to choose different IDPs from there. Check that the persistent service works by going back and choosing different organizations. You should be able to see the list of organizations that you have chosen and be able to edit them as well. Check that these functions work. https://use.thiss.io/manifest.json is supposed to show the latest version number. You can check the login works through https://demo.beta.seamlessaccess.org as well. 

Rollback

In order to rollback simply downgrade the version in cosmos-rules.yaml and follow the exact steps for committing and pushing the the changes to the git remote repo.

Backend (md.thiss.io)

  1. Clone the git repository in your computer (git@github.com:TheIdentitySelector/thiss-ops.git)
  2. Update the version under thiss::mdq for both 'md-1.thiss.io' and 'md-2.thiss.io'
Code Block
md-1.thiss.io:
   sunet_iaas_cloud:
   thiss::dockerhost:
      version: '5:20.10.12~3-0~ubuntu-bionic'
   thiss::mdq:
      version: 1.3.2
      src: https://a-1.thiss.io/metadata.json
      base_url: https://md.thiss.io
      post: /usr/sbin/service docker-thiss_mdq restart
   https:
   http:
  1. Do git add global/overlay/etc/puppet/cosmos-rules.yaml and git commit. You should ofcourse have right to commit in the repository.
  2. Run the script thiss-ops/bump-tag afterwards.
  3. To verify that the new version is installed, log in to the servers md-1.thiss.io and md-2.thiss.io and enter 'run-cosmo -v'. 
  4. Restart the service by running the command service sunet-thiss-mdq restart.
  5. You can check the status by running the command service sunet-thiss-mdq status.
  6. You can run  curl -k http://localhost/ to see that it shows right version, metadata information and no errors.
  7. You can also enter 'docker ps' in order to see if the new version is present on docker image tag. 

Verification

Check that https://md.thiss.io is up and showing the right version. Keep an eye on the metadata check alarm on the internal Nagios instance https://monitor.seamlessaccess.org

Rollback

Simply undo the  changes and go back to old changes and commit them in thiss-ops repository.

Production deployment

Frontend (service.seamlessaccess.org)

The current whitelist needs to be checked with the master list before each deploy. Note that use.thiss.io does not do whitelisting which means the WHITELIST variable is not set.

To deploy <version> to production (service.seamlessaccess.org)

  1. Clone the git repository in your computer (git@github.com:TheIdentitySelector/thiss-ops.git)
  2. Open global/overlay/etc/hiera/data/common.yaml and check that the whitelist is right or change it if needed.


    Warning
    titleWhitelisting

    It is important to set the WHITELIST environment variable to the comma-separated list of the current whitelisted domains before deploying. Ask Marina or Leif to verify the list. The list is updated here Seamless Access Configuration Parameters.


  3. Open global/overlay/etc/hiera/data/common.yaml, check the 's-maxage' under 'cache_control_prod. If the current s-max age is 2 days or 172800 seconds, you need to change it to 10 hours, two days before the deployment. If it is one day, you need to change it to 10 hours exactly 1 day before. We need to do it so the old cache gets cleared from non-fastly cache nodes and new files are requested from our backend during the deployment.
    Code Block
    cache_control_prod: 'public, max-age=36000, must-revalidate, s-maxage=172800, proxy-revalidate'
  4. Log in to static-*.org servers and enter 'run-cosmo -v'. 
  5. Restart the service by running the command service sunet-thiss_js restart.
  6. One liner to check that service is working by showing expected version and s-maxage.
    Code Block
    run-cosmos && service sunet-thiss_js restart && sleep 5 && curl http://localhost/manifest.json && echo "" && docker exec thiss_js-thiss-js-1 env |grep "s-maxage"
  7. It is good to see that backends are not overloaded, you can check the HIT ratio (observability > service overview)  from Fastly GUI for the service 'service.seamlessaccess.org'. Origin latency can be checked here https://manage.fastly.com/observability/dashboard/custom/1cHLXZhdo3OZyqpPmCTdYP/historic/5vqg1bmES8N0KNljosrwVX. The latency should be less than 5 seconds.
  8. Keep an eye on the transaction check in pingodm that it is not failing after lowering the s-maxage.
  9. Wait as long as the old 's-maxage' and then proceed to the next step. You can even check in the 'developer tools' of your browser (e.g. Chrome, Firefox) that the response headers have the new s-maxage of 10 hours. You can try by using https://demo.seamlessaccess.org/ 
  10. Update the ds_version_prod in global/overlay/etc/hiera/data/common.yaml.


    Code Block
    languagebash
    ds_version_prod: '2.1.51'


  11. Run git add

Rollback

In order to rollback simply downgrade the version in cosmos-rules.yaml and follow the exact steps for committing and pushing the the changes to the git remote repo.

Backend

  1. Clone the git repository in your computer (git@github.com:TheIdentitySelector/thiss-ops.git)
  2. Update the version under thiss::mdq for both 'md-1.thiss.io' and 'md-2.thiss.io'
Code Block
md-1.thiss.io:
   sunet_iaas_cloud:
   thiss::dockerhost:
      version: '5:20.10.12~3-0~ubuntu-bionic'
   thiss::mdq:
      version: 1.3.2
      src: https://a-1.thiss.io/metadata.json
      base_url: https://md.thiss.io
      post: /usr/sbin/service docker-thiss_mdq restart
   https:
   http:
  1. Do git add global/overlay/etc/puppet/cosmos-rules.yaml and git commit. You should ofcourse have right to commit in the repository.
  2. Run the script thiss-ops/bump-tag afterwards.
  3. To verify that the new version is installed, log in to the servers md-1.thiss.io and md-2.thiss.io and enter 'run-cosmo -v'. 
  4. You can check the status by running the command service docker-thiss_mdq status.
  5. You can also enter 'docker ps' in order to see if the new version is present on docker image tag. 

Verification

Check that https://md.thiss.io is up and showing the right version. Keep an eye on the metadata check alarm on the internal Nagios instance https://monitor.seamlessaccess.org

Rollback

Simply undo the  changes and go back to old changes and commit them in thiss-ops repository.

Production deployment (service.seamlessaccess.org)

Frontend

The current whitelist needs to be checked with the master list before each deploy. Note that use.thiss.io does not do whitelisting which means the WHITELIST variable is not set.

To deploy <version> to production (service.seamlessaccess.org)

  1. Clone the git repository in your computer (git@github.com:TheIdentitySelector/thiss-ops.git)
  2. Open global/overlay/etc/hiera/data/common.yaml and check that the whitelist is right or change it if needed.
    Warning
    titleWhitelisting

    It is important to set the WHITELIST environment variable to the comma-separated list of the current whitelisted domains before deploying. Ask Marina or Leif to verify the list. The list is updated here Seamless Access Configuration Parameters.

    Open .yaml and git commit. You should ofcourse have right to commit in the repository. Run git add global/overlay/etc/hiera/data/common.yaml, check the 's-maxage' under 'cache_control_prod. If the current s-max age is 2 days or 172800 seconds, you need to change it to 10 hours, two days before the deployment. If it is one day, you need to change it to 10 hours exactly 1 day before. We need to do it so the old cache gets cleared from non-fastly cache nodes and new files are requested from our backend during the deployment.
  3. It is good to see that backends are not overloaded, you can check the HIT ratio (observability > service overview) and origin latency under CDN services from Fastly GUI for the service 'service.seamlessaccess.org'.
  4. Keep an eye on the transaction check in pingodm that it is not failing after lowering the s-maxage.
  5. Wait as long as the old 's-maxage' and then proceed to the next step. You can even check in the 'developer tools' of your browser (e.g. Chrome, Firefox) that the response headers have the new s-maxage of 10 hours. You can try by using https://demo.seamlessaccess.org/ 
    6. Update the ds_version_prod in global/overlay/etc/hiera/data/common.yaml.
    Code Block
    languagebash
    ds_version_prod: '2.1.51'
cache_control_prod: 'public, max-age=36000, must-revalidate, s-maxage=172800, proxy-revalidate'
    Run git add global/overlay/etc/hiera/data/common.yaml and git commit. You should ofcourse have right to commit in the repository. Run git add global/overlay/etc/hiera/data/common.yamlas well if whitelist is changed. You should ofcourse have right to commit in the repository.Run the script thiss-ops/bump-tagafterwards.To verify that the new version is installed, log in to below servers and enter 'run-cosmo -v'. 

    static-1.aws1.geant.eu.seamlessaccess.org

    static-1.aws2.geant.eu.seamlessaccess.org

    static-1.ntx.sunet.eu.seamlessaccess.org

    static-1.se-east.sunet.eu.seamlessaccess.org (have to run service sunet-thiss_js restart manually)

    static-2.aws1.geant.eu.seamlessaccess.org

    static-2.aws2.geant.eu.seamlessaccess.org

    static-2.ntx.sunet.eu.seamlessaccess.org

    static-2.se-east.sunet.eu.seamlessaccess.org

  6. as well if whitelist is changed. You should ofcourse have right to commit in the repository.
  7. Run the script thiss-ops/bump-tagafterwards.
  8. To verify that the new version is installed, log in to below servers and enter 'run-cosmo -v'. 


    static-1.aws1.geant.eu.seamlessaccess.org

    static-1.aws2.geant.eu.seamlessaccess.org

    static-1.ntx.sunet.eu.seamlessaccess.org

    static-1.se-east.sunet.eu.seamlessaccess.org

    static-2.aws1.geant.eu.seamlessaccess.org

    static-2.aws2.geant.eu.seamlessaccess.org

    static-2.ntx.sunet.eu.seamlessaccess.org

    static-2.se-east.sunet.eu.seamlessaccess.org

  9. Restart the service by running the command service sunet-thiss_js restart.
  10. You can check the status by running the command service sunet-thiss_js status.
  11. You can run  curl -k https://localhost/manifest.json to see that it shows current version and no errors.
  12. You can also enter 'docker ps' in order to see if the new version is present on docker image tag. 
  13. One liner to check that service is working by showing expected version. (the seconds for sleeping may need to be adjusted as it will take time to download the new docker image)
    Code Block
    run-cosmos && service sunet-thiss_js restart && sleep 15 && curl http://localhost/manifest.json && echo "" && docker ps
  14. We may have to do more than one version upgrade to make it backward compatible, check https://github.com/TheIdentitySelector/thiss-js/blob/staging/RELEASE.md
  15. Check that the login works using demo.seamlessaccess.org. You can check the transaction check & page speed in Pingdom after each upgrade. Page speed shows the response headers and other information for each file loaded from https://service.seamlessaccess.org.
  16. It is also good to check the HIT ratio (observability > service overview) from Fastly GUI for the service 'service.seamlessaccess.org'. Chek that the HIT ratio is high enough, should be around 98 to 99 percent. Origin latency can be checked here https://manage.fastly.com/observability/dashboard/custom/1cHLXZhdo3OZyqpPmCTdYP/historic/5vqg1bmES8N0KNljosrwVX. The latency should be less than 5 seconds.
  17. After the verification step is done, everything looks good and few hours have passed, open global/overlay/etc/hiera/data/common.yaml and change 'max-age' under 'cache_control_prod' to original value.
  18. Log in to static-*.org servers and enter 'run-cosmo -v'. 
  19. Restart the service by running the command service sunet-thiss_js restart. One liner to check that service is working by showing expected version and s-maxage.
    Code Block
    run-cosmos && service sunet-thiss_js restart && sleep 5 && curl http://localhost/manifest.json && echo "" && docker exec thiss_js-thiss-js-1 env |grep "s-maxage"
  20. You can check the status by running the command service docker-thiss_js status.
  21. You can also enter 'docker ps' in order to see if the new version is present on docker image tag. 
  22. We may have to do more than one version upgrade to make it backward compatible, check https://github.com/TheIdentitySelector/thiss-js/blob/staging/RELEASE.md
  23. Check that the login works using demo.seamlessaccess.org. You can check the transaction check & page speed in Pingdom after each upgrade. Page speed shows the response headers and other information for each file loaded from https://service.seamlessaccess.org.
  24. It is also good to check the HIT ratio (observability > service overview) and origin latency under CDN services from Fastly GUI for the service 'service.seamlessaccess.org'. Chek that the HIT ratio is high enough, should be around 98 to 99 percent and latency is way below 5 seconds.                                                              Image Removed
    25. After the verification step is done, everything looks good and few hours have passed, open global/overlay/etc/hiera/data/common.yaml and change 'max-age' under 'cache_control_prod' to original value.
Verification
Verify that the changes have taken effect - this may take a while depending on how quickly the CDN picks up the changes. Find out which changes should be tested, check with the developer team or technical lead Leif Johansson. In addition to that, it should be checked that the discovery service works as usual by trying to login to a service for example wiki.sunet.se. It is good to do it in a private window of your browser in case your browser has cached the old version. https://service.seamlessaccess.org/manifest.json is supposed to show the latest version number. You can check the login works through https://demo.seamlessaccess.org as well. 
...
  1. Clone the git repository in your computer (git@github.com:TheIdentitySelector/thiss-ops.git)
  2. Open global/overlay/etc/hiera/data/common.yaml and check that the whitelist is right or change it if needed.

     Warning
    titleWhitelisting
    It is important to set the WHITELIST environment variable to the comma-separated list of the current whitelisted domains before deploying. Ask Marina or Leif to verify the list. The list is updated here Seamless Access Configuration Parameters.

  3. Do git add global/overlay/etc/hiera/data/common.yaml as well if whitelist is changed. You should ofcourse have right to commit in the repository.
  4. Run the script thiss-ops/bump-tag afterwards.
  5. Log in to below servers and enter 'run-cosmo -v'. 'thiss-js' docker container should get restarted. 

    static-1.aws1.geant.eu.seamlessaccess.org
    static-1.aws2.geant.eu.seamlessaccess.org
    static-1.ntx.sunet.eu.seamlessaccess.org
    static-1.se-east.sunet.eu.seamlessaccess.org
    static-2.aws1.geant.eu.seamlessaccess.org
    static-2.aws2.geant.eu.seamlessaccess.org
    static-2.ntx.sunet.eu.seamlessaccess.org
    static-2.se-east.sunet.eu.seamlessaccess.org.seamlessaccess.org
  6. Restart the service by running the command service sunet-thiss_js restart.
  7. You can check the status by running the command service 
    8.  docker
  8. sunet-thiss_js status.
  9. Log into Fastly management web GUI https://manage.fastly.com/ and purge all cache for the service 'service.seamlessaccess.org' otherwise it will take the amount of seconds set in 's-maxage' in the static servers for the Fastly servers to fetch the updated JSON JS page.
Verification
Visit https://service.seamlessaccess.org/ps.js should https://service.seamlessaccess.org/ps/index.html in browser and open the Developers tab. Find out the 'https://service.seamlessaccess.org/ps_<contetn hash>.js' link from the Network tab. That link should contain the new whitelisted domain(s). It may take a while to get updated depending on the age of the cache in your browser.
Image Added
It is also good to check that the service is working by visiting https://demo.seamlessaccess.org.
Backend (md.seamlessaccess.org)
  1. Clone the git repository in your computer (git@github.com:TheIdentitySelector/thiss-ops.git)
  2. Update the version under thiss::mdq for each site (ntx, se-east, aws1 and aws2)

     Code Block
    '^md-[0-9]\.ntx\.sunet\.eu\.seamlessaccess\.org$':
   thiss::dockerhost:
   thiss::mdq:
      version: 1.3.2
      src: https://meta.ntx.sunet.eu.seamlessaccess.org/metadata.json
      base_url: https://md.seamlessaccess.org
      post: /usr/sbin/service docker-thiss_mdq restart

  3. Do git add global/overlay/etc/puppet/cosmos-rules.yaml and git commit. You should ofcourse have right to commit in the repository.
  4. Run the script thiss-ops/bump-tag afterwards.
  5. To verify that the new version is installed, log in to below servers and enter 'run-cosmo -v'. 
    md-1.aws1.geant.eu.seamlessaccess.org
    md-1.aws2.geant.eu.seamlessaccess.org
    md-1.ntx.sunet.eu.seamlessaccess.org
    md-1.se-east.sunet.eu.seamlessaccess.org
    md-2.aws1.geant.eu.seamlessaccess.org
    md-2.aws2.geant.eu.seamlessaccess.org
    md-2.ntx.sunet.eu.seamlessaccess.org
    md-2.se-east.sunet.eu.seamlessaccess.org
  6. Restart the service by running the command service sunet-thiss-mdq restart.
  7. You can check the status by running the command service 
    8.  docker
  8. sunet-thiss
    9. _
  9. -mdq status.
  10. You can run  curl -k http://localhost/ to see that it shows right version, metadata information and no errors.
  11. You can also enter 'docker ps' in order to see if the new version is present on docker image tag. 
...