Page History

...

Conformance module repo: https://github.com/cicnavi/simplesamlphp-module-conformance

Running tests

Prior to running tests, of course, the SAML trust between the Conformance IdP and particular SP has to be established
(SAML metadata exchanged). Check the "SP metadata handling" on how to add the SP metadata you want to test.

Also, this instance of Conformance IdP is configured to require SP consent to run tests on it. When the test for particular SP is initiated,
Conformance IdP will check if it has consent for it. If not, it will send an invitation to accept the test to the SP contacts emails. Just to showcase
this scenario, a catch-all SMTP service is brought up at https://mailbox.maiv1.incubator.geant.org/, where all the invitations can be
inspected / accepted (emails won't actually be sent to SP contacts).

Manual testing

For manual testing and validation, SP can simply initiate the authentication using the Conformance IdP
(SP initiated flow), in which case the Conformance IdP will present the screen with available test options prior to 
returning the SAML response:

Image Added

After the test selection, Conformance IdP will return the (modified) SAML response so one can observe if the SP is
behaving appropriately.

Running single tests from Conformance module UI

On the other hand, administrator can use the Conformance IdP UI to initiate all tests using the Nuclei tool.
To do that, navigate to SimpleSAMLphp administration area > Configuration > Conformance (Details area) > Run Nuclei Test.

Choose the desired SP and click Run. This will invoke the Nuclei tool in the backend and start testing using the
predefined testing templates, and then stream all of its output to the screen.

After the testing is done, results can be seen on the Nuclei Results page.

Bulk testing

Bulk testing can be executed using [SimpleSAMLphp Cron module](https://github.com/simplesamlphp/simplesamlphp/blob/master/modules/cron/docs/cron.md).
As you can see in Cron documentation, a cron tag can be invoked using HTTP or CLI. Of course, with Conformance testing
using CLI is the preferred way, since testing can take a (relatively) long time depending on the number of SPs. 
However, you are free to test execution using the HTTP version, in which case the maximum execution time
will correspond to the 'max_execution_time' INI setting.

Only one test runner instance can run at given point in time. By maintaining internal state, test runner can first check
if there is another runner active. If yes, the latter will simply exit and let the active test runner do its work.
This way one is free to invoke the cron tag at any time, since only one test runner will ever be active.

For this test environment, feel free to initiate bult testing from the Cron module UI here:

https://conformance-idp.maiv1.incubator.geant.org/module.php/cron/info

Bulk testing uses the same Nuclei logic, and all test results will be available as described with single tests in UI.

SP metadata handling

Conformance IdP is configured with PDO metadata storage handler (it can use database to store SP metadata) in addition to plain PHP metadata files.

...

GET /resource HTTP/1.1
Host: server.example.com
Authorization: Bearer sometoken

Test

...

setup for next authentication event

Endpoint to define the next test for particular SP. This will determine the shape of the SAML response in the next
authentication event for the given service provider.

URI: https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/test/setup

HTTP method: GET

Parameters:

...

For example, to specify that the next test for the SP 'urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp' should be the one that doesn't sign the SAML Response:

https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/test/setup?testId=noSignature&spEntityId=urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp

Running Nuclei tests using predefined templates

Endpoint which can be used to run all Nuclei tests on a particular service provider using the predefined templates.
This is the same as running Nuclei tests from the Conformance module UI. 
The HTTP response is a streamed output from the Nuclei tool.

URI: https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/test/run

HTTP method: GET

Parameters:

• spEntityId
  • valid values: any trusted SP Entity ID
  • example: urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp
• acsUrl (optional) - if not provided, a default SP ACS URL will be used
  • valid values: ACS URL belonging to the SP defined with spEntityId
  • example: urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp
• enableDebug (optional) - parameter forwarded to Nuclei which determines if HTTP requests and responses should be shown
  • valid values: 1 (true), 0 (false, default)
  • example: 1
• enableVerbose (optional) - parameter forwarded to Nuclei which determines if verbose output should be shown
  • valid values: 1 (true), 0 (false, default)
  • example: 1

For example, to run tests for the SP `urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp`, make an
HTTP GET request to:

https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/test/run?spEntityId=urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp

Test results

Endpoint to fetch test results in JSON format. By default, all results from all SPs will be returned. This can be 
filter out using parameters below.

URI: https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/results/get

HTTP method: GET

Parameters:

• spEntityId (optional): can be used to fetch results for particular SP only
  • valid values: any trusted SP Entity ID
  • example: urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp
• latestOnly (optional): can be used to only fetch latest result per SP
  • valid values: `1` (true), `0` (false, default)
  • example: `1`

For example, to fetch all available results, make an HTTP GET request to:

https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/results/get

To fetch only latest results for all SPs:

https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/results/get?latestOnly=1

To fetch all results for the SP `urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp`:

https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/results/get?spEntityId=urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp

To fetch only the latest result for the SP `urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp`:

https://conformance-idp.maiv1.incubator.geant.org/module.php/conformance/nuclei/results/get?spEntityId=urn:x-simplesamlphp:geant:incubator:simplesamlphp-sp:good-sp&latestOnly=1

SP metadata provisioning

Endpoint to provision SP metadata which will be trusted by the Conformance IdP.

...