CloudPlugFest - Interoperability testing

Plan and procedures

There are a number of test plans available for testing interoperability of implementations of publicly defined interfaces.
For this time, we intend to use the material that is already available from ETSI for the various standards we expect to test:

OCCI, CDMI: we intend to use the ETSI reference testing procedure at http://www.etsi.org/deliver/etsi_ts/103100_103199/103142/01.01.01_60/ts_103142v010101p.pdf

Recommendations for ETSI OCCI Test Suite

Stand-ins

Reevaluate OCCI/CORE test cases covered by other test cases in extensions (e.g., OCCI/INFRA). Instead of dropping them we could introduce new descriptions to make them more understandable. Especially for tests using “abstract” parts of the standard such as “7.1.2.1 Create an OCCI Resource” usually not implemented in applications focusing on a specific OCCI extension. We would recommend the introduction of “stand-ins” where certain passing tests in extension test suites would automatically mark specified tests in core test suites as successful. Each test would specify its “stand-ins”. As an example, OCCI/CORE/CREATE tests would have “stand-ins” in OCCI/INFRA/CREATE. This way an implementation could have an “all-green” result without implementing abstract features of the standard which is currently not possible.

Detailed prerequisites

OCCI’s extensibility and dynamic declaration of available kinds, actions and mixins makes automated testing very difficult. Prerequisites for tests cases should be detailed and specify used kinds, mixins and actions very clearly. For example, the following formulations currently used in test case descriptions are too vague and lead to confusion and false positives/negatives.

• OCCI Client selects an OCCI Category provided by the discovery interface
• OCCI Client selects an OCCI Kind describing an OCCI Resource

Use cases should operate with identifiers (schemas, terms, etc.) strictly defined in the standard and therefore available in all compliant implementations. In cases where objects need to be created and subsequently queried, strict instructions should be given as well.

Missing test cases

Add new test cases for missing features (see the testing matrix, rows with Issue target: ETSI Test Case).

Invalid test cases

Multiple test cases in the current version of the ETSI OCCI Test Suite reference ‘application/occi+json’ as a valid rendering for OCCI 1.1, however this is not the case. A standardized JSON rendering for OCCI has never been released for OCCI 1.1 and therefore is not a valid alternative for interoperability testing.

Revisions for the upcoming OCCI 1.2

Revise the document with OCCI 1.2 in mind.  

Testing procedure

To maximise coverage, we intend to set up a matrix of testing implementations against each other, both remote and local.

For this we have set up a spreadsheet to capture who tests against who and what during the actual testing phase.

This document lives with your participation and contribution!

Please capture in that document the details of your tests and the individual test results. Please also create folders in the Google drive and store your logs to give evidence of the test results you capture in the spreadsheet.

Feel free to create any form of documents, spreadsheets and slides you wish to use during the reporting back in the workshop part of the plugfest.