Test Ideas¶

We need to test code, ideally on every checkin, and ideally fairly completely.

• Test Ideas
• Existing tools
• Automation
• Ideas for test scenarios
• Basic profile operations
• mapitest
• Fast Transfer Test Server
• Microsoft Test Suites

Existing tools¶

We already have some tools that can be used for testing: * mapitest * other command line tools (mapiprofile, openchangeclient, openchangepfadmin)

We previously had torture tests that used smbtorture from samba. The were removed in early 2011. mapitest is an easier-to-use standalone tool.

Automation¶

We could use buildbot to automate some of this. As a minimum, that would ensure the code builds, but could be easily extended to do at least some testing if the builders ("slaves" in buildbot terms) had access to a server.

We might need to roll up some of these tests with buildbot to avoid the waterfall display becoming too long. (e.g. run the tests as a script rather than a series of individual buildbot ShellCommands).

Ideas for test scenarios¶

Basic profile operations¶

Use mapiprofile to create a new database, add a profile to the account, manipulate the profile in some way (e.g. set it as default), add a second profile, change the default, run some tests, delete one of the profiles, run some more tests, delete the other profile.

mapitest¶

Use mapitest to run the whole suite.

Use mapitest to run the no-server tests on builders that don't have a server to talk to.

Perhaps run individual tests to make sure that the mapitest command line option (--mapi-calls) works.

Fast Transfer Test Server¶

Fast Transfer is important to support Outlook "cached" mode (Incremental Change Synchronization in the protocol documentation) type operations. It is fairly easy state machine to implement, but fairly difficult to test because Exchange (especially a clean install of Exchange 2010) sends a fairly small subset of the possible data types and modes, and that makes it hard to test failure paths.

Instead, we could adapt the OpenChange server code to run in a "test" mode. This would be useful to test our client side parser code, and could also be used to test intermediate processing (e.g. WAN accelerators).

The server would only need to support a very small number of ROPs, and may not need any backend support.

Important functionality:

• test all of the marker types
• test all of the property types
• stress test segmentation (e.g. breaking up variable length properties in unusual ways)
• add progress indications
• strings with and without terminating nulls.
• test handling of "legitimate" (in spec) error conditions
  • PidTagEcWarning
  • PidTagNewFxFolder
• test handling of FxDelProps operations in several variants
• really huge strings
• test handling of "crazy" (out of spec) error conditions (not expected to produce anything sensible, just to survive)
  • break the state machine logic (StartFolder in the middle of a Message; EndAttachment that isn't open, lots of Ends in a row, EndMessage where an EndAttachment was expected)
  • break "atoms"
  • just ending the transfer in the middle of a PtypString or PtypBinary, or an incomplete PtypMulti* array
  • trying to do RopFastTransferPutBuffer on a download context, and RopFastTransferGetBuffer on an upload context.
  • sending properties with non-recognised types (e.g. 0xFFFF)
  • Sending props with 0x000A type
  • breaking ordering rules (e.g. not sending PidTagAttachNumber as the first property of an attachment
  • not sending ExtendedErrorInfo structure when required
  • sending PidTagNewFxFolder then sending a bad FolderReplicaInfo struct
  • sending progress indications that contain bogus data, go backwards, aren't allowed in context
  • multivalue properties with zero length, and with length that could be intepreted as a negative value if signed.

It would probably be useful for proxy testing if the test client could be configured in various ways.

Microsoft Test Suites¶

Microsoft is developing test suites that check conformance of a given server implementation to the documented behaviour.
Some of the tests are more robust than others, but they mostly show interesting things about the server being tested (e.g. implementation errors, misunderstandings of the documents, and things we didn't check well enough). The tests we've seen so far correspond to:

• MS-OXCRPC
• MS-OXCROPS
• MS-OXCPRPT

One way to think of the test suite is shown below:

The test suite is essentially a client, and the OpenChange server (or another "System Under Test") is expected to respond in accordance with the requirements of the protocol documentation. It is an interesting approach (with a significant traceability burden), and can show errors in correctness and protocol compliance. Passing the tests doesn't mean that the server will be useful, and failing them doesn't mean that the server can't be used, but the results of failing tests are always worth investigating.

One aspect worth exploring is the "SUT Control Adapter", which is used for some (but not all) test suites. The "SUT Control Adapter" is used to reconfigure the server, or to do some other out-of-band action that is required to check certain behaviour. For example, the test suite might want to turn off Asynchronous notifications, then turn it back on. Microsoft provides two options for the "SUT Control Adapter" (one using PowerShell remote management, and the other using pop-up dialogs that instruct the tester to do some action manually). Neither is suitable for automated testing of the OpenChange server. Instead, we'll probably reimplement the interface (IOXCRPCServerControlAdapter, only five methods) to do SSH commands that implement the required configuration actions.