OpenChange

h1. MAPIStore 1.0 Development Guide

{{>toc}}

h2. What is MAPIStore?

MAPIStore is the SAL component of OpenChange server. SAL stands for Storage Abstraction Layer. It is the component used by OpenChange Server to push/get information (messages, folders) to/from storage backends. It is designed as a library called within OpenChange Server and it accesses backends compiled as dynamic shared object (DSO) and loaded when mapistore is initialized.

The main objective of mapistore is to provide an interface layer with a common set of atomic functions (operations) used to trigger and dispatch data and commands to the appropriate backend. MAPIStore relies on a backend mechanism specifically designed to transparently handle some of the MAPI semantics required by any Exchange compatible server.

The initial idea was to provide to OpenChange a highly customizable storage backend mechanism which would fit in any situation and any environments. One of the greatest limitation we have found with existing groupware is the storage layer which is generally limited to a single solution, service or format and is neither scalable nor modifiable when user requirements evolve upon time.

MAPIStore solves this problem and go beyond classical limitations. It is not a revolutionary concept, but the way openchange uses it makes the whole difference and offer administrators an innovative way to customize storage.

MAPIStore allows you to:

* use a different backend for any top-folder

* transparently move/copy data across backends

* develop new backends quickly

* access all the different backends through an unique API

For example (assuming all associated backends were developed) a user could have the following storage organization for his mailbox:

* Mails stored using an IMAP backend (Cyrus-IMAP or dovecot)

* Calendar items stored in CalDAV or pushed in Google calendar

* Sent emails and archives/backup stored in a compression backend

* Tasks stored in a MySQL database

* Notes stored on the filesystem

If the user is not satisfied with one of the backend's performance, they would just have to use an administration tool, change the backend, wait for the replication, synchronization to finish and there data will be available from the new backend.

Information can be completely decentralized, stored on one of several servers and still be accessible transparently from OpenChange server.

h2. Getting started

h3. Getting Source code

Using mapistore v1 requires to use a different development tree than trunk. You will need to checkout the code from our current mapistore v1 development branches:

<pre>

svn co http://svnmirror.openchange.org/openchange/branches/sogo-good

</pre>

Refer to [[HowTo Install OpenChange From Source]] (but use svn command above instead of trunk) for instructions on requirements and how to build openchange. 

This branch also relies on the sogo backend developed by inverse.ca which provides the most up to date backend's implementation using mapistore v1. 

For instructions on how to download and build the backend, refers to [[HowTo Setup SOGo with OpenChange Server]].

h3. Why isn't mapistore v1 available in trunk?

Back in February 2010, we have decided to move from our existing mapistore v1 interface to a brand new and completely redesigned mapistore v2 interface. This new interface was introducing new concepts, fixing limitation of v1 implementation. This was a massive work and while it started very well, the effort was too big for the small number of people working on it and became abandoned, while v1 implementation kept progressing.

It means that mapistore v2 in trunk is not usable in current state and will progressively be replaced step by step with an updated version of mapistore v1 covered in this documentation, introducing new concepts progressively.

h2. MAPIStore architecture overview

h3. URI and scope

A URI is the key (and only) element mapistore and calling applications rely on in order to access and perform operation on a specific backend.

It is made of a prefixing namespace followed by data specific to the backend:

* *namespace*: This is the unique identifier associated to your backend. It is made of your backend's name followed by *://*

* *backend's specific data*: This is a set of data that doesn't have any meaning for the calling application but is relevant for the backend. While the format of this data is completely specific to your backend, backends generally set information such as username, password or folder name within the destination system the backend manages.

<pre>

{namespace}{backend's specific data}

sogo://user:password@folder/

java://user##folder/

</pre>

h3. Folder and Message identifiers uniqueness

In Exchange world, folder and message identifiers - respectively called FID and MID - are unsigned 64-bit integers that represents a physical object (folder, message) on the server. 

This is the only link clients and OpenChange server are using to refer to an object. Given that identifiers are stored on 64-bit integers, there is a virtual allocation pool of 2^16 elements (folder, messages) to cover all users needs registered on the server for its lifetime.

*Folder and Messages IDs are unique to the entire system*.

Furthermore all these ids are only used once and can't be allocated again for another (new) object. This is a requirement to get offline synchronization (FXICS) working properly and prevent form collision with paste elements.

h3. Contexts

An Exchange user mailbox is (roughly) represented as a top folder (The Mailbox name) holding root (sub) folders (Inbox, Outbox, Calendar, Contacts, Notes, Tasks etc.). In MAPIStore, we consider that we shouldn't have to use the same backend for everything. We should be able to use a backend for Inbox, another for Calendar and so on.

This is where the notion of mapistore context steps in. A context is a sandbox created by a backend when you access root folder. For every operation you perform on this root folder or any data within this folder, the same created context will be used. For example, if we want to open the message _Rates_ in /Inbox/Holidays/, we will:

# Create a context on Inbox (it opens the folder)

# Open the Holidays folder using the Inbox context

# Access the message using the Inbox context

If the backend attached to my Inbox points to an IMAP server, all the operation performed on Inbox and subfolders will be propagated to this IMAP server. In this case, creating a context would mean:

# Open a connection on the IMAP server

# Open the Inbox folder

# Keep the connection opened and save parameters this IMAP backend needs to access/interact with physical (folder, message) or virtual (tables) elements stored within this Inbox folder.

h3. openchange.ldb URI wrapper

We have just seen that a context has to be created when we access a root folder and the only key we need to call the proper backend and create the context is the URI.

It means we need a mapping to associate the Inbox folder of a user to a specific URI (e.g.: imap://user:pass@Inbox/). This is done through the openchange.ldb database which maintains the mapping between Exchange mailbox root folders and URI used to access them.

This file does not (yet) belongs to mapistore, but is widely used by OpenChange Server when serving users mailboxes. It is created within _/usr/local/samba/private/openchange.ldb_ when OpenChange server is provisioned and can be read/edited using ldbedit installed in _/usr/local/samba/bin/ldbedit_.

<pre>

/usr/local/samba/bin/ldbedit -H /usr/local/samba/private/openchange.ldb

</pre> 

h3. indexing.tdb URI/FMID wrapper

What we call FMID is a shortcut combination of FID (Folder Identifier) and MID (Message Identifier). It means either Folder or Message identifier.

While _openchange.ldb_ handles the mapping between Mailbox root folders and mapistore URI, *indexing.tdb handles the mapping for all the other elements (folder, message) of a user that are NOT root folders*.

Each time a user creates a folder within a root folder (Inbox, Calendar etc.) or create a message (Appointment, Drafts, Task etc.), the FID or MID is associated to a mapistore URI within the indexing.tdb database of the user. It also means that indexing.tdb is user specific and each user of the system has its own indexing.tdb. This database is similar to a hash table: it stores the FID/MID as key and the mapistore URI as value.

To make things clear:

* openchange.ldb indexes and map root folders of user mailboxes and is common to all users

* indexing.tdb indexes everything else but one database is created for each user

h2. MAPIStore Backend

A backend has to be developed as a dynamic shared library and installed in the folder mapistore uses to look up and load backends:

<pre>

/usr/local/samba/lib/mapistore_backends/

</pre>

For mapistore to be able to load the backend, it is required to implement an entry point called *mapistore_init_backend*.

This function fills a mapistore backend structure and registers it. This makes backends available for the initialized mapistore instance.

All the required parameters are grouped into sub structures, but the key one to proper backend registration is the backend one. 

A very preliminary backend would looks like:

<pre>

#include <stdbool.h>

#include <talloc.h>

#include <gen_ndr/exchange.h>

#include <mapistore/mapistore.h>

#include <mapistore/mapistore_errors.h>

static int example_backend_init(void)

{

     return MAPISTORE_SUCCESS;

}

static int example_backend_create_context(TALLOC_CTX *mem_ctx,

                                          struct mapistore_connection_info *conn_info,

                                          struct tdb_wrap *indexing_tdb,

                                          const char *uri,

                                          void **context_object)

{

    return MAPISTORE_SUCCESS;

}

int mapistore_init_backend(void)

{

     struct mapistore_backend backend;

     int                      ret;

     static bool              registered = false;

     if (registered == true) return MAPISTORE_SUCCESS;

     backend.backend.name = "Example";

     backend.backend.description = "An example backend";

     backend.backend.namespace = "example://";

     backend.backend.init = example_backend_init;

     backend.backend.create_context = example_backend_create_context;

     [...]

     ret = mapistore_backend_register(&backend);

     if (ret != MAPISTORE_SUCCESS) {

        DEBUG(0, ("Failed to register the '%s' mapistore backend!\n", backend.backend.name));

     }

     registered = true;

     return ret;

}

</pre>

h3. void * and void ** parameters in backend calls

These are parameters passed along mapistore that are initially instantiated and manager by backends. 

These structures are specific to the backend and OpenChange server never makes any use of this data nor does it care about its content. It just passes it along further mapistore calls whenever necessary.

From a general pespective:

* _void *_ is an existing object the backend already instantiated and which mapistore pass along to the backend. The nature of the object depends on the mapistore call made.

* _void **_ is a new object returned by the backend and which will be used in further mapistore call whenever it relates to the correct subset of operations.

MAPIStore v1 provides an object-oriented implementation and offers backend the ability to pass objects along. Objects depends on each other. For example:

* add_context creates a context object

* root_folder returns the folder object of a context object (just morphing)

* open_folder takes the parent folder object returns the request folder object

* open_message: takes a folder object and returns a message object

Backends are sole responsible for these handling these objects. They can create and store any data structure to keep and save information among call and find data they need. They can also store pointers on function associated to the object to make direct calls within the object context rather than having to maintain an internal list.

While the content of the object is specific to the backend, it is however require to create and use them since most mapistore calls don't provide any context but these specific backend void pointers.

h3. [[MAPIStore 1.0 backend.backend structure|backend.backend structure]]

h3. [[MAPIStore 1.0 backend.context structure|backend.context structure]]

h3. [[MAPIStore 1.0 backend.folder structure|backend.folder structure]]

h3. [[MAPIStore 1.0 backend.manager structure|backend.manager structure]]