OpenChange

h1. MAPIStore 1.0 Development Guide

{{>toc}}

h2. What is MAPIStore?

MAPIStore is the SAL component of OpenChange server. SAL stands for Storage Abstraction Layer. It is the component used by OpenChange Server to push/get information (messages, folders) to/from storage backends. It is designed as a library called within OpenChange Server and it accesses backends compiled as dynamic shared object (DSO) and loaded when mapistore is initialized.

The main objective of mapistore is to provide an interface layer with a common set of atomic functions (operations) used to trigger and dispatch data and commands to the appropriate backend. MAPIStore relies on a backend mechanism specifically designed to transparently handle some of the MAPI semantics required by any Exchange compatible server.

The initial idea was to provide to OpenChange a highly customizable storage backend mechanism which would fit in any situation and any environments. One of the greatest limitation we have found with existing groupware is the storage layer which is generally limited to a single solution, service or format and is neither scalable nor modifiable when user requirements evolve upon time.

MAPIStore solves this problem and go beyond classical limitations. It is not a revolutionary concept, but the way openchange uses it makes the whole difference and offer administrators an innovative way to customize storage.

MAPIStore allows you to:

* use a different backend for any top-folder

* transparently move/copy data across backends

* develop new backends quickly

* access all the different backends through an unique API

For example (assuming all associated backends were developed) a user could have the following storage organization for his mailbox:

* Mails stored using an IMAP backend (Cyrus-IMAP or dovecot)

* Calendar items stored in CalDAV or pushed in Google calendar

* Sent emails and archives/backup stored in a compression backend

* Tasks stored in a MySQL database

* Notes stored on the filesystem

If the user is not satisfied with one of the backend's performance, they would just have to use an administration tool, change the backend, wait for the replication, synchronization to finish and there data will be available from the new backend.

Information can be completely decentralized, stored on one of several servers and still be accessible transparently from OpenChange server.

h2. Getting started

h3. Getting Source code

Using mapistore v1 requires to use a different development tree than trunk. You will need to checkout the code from our current mapistore v1 development branches:

<pre>

svn co http://svnmirror.openchange.org/openchange/branches/sogo-good

</pre>

Refer to [[HowTo Install OpenChange From Source]] (but use svn command above instead of trunk) for instructions on requirements and how to build openchange. 

This branch also relies on the sogo backend developed by inverse.ca which provides the most up to date backend's implementation using mapistore v1. 

For instructions on how to download and build the backend, refers to [[HowTo Setup SOGo with OpenChange Server]].

h3. Why isn't mapistore v1 available in trunk?

Back in February 2010, we have decided to move from our existing mapistore v1 interface to a brand new and completely redesigned mapistore v2 interface. This new interface was introducing new concepts, fixing limitation of v1 implementation. This was a massive work and while it started very well, the effort was too big for the small number of people working on it and became abandoned, while v1 implementation kept progressing.

It means that mapistore v2 in trunk is not usable in current state and will progressively be replaced step by step with an updated version of mapistore v1 covered in this documentation, introducing new concepts progressively.

h2. MAPIStore architecture overview

h3. URI and scope

A URI is the key (and only) element mapistore and calling applications rely on in order to access and perform operation on a specific backend.

It is made of a prefixing namespace followed by data specific to the backend:

* *namespace*: This is the unique identifier associated to your backend. It is made of your backend's name followed by *://*

* *backend's specific data*: This is a set of data that doesn't have any meaning for the calling application but is relevant for the backend. While the format of this data is completely specific to your backend, backends generally set information such as username, password or folder name within the destination system the backend manages.

<pre>

{namespace}{backend's specific data}

sogo://user:password@folder/

java://user##folder/

</pre>

h3. Contexts

An Exchange user mailbox is (roughly) represented as a top folder (The Mailbox name) holding root (sub) folders (Inbox, Outbox, Calendar, Contacts, Notes, Tasks etc.). In MAPIStore, we consider that we shouldn't have to use the same backend for everything. We should be able to use a backend for Inbox, another for Calendar and so on.

This is where the notion of mapistore context steps in. A context is a sandbox created by a backend when you access root folder. For every operation you perform on this root folder or any data within this folder, the same created context will be used. For example, if we want to open the message _Rates_ in /Inbox/Holidays/, we will:

# Create a context on Inbox (it opens the folder)

# Open the Holidays folder using the Inbox context

# Access the message using the Inbox context

If the backend attached to my Inbox points to an IMAP server, all the operation performed on Inbox and subfolders will be propagated to this IMAP server. In this case, creating a context would mean:

# Open a connection on the IMAP server

# Open the Inbox folder

# Keep the connection opened and save parameters this IMAP backend needs to access/interact with physical (folder, message) or virtual (tables) elements stored within this Inbox folder.

h3. openchange.ldb URI wrapper

We have just seen that a context has to be created when we access a root folder and the only key we need to call the proper backend and create the context is the URI.

It means we need a mapping to associate the Inbox folder of a user to a specific URI (e.g.: imap://user:pass@Inbox/). This is done through the openchange.ldb database which maintains the mapping between Exchange mailbox root folders and URI used to access them.

This file does not (yet) belongs to mapistore, but is widely used by OpenChange Server when serving users mailboxes. It is created within _/usr/local/samba/private/openchange.ldb_ when OpenChange server is provisioned and can be read/edited using ldbedit installed in _/usr/local/samba/bin/ldbedit_.

<pre>

/usr/local/samba/bin/ldbedit -H /usr/local/samba/private/openchange.ldb

</pre> 

h3. indexing.tdb URI/FMID wrapper

What we call FMID is a shortcut combination of FID (Folder Identifier) and MID (Message Identifier). It means either Folder or Message identifier.

While _openchange.ldb_ handles the mapping between Mailbox root folders and mapistore URI, *indexing.tdb handles the mapping for all the other elements (folder, message) of a user that are NOT root folders*.

Each time a user creates a folder within a root folder (Inbox, Calendar etc.) or create a message (Appointment, Drafts, Task etc.), the FID or MID is associated to a mapistore URI within the indexing.tdb database of the user. It also means that indexing.tdb is user specific and each user of the system has its own indexing.tdb. This database is similar to a hash table: it stores the FID/MID as key and the mapistore URI as value.

To make things clear:

* openchange.ldb indexes and map root folders of user mailboxes and is common to all users

* indexing.tdb indexes everything else but one database is created for each user

h2. MAPIStore Backend

A backend has to be developed as a dynamic shared library and installed in the folder mapistore uses to look up and load backends:

<pre>

/usr/local/samba/lib/mapistore_backends/

</pre>

For mapistore to be able to load the backend, it is required to implement an entry point called *mapistore_init_backend*.

This function fills a mapistore backend structure and registers it. This makes backends available for the initialized mapistore instance.

All the required parameters are grouped into sub structures, but the key one to proper backend registration is the backend one. 

A very preliminary backend would looks like:

<pre>

#include <stdbool.h>

#include <talloc.h>

#include <gen_ndr/exchange.h>

#include <mapistore/mapistore.h>

#include <mapistore/mapistore_errors.h>

static int example_backend_init(void)

{

     return MAPISTORE_SUCCESS;

}

static int example_backend_create_context(TALLOC_CTX *mem_ctx,

                                          struct mapistore_connection_info *conn_info,

                                          struct tdb_wrap *indexing_tdb,

                                          const char *uri,

                                          void **context_object)

{

    return MAPISTORE_SUCCESS;

}

int mapistore_init_backend(void)

{

     struct mapistore_backend backend;

     int                      ret;

     static bool              registered = false;

     if (registered == true) return MAPISTORE_SUCCESS;

     backend.backend.name = "Example";

     backend.backend.description = "An example backend";

     backend.backend.namespace = "example://";

     backend.backend.init = example_backend_init;

     backend.backend.create_context = example_backend_create_context;

     [...]

     ret = mapistore_backend_register(&backend);

     if (ret != MAPISTORE_SUCCESS) {

        DEBUG(0, ("Failed to register the '%s' mapistore backend!\n", backend.backend.name));

     }

     registered = true;

     return ret;

}

</pre>

h3. backend.backend structure

This structure is required to proper backend registration. It holds the following parameters:

* *backend.name*: the name of the backend

* *backend.description*: A short description for the backend

* *backend.namespace*: the namespace used to reference this backend

* *backend.init*: a pointer on the function to call for initialization

* *backend.create_context*: a pointer on the function to call to create a mapistore context

h4. Deleting context

You will notice there is no function to delete a context explicitly within the backend. This is because *the mapistore interface deletes a backend on its own* by free'ing the memory context used by this backend.

However, backend's developers are required to *allocate their internal data with the memory context passed during create_context and associate a talloc destructor function*. If additional operations need to be performed such as close a file descriptor, socket or call the language specific memory release system (garbage collector, pool etc.) 

When mapistore will delete the context (free up memory), the hierarchy will automatically be processed and the destructor function for the backend will be called. This is in this internal function that you need to implement actions such as close (fd, socket) etc.

To create a talloc destructor, call the following function:

<pre>

talloc_set_destructor((void *)context, (int (*)(void *))destructor_fct);

</pre>

In this example context is the structure allocated with the memory context passed in parameter of the create_context function. An example of the destructor function would looks like:

<pre>

static int desctructor_fct(void *data)

{

    /* perform operations to clean-up everything properly */

   return 0;

}

</pre>

h4. backend.init function

This is the function called right after the backend is registered (mapistore_backend_register). This function will be executed each time mapistore_init function is called from the calling application. 

In the context of OpenChange server and Samba forked model, it means the backend would be called each time a new instance of the server is created. 

To keep init only called once in the server's lifetime, it is required to use a static variable within mapistore_init_backend which value changes when backend gets initialized the first time.

The init function is used to initialize your backend with data (or environment) it will need all along mapistore's lifetime. It is a one-time operation and is generally used to:

* open a connection to a remote system

* load a virtual machine

It results in having a static structure, local to the backend's file which is used along calls to access an environment or file descriptor.

It prevents from having to open or load a virtual machine or open a connection to a system each time a context is created.

h4. backend.create_context function

This is the function called when we are creating a context (see Contexts subsection). Its prototype uses the following parameters:

* *TALLOC_CTX ** Pointer to the memory context used for this backend. This is the memory context you need to use to allocate memory for your specific backend's context structure and for which you may want to set a destructor.

* *struct mapistore_connection_info ** Pointer to a data structure holding information about the incoming user and wrapping structures such as the mapistore_context structure pointer. It is needed for a backend for example to register messages on its own.

* *struct tdb_wrap ** Pointer to a wrapped TDB database (sort of hash table) which associates for all folder/messages an Exchange specific folder and message identifier to a mapistore URI.

* *const char *uri* Pointer to the URI for this context

* *void **context_object* Pointer to a context object to return and which the backend uses to associate backend's specific information on the context.

h3. backend.context structure

This structure holds couple of pointer on functions which perform operations specific to mapistore contexts:

* backend.context.get_path

* backend.context.get_root_folder

h4. context.get_path

*In MAPIStore, backends can't register FMID to mapistore URI themselves. This is the role of mapistore middleware.*

To understand this choice, an overview of message creation provides a good example. When a MAPI client creates a message, it doesn't do it through a single operation. It may even not do it through a single network transaction. This is different from an IMAP message where the email is composed and pushed at once. In the case of MAPI, a client will successively call (for a draft email): CreateMessage, SetProps, ModifyRecipients, SaveChangesMessage. If a client releases the message before it reaches the SaveChangesMessage, the message is destroyed and won't be anymore available on the system. However, during the message creation process, the message virtually exists and has a MID associated. It is just that nobody except the client creating the message can access it.

From an openchange perspective, it means that the MID exists but is not yet stored within the indexing.tdb database of the user and is not permanently associated to a mapistore URI:

* When the createmessage's backend function is called, it receives a MID.

* It generates a mapistore URI for this MID

* It stores internally this MID to mapistore URI mapping

* It goes this way through the entire message creation

When the message is finally save, it is time to expose the message publicly and reference it. This is where the context.get_path functions matters:

* The mapistore middleware will register the MID (or FID) within the indexing.tdb of the user

* However it doesn't know the mapistore URI which was associated

* It queries the backend using context.get_path to retrieve the URI associated to the FMID

* It registers the message within the indexing.tdb database

You have also figured out that such behavior doesn't only apply to messages but also to folders. When a CreateFolder is called, the FID is passed to CreateFolder, the backend generates a URI and internally associates it to this FID. When mapistore middleware registers the folder, it queries context.get_path with the FID and retrieve the URI.

This is an indirect way to retrieve data, but it is also useful when a call returns an error etc.

h4. context.get_root_folder

mapistore v1 is using object concepts. For example you may have noticed that we have divided backend functions from context functions. Forthcoming documentation also describe folder, message, table or property functions. Each of these subsets of functions represent an objet. So we have:

* a backend object

* a context object

* a fodler object

* a message object

* etc.

When we create a context, we are in fact opening a folder, except that this folder is a root folder. For this very special case, we are only creating a context but the folder is however opened. It means that a context is also a folder. The get_root_folder functions returns a folder representation of the context object created. It lets backends directly call folder operations on contexts rather than having to open (again) the folder to call its operations.