OpenChange

h1. MAPIStore 1.0 Development Guide

{{>toc}}

h2. What is MAPIStore?

MAPIStore is the SAL component of OpenChange server. SAL stands for Storage Abstraction Layer. It is the component used by OpenChange Server to push/get information (messages, folders) to/from storage backends. It is designed as a library called within OpenChange Server and it accesses backends compiled as dynamic shared object (DSO) and loaded when mapistore is initialized.

The main objective of mapistore is to provide an interface layer with a common set of atomic functions (operations) used to trigger and dispatch data and commands to the appropriate backend. MAPIStore relies on a backend mechanism specifically designed to transparently handle some of the MAPI semantics required by any Exchange compatible server.

The initial idea was to provide to OpenChange a highly customizable storage backend mechanism which would fit in any situation and any environments. One of the greatest limitation we have found with existing groupware is the storage layer which is generally limited to a single solution, service or format and is neither scalable nor modifiable when user requirements evolve upon time.

MAPIStore solves this problem and go beyond classical limitations. It is not a revolutionary concept, but the way openchange uses it makes the whole difference and offer administrators an innovative way to customize storage.

MAPIStore allows you to:

* use a different backend for any top-folder

* transparently move/copy data across backends

* develop new backends quickly

* access all the different backends through an unique API

For example (assuming all associated backends were developed) a user could have the following storage organization for his mailbox:

* Mails stored using an IMAP backend (Cyrus-IMAP or dovecot)

* Calendar items stored in CalDAV or pushed in Google calendar

* Sent emails and archives/backup stored in a compression backend

* Tasks stored in a MySQL database

* Notes stored on the filesystem

If the user is not satisfied with one of the backend's performance, they would just have to use an administration tool, change the backend, wait for the replication, synchronization to finish and there data will be available from the new backend.

Information can be completely decentralized, stored on one of several servers and still be accessible transparently from OpenChange server.

h2. Getting started

h3. Getting Source code

Using mapistore v1 requires to use a different development tree than trunk. You will need to checkout the code from our current mapistore v1 development branches:

<pre>

svn co http://svnmirror.openchange.org/openchange/branches/sogo-good

</pre>

Refer to [[HowTo Install OpenChange From Source]] (but use svn command above instead of trunk) for instructions on requirements and how to build openchange. 

This branch also relies on the sogo backend developed by inverse.ca which provides the most up to date backend's implementation using mapistore v1. 

For instructions on how to download and build the backend, refers to [[HowTo Setup SOGo with OpenChange Server]].

h3. Why isn't mapistore v1 available in trunk?

Back in February 2010, we have decided to move from our existing mapistore v1 interface to a brand new and completely redesigned mapistore v2 interface. This new interface was introducing new concepts, fixing limitation of v1 implementation. This was a massive work and while it started very well, the effort was too big for the small number of people working on it and became abandoned, while v1 implementation kept progressing.

It means that mapistore v2 in trunk is not usable in current state and will progressively be replaced step by step with an updated version of mapistore v1 covered in this documentation, introducing new concepts progressively.

h2. MAPIStore architecture overview

h3. URI and scope

A URI is the key (and only) element mapistore and calling applications rely on in order to access and perform operation on a specific backend.

It is made of a prefixing namespace followed by data specific to the backend:

* *namespace*: This is the unique identifier associated to your backend. It is made of your backend's name followed by *://*

* *backend's specific data*: This is a set of data that doesn't have any meaning for the calling application but is relevant for the backend. While the format of this data is completely specific to your backend, backends generally set information such as username, password or folder name within the destination system the backend manages.

<pre>

{namespace}{backend's specific data}

sogo://user:password@folder/

java://user##folder/

</pre>

h3. Contexts

An Exchange user mailbox is (roughly) represented as a top folder (The Mailbox name) holding root (sub) folders (Inbox, Outbox, Calendar, Contacts, Notes, Tasks etc.). In MAPIStore, we consider that we shouldn't have to use the same backend for everything. We should be able to use a backend for Inbox, another for Calendar and so on.

This is where the notion of mapistore context steps in. A context is a sandbox created by a backend when you access root folder. For every operation you perform on this root folder or any data within this folder, the same created context will be used. For example, if we want to open the message _Rates_ in /Inbox/Holidays/, we will:

# Create a context on Inbox (it opens the folder)

# Open the Holidays folder using the Inbox context

# Access the message using the Inbox context

If the backend attached to my Inbox points to an IMAP server, all the operation performed on Inbox and subfolders will be propagated to this IMAP server. In this case, creating a context would mean:

# Open a connection on the IMAP server

# Open the Inbox folder

# Keep the connection opened and save parameters this IMAP backend needs to access/interact with physical (folder, message) or virtual (tables) elements stored within this Inbox folder.

h3. openchange.ldb URI wrapper

We have just seen that a context has to be created when we access a root folder and the only key we need to call the proper backend and create the context is the URI.

It means we need a mapping to associate the Inbox folder of a user to a specific URI (e.g.: imap://user:pass@Inbox/). This is done through the openchange.ldb database which maintains the mapping between Exchange mailbox root folders and URI used to access them.

This file does not (yet) belongs to mapistore, but is widely used by OpenChange Server when serving users mailboxes. It is created within _/usr/local/samba/private/openchange.ldb_ when OpenChange server is provisioned and can be read/edited using ldbedit installed in _/usr/local/samba/bin/ldbedit_.

<pre>

/usr/local/samba/bin/ldbedit -H /usr/local/samba/private/openchange.ldb

</pre> 

h2. MAPIStore Backend

A backend has to be developed as a dynamic shared library and installed in the folder mapistore uses to look up and load backends:

<pre>

/usr/local/samba/lib/mapistore_backends/

</pre>

For mapistore to be able to load the backend, it is required to implement an entry point called *mapistore_init_backend*.

This function fills a mapistore backend structure and registers it. This makes backends available for the initialized mapistore instance.

All the required parameters are grouped into sub structures, but the key one to proper backend registration is the backend one. 

A very preliminary backend would looks like:

<pre>

#include <stdbool.h>

#include <talloc.h>

#include <gen_ndr/exchange.h>

#include <mapistore/mapistore.h>

#include <mapistore/mapistore_errors.h>

static int example_backend_init(void)

{

     return MAPISTORE_SUCCESS;

}

static int example_backend_create_context(TALLOC_CTX *mem_ctx,

                                          struct mapistore_connection_info *conn_info,

                                          struct tdb_wrap *indexing_tdb,

                                          const char *uri,

                                          void **context_object)

{

    return MAPISTORE_SUCCESS;

}

int mapistore_init_backend(void)

{

     struct mapistore_backend backend;

     int                      ret;

     static bool              registered = false;

     if (registered == true) return MAPISTORE_SUCCESS;

     backend.backend.name = "Example";

     backend.backend.description = "An example backend";

     backend.backend.namespace = "example://";

     backend.backend.init = example_backend_init;

     backend.backend.create_context = example_backend_create_context;

     [...]

     ret = mapistore_backend_register(&backend);

     if (ret != MAPISTORE_SUCCESS) {

        DEBUG(0, ("Failed to register the '%s' mapistore backend!\n", backend.backend.name));

     }

     registered = true;

     return ret;

}

</pre>

h3. backend structure

This structure is required to proper backend registration. It holds the following parameters:

* *backend.name*: the name of the backend

* *backend.description*: A short description for the backend

* *backend.namespace*: the namespace used to reference this backend

* *backend.init*: a pointer on the function to call for initialization

* *backend.create_context*: a pointer on the function to call to create a mapistore context

h4. Deleting context

You will notice there is no function to delete a context explicitly within the backend. This is because *the mapistore interface deletes a backend on its own* by free'ing the memory context used by this backend.

However, backend's developers are required to *allocate their internal data with the memory context passed during create_context and associate a talloc destructor function*. If additional operations need to be performed such as close a file descriptor, socket or call the language specific memory release system (garbage collector, pool etc.) 

When mapistore will delete the context (free up memory), the hierarchy will automatically be processed and the destructor function for the backend will be called. This is in this internal function that you need to implement actions such as close (fd, socket) etc.

To create a talloc destructor, call the following function:

<pre>

talloc_set_destructor((void *)context, (int (*)(void *))destructor_fct);

</pre>

In this example context is the structure allocated with the memory context passed in parameter of the create_context function. An example of the destructor function would looks like:

<pre>

static int desctructor_fct(void *data)

{

    /* perform operations to clean-up everything properly */

   return 0;

}

</pre>

h4. backend.init function

This is the function called right after the backend is registered (mapistore_backend_register). This function will be executed each time mapistore_init function is called from the calling application. 

In the context of OpenChange server and Samba forked model, it means the backend would be called each time a new instance of the server is created. 

To keep init only called once in the server's lifetime, it is required to use a static variable within mapistore_init_backend which value changes when backend gets initialized the first time.

The init function is used to initialize your backend with data (or environment) it will need all along mapistore's lifetime. It is a one-time operation and is generally used to:

* open a connection to a remote system

* load a virtual machine

It results in having a static structure, local to the backend's file which is used along calls to access an environment or file descriptor.

It prevents from having to open or load a virtual machine or open a connection to a system each time a context is created.

h4. backend.create_context function