Openchange Tools done

I a writting this piece to mark the last efforts in the development of openchangetools extensions for gsoc 2009 which I conluded a week ago. The is now over and further efforts may include gui for other tools in next years gsoc. But let all wait till then. The code for the project is available from the openchange repo and will also be available from google soon, after code sample submissions.

Mapiprofile Tool done

Hello all,

I am very delighted to say today that I have finished making the mapiprofile tool. This is a tool that directly mimics the commandline options provided by mapiprofile. It has been a long journey, sometimes rough, but we are here now and its done. The next tool which I am already working on is the mapiadmin tool which has an few more functionalties compared to the mapiprofie tool that I just finished. This is indeed a great experience especually the pressure mounts towards the evaluation deadlines, I got to work continusly for the longest time in my life. But the ultimate feeling after cracking it is really good. I will keep you posted on the developments of the mapiadmin tool. Now I show some shots of the new tool just finished. This tool is integrated into the desktop using a deskbar applet written again in Qt.This is the welcome screen from which the user selects to either create a new database stored as dbname.ldb in $HOME/.openchange/ or select one that is already existing.
This is the create new database option, where the user gets to enter a name of the database to create. This is what the user gets in case he/she selects the select an existing database option, one can then navigate to select the database of choice in the folder of choice. This is the result after selecting a database, where a user gets to view all profiles in that database and gets the options to manipulate these profiles accordingly. In case of a new database, with no profile, only the add profile option is available.

This is about the main features of the tool.

Major functionalities identified

Following the start of the project work. I have identified major functionalities of the gui tools that I am about to build. The following are the major functionalities identified;

mapiprofile gui:

• Should instantiate a mapi profile by creating a new database
• Instantiate users for a given profile mentioned (these include details such as username, password, workstation, domain...) which are used to create a user
• List all users, profiles with associated elements
• Perform profile dump
• Manage default profile (be able to set a certain profile to be default)
• Deleting profiles

openchangepfadmin:

• Public folder management (creation, removal and listing)
• User management (addition and removal of users)
• Permissions and Delegation (addition, removal and modification of user permissions)

The third gui for openchangeproxy will be posted lator

About Openchange Project

Administrating Public Folders with OpenChange
OpenChange MAPI Library

The OpenChange project provides a portable Open Source
implementation of the protocols used by Microsoft Exchange. The
client-side libmapi and libmapiadmin libraries are an
early result, and usable today.


Microsoft Exchange is probably the most popular groupware system
in the world, and many people never use any other tool of its
type. Exchange-centric work habits are central to the way millions of
people perform their daily work. However Microsoft Exchange is
currently fully utilised only in conjunction with Microsoft groupware
clients running on Microsoft operating systems.  End users, system
administrators and developers need alternative tools so they can
continue to use Microsoft Exchange working methods, but improve their
efficiency and safety, and reduce their cost. This article offers an
overview of the OpenChange MAPI library features and its related tools
needed to administrate and use Microsoft Exchange Server and its
Public Folders.

Why Use Microsoft Exchange Protocols?

The main reason today for a client to use Exchange protocols is to
communicate with Microsoft Exchange as if it were Outlook. Microsoft
have even recently started to licence limited access to these
protocols so that developers can emulate Outlook functions. The
fundamental problem is interoperability, normally best addressed by
Internet standards. But Exchange may be a special case. There is a
huge installed based which has been trained to expect a certain kind
of functionality, and there is no single Internet standard that can
deliver that functionality -- many are needed, largely unrelated to
each other.

The OpenChange project believes the fastest way to a
global groupware standard is to first provide an open implementation
of Exchange, and move on from there. The OpenChange project is
encouraging mail, calendaring and other groupware client developers to
consider integrating libmapi and has produced functional commandline
and GUI reference examples. We are developing OpenChange Server, a
drop-in replacement to Exchange. We are planning a gateway strategy to
integrate with Internet and other standards. All this improves the
number and quality of options available to people with Microsoft
Exchange-centric work habits.


What is MAPI?

MAPI originally referred to a proprietary set of function call
interfaces developed by Microsoft before Microsoft Exchange
existed. When Microsoft Exchange 5.5 was developed in 1997, the
decision was taken to create a proprietary transport protocol for MAPI
which closely matches the MAPI calling interface. This protocol is
often called ExchangeRPC and is used in Outlook-Exchange
communications.

The term "MAPI application" on Windows means an application that
is using the MAPI API to call Microsoft's MAPI provider, usually
called MAPI.DLL. These applications know nothing about how MAPI
works. In exactly the same way, libmapi isolates applications
from knowledge about the underlying Microsoft networking. While the
OpenChange project has written a number of clients that use
libmapi, the intention is to enable groupware application
authors to use libmapi to let them deliver Exchange-style
groupware functions efficiently.

Why Not Buy MAPI from Microsoft?

OpenChange could have partially implemented libmapi by
buying access to some of the specifications from Microsoft. Due to EU
requests Microsoft now license Exchange protocols [1]. The license is
of course incompatible with Open Source Software, and in addition it
is not possible to buy access at any price to a full top-to-bottom
protocol stack. Neither is Microsoft's work supported on non-Microsoft
operating systems. These are yet more good reasons why libmapi
needs to exist.

Exchange Protocols (Briefly)

When OpenChange team members first looked at the network network
traffic these generated by calling MAPI functions on Windows operating
systems, we noticed blobs of data first either compressed or
obfuscated, then encapsulated by an MSRPC transport protocol function
and finally pushed on the wire. As it happened, a large amount of the
decoding work had already been done by the Samba project.

The Samba project has a great deal of experience handling
Interface Definition Language (IDL) descriptions of RPC traffic, and
encapsulation of the Network Data Representation (NDR) format used
with RPC, and debugging these.  In 2007, Samba is the result of
fifteen years of continuous implementation of Microsoft networking
protocols, and hundreds of man-years of effort.

libmapi Design

The libmapi design goals were developed after carefully studying
the public Microsoft MAPI programming documentation and considering
Exchange protocols on the wire. The goals are:

* To provide a set of functions with similar semantics to the
 Microsoft C++ API, maximising the benefit of shared information
 across the two implementations. libmapi needs to be easily usable by
 Linux developers and Windows developers alike, including Windows
 developers who already know MAPI.

* To improve on the state of the art in MAPI implementations,
 maximising the efficiency of programmers who use libmapi. Wherever
 possible libmapi hides implementation details from the user, so that
 all they have to know is that they are passing in a
 groupware-related object and getting back a response in order to
 have a groupware conversation.

The similar API function names help in comparing programming
resources, from the Microsoft MSDN website, the online OpenChange
resources and in discussion groups. They also help administrators
track down errors.

The implementation involved some difficult decisions. Exchange is
implemented by instantiating MAPI objects on the server and then
operating on these objects by means of unique handles.  libmapi has
retained this design, but the implementation is significantly
different in two ways:

* libmapi has a global context, rather than separately instantiated
 objects with private members.  This gives less work for libmapi
 users and reduced risk of making common programming errors. It also
 allows libmapi to have substantial internal reworking without
 impacting public function prototypes.

* libmapi hides internal mechanisms. MAPI objects are opaque, and
 passed by reference between MAPI functions. Although the API is very
 similar to Microsoft MAPI, the Microsoft MAPI C++ object hierarchy
 is not implemented.

This architecture has served OpenChange well so far. The
development team concentrates on implementing core MAPI functionality
rather than building an object hierarchy that may later turn out to be
suboptimal.





Installation

The OpenChange project is quite young and has not yet produced any
binary distributions. To compile from source you'll need a reasonably
modern Linux distribution. To test you will also need Microsoft
Exchange Server 2000, 2003 or 2007. You need to compile both libmapi
and the Samba4 suite whose libraries libmapi uses. We will install
Samba4 first, which will give you the libraries, headers and tools
required to build OpenChange.

Before attempting the Samba build, make sure you have installed the
necessary development tools including automake. The Samba wiki has
more details[3].

$ rsync -avz samba.org::ftp/unpacked/samba4 .
$ ./configure --prefix=/usr/local/samba
$ make
# make installlib && make installheader  # <- as root!

Since the libraries will be installed in /usr/local/samba/lib,
ldconfig needs to know about it. Make sure this directory is listed in
/etc/ld.so.conf and run ldconfig -v. Similarly, you need
/usr/local/samba/bin in your PATH for the pidl[4] and OpenChange
binaries. You will also need to set the path for the man pages.
If your shell is bash:

$ export PATH=$PATH:/usr/local/samba/bin
$ export MANPATH=$MANPATH:/usr/local/samba/share/man

You need to install pidl from the Samba4 source directory, as root:

# cd pidl && perl Makefile.PL && make && make install

Before attempting the OpenChange build, install libmagic using your
distribution's package mangement tools, for example with Debian/Ubuntu:

$ apt-get install libmagic libmagic-dev

This should be the only development library you need, although you will
also need the packages automake,flex and libpopt.
The OpenChange wiki has more details.[5] First you need to get the
OpenChange source code, then the usual configuration steps.

$ svn co https://svn.openchange.org/openchange
$ cd trunk
$ ./autogen.sh
$ ./configure --prefix=/usr/local/samba
$ make
# make install    #   <-- as root!

If all went well, you should be able to test some commands:

$ openchangeclient --help
$ mapiprofile --help
$ man openchangeclient

Make sure you have a working Exchange Administrator[6] account --
can you see your inbox from Outlook? You are now ready to use libmapi
for the first time.

Initialising libmapi

The job of the MAPI library is to use Microsoft Exchange protocols
to send and receive groupware information. As designed by Microsoft,
MAPI has no knowledge of how to locate an Exchange server, or store
information. This is the purpose of MAPI profiles: to store the
information MAPI needs about servers, users and networks. libmapi
implements functions to do this work, which have been exposed in the
commandline tool mapiprofile.

First, initialise the mapiprofile database:

$ mapiprofile --database=$HOME/.openchange/profiles.ldb   \
             --ldif=/usr/local/samba/share/setup \
             --newdb

This uses template information stored in the LDAP dump files to
create a new mapiprofile database. Any destination filename can be
given, however the one specified is the default. If you store your
mapiprofiles here, you do not need to supply the name to any libmapi
utilities. The location of the ldif files is also the default, set (it
is relative to the --prefix parameter specified at build
time. As you will see later, these defaults can save some very long
commandlines!

Now you need to initialise a new user. Note that the --database
parameter is not required here either, it will use the default
location.)

$ mapiprofile --profile=MAPIAdmin               \
             --username=Administrator          \
             --password=openchange             \
             --workstation=LOCALHOST           \
             --domain=OPENCHANGE               \
             -I 192.168.194.22                 \
             --create

This specifies all the information required to find and use a
Microsoft Exchange server. The parameters --workstation and
--domain refer to Microsoft networking names, as used by Samba
and Windows workstations. This command attempts to contact the
Exchange server and verify the credentials, so if there is a network
problem or you specify the wrong password, the profile creation will
fail.

libmapi asks the Exchange server for any usernames that match the
one you supplied, so in this example if the accounts
Administrator and Administrator2 both exist you
will be asked to select which one you mean. This is implemented in
libmapi as a callback, so if you are developing a GUI groupware client
libmapi will pass a list of usernames to your dialog function, and
then resume control when the user has made a selection.

mapiprofile has other actions, all of which should work now:

$ mapiprofile --list
$ mapiprofile --MAPIAdmin --dump
$ mapiprofile --MAPIAdmin --default

The last command sets the default profile, so when some other
libmapi application attempts to perform an operation without
specifying any details, this is the one that is used.

Using libmapi

We have already used one libmapi application,
mapiprofile.  openchangeclient is another example, a
commandline utility that offers all the functions of Outlook as
atoms. Email, folders, calendaring, notes and newmail notification,
with a single function callable per invocation.  For example:

$ openchangeclient --mailbox

will list the folder summary for the mailbox associated (at the
Exchange server) with the default profile (as set in the database
stored in the default database.) We recommend you experiment with the
openchangeclient options to get a feel for how complete the
support is.

Introducing libmapiadmin

libmapi implements MAPI functions as Microsoft intended. But
what about administration? A Windows administrator has a graphical
Microsoft Management Console plugin for adding and deleting users,
assigning rights to users and folders and so on. OpenChange provides
an alternative approach through libmapiadmin. This library makes
Active Directory calls to manage user properties, using knowledge from
OpenChange research to add Exchange semantics to normal Active
Directory objects. This ability to manipulate Exchange Objects is
exposed in the sample OpenChange program openchangepfadmin.

About Exchange Objects

Exchange objects are assigned to classes.  E-mails typically use
the message class IPM.Note [8], while posts to a public folder have
IPM.Post. Message classes are used by Outlook to determine how to
present a message to a user (eg a note is very different to an
appointment!) but also what logical operations should be permitted or
automatically applied (comparing with other appointments, expiring,
etc.)

Collection Classes, or Public Folders, are a kind of Exchange
object that enable many of the most-used groupware functions. Public
Folders have classnames that tell Exchange clients what kinds of
message classes it will find within them:
- IPF.Note for emails
- IPF.Appointment for calendaring
- IPF.Task for task
- IPF.Contact for contacts

You have seen collection classes already, listed in the response to
the openchangeclient --mailbox command.

Therefore, despite the very ordinary-sounding name, Exchange
Public folders are the primary technical mechanism by which Microsoft
Exchange stops non-MAPI groupware systems from interoperability. They
are also the prime mechanism for delivering the functionality that
millions of people rely on every day.

openchangepfadmin overview

We will now use openchangepfadmin to setup our test environment and
operate on Public Folders. To do this we need a Windows user account
with sufficient access rights. We will use the MAPIAdmin profile we
previously created with mapiprofile, which is associated with
the Administrator account. We made this the default profile in the
default profile database, so we will not need to specify these details
again in this article.

We need to create two users account in Active Directory:
linuxowner and linuxreader. openchangepfadmin will automate the whole
process through public functions from OpenChange libmapiadmin and its
adduser command:
* AD user account created through SAMR [7]
* User attributes extended in LDAP to match what Exchange needs
* Exchange mailbox created for the new user.

This account creation operation operation is performed through an
asynchronous call and can take up to 60 seconds. The Exchange account is
usable once openchangepfadmin --adduser returns.

The only mandatory parameter is the username.  If no user-defined
account password is specified a random generated password will be used
and its value displayed on standard output if the operation
succeeds. Default values are supplied for any parameters omitted
including description (--adesc), comment (--acomment) or
full name (--afullname).

You can verify that the operation succeeded in the Windows Active
Directory Users and Computers console (Figure 1).

$ openchangepfadmin --adduser=linuxowner --apassword=linuxowner \
                   --adesc="Linux Owner Test account"  \
      --afullname="Linux Owner"
   mapiadmin_user_add       : MAPI_E_SUCCESS (0x0)
username: linuxowner
password: linuxowner
$ openchangepfadmin --adduser=linuxreader --apassword=linuxreader \
                   --adesc="Linux Reader Test account"    \
      --afullname="Linux Reader"
   mapiadmin_user_add       : MAPI_E_SUCCESS (0x0)
username: linuxreader
password: linuxreader

Bi: screenshots/01_openchangepfadmin_accounts.png
B: Figure 1: User add operation results in Windows Active Directory
Users and Computers console

For the second step of this demonstration we will create a public
folder under All Public Folders. openchangepfadmin
doesn't yet allow Administrators to create subdirectories, but
you can specify the class of collection the folder will handle.
In this example, we will create a public and shared calendar named
public_events (Figure 2). If you ommit dirclass, the
folder collection class will be set to IPM.Post by default.

$ openchangepfadmin --mkdir --folder=public_events --dirclass=IPF.Appointment

Bi: screenshots/02_openchangepfadmin_newcalendar.png
B: Figure 2: Public calendar creation in Public Folder store


We will now set permissions for our two new users with different
access rights to thepublic_events public folder. The full list
of Exchange access rights is very long, so there are some predefined
roles that users can have. In this case we use the roles of Owner (can
do anything with a calendar entry) and Reviewer (can read but not
modify calendar entries.). We will also assign the role of None (no
rights at all) to Default and Anonymous preventing any
users but linuxowner and linuxreader from accessing the folder
content. These two particular names are not mapped to any AD accounts,
are language dependent (Default is generally represented with an
empty string while Anonymous depends on the Windows language),
and their permissions can only be modified. Any attempt to delete or
add permissions for these particular names will cause
openchangepfadmin to return an error.

$ openchangepfadmin --username=linuxowner --folder=public_events \
                   --addright=RoleOwner
Permission RoleOwner added for linuxowner on folder public_events

$ openchangepfadmin --username=linuxreader --folder=public_events \
       --addright=RoleReviewer
Permission RoleReviewer added for linuxreader on folder public_events

$ openchangepfadmin --username="" --folder=public_events \
       --modright=RoleNone
Permission changed to RoleNone for  on folder appointment

$ openchangepfadmin --username=Anonymous --folder=public_events \
       --modright=RoleNone
Permission changed to RoleNone for Anonymous on folder appointment

Bi: screenshots/03_openchangepfadmin_acls.png
B: Figure3: public_events folder sharing window


openchangeclient Revisited

openchangeclient provides atomic operations for all Outlook
functionality. Many of these features deal with the message class,
such as the contents of the Inbox (which is a container class of type
IPM.Post and holds IPM.Note items):

$ openchangeclient --fetchmail

Now we are going to operate on Public Folders, specifically the
container class IPF.Appointment. To do this we need to be two users in
succession, both having write access to this shared object, so we will
no longer use the default profile feature of openchangeclient.

$ openchangeclient --profile=linuxowner --pf --folder=public_events \
      --sendappointment --subject="Coffee Break"      \
     --location="Coffee Machine" --busystatus=BUSY    \
     --dtstart="2007-09-01 10:00:00"      \
     --dtend="2007-09-01 10:45:00"
   sendappointment          : MAPI_E_SUCCESS (0x0)

Bi: screenshots/04_outlook_calview.png
B: sendappointment operation result in Microsoft Outlook

$ openchangeclient --profile=linuxreader --pf --folder=public_events \
      --sendappointment --subject="Coffee Break Extension" \
     --location="Coffee Machine" --busystatus=BUSY  \
     --dtstart="2007-09-01 10:45:00"   \
     --dtend="2007-09-01 12:00:00"
   sendappointment          : UNKNOWN_ENUM_VALUE (0x4FF)

$ openchangeclient --profile=linuxreader --pf --folder=public_events \
      --fetch-items=Appointment
|== Coffee Break ==|
       Location: Coffee Machine
       Start time     :   Sat Sep  1 10:00:00 2007 CEST
       End time       :   Sat Sep  1 10:45:00 2007 CEST
       Private: False
   fetchitems               : MAPI_E_SUCCESS (0x0)


In the console output above, we can see linuxowner can create
appointment entries in public_events while linuxreader can't
(error returned). In the meantime linuxreader can fetch folder
content.


Cleaning up the test environment

You would never perform testing on an important server, of course, but in
case you do want to reset the Exchange server, here's how:

$ openchangepfadmin --rmdir --folder=public_events
$ openchangepfadmin --rmuser=linuxowner
   mapiadmin_user_del       : MAPI_E_SUCCESS (0x0)
$ openchangepfadmin --rmuser=linuxreader
   mapiadmin_user_del       : MAPI_E_SUCCESS (0x0)


And After All That...

In our tour of libmapi and related code we have always been
working relative to Microsoft Exchange Server. As the OpenChange
project progresses, and as other projects start to use our code, that
situation will change. Interoperability is when many dissimilar
systems can communicate using a common language. By making the
language of MAPI easily accessible, we hope the number of dissimilar
systems speaking it will increase greatly.

And in practical terms for the near future? OpenChange is planning
to release libmapi around November 2007, with support from at
least one GUI groupware tool. Work is continuing on the server, and a
demonstration release should be ready some time in 2007. In the
meantime, the client code we already have is likely to find a role
with Microsoft Exchange administrators who are looking for a simple
tool to automate simple tasks -- perhaps especially now there are Perl
bindings for libmapi.


Julien Kerihuel leads the OpenChange project, which he founded in
2003. Dan Shearer handles technical interoperability and strategy issues
for OpenChange, drawing on his twenty years of experience in Open Source.

I   currently work with OpenChange to build graphical front ends for
OpenChangeTools such as mapiprofile, mapiproxy, openchangepfadmin and
integrating these front ends into the gnome desktop.

For questions contact us on freenode at #openchange or write me an email
at engmakokal@gmail.com or visit openchange.org

Development Environment Set, ...

I have managed to set up the development environment for this project successfully. I also tested basic openchange features such as openchange client, creating profiles, sending mails (variety of types). I am now working on the base structure for mapiproxy gui which will be my next task. For this, I am in the process of identifying the capabilties of mapiproxy, what it does, and what can do but does not do, and collect all these info to use in forming the structure of the gui.

Introducing the project

OpenChange aims to provide a portable Open Source implementation of Microsoft Exchange Server and Exchange protocols

Exchange is a groupware server designed to work with Microsoft Outlook, and providing features such as a messaging server, shared calendars, contact databases, public folders, notes and tasks.During this year's SoC, I will be building the graphical front end to openchangetools. This will encompass a number of tasks such as building GUI for mapiproxy, openchangepfadmin and openchangeproxy. These guis then need to be integrated into the gnome desktop under System > Administration. Some of the changes by on these guis need to be kept in a database accessible to other programs such as evolution. Look at the full proposal here.

Check out the openchange wiki here.

I will be posting relevant updates on the project as I progress.