MAPI Documentation
Description
MAPI is a solution to a specific problem. The problem in our case is that current solutions for network monitoring don't scale well under heavy load. There are many solutions (especially for Unix systems) which are truly successful for small installations and SOHO networks. All these solutions are purely userspace meaning that they consist of libraries and applications which communicate with the OS kernel in order to get information. For faster networks however the performance is not always optimal. MAPI tries to address this problems by integrating into the OS kernel itself.
Consider a Linux application which just needs to count the incoming network packets from a fast gigabit connection. Since there is no additional processing the packets are just discarded by the application after they are accounted for. The cost however, to transfer the actual packet from kernel space (the Linux kernel) to userspace (the application) is still the same for each individual packet. Therefore the application has efficiency problems for extreme network conditions even though its function (counting packets) is simple enough.
MAPI splits the monitoring software into a Linux kernel module and userspace API for interfacing with the module. Therefore a MAPI application which needs to count packets actually asks the kernel module "count the packets for me". The packets are discarded (after accounted for) by the kernel before reaching userspace. Therefore there is no penalty from transfering the data of the packets to the application level (userspace).
MAPI of course can do many more things than just packet counting. In fact MAPI is a programming framework rather than an application, allowing the user to program his/her requirements from the monitoring process. The MAPI distribution includes some sample network functions so that one can start experimenting with the software before deploying custom solutions for specific network installations.
You can download as PDF files a general presentation about MAPI, some sample test runs, and a high-level overview of MAPI.
Supported Functions
| Count Packets | Counts total packets received until now. |
|---|---|
| Count Bytes | Counts total bytes received until now. |
| Packets in Interval | Counts all packets received in a specified interval. |
| Bytes in Interval | Counts all bytes received in a specified interval. |
| Rotating and Additive Hash | Receives or ignores packets depending on the return value of the hash function. |
| EXB String Search Algorithm | Receives only the packets with the given payload. |
| Boyer-Moore String Search Algorithm | Receives only the packets with the given payload. |
| Log Packets | Logs all packets in the specified file. |
| Sample Packets | Samples the stream of packets. Application can select between probabilistic or deterministic sampling. |
| Packet Save | Trims packets according to the specified parameters. |
| SubFlows | Categorizes IP traffic into quads with same source IP, destination IP, source port, destination port and protocol. It keeps statistics about quads such as total packets seen, total bytes seen, mean packet size etc. Also it has an expiration mechanism about this quads. |
| IP Reassembly | Reassembles IP traffic, ignores packet if IP checksum is not valid etc (almost everything IP stack does). |
| UDP Reassembly | Removes UDP headers after the packet has already succesfully passed from IP reassembly function. |
Installation
MAPI consists of a Linux kernel module and a userspace library which interfaces with the module providing an abstraction layer for the user. To learn how to compile Linux kernels (and install modules) please see the Linux kernel howto. The MAPI package is built with autoconf/automake so the following command will compile and install MAPI.
./configure
make
make install