Copy the KDE 3.5 branch to branches/trinity for new KDE 3.5 features.

BUG:215923


git-svn-id: svn://anonsvn.kde.org/home/kde/branches/trinity/kdepim@1054174 283d02a7-25f6-0310-bc7c-ecb5cbfe19da

1 files changed, 871 insertions, 0 deletions

diff --git a/kmail/Mainpage.dox b/kmail/Mainpage.dox
new file mode 100644
index 000000000..0b15ebfbc
--- /dev/null
+++ b/kmail/Mainpage.dox
@@ -0,0 +1,871 @@
+/** \mainpage KMail architectural overview
+
+\section KMail design principles
+
+This file is intended to guide the reader's way through the KMail
+codebase. It should esp. be handy for people not hacking full-time on
+KMail as well as people that want to trace bugs in parts of KMail
+which they don't know well.
+
+Contents:
+- Kernel
+- Identity
+- Filters
+- ConfigureDialog
+- MDNs
+- Folders
+- Index
+- Headers
+- Display
+
+TODO: reader, composer, messages, accounts, ...
+
+\section kernel KERNEL
+
+Files: kmkernel.h, kmkernel.cpp
+
+Contact Zack Rusin <zack@kde.org> with questions...
+
+The first thing you'll notice about KMail is the extensive use of
+kmkernel->xxx() constructs. The "kmkernel" is a define in kmkernel.h
+declared as :
+#define kmkernel KMKernel::self() 
+KMKernel is the central object in KMail. It's always created before
+any other class, therefore you are _guaranteed_ that KMKernel::self()
+(and therefore "kmkernel" construct) won't return 0 (null).
+
+KMKernel implements the KMailIface (our DCOP interface) and gives
+access to all the core KMail functionality. 
+
+
+\section identity IDENTITY
+
+FIXME this has moved to libkpimidentities, right?
+
+Files: identity*, kmidentity.{h,cpp}, configuredialog.cpp,
+       signatureconfigurator.{h,cpp}
+
+Contact Marc Mutz <mutz@kde.org> on questions...
+
+Identities consists of various fields represented by
+QStrings. Currently, those fields are hardcoded, but feel free to
+implement KPIM::Identity as a map from strings to QVariants or somesuch.
+
+One part of identities are signatures. They can represent four modes
+(Signature::Type) of operation (disabled, text from file or command
+and inline text), which correspond to the combo box in the
+identitydialog.
+
+Identities are designed to be used through the KPIM::IdentityManager:
+const KPIM::Identity & ident =
+	 kmkernel->identityManager()->identityForUoidOrDefault(...)
+Make sure you assign to a _const_ reference, since the identityForFoo
+methods are overloaded with non-const methods that access a different
+list of identities in the manager that is used while configuring. That
+is known source of errors when you use identityForFoo() as a parameter
+to a method taking const KPIM::Identity &.
+
+WARNING: Don't hold the reference longer than the current functions
+scope or next return to the event loop. That's b/c the config dialog
+is not modal and the user may hit apply/ok anytime between calls to
+function that want to use the identity reference. Store the UOID
+instead if you need to keep track of the identity. You may also want
+to connect to one of the KPIM::IdentityManager::changed() or ::deleted()
+signals, if you want to do special processing in case the identity
+changes.
+
+Thus, in the ConfigureDialog, you will see non-const KPIM::Identity
+references being used, while everywhere else (KMMessage,
+IdentityCombo) const references are used.
+
+The KPIM::IdentityCombo is what you see in the composer. It's a
+self-updating combo box of KPIM::Identity's. Use this if you want the user
+to choose an identity, e.g. in the folder dialog.
+
+Ihe IdentityListView is what you see in the config dialog's identity
+management page. It's not meant to be used elsewhere, but is DnD
+enabled (well, at the time of this writing, only drag-enabled). This
+is going to be used to dnd identities around between KNode and KMail,
+e.g.
+
+The SignatureConfigurator is the third tab in the identity
+dialog. It's separate since it is used by the identity manager to
+request a new file/command if the current value somehow fails.
+
+
+
+\section filter FILTER
+
+Contact Marc Mutz <mutz@kde.org> on questions...
+
+Filters consist of a search pattern and a list of actions plus a few
+flags to indicate when they are to be applied (kmfilter.h).
+  They are managed in a QPtrList<KMFilter>, called KMFilterMgr. This
+filter magnager is responsible for loading and storing filters
+(read/writeConfig) and for executing them (process). The unique
+instance of the filter manager is held by the kernel
+(KMKernel::filterMgr()).
+
+The search pattern is a QPtrList of search rules (kmsearchpattern.h) and a
+boolean operator that defines their relation (and/or).
+
+A search rule consists of a field-QString, a "function"-enum and a
+"contents" or "value" QString. The first gives the header (or
+pseudoheader) to match against, the second says how to match (equals,
+consists, is less than,...) and the third holds the pattern to match
+against.
+  Currently, there are two types of search rules, which are mixed
+together into a single class: String-valued and int-valued. The latter
+is a hack to enable \verbatim<size>\endverbatim and
+\verbatim<age in days>\endverbatim pseudo-header matching.
+  KMSearchRules should better be organized like KMFilterActions are.
+
+A filter action (kmfilteraction.h) inherits from KMFilterAction or one
+of it's convenience sub-classes. They have three sub-interfaces: (1)
+argument handling, (2) processing and (3) parameter widget handling.
+  Interface (1) consists of args{From,As}String(), name() and
+isEmpty() and is used to read and write the arguments (if any) from/to
+the config.
+  Interface (2) is used by the filter manager to execute the action
+(process() / ReturnCode).
+  Interface (3) is used by the filter dialog to allow editing of
+actions and consists of name(), label() and the
+*ParamWidget*(). Complex parameter widgets are collected in
+kmfawidget.{h,cpp}.
+
+A typical call for applying filters is 
+
+KMKernel::filterMgr()
+foreach message {
+  KMFilterMgr::process():
+}
+
+
+\section configuration CONFIGURE DIALOG
+
+Files: configuredialog*.{h,cpp} ( identitylistview.{h,cpp} )
+
+Contact Marc Mutz <mutz@kde.org> on questions...
+
+The configuredialog is made up of pages that in turn may consist of a
+number of tabs. The genral rule of thumb is that each page and tab is
+responsible for reading and writing the config options presented on
+it, although in the future, that may be reduced to interacting with
+the corresponding config manager instead. But that won't change the
+basic principle.
+
+Thus, there is an abstract base class ConfigurePage (defined in
+configuredialog_p.h), which derives from QWidget. It has four methods
+of which you have to reimplement at least the first two:
+
+- void setup()
+  Re-read the config (from the config file or the manager) and update
+  the widgets correspondingly. That is, you should only create the
+  widgets in the ctor, not set the options yet. The reason for that is
+  that the config dialog, once created, is simply hidden and shown
+  subsequently, so we need a reset-like method anyway.
+
+- void apply()
+  Read the config from the widgets and write it into the config file
+  or the corresponding config manager.
+
+- void installProfile()
+  This is called when the user selected a profile and hit apply. A
+  profile is just another KConfig object. Therefore, this method
+  should be the same as setup(), except that you should only alter
+  widgets for configs that really exist in the profile.
+
+For tabbed config pages, there exists a convenience class called
+TabbedConfigurationPage, which (as of this writing only offers the
+addTab() convenience method. It is planned to also provide
+reimplementations of setup, dismiss, apply and installProfile that just
+call the same functions for each tab.
+
+\section mdn MDNs
+
+Files: libkdenetwork/kmime_mdn.{h,cpp} and kmmessage.{h,cpp}, mostly
+
+Contact Marc Mutz <mutz@kde.org> on questions...
+
+MDNs (Message Disposition Notifications; RFC 2298) are a way to send
+back information regarding received messages back to their
+sender. Examples include "message read/deleted/forwarded/processed".
+
+The code in kmime_mdn.{h,cpp} is responsible for creating the
+message/disposition-notification body part (2nd child of
+multipart/report that makes the MDN) and for providing the template
+for human-readable text that goes into the text/plain part (1st child
+of the multipart/report).
+
+The code in KMMessage::createMDN() actually constructs a message
+containing a MDN for this message, using the kmime_mdn helper
+functions. It starts by checking the index for an already sent MDN,
+since the RFC demands that MDNs be sent only once for every
+message. If that test succeeds, it goes on to check various other
+constraints as per RFC and if all goes well the message containing the
+multipart/report is created.
+
+If you need to use this functionality, see KMReaderWin::touchMsg() and
+KMFilterAction::sendMDN() for examples. The touchMsg() code is invoked
+on display of a message and sends a "displayed" MDN back (if so
+configured), whereas the KMFilterAction method is a convenience helper
+for the various filter actions that can provoke a MDN (move to trash,
+redirect, forward, ...).
+
+
+\section folders Folders
+
+Files: kmfolder*.{h,cpp}, folderstorage.{h,cpp} and *job.{h,cpp}
+
+Contact Zack Rusin <zack@kde.org> with questions...
+
+The collaboration among KMail folder classes looks
+as follows :
+               
+                     KMFolderNode
+                     /          \
+                    /            \
+               KMFolderDir        \
+                                  KMFolder
+                                    .
+                                    .
+                                    v
+                               FolderStorage
+                                    |
+                                    |
+                               KMFolderIndex
+                                    |
+                                    |
+        ---< actual folder types: KMFolderImap, KMFolderMbox... >--
+
+At the base KMail's folder design starts with KMFolderNode which
+inherits QObject. KMFolderNode is the base class encapsulating
+common folder properties such as the name and a boolean signifying whether
+the folder is a folder holding mail directly or a KMFolderDir.
+KMFolderNode's often do not have an on-disk representation, they are
+entities existing only within KMail's design. 
+
+KMFolder acts as the runtime representation of a folder with the physical
+storage part being represented by a member of type FolderStorage.
+KMFolder and FolderStorage have many functions with the same names and
+signatures, but there is no inheritance.
+KMFolderIndex contains some common indexing functionality for physical folders.
+Subclasses of KMFolderIndex finally interact directly with physical storage
+or with storage providers over the network.
+
+KMFolderDir is a directory abstraction which holds KMFolderNode's.
+It inherits KMFolderNode and KMFolderNodeList which is a QPtrList<KMFolderNode>.
+A special case of a KMFolderDir is KMFolderRootDir; it represents
+the toplevel KMFolderDir in KMail's folder hierarchy. 
+
+KMFolderDir's contents are managed by KMFolderMgr's.
+KMail contains three main KMFolderMgr's. They can be
+accessed via KMKernel ( the "kmkernel" construct ). Those methods are :
+1) KMFolderMgr *folderMgr() - which returns the folder manager for
+   the folders stored locally.
+2) KMFolderMgr *imapFolderMgr() - which returns the folder manager
+   for all imap folders. They're handled a little differently because
+   for all imap messages only headers are cached locally while the
+   main contents of all messages is kept on the server.
+3) KMFolderMgr *dimapFolderMgr() - which returns disconnected IMAP (dimap) 
+   folder manager. In dimap, both the headers and a copy of the full message
+   are cached locally.
+4) KMFolderMgr *searchFolderMgr() - which returns the folder manager
+   for search folders (folders created by using the "find
+   messages" tool). Other email clients call this type of folder
+   "virtual folders".
+
+FolderJob classes - These classes allow asynchronous operations on
+KMFolder's. You create a Job on the heap, connect to one of its 
+signals and wait for the job to finish. Folders serve as FolderJob
+factories. For example, to retrieve the full message from a folder
+you do :
+
+FolderJob *job = folderParent->createJob( aMsg, tGetMessage );
+connect( job, SIGNAL(messageRetrieved(KMMessage*)),
+         SLOT(msgWasRetrieved(KMMessage*)) );
+job->start();
+
+
+\section index Index (old)
+
+Files: kmfolderindex.{h,cpp} and kmmsg{base,info}.{h,cpp}
+
+Contact Marc Mutz <mutz@kde.org> or
+	Till Adam <adam@kde.org> or
+	Don Sanders <sanders@kde.org>
+with questions...
+
+   index := header *entry
+
+
+   header := magic LF NUL header-length byte-order-marker sizeof-long
+
+   magic := "# KMail-Index V" 1*DIGITS
+
+   header-length := Q_UINT32
+
+   byte-order-marker := Q_UINT32( 0x12345678 )
+
+   sizeof-long := Q_UINT32( 4 / 8 )
+
+
+   entry := tag length value
+
+   tag := Q_UINT32 ; little endian (native?)
+
+   length := Q_UINT16 ; little endian (native?)
+
+   value := unicode-string-value / ulong-value
+
+   unicode-string-value := 0*256QChar ; network-byte-order
+
+   ulong-value := unsigned_long ; little endian
+
+Currently defined tag values are:
+
+   Msg*Part      num. val  type obtained by:
+
+   No		 0	   u	-
+   From		 1	   u	fromStrip().stripWhitespace()
+   Subject	 2	   u	subject().stripWhitespace()
+   To		 3	   u	toStrip().stripWhiteSpace()
+   ReplyToIdMD5	 4	   u	replyToIdMD5().stripWhiteSpace()
+   IdMD5	 5	   u	msgIdMD5().stripWhiteSpace()
+   XMark	 6	   u	xmark().stripWhiteSpace()
+   Offset	 7	   l	folderOffset() (not only mbox!)
+   LegacyStatus	 8	   l	mLegacyStatus
+   Size		 9	   l	msgSize()
+   Date		 10	   l	date()
+   File		 11	   u	fileName() (not only maildir!)
+   CryptoState	 12	   l	(signatureState() << 16) | encryptionState())
+   MDNSent	 13	   l	mdnSentState()
+   ReplyToAuxIdMD5 14	   u	replyToAuxIdMD5()
+   StrippedSubject 15	   u	strippedSubjectMD5().stripWhiteSpace()
+   Status	   16	   l	status()
+
+   u: unicode-string-value; l: ulong-value
+
+Proposed new (KDE 3.2 / KMail 1.6) entry format:
+
+   index := header *entry
+
+   entry := sync 1*( tag type content ) crc-tag crc-value
+
+   sync := Q_UINT16 (32?) ; resync mark, some magic bit pattern
+			  ; (together with preceding crc-tag provides
+			  ;  24(40)bits to resync on in case of index
+			  ;  corruption)
+
+   tag := Q_UINT8
+
+   type := Q_UINT8
+
+   content := variable-length-content / fixed-length-content
+
+   crc-tag := tag type   ; tag=CRC16, type=CRC16
+
+   crc-value := Q_UINT16 ; the CRC16 sum is calculated over all of
+			 ; 1*( tag type content )
+
+   variable-length-content := length *512byte padding
+
+   padding := *3NUL ; make the string a multiple of 4 octets in length
+
+   fixed-length-content := 1*byte
+
+   length := Q_UINT16 (Q_UINT8?)
+
+   byte := Q_UINT8
+
+The type field is pseudo-structured:
+
+bit: 7     6     5     4     3     2     1     0
+  +-----+-----+-----+-----+-----+-----+-----+-----+
+  |      uniq       |   chunk   |       len       |
+  +-----+-----+-----+-----+-----+-----+-----+-----+
+
+uniq: 3 bits = max. 8 different types with same chunk size:
+
+      for chunk = (0)00 (LSB(base)=0: octets):
+	00(0) Utf8String
+	01(0) BitField
+	10(0) reserved
+	11(0) Extend
+
+      for chunk = (1)00 (LSB(base)=1: 16-octet blocks):
+	00(1) MD5(Hash/List)
+	01(1) reserved
+	10(1) reserved
+	11(1) Extend
+
+      for chunk = 01 (shorts):
+	000 Utf16String
+	001-110 reserved
+	111 Extend
+
+      for chunk = 10 (int32):
+	000 Utf32String (4; not to be used yet)
+	001 Size32
+	010 Offset32
+        011 SerNum/UOID
+	100 DateTime
+	101 Color (QRgb: (a,r,g,b))
+	110 reserved
+	111 Extend
+
+      for chunk = 11 (int64):
+        000 reserved
+        001 Size64
+        010 Offset64
+	011-110 reserved
+	111 Extend
+
+len: length in chunks
+     000 (variable width -> Q_UINT16 with the real width follows)
+     001..111: fixed-width data of length 2^len (2^1=2..2^6=128)
+
+You find all defined values for the type field in indexentrybase.cpp
+
+Currently defined tags are:
+
+   tag               type       content
+
+   DateSent	     DateTime   Date:
+   DateReceived	     DateTime   last Received:'s date-time
+   FromDisplayName   String     decoded display-name of first From: addr
+   ToDisplayName     String	dto. for To:
+   CcDisplayName     String	dto. for Cc:
+   FromAddrSpecs     String	possibly IMAA-encoded, comma-separated addr-spec's of From:
+   ToAddrSpecs	     String	dto. for To:
+   CcAddrSpecs	     String	dto. for Cc:
+   Subject	     String	decoded Subject:
+   BaseSubjectMD5    String	md5(utf8(stripOffPrefixes(subject())))
+   BodyPeek	     String	body preview
+   MaildirFile	     String	Filename of maildir file for this messagea
+   MBoxOffset	     Offset(64) Offset in mbox file (pointing to From_)
+   MBoxLength        Size(64)   Length of message in mbox file (incl. From_)
+   Size		     Size(64)	rfc2822-size of message (in mbox: excl. From_)
+   Status	     BitField	(see below)
+   MessageIdMD5	     MD5Hash	MD5Hash of _normalized_ Message-Id:
+   MDNLink	     SerialNumber SerNum of MDN received for this message
+   DNSLink	     SerialNumber SerNUm of DSN received for this message
+   ThreadHeads	     SerialNumberList	MD5Hash's of all (so far discovered)
+					_top-level thread parents_
+   ThreadParents     SerialNumberList	MD5Hash's of all (so far discovered)
+					thread parents
+
+
+   "String" is either Utf8String or (Utf16String or Latin1String),
+	    depending on content
+
+Currently allocated bits for the Status BitField are:
+
+    Bit              Value: on(/off) (\\imapflag)
+
+    # "std stati":
+    0		     New  (\\Recent)
+    1		     Read (\\Seen)
+    2		     Answered (\\Answered)
+    3		     Deleted (\\Deleted)
+    4		     Flagged (\\Flagged)
+    5		     Draft (\\Draft)
+    6		     Forwarded
+    7		     reserved
+
+    # message properties:
+    8		     HasAttachments
+    9		     MDNSent ($MDNSent)
+    10..11	     00: unspecified, 01: Low, 10: Normal, 11: High Priority
+    12..15	     0001: Queued, 0010: Sent, 0011: DeliveryDelayed,
+		     0100: Delivered, 0101: DeliveryFailed,
+		     0110: DisplayedToReceiver, 0111: DeletedByReceiver,
+		     1001: ProcessedByReceiver, 1010: ForwardedByReceiver,
+		     1011-1110: reserved
+		     1111: AnswerReceived
+
+    # signature / encryption state:
+    16..19	     0001: FullyEncrypted, 0010: PartiallyEncrypted, 1xxx: Problem
+    20..23	     0001: FullySigned, 0010: PartiallySigned, 1xxx: Problem
+
+    # "workflow stati":
+    24..25	     01: Important, 10: ToDo, 11: Later (from Moz/Evo)
+    26..27	     01: Work, 10: Personal, 11: reserved (dto.)
+    28..29	     01: ThreadWatched, 10: ThreadIgnored, 11: reserved
+    30..31	     reserved
+
+All bits and combinations marked as reserved MUST NOT be altered if
+set and MUST be set to zero (0) when creating the bitfield.
+
+
+\section headers Headers (Threading and Sorting)
+
+Contact Till Adam <adam@kde.org> or
+	Don Sanders <sanders@kde.org>
+with questions...
+
+Threading and sorting is implemented in kmheaders.[cpp|h] and headeritem.[cpp|h] 
+and involves a handfull of players, namely:
+
+class KMHeaders: 
+  this is the listview that contains the subject, date etc of each mail. 
+  It's a singleton, which means there is only one, per mainwidget headers 
+  list. It is actually a member of KMMainwidget and accessible there.
+
+class HeaderItem: 
+  these are the [Q|K]ListViewItem descendend items the KMHeaders listview 
+  consists of. There's one for each message.
+
+class SortCacheItem: 
+  these are what the threading and sorting as well as the caching of those
+  operate on. Each is paired with a HeaderItem, such that each holds a
+  pointer to one, and vice versa, each header item holds a pointer to it's
+  associated sort cache item.
+  
+.sorted file:
+  The order of the sorted and threaded (if threading is turned on for this
+  folder) messages is cached on disk in a file named .$FOLDER.index.sorted 
+  so if, for example the name of a folder is foo, the associated sorting 
+  cache file would be called ".foo.index.sorted".
+  For each message, its serial number, that of its parent, the length of its
+  sorting key, and the key itself are written to this file. At the start of
+  the file several per folder counts and flags are cached additionally,
+  immediately after a short file headers. The entries from the start of the
+  file are:
+  - "## KMail Sort V%04d\n\t" (magic header and version string)
+  - byteOrder flag containing 0x12345678 for byte order testing
+  - column, the sort column used for the folder
+  - ascending, the sort direction
+  - threaded, is the view threaded or is it not?
+  - appended, have there been items appended to the file (obsolete?)
+  - discovered_count, number of new mail discovered since the last sort file
+    generation
+  - sorted_count, number of sorted messages in the header list
+
+What is used for figuring out threading?
+  - messages can have an In-Reply-To header that contains the message id of
+    another message. This is called a perfect parent.
+  - additionally there is the References header which, if present, holds a 
+    list of message ids that the current message is a follow up to. We
+    currently use the second to last entry in that list only. See further 
+    down for the reasoning behind that.
+  - If the above two fail and the message is prefixed (Re: foo, Fwd: foo etc.)
+    an attempt is made to find a parent by looking for messages with the same
+    subject. How that is done is explained below as well.
+  
+  For all these comparisons of header contents, the md5 hashes of the headers
+  are used for speed reasons. They are stored in the index entry for each 
+  message. All data structures described below use md5 hash strings unless
+  stated otherwise.
+
+Strategy:
+  When a folder is opened, updateMessageList is called, which in turn calls
+  readSortOrder where all the fun happens. If there is a .sorted file at the 
+  expected location, it is openend and parsed. The header flags are read in
+  order to figure out the state in which this .sorted file was written. This
+  means the sort direction, whether the folder is threaded or not etc. 
+  FIXME: is there currently sanity checking going on?
+  Now the file is parsed and for each message in the file a SortCacheItem is
+  created, which holds the offset in the .sorted file for this message as well
+  as it's sort key as read from the file. That sort cache item is entered into
+  the global mSortCache structure (member of the KMHeaders instance), which is
+  a  QMemArray<SortCacheItem *> of the size mFolder->count(). Note that this
+  is the size reported by the folder, not as read from the .sorted file. The
+  mSortCache (along with some other structures) is updated when messages are
+  added or removed. More on that further down.
+  As soon as the serial number is read from the file, that number is looked up
+  in the message dict, to ensure it is still in the current folder. If not, it
+  has been moved away in the meantime (possibly by outside forces such as
+  other clients) and a deleted counter is incremented and all further
+  processing stopped for this message.
+  The messages parent serial number, as read from the sorted file is then
+  used to look up the parent and reset it to -1 should it not be in the 
+  current folder anymore. -1 and -2 are special values that can show up
+  as parent serial numbers and are used to encode the following:
+  -1 means this message has no perfect parent, a parent for it needs to
+  be found from among the other messages, if there is a suitable one
+  -2 means this message is top level and will forever stay so, no need
+  to even try to find a parent. This is also used for the non-threaded
+  case. These are messages that have neither an In-Reply-To header nor
+  a References header and have a subject that is not prefixed.
+  In case there is a perfect parent, the current sort cache item is
+  appended to the parents list of unsorted children, or to that of
+  root, if there is not. A sort cache item is created in the mSortCache
+  for the parent, if it is not already there. Messages with a parent of
+  -1 are appended to the "unparented" list, which is later traversed and
+  its elements threaded. Messages with -2 as the parent are children of
+  root as well, as noted above, and will remain so. 
+
+  Once the end of the file is reached, we should have a nicely filled 
+  mSortCache, containing a sort cache item for each message that was in the 
+  sorted file. Messages with perfect parents know about them, top level 
+  messages know about that as well, all others are on a list and will be
+  threaded later.
+
+  Now, what happens when messages have been added to the store since the last
+  update of the .sorted file? Right, they don't have a sort cache item yet, 
+  and would be overlooked. Consequently all message ids in the folder from 0
+  to mFolder->count() are looked at and a SortCacheItem is created for the
+  ones that do not have one yet. This is where all sort cache items are created
+  if there was no sorted file. The items created here are by definition un-
+  sorted as well as unparented. On creation their sort key is figured out as
+  well.
+
+  The next step is finding parents for those messages that are either new, or
+  had a parent of -1 in the .sorted file. To that end, a dict of all sort
+  cache items indexed by the md5 hash of their messsage id headers is created,
+  that will be used for looking up sort cache items by message id. The list of
+  yet unparented messages is then traversed and findParent() called for each 
+  element wihch checks In-Reply-To and References headers and looks up the 
+  sort cache item of those parents in the above mentioned dict. Should none be
+  found, the item is added to a second list the items of which will be subject
+  threaded. 
+  
+  How does findParent() work, well, it tries to find the message pointed to by
+  the In-Reply-To header and if that fails looks for the one pointed to by the
+  second to last entry of the References header list. Why the second to last?
+  Imagine the following situation in Bob's kmail:
+  - Bob get's mail from Alice
+  - Bob replies to Alice's mail, but his mail is stored in the Outbox, not the
+    Inbox.
+  - Alice replies again, so Bob now has two mails from Alice which are part of
+    the same thread. The mail in the middle is somewhere else. We want that to
+    thread as follows:
+      Bob             <- In-Reply-To: Alice1
+    ============
+      Alice1          
+        |_Alice       <- In-Reply-To: Bob (not here), References: Alice, Bob
+
+  - since the above is a common special case, it is worth trying. I think. ;)
+
+  If the parent found is perfect (In-Reply-To), the sort cache items is marked
+  as such. mIsImperfectlyThreaded is used for that purposer, we will soon see
+  why that flag is needed.
+
+  On this first pass, no subject threading is attempted yet. Once it is done,
+  the messages that are now top-level, the current thread heads, so to speak, 
+  are collected into a second dict ( QDict< QPtrList< SortCacheItem > > )
+  that contains for each different subject an entry holding a list of (so far
+  top level) messages with that subject, that are potential parents for 
+  threading by subjects. These lists are sorted by date, so the parent closest
+  by date can be chosen. Sorting of these lists happens via insertion sort 
+  while they are built because not only are they expected to be short (apart
+  from hard corner cases such as cvs commit lists, for which subject threading 
+  makes little sense anyhow and where it should be turned off), but also since
+  the messages should be roughly pre sorted by date in the store already. 
+  Some cursory benchmarking supports that assumption. 
+  If now a parent is needed for a message with a certain subject, the list of
+  top level messages with that subject is traversed and the first one that is
+  older than our message is chosen as it's parent. Parents more than six weeks
+  older than the message are not accepted. The reasoning being that if a new
+  message with the same subject turns up after such a long time, the chances
+  that it is still part of the same thread are slim. The value of six weeks
+  is chosen as a result of a poll conducted on #kde-devel, so it's probably
+  bogus. :) All of this happens in the aptly named: findParentBySubject().
+
+  Everthing that still has no parent after this ends up at toplevel, no further
+  attemp is made at finding one. If you are reading this because you want to
+  implement threading by body search or somesuch, please go away, I don't like
+  you. :)
+
+  Ok, so far we have only operated on sort cache items, nothing ui wise has
+  happened yet. Now that we have established the parent/child relations of all
+  messages, it's time to create HeaderItems for them for use in the header
+  list. But wait, you say, what about sorting? Wouldn't it make sense to do
+  that first? Exactly, you're a clever bugger, ey? Here, have a cookie. ;)
+  Both creation of header items and sorting of the as of yet unsorted sort
+  cache items happen at the same time. 
+  
+  As previously mentioned (or not) each sort cache item holds a list of its
+  sorted and one of its unsorted children. Starting with the root node the
+  unsorted list is first qsorted, and then merged with the list of already
+  sorted children. To achieve that, the heads of both lists are compared and
+  the one with the "better" key is added to the list view next by creating a
+  KMHeaderListItem for it. That header item receives both its sort key as well
+  as its id from the sort cache item. Should the current sort cache item have
+  children, it is added to the end of a queue of nodes to repeat these steps
+  on after the current level is sorted. This way, a breadth first merge sort
+  is performed on the sort cache items and header items are created at each
+  node.
+ 
+  What follows is another iteration over all message ids in the folder, to
+  make sure that there are HeaderItems for each now. Should that not be the
+  case, top level emergency items are created. This loop is also used to add
+  all header items that are imperfectly threaded to a list so they can be
+  reevalutated when a new message arrives. Here the reverse mapping from
+  header items to sort cache items are made as well. Those are also necessary
+  so the sort cache item based data structures can be updated when a message
+  is removed. 
+
+  The rest of readSortOrder should make sense on itself, I guess, if not, drop
+  me an email.
+
+What happens when a message arrives in the folder?
+  Among other things, the msgAdded slot is called, which creates the necessary
+  sort cache item and header item for the new message and makes sure the data 
+  structures described above are updated accordingly. If threading is enabled,
+  the new message is threaded using the same findParent and findParentBySubject
+  methods used on folder open. If the message ends up in a watched or ignored
+  thread, those status bits are inherited from the parent. The message is also
+  added to the dict of header items, the index of messages by message id and,
+  if applicable and if the message is threaded at top level, to the list of
+  potential parents for subject threading. 
+
+  After those house keeping tasks are performed, the list of as of yet imper-
+  fectly threaded messages is traversed and our newly arrived message is 
+  considered as a new parent for each item on it. This is especially important
+  to ensure that parents arriving out of order after their children still end
+  up as parents. If necessary, the entries in the .sorted file of rethreaded
+  messages are updated. An entry for the new message itself is appended to the
+  .sorted file as well.
+
+  Note that as an optimization newly arriving mail as part of a mailcheck in
+  an imap folder is not added via msgAdded, but rather via complete reload of
+  the folder via readSortOrder(). That's because only the headers are gotten
+  for each mail on imap and that is so fast, that adding them individually to
+  the list view is very slow by comparison. Thus we need to make sure that
+  writeSortOrder() is called whenever something related to sorting changes, 
+  otherwise we read stale info from the .sorted file. The reload is triggered
+  by the folderComplete() signal of imap folders.
+
+What happens when a message is removed from the folder?
+  In this case the msgRemoved slot kicks in and updates the headers list. First
+  the sort cache item and header item representing our message are removed from
+  the data structures and the ids of all items after it in the store decre-
+  mented. Then a list of children of the message is assembled containing those
+  children that have to be reparented now that our message has gone away. If 
+  one of those children has been marked as toBeDeleted, it is simply added to 
+  root at top level, because there is no need to find a parent for it if it is
+  to go away as well. This is an optimization to avoid rethreading  all 
+  messages in a thread when deleting several messages in a thread, or even the
+  whole thread. The KMMoveCommand marks all messages that will be moved out of
+  the folder as such so that can be detected here and the useless reparenting
+  can be avoided. Note that that does not work when moving messages via filter
+  action. 
+
+  That list of children is then traversed and a new parent found for each one
+  using, again, findParent and findParentBySubject. When a message becomes
+  imperfectly threaded in the process, it is added to the corresponding list.
+
+  The message itself is removed from the list of imperfectly threaded messages
+  and its header item is finally deleted. The HeaderItem destructor destroys
+  the SortCacheItem as well, which is hopefully no longer referenced anywhere
+  at this point.
+
+
+
+\section display DISPLAY (reader window - new)
+
+  Contact Marc Mutz <mutz@kde.org> with questions...
+
+What happens when a message is to displayed in the reader window?
+  First, since KMMessagePart and KMMessage don't work with nested body
+  parts, a hierarchical tree of MIME body parts is constructed with
+  partNode's (being wrappers around KMMessagePart and
+  DwBodyPart). This is done in KMReaderWin::parseMsg(KMMessage*).
+  After some legacy handling, an ObjectTreeParser is instantiated and
+  it's parseObjectTree() method called on the root
+  partNode. ObjectTreeParser is the result of an ongoing refactoring
+  to enhance the design of the message display code. It's an
+  implementation of the Method Object pattern, used to break down a
+  huuuge method into many smaller ones without the need to pass around
+  a whole lot of paramters (those are made members
+  instead). parseObjectTree() recurses into the partNode tree and
+  generates an HTML representation of each node in the tree (although
+  some can be hidden by setting them to processed beforehand or - the
+  new way - by using AttachmentStrategy::Hidden). The HTML generation
+  is delegated to BodyPartFormatters, while the HTML is written to
+  HTMLWriters, which abstract away HTML sinks. One of those is
+  KHTMLPartHTMLWriter, which writes to the KHTMLPart in the
+  readerwindow. This is the current state of the code. The goal of the
+  ongoing refactoring is to make the HTML generation blissfully
+  ignorant of the readerwindow and to allow display plugins that can
+  operate against stable interfaces even though the internal KMail
+  classes change dramatically.
+
+  To this end, we designed a new set of interfaces that allows plugins
+  to be written that need or want to asynchronously update their HTML
+  representation. This set of interfaces consists of the following:
+
+  - @em BodyPartFormatterPlugin
+	the plugin base class. Allows the BodyPartFormatterFactory to
+	query it for an arbitray number of BodyPartFormatters and
+	their associated metadata and url handlers.
+
+  - @em BodyPartFormatter
+	the formatter interface. Contains a single method format()
+	that takes a BodyPart to process and an HTMLWriter to write the
+	generated HTML to and returns a result code with which it can
+	request more information or delegate the formatting back to
+	some default processor. BodyPartFormatters are meant to be
+	Flyweights, implying that the format() method is not allowed
+	to maintain state between calls, but see Mememto below.
+
+  - @em BodyPart
+	body part interface. Contains methods to retrieve the content
+	of a message part and some of the more interesting header
+	information. This interface decouples the bodypart formatters
+	from the implementation used in KMail (or KNode, for that
+	matter). Also contains methods to set and retrieve a Memento,
+	which is used by BodyPartFormatters to maintain state between
+	calls of format().
+
+  - @em BodyPartMemento
+	interface for opaque state storage and event handling.
+	Contains only a virtual destructor to enable the backend
+	processing code to destroy the object without the help of the
+	bodypart formatter.
+
+
+During the design phase we identified a need for BodyPartFormatters to
+request their being called on some form of events, e.g. a dcop
+signal. Thus, the Memento interface also includes the IObserver and
+ISubject interfaces. If a BodyPartFormatter needs to react to a signal
+(Qt or DCOP), it implements the Memento interface using a QObject,
+connects the signal to a slot on the Memento and (as an ISubject)
+notifies it's IObservers when the slot is called. If a Memento is
+created, the reader window registers itself as an observer of the
+Memento and will magically invoke the corresponding BodyPartFormatter,
+passing along the Memento to be retrieved from the BodyPart interface.
+
+An example should make this clearer. Suppose, we want to update our
+display after 10 seconds. Initially, we just write out an icon, and
+after 10 seconds, we want to replace the icon by a "Hello world!"
+line. The following code does exactly this:
+
+@code
+class DelayedHelloWorldBodyPartFormatter
+      : public KMail::BodyPartFormatter {
+public:
+  Result format( KMail::BodyPart * bodyPart,
+		 KMail::HTMLWriter * htmlWriter ) {
+    if ( !bodyPart->memento() ) {
+      bodyPart->registerMemento( new DelayedHelloWorldBodyPartMemento() );
+      return AsIcon;
+    } else {
+      htmlWriter->write( "Hello, world!" );
+      return Ok;
+    }
+  }
+};
+
+class DelayedHelloWorldBodyPartMemento
+      : public QObject, public KMail::BodyPartMemento {
+public:
+  DelayedHelloWorldBodyPartMemento()
+    : QObject( 0, "DelayedHelloWorldBodyPartMemento" ),
+      KMail::BodyPartMemento()
+  {
+    QTimer::singleShot( 10*1000, this, SLOT(slotTimeout()) );
+  }
+
+private slots:
+  void slotTimeout() { notify(): }
+
+private:
+  // need to reimplement this abstract method...
+  bool update() { return true; }
+};
+@endcode
+
+*/