The problem the component solves
mailbagit should preserve the original arrangement of email exports, particularly the accounts folder structure, and use that arrangement when writing derivatives. mailbagit currently supports this by parsing the internal folder structure of PST files and reading the X-Folder header when present for MBOX and EML sources. This information is stored in the model using the Message-Path field (which is then converted to Derivatives-Path, as documented in the Input-Output Examples doc.
While X-Folder is often used to preserve email folder information, Gmail instead seems to us a X-Gmail-Labels header, which mailbagit does not currently parse. Unlike X-Folder (which could be Inbox/Listservs, X-Gmail-Labels appears to be a comma separated list that includes both the folder of the email as well as labels like "Important" or "Unread," which we need to exclude Common examples are:
X-Gmail-Labels: Unread,InboxX-Gmail-Labels: Unread,Important,InboxX-Gmail-Labels: Important,InboxX-Gmail-Labels: InboxX-Gmail-Labels: Inbox,Category Promotions,UnreadX-Gmail-Labels: Inbox,Important,Category Updates,UnreadX-Gmail-Labels: Spam,Category Personal,UnreadX-Gmail-Labels: Inbox,Opened,Category UpdatesX-Gmail-Labels: Inbox,Category Social,UnreadX-Gmail-Labels: Inbox,Category Forums,UnreadX-Gmail-Labels: Inbox,Important,Opened,Category PersonalX-Gmail-Labels: Archived,Sent,OpenedX-Gmail-Labels: Sent,NotesX-Gmail-Labels: Sent
The Spam label appears to be a folder, not a label:
X-Gmail-Labels: Unread,Spam
While the Category labels are arguably arrangement, I don't think we can treat them as such reasonably. Thus, I expect to ignore labels starting with "Category."
It looks like the X-GM-LABELS header is also used for this, so we should try to read both.
It does not appear that the folder is consistently the first or last item in the list or anything reliable like that to help with parsing.
I can't find clear documentation on these headers to be confident that this is consistent practice. Thus, I'm thinking we need to make this user-overridable, probably along the lines of the plugins. Perhaps a user could put a list of labels to exclude in with the plugin directory. This way, if there's another label we're not excluding a user could put that in a Gmail-Labels.txt file and mailbagit would use those labels instead of the default excluded labels. Just a line separated list could be fine across platforms just by relying in Python's .readlines() or similar. That way the line breaks should be whatever is native to the OS.
Relevant part of mailbag spec?
N/A
Type of component
Expected contribution
Major challenges or things to keep in mind
Inconsistent arrangement structure is the worst.
The problem the component solves
mailbagitshould preserve the original arrangement of email exports, particularly the accounts folder structure, and use that arrangement when writing derivatives.mailbagitcurrently supports this by parsing the internal folder structure of PST files and reading theX-Folderheader when present for MBOX and EML sources. This information is stored in the model using theMessage-Pathfield (which is then converted toDerivatives-Path, as documented in the Input-Output Examples doc.While
X-Folderis often used to preserve email folder information, Gmail instead seems to us aX-Gmail-Labelsheader, whichmailbagitdoes not currently parse. UnlikeX-Folder(which could beInbox/Listservs,X-Gmail-Labelsappears to be a comma separated list that includes both the folder of the email as well as labels like "Important" or "Unread," which we need to exclude Common examples are:X-Gmail-Labels: Unread,InboxX-Gmail-Labels: Unread,Important,InboxX-Gmail-Labels: Important,InboxX-Gmail-Labels: InboxX-Gmail-Labels: Inbox,Category Promotions,UnreadX-Gmail-Labels: Inbox,Important,Category Updates,UnreadX-Gmail-Labels: Spam,Category Personal,UnreadX-Gmail-Labels: Inbox,Opened,Category UpdatesX-Gmail-Labels: Inbox,Category Social,UnreadX-Gmail-Labels: Inbox,Category Forums,UnreadX-Gmail-Labels: Inbox,Important,Opened,Category PersonalX-Gmail-Labels: Archived,Sent,OpenedX-Gmail-Labels: Sent,NotesX-Gmail-Labels: SentThe
Spamlabel appears to be a folder, not a label:X-Gmail-Labels: Unread,SpamWhile the
Categorylabels are arguably arrangement, I don't think we can treat them as such reasonably. Thus, I expect to ignore labels starting with "Category."It looks like the
X-GM-LABELSheader is also used for this, so we should try to read both.It does not appear that the folder is consistently the first or last item in the list or anything reliable like that to help with parsing.
I can't find clear documentation on these headers to be confident that this is consistent practice. Thus, I'm thinking we need to make this user-overridable, probably along the lines of the plugins. Perhaps a user could put a list of labels to exclude in with the plugin directory. This way, if there's another label we're not excluding a user could put that in a Gmail-Labels.txt file and
mailbagitwould use those labels instead of the default excluded labels. Just a line separated list could be fine across platforms just by relying in Python's.readlines()or similar. That way the line breaks should be whatever is native to the OS.Relevant part of mailbag spec?
N/A
Type of component
Expected contribution
Major challenges or things to keep in mind
Inconsistent arrangement structure is the worst.