Paul Hammant's Blog: Reducing My Robo-Emails To A Handful of Rollup Emails Every Few Minutes (IMAP Rewriting With Python)

I have been wanting to release this thing for ages. I have a Python2 script that reads from one throwaway inbox that I use for robot-sent email from online services (a tsunami at times), distills some wisdom from those, and writes/updates a single ‘Rollup’ email in another throwaway email inbox, that I point my phone at. I then configure my phone’s email client to show that Rollup inbox (which only ever has three emails in it), as well as my regular from-humans Gmail inbox.

Here is the repo - ImapDigester on Github. I’ve written this in Python2, and rested on a library called imapclient. I am not as proficient in Python as I am in Java, but this library is more elegant for IMAP handling than the Java equivalents. Yup, this technology is about IMAP processing only, it does no SMTP at all. I’ve deployed it to a $5 Raspberry PI Zero which only does the job of imap-digesting every few minutes as scheduled by cron. It is just headlessly dangling out a wall socket, doing it’s job 24/7.

Emails, if things are happening, hover near the top of the inbox. That is the general rule. They (rollups) get rewritten whether I have read them or not, though there will be a clue as to what I’ve previously looked at:

There’s only ever one email per processor. I only have processors for my credit cards (all credit cards - regardless of provider - in one email), Github notifications fly at you as fast as people update repos. StackExchange filters can otherwise spam you every 15 mins).

Machine Emails This Can Rollup

Citibank, Chase and Amex Credit Cards

Various transaction types from all three of those get consolidates into one rollup email:

I can’t add BarclayCard, CapitalOne, JPMorgan, BofA, Discover and others, as I do not have those cards from those issuers. If I did, they’d get sucked into the same single-email rollup.

A Credit Card rollup is the biggie for me, but I can’t get everything I want for now. None of the three providers that I deal with will keep me updated by default on transactions that are “pending” or “on hold” as they transition to finalized on the statement. I mean you get the provisional amount sent to you in an alert, but nothing after that. Sucks. Id like a non-login URL to programatically click on, from the email, like - http://americanexpress.com/activate_pending_txn_alert/C1CB6635-0D9B-4FEC-86D4-D314B95212B3 - that silently adds one spending item to the set I’d like further updates too. I’d also like a payment ID passed up in all alerts from card issuers. Really - a mimetype addition to the set multi-part parts that just gives me JSON for all the fields, so I don’t have to snip them out the page.

Github Notifications (From Repo’s You’re Watching)

I used Homebrew to test this - jeez those guys and their community are trying to break Github. That’s a lot of notification emails, and I would not want that it reality, so I’ll change my watching vs not watching when I’m comfortable with the functionality I am getting. For now though, the seen/unseen boundary in my Github Rollup (I scrolled down to this):

(Microsoft’s Outlook client on iOS)

Pull-requests, Issues, and comments outside of those are ‘topics’. Everything for one topic is always together. A topic can comprise bits you seen (unbolded) and bits you have not seen yet (bolded). Topics rise to the top based on the activity of the conversations on them.

I would like it if Github could reach into my inbox (securely/safely) and rewrite the rollup themselves.

Stack Exchange Questions You’re Interested In

You can setup more than one filter with Stack Exchange, and each of them will make a rollup email.

Note: because of design choice from the StackExchange people, you have to have at least two tags to filter on, so that the message will be recognized by this script. ImapDigester could be changed to work with StackExchanges inconsistent subject-line design, but that’s a job for another day.

The implementation of this is that most recent is always at the top. There’s no parsing - the div for each StackExchange notification is just snipped out of the HTML-email they sent, and recomposed in a longer on that’s the rollup. For testing purposes I’ve subscribed one StackExchange filter to “java” + “java-8” , and another one to “angularjs-directive” “angularjs”.

This one is less interesting than the Github notification one. As before, I would like it if StackExchange could securely reach into my inbox and rewrite the rollup themselves.

Behaviors That Are Non Obvious

The Email’s “SEEN” Flag.

As the emails are rewritten, the daemon that does that detects whether you’ve read the previous one or not. If you have it moved a divisor up show the limit to which you’ve previously been reading. Yes, that even works through multiple email rewrites for situations where you were not watching.

Deletion of Rollup Email

If you delete a rollup email, you’re indicating to the ImapDigester daemon that everything in it has not only be read, but can be omitted from future rollups. This is imperfect - probably for financial transactions you’d prefer a simple three-month limit on what’s shown regardless of deletion of the rollup email.

Comparison to Unnroll.Me and SaneBox

I have not tried either of these. Sadly I only discovered them as I prepped this blog entry :-)

Unroll.me is different in that it focuses on the inbox you already have, and does a daily rollup email. ImapDigester constantly rewrites the rollup email(s), pays attention to the read/unread flag of each and knows the deletion of a rollup email has a meaning. To use ImapDigester you need each service provider to send emails to your secret for-robots email account. It helps if the service has an account email address and a separate notification email address.

Sanebox similarly reaches into your mail email account’s inbox and moves things around.

I’m not brave enough to run ImapDigester on my mail (gmail) mail account, and I’m only running every 5 mins, so there’s a chance that email could languish in my inbox before ImapDigester would pull it out for digesting.

What in the future?

I’d email accounts with sub-accounts that would allow an online service (like github) to reach into an IMAP account of my choosing and rewrite their own rollup email. Ideally that would be via a separate id/password for a sub-account would limit the emails that the service provider can rewrite to just the ones they wrote themselves. Meanwhile, underneath that I’d be able to see all rollup emails in an email client app as usual.

Related Postings

Published

April 4th, 2016
Reads:

Categories