Paul Hammant's Blog: Scraping Leankit for analytics and pretty urls

Of course, all story/issue trackers should store their canonical data as JSON and in source-control, but I’ve sadly not managed to convince any of the vendors yet. I’ve been playing with Leankit recently, and some of the visuals are alluring. A peeve though is the lack of access to my data (JSON, source-control as mentioned) and the lack of pretty incoming URLs like JIRA (and others) have.

I can fix both, from the comfort of my couch :)

Installs

brew install jq
brew install jshon
pip install csvkit

The scripts below will extract CSV cards, turn them into JSON, and also make redirect links from the user-entered Card-ID to the lengthy Leankit URL. Both the JSON and the redirect shims could be deployed to an Apache-style web server for serving up again for presentation, analyitics, and redirect fun.

scrape_leankit.sh

Change YOURACCOUNTCODE in this script appropriately.

Run like so: ./scrape_leankit.sh you@yourcompany.com yourpassword.

#!/bin/sh

download_cards_and_make_redirect_files () { 
  curl -L -b .headers "https://YOURACCOUNTCODE.leankit.com/Cards/GetCardsFile?BoardId=$1" | csvjson | jshon -S > "$2.json"
  cat "$2.json" | jq '[.[] | {ext: .ExternalCardID, int: .Card_Id}]' | in2csv -f json | ruby write_redirect_html.rb "$1" "$2"
} 

# escape @ char of email to %40
email=${1/@/%40}

#login - store cookies n all to .headers
curl -d "userName=$1&password=$2&rememberMe=true&rememberMe=false&Login=Login&returnUrl=&rememberMe=" --dump-header .headers https://YOURACCOUNTCODE.leankit.com/Account/Membership/Authenticate > /dev/null

# first param, is the LeanKit board number, second is the short code for the board. 
# Do as many boards as you want to process....
download_cards_and_make_redirect_files 123456789 ABCD
download_cards_and_make_redirect_files 112233445 PROJ2
download_cards_and_make_redirect_files 555555555 XYZ

write_redirect_html.rb

Change YOURACCOUNTCODE in this script as above.

require 'fileutils'
$stdin.each_line do |line|
  parts = line.split(',')
  if parts[0] != "" && parts[0] != "ext" then 
    FileUtils.mkdir_p ARGV[1] + "/" + parts[0]
    File.open(ARGV[1] + "/" + parts[0] + "/index.html", 'w') do |file| 
      file.write("<html><head>") 
      file.write("<meta http-equiv=\"refresh\" content=\"0; url=https://YOURACCOUNTCODE.leankit.com/Boards/View/" + ARGV[0] + "/" + parts[1].strip + "\" />")
      file.write("</head></html>\n") 
    end
  end
end

What’s next?

If you were using Leankit for your workplace, you could run the scraper each night and check everything into source-control. The diffs shouldn’t kill your SCM, and if nothing has changed, there’s no commit. Apart from anything else, you could watch the diffs to see how things are progressing in your team(s). I’ve no idea whether that would be workable or not. Can someone report back?

Published

May 2nd, 2015
Reads:

Categories