Skip to content

Creating a new connector

Jump to bottom
reZach edited this page Apr 24, 2019 · 5 revisions

This page is meant to guide developers creating a connector for My Budget. Connectors are used by My Budget to connect into a particular bank or institution in order to pull transaction data.

The pitch

Since My Budget is free, we can't afford to pay for a service like Plaid to retrieve your transactions through an api. That's still okay - we are able to pull and retrieve transaction details manually through puppeteer. However, the amount of work to add connectors to every bank is too enormous for just me; this guide's goal is to walk you through how to write a connector for a bank you use. Once it's approved and merged into the application, others can use the connector you built (and maintain on a volunteer basis). You'll be added to the maintainers page as well.

Getting started

Click the "fork" button in my repo to fork the repo to your account. Clone a local version of the repo by running these commands. Once you've ran these commands, move on to the next section.

git clone https://github.com/[yourgithubname]/my-budget.git
git remote add upstream https://github.com/reZach/my-budget.git
git checkout -b [branchname]
cd my-budget
npm i
npm run dev

Understand your bank's "flow"

In order to write a connector, it's important to understand how you can access your transactions from your bank. By this statement, I am referring to "what steps does it take for you to log in, and view your transactions from your bank's website?" Take for example the steps it takes to view the transaction data for an account with Discover.

  1. Navigate to https://www.discover.com.
  2. Click on the "Log In" button (this opens a login modal).
  3. Enter User ID.
  4. Enter Password.
  5. Click on the "Log In" button.

(I had to do some navigating around here, but found a transaction search page)

  1. Navigate to https://card.discover.com/cardmembersvcs/statements/app/search.
  2. Click on the Search button.
  3. View transaction details on the page.

Once you understand the flow how to navigate to your transactions, we need to turn these steps into code with puppeteer. Proceed to the next section.

Make necessary prep work/changes

If you've done the steps above, you should have the repo locally on your computer. You'll have to do a number of steps to add your connector code to the solution.

  1. Create a [myname].js file in the app/utils/banks folder. If I was writing a connector for Chase, I would name my file chase.js and the path to this file would be app/utils/banks/chase.js.
  2. Add an entry in the validBanks property in the constructor of app/components/Save/Save.js. The name of this object should be a lowercase name of the bank, ie "chase" and the url property should be the url of the bank except the https://. For Chase bank, my changes would look like this.
validBanks: [
    {
        name: "discover",
        url: "www.discover.com"
    },
    {
        name: "chase",
        url: "www.chase.com"
    }
],

Write puppeteer code

Take the contents of app/utils/banks/template.js and paste it in your file - you can use this as an example, or look at an existing connector to write the puppeteer code. Looking at our existing code for Discover, our puppeteer code will look like this.

await page.goto('https://www.discover.com'); // Step 1
await page.click(".log-in-link[role='button']"); // Step 2
await page.type("#userid", username); // Step 3
await page.type("#password", password); // Step 4
await page.click("#log-in-button"); // Step 5
await page.waitForNavigation(); // necessary after navigating to a new page
await page.goto("https://card.discover.com/cardmembersvcs/statements/app/search"); // Step 6
await page.click("#submit-trigger"); // Step 7
await page.waitForNavigation(); // necessary after navigating to a new page

Write HTML parsing code

Once we've used puppeteer to navigate to our transactions, we must use css selectors to pull out the HTML that contains our transaction data. For Discover as of this writing, the HTML that contains transaction data looks like this. I am able to see this HTML by right-clicking my bank's website and clicking Inspect.

Finding transactions in the HTML

<tbody>
<!-- ... more transactions ... -->
<tr id="transaction-3" title="See Transaction Details" class="transaction-detail-toggler">
        <td class="trans-date" scope="row"><img src="/recent-activity-statements/images/collapsible-arrow-plus-off.png" height="8" width="8" class="transaction-detail-toggler" alt="View Details">04/17/17</td>
<!-- The Description Column is treated like Preformatted text, therefore any whitespace or carriage returns will be taken into effect.  -->
        <td class="desc"><a href="?statementDate=20170516&amp;transaction=3" class="transaction-detail-toggler">BLIZZARD ENT*ONL STORE 800-592-5499 CA<br>BDNLZTZ

</a>
		<!-- display tagList if available ---START--- -->
			<form class="tag-submit" method="post" action="/cardmembersvcs/statements/app/search_results">
				<ul class="transaction-tag-list">
				</ul>
				<input type="hidden" name="tagId" value="">
				<input type="hidden" name="tagsSelector" value="selected">				
			</form>
			<a href="javascript:void(0)" class="see-all hide" aria-expanded="false">See All Tags</a>
	<!-- display tagList if available ---END--- -->
<!-- tm3 -->		</td>
    <td class="ctg" nowrap="nowrap">Merchandise</td>
    <td class="amt">$49.99</td>
    <td class=""><span style="display: none;">$0.00</span></td>
</tr>
<!-- ... more transactions ... -->
</tbody>

For banks that don't allow you to right-click, try capturing the entire page source by using Ctrl + U in Chrome - or other shortcuts in different browsers. For banks that don't work with capturing the page source (ie. if the page displaying the transactions is shown as a result of a form submission, try capturing the entire HTML using the method at the bottom of the page, and then adjusting the css selector accordingly in the following example).

var transactions = await page.evaluate(function(){
    var raw = [];
    console.log(document.querySelector("body"));
    // commented
    return raw;
});

From the above HTML, I identified that I can see the date (04/17/17), transaction name (BLIZZARD ENT*ONL STORE 800-592-5499 CA), category (Merchandise) and amount ($49.99) for a given transaction. Each <tr> also has an id that begins with transaction-, so I'd like to use that in my css selector to be safer that I'll select just the transaction HTML and nothing else.

What ends up being my code after my puppeteer code is this. This code will query the page's HTML and pull out all <tr> nodes with ids that begin with transaction-.

var transactions = await page.evaluate(function(){
    var raw = [];
    document.querySelectorAll("tr[id^=transaction-]")
        .forEach(function(current, index, list){
            raw.push(current.innerHTML);
        }
    );
    return raw;
});

Parse the values with RegExp

Now that we have the HTML that holds our transaction data, we will be using regular expressions to pull out the values we want. I like to use regexr.com for testing regex. As an example, I can use the following regex to pull out the date value for my transaction. (\d{2}\/\d{2}\/\d{2})<\/td>.

Using regex to parse out values

I continue to write regex for the following fields of a transaction.

  • day.
  • month.
  • year.
  • category (optional).
  • subcategory (optional).
  • amount.
  • note (optional).

At the end of my code, I save each transaction in the actualTransactions object that is returned. A good example of how I do this is the Discover connector, found at app/utils/banks/discover.js.

Test

You'll want to test your connector thoroughly before you submit a pull request, so be sure you do these things.

  1. Start the application, using the npm run dev command as shown up above. Click the sync button in the app and type the name of the bank you are adding (it is the same value as the "name" property in the validBanks property). Enter in your user name and password and watch the flow of puppeteer; ensure it is finding transactions in the shortest way possible. (There is no need to navigate between pages if you can simply navigate to a page directly). As long as you get to the import transactions modal, you are good to go.
  2. Make sure your puppeteer code isn't throwing exceptions, be sure to handle errors gracefully.

Make a pull request

If everything passes on your side, run the following commands to merge the latest changes to your fork, the submit a pull request.

git fetch upstream
git merge upstream/[branchname]
git push -u origin [branchname]

Then, create a pull request from your forked repo, and we will review your changes.

Clone this wiki locally