Capone

Never let your books land you in the pen.

Capone is a library that provides double-entry bookkeeping (the foundation of all modern accounting) for Django with the ability to link each recorded transaction to zero or more other Django models as evidence for that transaction.

Introduction

In double-entry bookkeeping (DEB), all recordable events (purchases, sales, equipment depreciation, bad debt markdowns, etc.) are tracked as "ledger entries" or "transactions" in "ledgers". Each ledger entry is made up of one or more "credit" and one or more "debit" entries. For the sake of this brief example, you can think of credits as increasing the amount of money recorded in a ledger and a debit decreasing it. With that assumption, the central idea behind double-entry bookkeeping is that the sum of every ledger entry's debits must equal the sum of its credits. capone implements a double-entry bookkeeping system by providing an API for checking that all created entries satisfy this condition or rolling back the transaction if not.

In addition to this standard bookkeeping functionality, capone also allows any number of arbitrary objects to be attached, via generic foreign keys, to a ledger entry as "evidence" for that transaction's having happened. For instance, a transaction recording a bank deposit paying for several medical tests at a time from an insurance company to your medical testing company could be linked to the original Order objects that recorded the test. capone also provides an API for the efficient querying of ledger entries by evidence.

For more information on the concept of double-entry bookkeeping itself, we recommend the Wikipedia article: https://en.wikipedia.org/wiki/Double-entry_bookkeeping_system.

Local Development

Setup:

First, you must set up your working environment:

make setup

This will build a local virtualenv and all other requirements for local development.

Running Commands:

The following commands are available for interacting with the app:

To start a shell instance so that you can interact with the app via the ORM:

make shell

Note: before any of the following instructions, you may have to run make develop to set up a postgres database for this app.

First, activate a virtualenv so that your commands have access to the environment built by make setup:

From the repository root, run:

source .venv/bin/activate

Then you should be free to run

./manage.py makemigrations --settings=capone.tests.settings

or any other manage.py command, even those in the Makefile.

To run individual tests, use the following:

./manage.py test --settings=capone.tests.settings capone.tests

Notice the --settings=capone.tests.settings argument: because this repository is a django sub-module, it wouldn't make sense for it to come with its own default settings.py file. Instead, it ships with one used to run its tests. To use manage.py, we have to pass an import path to the settings file explicitly.

Models

Let's introduce the models provided by capone and how they relate to one another.

Note that all objects in this library have created_at and modified_at fields that are auto_now_add and auto_now, respectively.

Accounting Models

The models in this section are those that correspond most to well known accounting concepts, i.e. those involved in keeping accounts using the principles of double-entry bookkeeping. They model ledgers, journal entries, credits and debits, and any metadata one wishes to store with these objects.

Ledger

A Ledger is the top-most level of organization of information in double-entry bookkeeping as well as the capone app. Most ledgers have names familiar to those with any knowledge of accounting, such as "revenue" or "accounts receivable".

Ledgers are synonymous with the accounting concept of an "account", so you may see references to accounts in this documentation or elsewhere in the accounting literature.

As a data structure, a Ledger in this library is little more than a name, description, and unique number: LedgerEntries (see below) point to a Ledger to represent their being "in" a Ledger. Transactions (see below also) that are "between" two Ledgers have a LedgerEntry pointing to one Ledger and another LedgerEntry pointing to the other Ledger.

increased_by_debits

Ledger also has the sometimes confusing field increased_by_debits. All Ledgers are of one of two types: either debits increase the "value" of an account or credits do. By convention, asset and expense accounts are of the former type, while liabilities, equity, and revenue are of the latter: in short, an increase to an "asset"-type account is a debit, and an increase to a "liability" or "equity"-type account is a credit.

Here's a handy mnemonic for the two types of accounts: The accounting equation says (by definition) that:

assets == liabilities + owner equity

The terms on the right of the equals sign are increased by debits, and terms on the left of the equals sign are decreased by debits. We can therefore use the accounting equation to know whether to use debits or credits to model an increase in a ledger.

So because debits and credits mean different things in different types of accounts, we can have a transaction with an "equal and opposite" credit and debit pair of the same currency amount, but that still represents a net increase in the value of a company: a debit in Accounts Receivable and a credit in Revenue increases both accounts while satisfying the accounting equation.

Currently, field increased_by_debits is not used by the code in capone but is provided as a convenience to users who might wish to incorporate this information into an external report or calculation.

Transaction

A Transaction is a record of a discrete financial action, represented by a collection of debits and credits whose sums equal one another. Practically all models in capone link to or through Transaction: in a sense you could say it's the main model provided by capone. A Transaction can sometimes be referred to as a "journal entry".

The Transaction model records debits and credits by linking to LedgerEntries, which include currency amounts of the proper sign, and those LedgerEntries themselves point to Ledger. In other words, Transaction and Ledger are linked in a many-to-many fashion by going through LedgerEntry as a custom through model. The "proper sign" part is taken care of by the credit and debit convenience methods (see examples below).

Transactions should never be deleted. Instead, a new Transaction with debits and credits swapped should be created using capone.api.actions.void_transaction to negate the effect of the Transaction you'd like to remove. The voids field on the new Transaction will automatically be filled in with the old Transaction you wish to remove. By this method, you'll never have to delete data from your system as a part of normal operation, which mimics one of the many benefits of traditional, non-computerized double-entry bookkeeping.

Transaction also has the following fields to provide metadata for each transaction:

• created_by: The user who created this Transaction.
• notes: A free-form text field for adding to a Transaction any information not expressed in the numerous metadata fields.
• posted_timestamp: The time a Transaction should be considered valid from. capone.api.actions.create_transaction automatically deals with filling in this value with the current time. You can change this value to post-date or back-date Transactions because created_at will always represent the true object creation time.
• transaction_id: A Universally Unique Identifier (UUID) for the Transaction, useful for unambiguously referring to a Transaction without using primary keys or other database internals.
• type: A user-defined type for the Transaction (see the TransactionType model below).

TransactionType

A TransactionType is a user-defined, human-readable "type" for a Transaction, useful for sorting, aggregating, or annotating Transactions. The default TransactionType is MANUAL, which is created automatically by the library, but you can define others, say for bots or certain classes of users.

Currently, TransactionType is not used by the code in capone but is provided as a convenience to users who might wish to incorporate this information into an external report or calculation.

LedgerEntry

LedgerEntries represent single debit or credit entries in a single Ledger. LedgerEntries are grouped together into Transactions (see above) with the constraint that the sum of all credit and debit LedgerEntries for a given Transaction must equal zero.

LedgerEntries have a field entry_id, which is a UUID for unambiguously referring to a single LedgerEntry.

Evidence Models

The models in this section deal with adding evidence to Transactions and searching over that evidence.

TransactionRelatedObject

A TransactionRelatedObject (TRO) represents the "evidence" relationship that makes the capone library more useful. A TRO links a Transaction to an arbitrary object in the larger app that this library is used in using a generic foreign key. One TRO links one Transaction and one arbitrary object, so we make as many TROs as we want pieces of evidence. There are several convenience methods in capone.api.queries for efficiently querying over Transactions based on evidence and evidence objects based on their Transactions (see examples below).

LedgerBalance

A LedgerBalance is similar to a TRO in that it allows linking ledger objects with objects from the wider app that the library is used in via generic foreign keys. The purpose of LedgerBalance is to denormalize for more efficient querying the current sum of debits and credits for an object in a specific Ledger. Therefore, there is only one LedgerBalance for each (ledger, related_object) tuple.

You should never have to manually create or edit a LedgerBalance: doing so, as well as keeping them up-to-date, is handled by capone internals. For the same reasons, deleting them is not necessary or a good idea.

The purpose of LedgerBalance can best be demonstrated by considering the deceptively simple query, "how many Orders (a non-capone model we presumably created in the app where we include capone as a library) have an Accounts Receivable balance greater than zero?" One would have to calculate the ledger balance over literally the product of all ledgers and all non-capone objects in the database, and then filter them for all those with balances above zero, to answer this question, which is obviously too expensive. By keeping track of the per-Ledger balance for each object used as evidence in a Transaction, we can much more easily make these queries with reasonable overhead.

Usage

Creating Ledgers

Let's start by creating two common ledger types, "Accounts Receivable" and "Revenue", which usually have transactions between themselves:

>>> from capone.models import Ledger
>>> ar = Ledger.objects.create(name='Accounts Receivable', number=1, increased_by_debits=True)
<Ledger: Ledger Accounts Receivable>
>>> revenue = Ledger.objects.create(name='Revenue', number=2, increased_by_debits=True)
<Ledger: Ledger Revenue>

Both of these accounts are asset accounts, so they're both increased by debits. Please consult the double-entry bookkeeping Wikipedia article or the explanation for increased_by_debits above for a more in-depth explanation of the "accounting equation" and whether debits increase or decrease an account.

Also, note that the default convention in capone is to store debits as positive numbers and credits as negative numbers. This convention is common but completely arbitrary. If you want to switch the convention around, you can set DEBITS_ARE_NEGATIVE to True in your settings.py file. By default, that constant doesn't need to be defined, and if it remains undefined, capone will interpret its value as False.

Faking Evidence Models

Now let's create a fake Order, so that we have some evidence for these ledger entries, and a fake User, so we'll have someone to blame for these transactions:

>>> from capone.tests.factories import OrderFactory
>>> order = OrderFactory()
>>> from capone.tests.factories import UserFactory
>>> user = UserFactory()

Creating Transactions

We're now ready to create a simple transaction:

>>> from capone.api.actions import create_transaction
>>> from capone.api.actions import credit
>>> from capone.api.actions import debit
>>> from decimal import Decimal
>>> from capone.models import LedgerEntry
>>> txn = create_transaction(user, evidence=[order], ledger_entries=[LedgerEntry(amount=debit(Decimal(100)), ledger=ar), LedgerEntry(amount=credit(Decimal(100)), ledger=revenue)])
>>> txn.summary()
{
    u'entries': [
        'LedgerEntry: $100.0000 in Accounts Receivable',
        'LedgerEntry: $-100.0000 in Revenue',
    ],
    u'related_objects': [
        'TransactionRelatedObject: Order(id=1)',
    ]
}

Note that we use the helper functions credit and debit with positive numbers to keep the signs consistent in our code. There should be no reason to use negative numbers with capone.

Note also that the value for the credit and debit is the same: $100. If we tried to create a transaction with mismatching amounts, we would get an error:

>>> create_transaction(user, evidence=[order], ledger_entries=[LedgerEntry(amount=debit(Decimal(100)), ledger=ar), LedgerEntry(amount=credit(Decimal(101)), ledger=revenue)])
---------------------------------------------------------------------------
TransactionBalanceException               Traceback (most recent call last)

[...]

TransactionBalanceException: Credits do not equal debits. Mis-match of -1.

So the consistency required of double-entry bookkeeping is automatically kept.

There are many other options for create_transaction: see below or its docstring for details.

Ledger Balances

capone keeps track of the balance in each ledger for each evidence object in a denormalized and efficient way. Let's use this behavior to get the balances of our ledgers as well as the balances in each ledger for our order object:

>>> from capone.api.queries import get_balances_for_object

>>> get_balances_for_object(order)
defaultdict(<function <lambda> at 0x7fd7ecfa96e0>, {<Ledger: Ledger Accounts Receivable>: Decimal('100.0000'), <Ledger: Ledger Revenue>: Decimal('-100.0000')})

>>> ar.get_balance()
Decimal('100.0000')

>>> revenue.get_balance()
Decimal('-100.0000')

Voiding Transactions

We can also void that transaction, which enters a transaction with the same evidence but with all values of the opposite sign:

>>> from capone.api.actions import void_transaction
>>> void = void_transaction(txn, user)
<Transaction: Transaction 9cd85014-c588-43ff-9532-a6fc2429069e>

>>> void_transaction(txn, user)
---------------------------------------------------------------------------
UnvoidableTransactionException            Traceback (most recent call last)

[...]

UnvoidableTransactionException: Cannot void the same Transaction #(e0842107-3a5b-4487-9b86-d1a5d7ab77b4) more than once.

>>> void.summary()
{u'entries': ['LedgerEntry: $-100.0000 in Accounts Receivable',
  'LedgerEntry: $100.0000 in Revenue'],
 u'related_objects': ['TransactionRelatedObject: Order(id=1)']}

>>> txn.voids

>>> void.voids
<Transaction: Transaction e0842107-3a5b-4487-9b86-d1a5d7ab77b4>

Note the new balances for evidence objects and Ledgers:

>>> get_balances_for_object(order)
defaultdict(<function <lambda> at 0x7fd7ecfa9758>, {<Ledger: Ledger Accounts Receivable>: Decimal('0.0000'), <Ledger: Ledger Revenue>: Decimal('0.0000')})

>>> ar.get_balance()
Decimal('0.0000')

>>> revenue.get_balance()
Decimal('0.0000')

Transaction Types

You can label a Transaction using a foreign key to the TransactionType to, say, distinguish between manually made Transactions and those made by a bot, or between Transactions that represent two different types of financial transaction, such as "Reconciliation" and "Revenue Recognition".

By default, Transactions are of a special, auto-generated "manual" type:

>>> txn.type
<TransactionType: Transaction Type Manual>

but you can create and assign TransactionTypes when creating Transactions:

>>> from capone.models import TransactionType
>>> new_type = TransactionType.objects.create(name='New type')
>>> txn = create_transaction(user, evidence=[order], ledger_entries=[LedgerEntry(amount=debit(Decimal(100)), ledger=ar), LedgerEntry(amount=credit(Decimal(100)), ledger=revenue)], type=new_type)
>>> txn.type
<TransactionType: Transaction Type New type>

Querying Transactions

Getting Balances

Transaction has a summary method to summarize the data on the many models that can link to it:

>>> txn.summary()
{u'entries': ['LedgerEntry: $100.0000 in Accounts Receivable',
  'LedgerEntry: $-100.0000 in Revenue'],
 u'related_objects': ['TransactionRelatedObject: Order(id=1)']}

To get the balance for a Ledger, use its get_balance method:

>>> ar.get_balance()
Decimal('100.0000')

To efficiently get the balance of all transactions with a particular object as evidence, use get_balances_for_objects:

>>> get_balances_for_object(order)
defaultdict(<function <lambda> at 0x7fd7ecfa9230>, {<Ledger: Ledger Accounts Receivable>: Decimal('100.0000'), <Ledger: Ledger Revenue>: Decimal('-100.0000')})

Transactions are validated before they are created, but if you need to do this manually for some reason, use the validate_transaction function, which has the same prototype as create_transaction:

>>> validate_transaction(user, evidence=[order], ledger_entries=[LedgerEntry(amount=debit(Decimal(100)), ledger=ar), LedgerEntry(amount=credit(Decimal(100)), ledger=revenue)], type=new_type)
>>> validate_transaction(user, evidence=[order], ledger_entries=[LedgerEntry(amount=debit(Decimal(100)), ledger=ar), LedgerEntry(amount=credit(Decimal(101)), ledger=revenue)], type=new_type)
---------------------------------------------------------------------------
TransactionBalanceException               Traceback (most recent call last)
<ipython-input-64-07b6d139bb37> in <module>()
----> 1 validate_transaction(user, evidence=[order], ledger_entries=[LedgerEntry(amount=debit(Decimal(100)), ledger=ar), LedgerEntry(amount=credit(Decimal(101)), ledger=revenue)], type=new_type)

/home/hunter/capone/capone/api/queries.pyc in validate_transaction(user, evidence, ledger_entries, notes, type, posted_timestamp)
     67     if total != Decimal(0):
     68         raise TransactionBalanceException(
---> 69             "Credits do not equal debits. Mis-match of %s." % total)
     70
     71     if not ledger_entries:

TransactionBalanceException: Credits do not equal debits. Mis-match of -1.

Queries

Along with the query possibilities from the Django ORM, capone provides Transaction.filter_by_related_objects for finding Transactions that are related to certain models as evidence.

>>> Transaction.objects.count()
5

>>> Transaction.objects.filter_by_related_objects([order]).count()
5

>>> order2 = OrderFactory()

>>> create_transaction(user, evidence=[order2], ledger_entries=[LedgerEntry(amount=debit(Decimal(100)), ledger=ar), LedgerEntry(amount=credit(Decimal(100)), ledger=revenue)])
<Transaction: Transaction 68a4adb1-b898-493f-b5f3-4fe7132dd28d>

>>> Transaction.objects.filter_by_related_objects([order2]).count()
1

filter_by_related_objects is defined on a custom QuerySet provided for Transaction, so calls to it can be chained like ordinary QuerySet function calls:

>>> create_transaction(user, evidence=[order2], ledger_entries=[LedgerEntry(amount=debit(Decimal(100)), ledger=ar), LedgerEntry(amount=credit(Decimal(100)), ledger=revenue)])
<Transaction: Transaction 92049712-4982-4718-bc71-a405b0d762ac>

>>> Transaction.objects.filter_by_related_objects([order2]).count()
2

>>> Transaction.objects.filter_by_related_objects([order2]).filter(transaction_id='92049712-4982-4718-bc71-a405b0d762ac').count()
1

filter_by_related_objects takes an optional match_type argument, which is of type MatchType(Enum) that allows one to filter in different ways, namely whether the matching transactions may have "any", "all", "none", or "exactly" the evidence provided, determined by MatchTypes ANY, ALL, NONE, and EXACT, respectively.

Asserting over Transactions

For writing tests, the method assert_transaction_in_ledgers_for_amounts_with_evidence is provided for convenience. As its name implies, it allows asserting the existence of exactly one Transaction with the ledger amounts, evidence, and other fields on Ledger provided to the method.

>>> create_transaction(user, evidence=[order], ledger_entries=[LedgerEntry(amount=debit(Decimal(100)), ledger=ar), LedgerEntry(amount=credit(Decimal(100)), ledger=revenue)])
<Transaction: Transaction b3e73f1d-6b10-4597-b19b-84800839d5b3>
>>> with assert_raises(Transaction.DoesNotExist):
...     assert_transaction_in_ledgers_for_amounts_with_evidence(ledger_amount_pairs=[(revenue.name, credit(Decimal(100))), (ar.name, debit(Decimal(100)))], evidence=[])
...
>>> assert_transaction_in_ledgers_for_amounts_with_evidence(ledger_amount_pairs=[(revenue.name, credit(Decimal(100))), (ar.name, debit(Decimal(100)))], evidence=[order])
>>> with assert_raises(Transaction.DoesNotExist):
...     assert_transaction_in_ledgers_for_amounts_with_evidence(ledger_amount_pairs=[(revenue.name, credit(Decimal(100))), (ar.name, debit(Decimal(100)))], evidence=[order])
...
Traceback (most recent call last):
  File "<console>", line 2, in <module>
    File "/usr/lib/python2.7/unittest/case.py", line 116, in __exit__
        "{0} not raised".format(exc_name))
        AssertionError: DoesNotExist not raised

You can see capone.tests.test_assert_transaction_in_ledgers_for_amounts_with_evidence for more examples!

Image Credits

Image courtesy Officer on Wikipedia. This work was created by a government unit (including state, county, and municipal government agencies) of the U.S. state of Florida. It is a public record that was not created by an agency which state law has allowed to claim copyright and is therefore in the public domain in the United States.