Skip to content

Commit

Permalink
Fix lack of idempotency on bump_iap_set_expiry
Browse files Browse the repository at this point in the history
This commit fixes an important bug in the logic of bump_iap_set_expiry.
I discovered that it lacks idempotency when sending the same IAP (In-app purchase) receipt more than once.

It would cause the end user to end up with a higher expiry date than
entitled to them.

Since there was no way to create a logic that solves this by only using
a simple expiry date number, the system was changed to keep track of
transactions, and then calculate the expiry date on the fly.

Furthermore, these changes improve the overall system as mistakes such
as this one become more easily fixable, as the transaction data is a
much better source of truth from which we can calculate expiry.

Multiple changes were made to support this logical and structural change:
1. A new set of functions were introduced to perform the calculation in O(N) time
2. New tests and test cases were introduced to cover both logic and the original bug
3. bump_expiry and bump_iap_set_expiry were replaced by a single function that appends a transaction to the user's transaction list
4. A new standard transaction data type was introduced
5. Low-level functions that fetch and write user data on the database were changed to calculate or handle expiry dates on the fly to reduce other API changes to a minimum
6. IAP and LN successful purchase logics were modified to generate transaction data for the transaction history.
7. Logic was added to handle the old legacy expiry date data as a "legacy" transaction.

Testing 1
----------

PASS

Made sure all automated tests are passing (They are quite extensive and realistic at this point)

Testing 2
----------

PASS

Devices: iPhone 13 Mini (physical device), iPhone 15 simulator
iOS: 17.3.1 and 17.2
damus-api: This commit
Damus: Tip of master (5e530bfc9c584dcb93729b6a21863c97125ffb21, very close to 1.8(1))
Damus-website: 927bf8bacae71d59ab0c5f2386e1c50b8e172790 (Equivalent to the one released with 1.7.2)
Setup: Local server setup, App Store sandbox environment.
Coverage:
1. Loaded a copy of the production DB on the local test environment, and verified:
  1. A lot of the users I knew had Damus Purple membership still had their star next to them after loading the new version.
  2. My own account was displaying correct information
  3. I still saw reminder notifications to renew as expected
  4. Going through LN checkout worked and lead to the correct expiry date to be shown
  5. Going through the LN checkout a second time worked and lead to the expected extension of expiry
  6. Saw no crashes on the server
  7. Saw no lags on the server
2. Loaded an older test database I used a while ago with an older version
  1. My own account was displaying correct information
  2. Tried renewing with an IAP purchase via Sandbox. It worked as expected
  3. Expiry date is as expected

Changelog-Fixed: Fixed IAP receipt verification idempotency problems
Changelog-Changed: Changed structure of expiry to be a function of transaction data
Signed-off-by: Daniel D’Aquino <[email protected]>
Signed-off-by: William Casarin <[email protected]>
  • Loading branch information
danieldaquino authored and jb55 committed Mar 12, 2024
1 parent 8e7f076 commit 346acfb
Show file tree
Hide file tree
Showing 9 changed files with 736 additions and 92 deletions.
56 changes: 39 additions & 17 deletions src/app_store_receipt_verifier.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@
/**
* @typedef {import('@apple/app-store-server-library').JWSTransactionDecodedPayload} JWSTransactionDecodedPayload
*/
/**
* @typedef {import('./transaction_management.js').Transaction} Transaction
*/

const { AppStoreServerAPIClient, Environment, ReceiptUtility, Order, ProductType, SignedDataVerifier } = require("@apple/app-store-server-library")
const { current_time } = require("./utils")
Expand All @@ -14,15 +17,22 @@ const debug = require('debug')('api:iap')
*
* @param {string} receipt_data - The receipt data to verify in base64 format.
* @param {string} authenticated_account_token - The UUID account token of the user who is authenticated in this request.
*
* @returns {Promise<number|null>} The expiry date of the receipt if valid, null otherwise.
*
* @returns {Promise<Transaction[]|null>} The validated transactions
*/
async function verify_receipt(receipt_data, authenticated_account_token) {
debug("Verifying receipt with authenticated account token: %s", authenticated_account_token);
// Mocking logic for testing purposes
if (process.env.MOCK_VERIFY_RECEIPT == "true") {
debug("Mocking verify_receipt with expiry date 30 days from now");
return current_time() + 60 * 60 * 24 * 30;
return [{
type: "iap",
id: "1",
start_date: current_time(),
end_date: current_time() + 60 * 60 * 24 * 30,
purchased_date: current_time(),
duration: null
}]
}

// Setup the environment and client
Expand All @@ -37,25 +47,32 @@ async function verify_receipt(receipt_data, authenticated_account_token) {

// If the transaction ID is present, fetch the transaction history, verify the transactions, and return the latest expiry date
if (transactionId != null) {
return await fetchLastVerifiedExpiryDate(client, transactionId, rootCaDir, environment, bundleId, authenticated_account_token);
return await fetchValidatedTransactions(client, transactionId, rootCaDir, environment, bundleId, authenticated_account_token);
}
return Promise.resolve(null);
}

/**
* Verifies the transaction id and returns the expiry date if the transaction is valid.
* Verifies the transaction id and returns the verified transaction history if valid
*
* @param {number} transaction_id - The transaction id to verify.
* @param {string} authenticated_account_token - The UUID account token of the user who is authenticated in this request.
*
* @returns {Promise<number|null>} The expiry date of the receipt if valid, null otherwise.
* @returns {Promise<Transaction[]|null>} The validated transactions
*/
async function verify_transaction_id(transaction_id, authenticated_account_token) {
debug("Verifying transaction id '%d' with authenticated account token: %s", transaction_id, authenticated_account_token);
// Mocking logic for testing purposes
if (process.env.MOCK_VERIFY_RECEIPT == "true") {
debug("Mocking verify_receipt with expiry date 30 days from now");
return current_time() + 60 * 60 * 24 * 30;
return [{
type: "iap",
id: "1",
start_date: current_time(),
end_date: current_time() + 60 * 60 * 24 * 30,
purchased_date: current_time(),
duration: null
}];
}

// Setup the environment and client
Expand All @@ -66,14 +83,14 @@ async function verify_transaction_id(transaction_id, authenticated_account_token

// If the transaction ID is present, fetch the transaction history, verify the transactions, and return the latest expiry date
if (transaction_id != null) {
return await fetchLastVerifiedExpiryDate(client, transaction_id, rootCaDir, environment, bundleId, authenticated_account_token);
return await fetchValidatedTransactions(client, transaction_id, rootCaDir, environment, bundleId, authenticated_account_token);
}
return Promise.resolve(null);
}


/**
* Fetches transaction history with the App Store API, verifies the transactions, and returns the last valid expiry date.
* Fetches transaction history with the App Store API, verifies the transactions, and returns formatted transactions.
* It also verifies if the transaction belongs to the account who made the request.
*
* @param {AppStoreServerAPIClient} client - The App Store API client.
Expand All @@ -82,10 +99,10 @@ async function verify_transaction_id(transaction_id, authenticated_account_token
* @param {Environment} environment - The App Store environment.
* @param {string} bundleId - The bundle ID of the app.
* @param {string} authenticatedAccountToken - The UUID account token of the user who is authenticated in this request.
* @returns {Promise<number|null>} The expiry date (As Unix timestamp measured in seconds) of the receipt if valid, null otherwise.
* @returns {Promise<Transaction[]|null>} The validated transactions
*/
async function fetchLastVerifiedExpiryDate(client, transactionId, rootCaDir, environment, bundleId, authenticatedAccountToken) {
async function fetchValidatedTransactions(client, transactionId, rootCaDir, environment, bundleId, authenticatedAccountToken) {
const transactions = await fetchTransactionHistory(client, transactionId);
debug("[Account token: %s] Fetched transaction history for transaction ID: %s; Found %d transactions", authenticatedAccountToken, transactionId, transactions.length);
const rootCAs = readCertificateFiles(rootCaDir);
Expand All @@ -96,11 +113,16 @@ async function fetchLastVerifiedExpiryDate(client, transactionId, rootCaDir, env
if (validDecodedTransactions.length === 0) {
return null;
}
const expiryDates = decodedTransactions.map((decodedTransaction) => decodedTransaction.expiresDate);
debug("[Account token: %s] Found expiry dates: %o", authenticatedAccountToken, expiryDates);
const latestExpiryDate = Math.max(...expiryDates);
debug("[Account token: %s] Latest expiry date: %d", authenticatedAccountToken, latestExpiryDate);
return latestExpiryDate / 1000; // Return the latest expiry date in seconds
return validDecodedTransactions.map((decodedTransaction) => {
return {
type: "iap",
id: decodedTransaction.transactionId,
start_date: decodedTransaction.purchaseDate / 1000,
end_date: decodedTransaction.expiresDate / 1000,
purchased_date: decodedTransaction.purchaseDate / 1000,
duration: null
}
})
}

/**
Expand Down
17 changes: 13 additions & 4 deletions src/invoicing.js
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
const LNSocket = require('lnsocket')
const { bump_expiry } = require('./user_management')
const { add_successful_transactions_to_account } = require('./user_management')
const { nip19 } = require('nostr-tools')
const { v4: uuidv4 } = require('uuid')
const { current_time } = require('./utils')

const PURPLE_ONE_MONTH = "purple_one_month"
const PURPLE_ONE_YEAR = "purple_one_year"
Expand Down Expand Up @@ -141,7 +142,7 @@ class PurpleInvoiceManager {
if (checkout_object?.invoice) {
checkout_object.invoice.paid = await this.check_invoice_is_paid(checkout_object.invoice.label)
if (checkout_object.invoice.paid) {
this.handle_successful_payment(checkout_object.invoice.bolt11)
this.handle_successful_payment(checkout_object)
checkout_object.completed = true
await this.checkout_sessions_db.put(checkout_id, checkout_object) // Update the checkout object since the state has changed
}
Expand Down Expand Up @@ -174,7 +175,8 @@ class PurpleInvoiceManager {
}

// This is called when an invoice is successfully paid
async handle_successful_payment(bolt11) {
async handle_successful_payment(checkout_object) {
const bolt11 = checkout_object.invoice.bolt11;
const invoice_request_info = this.invoices_db.get(bolt11)
if (!invoice_request_info) {
throw new Error("Invalid bolt11 or not found")
Expand All @@ -183,7 +185,14 @@ class PurpleInvoiceManager {
this.invoices_db.put(bolt11, invoice_request_info)
const npub = invoice_request_info.npub
const pubkey = nip19.decode(npub).data.toString('hex')
const result = bump_expiry(this.api, pubkey, this.invoice_templates[invoice_request_info.template_name].expiry)
const result = add_successful_transactions_to_account(this.api, pubkey, [{
type: "ln",
id: checkout_object.id,
start_date: null,
end_date: null,
purchased_date: current_time(),
duration: this.invoice_templates[invoice_request_info.template_name].expiry
}]);
if (!result.account) {
throw new Error("Could not bump expiry")
}
Expand Down
18 changes: 9 additions & 9 deletions src/router_config.js
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
const { json_response, simple_response, error_response, invalid_request, unauthorized_response } = require('./server_helpers')
const { create_account, get_account_info_payload, check_account, get_account, put_account, get_account_and_user_id, get_user_uuid, bump_iap_set_expiry, delete_account } = require('./user_management')
const { create_account, get_account_info_payload, check_account, get_account, put_account, get_account_and_user_id, get_user_uuid, delete_account, add_successful_transactions_to_account } = require('./user_management')
const handle_translate = require('./translate')
const { verify_receipt, verify_transaction_id } = require('./app_store_receipt_verifier');
const bodyParser = require('body-parser')
Expand Down Expand Up @@ -110,14 +110,14 @@ function config_router(app) {
unauthorized_response(res, 'The account UUID is not valid for this account. Expected: "' + account_uuid + '", got: "' + alleged_account_uuid + '"')
return
}
let expiry_date = await verify_receipt(receipt_base64, account_uuid)
if (!expiry_date) {

let verified_transaction_history = await verify_receipt(receipt_base64, account_uuid)
if (!verified_transaction_history) {
unauthorized_response(res, 'Receipt invalid')
return
}
const { account: new_account, request_error } = bump_iap_set_expiry(app, req.authorized_pubkey, expiry_date)

const { account: new_account, request_error } = add_successful_transactions_to_account(app, req.authorized_pubkey, verified_transaction_history)
if (request_error) {
error_response(res, request_error)
return
Expand Down Expand Up @@ -169,13 +169,13 @@ function config_router(app) {
return
}

let expiry_date = await verify_transaction_id(transaction_id, account_uuid)
if (!expiry_date) {
let verified_transaction_history = await verify_transaction_id(transaction_id, account_uuid)
if (!verified_transaction_history) {
unauthorized_response(res, 'Transaction ID invalid')
return
}

const { account: new_account, request_error } = bump_iap_set_expiry(app, req.authorized_pubkey, expiry_date)
const { account: new_account, request_error } = add_successful_transactions_to_account(app, req.authorized_pubkey, verified_transaction_history)
if (request_error) {
error_response(res, request_error)
return
Expand Down
123 changes: 123 additions & 0 deletions src/transaction_management.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
"use strict";
// @ts-check


/** @typedef {Object} Transaction
* @property {"iap" | "ln" | "legacy"} type - The type of transaction
* @property {string} id - The id of the transaction (Apple IAP transaction ID for iap, Checkout ID for ln, `0` for legacy)
* @property {number | null} start_date - The start date of the transaction if transaction has a fixed start and end (IAP and legacy), null otherwise (Unix timestamp in seconds)
* @property {number | null} end_date - The end date of the transaction if transaction has a fixed start and end (IAP and legacy), null otherwise (Unix timestamp in seconds)
* @property {number | null} purchased_date - The date of the transaction (Applies to LN and IAP only, Unix timestamp in seconds)
* @property {number | null} duration - The duration of the transaction (Applies to LN only, measured in seconds)
*/


/** Calculates the expiry date given a transaction history
* @param {Transaction[]} transaction_history - The transaction history
* @returns {number | null} - The expiry date of the transaction history (Unix timestamp in seconds), null if no expiry date can be calculated
*/
function calculate_expiry_date_from_history(transaction_history) {
// make a deep copy of the transaction history so that we don't modify the original objects
const transaction_history_copy = deep_copy_unique_transaction_history(transaction_history);

// sort the transaction history by earliest date
var remaining_transactions = transaction_history_copy.sort((a, b) => get_earliest_date_from_transaction(a) - get_earliest_date_from_transaction(b));
var time_cursor = null;
var flexible_credits_remaining = 0;

for (var i = 0; i < remaining_transactions.length; i++) {
var transaction = remaining_transactions[i];

// Move time cursor and count flexible credits available.
if (is_transaction_fixed_schedule(transaction)) {
time_cursor = Math.max(time_cursor, transaction.end_date);
}
else if (is_transaction_flexible_schedule(transaction)) {
flexible_credits_remaining += transaction.duration;
time_cursor = Math.max(time_cursor, transaction.purchased_date);
}

// Check if there is a gap between the time cursor and the next transaction, then consume flexible credits.
if (i < remaining_transactions.length - 1) {
var next_transaction = remaining_transactions[i + 1];
let earliest_date_from_next_transaction = get_earliest_date_from_transaction(next_transaction);
if (time_cursor < earliest_date_from_next_transaction && flexible_credits_remaining > 0) {
flexible_credits_remaining -= Math.min(earliest_date_from_next_transaction - time_cursor, flexible_credits_remaining);
time_cursor = earliest_date_from_next_transaction;
}
}
else {
// This is the last transaction. Consume all remaining flexible credits.
time_cursor += flexible_credits_remaining;
}
}

return time_cursor;
}


/** Checks if a transaction is fixed schedule
* @param {Transaction} transaction - The transaction to be checked
*/
function is_transaction_fixed_schedule(transaction) {
return transaction.start_date !== null && transaction.start_date !== undefined && transaction.end_date !== null && transaction.end_date !== undefined;
}


/** Checks if a transaction is flexible schedule
* @param {Transaction} transaction - The transaction to be checked
*/
function is_transaction_flexible_schedule(transaction) {
return transaction.purchased_date !== null && transaction.purchased_date !== undefined && transaction.duration !== null && transaction.duration !== undefined && !is_transaction_fixed_schedule(transaction);
}


/** Returns the earliest date from a transaction
* @param {Transaction} transaction - The transaction
* @returns {number | null} - The earliest date from the transaction (Unix timestamp in seconds), null if the transaction is null
*/
function get_earliest_date_from_transaction(transaction) {
if (is_transaction_fixed_schedule(transaction)) {
return transaction.start_date;
}
else if (is_transaction_flexible_schedule(transaction)) {
return transaction.purchased_date;
}
else {
return null;
}
}


/** Deep copies a transaction history. More efficient than a JSON parsing roundtrip
* @param {Transaction[]} transaction_history - The transaction history
* @returns {Transaction[]} - The deep copied transaction history
*/
function deep_copy_unique_transaction_history(transaction_history) {
// @type {Transaction[]}
var new_transaction_history = [];
// @type {Set<string>}
var unique_transaction_ids = new Set();
for (var i = 0; i < transaction_history.length; i++) {
const unique_id = transaction_history[i].type + "_" + transaction_history[i].id;
if (unique_transaction_ids.has(unique_id)) {
continue;
}
unique_transaction_ids.add(unique_id);
new_transaction_history.push({
type: transaction_history[i].type,
id: transaction_history[i].id,
start_date: transaction_history[i].start_date,
end_date: transaction_history[i].end_date,
purchased_date: transaction_history[i].purchased_date,
duration: transaction_history[i].duration
});
}

return new_transaction_history;
}


module.exports = {
calculate_expiry_date_from_history, deep_copy_unique_transaction_history
}
Loading

0 comments on commit 346acfb

Please sign in to comment.