Fixes #3682 - Fix $ne filters incorrectly excluding null values

[joel-rich]

Fixes #3682

As described in the ticket, any "is not" or "not one of" filters currently exclude the specified value(s) and also exclude null values.

Reproduce by filtering All Transactions to exclude any category - all off-budget or categorized transactions also get filtered out.

Given, example list dataset of Items (name, colour)

"Banana", "Yellow"
"Lime", "Yellow"
"Air", null
"Car", "Red"

And query of select * from Items where colour != "Yellow",

Old result set (incorrectly excluding both null and "Yellow" Items):
"Car", "Red"

After my change, the new result set returns all Items which aren't yellow:

"Air", null
"Car", "Red"

---

I wrote a couple tests for namedParameter queries and no existing tests are failing, but even still I'm not overly confident in the changes on lines 709-710 due to it being a weird bit of sql and I'm not a sql expert. I would appreciate it if someone with more sql knowledge confirms it won't have any weird edge-cases I didn't think to test for.

[netlify]

✅ Deploy Preview for actualbudget ready!

Name	Link
🔨 Latest commit	c336e9a
🔍 Latest deploy log	https://app.netlify.com/sites/actualbudget/deploys/6710c225f7a933000862d784
😎 Deploy Preview	https://deploy-preview-3686.demo.actualbudget.org
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[github-actions]

Bundle Stats — desktop-client

Hey there, this message comes from a GitHub action that helps you and reviewers to understand how these changes affect the size of this project's bundle.

As this PR is updated, I'll keep you updated on how the bundle size is impacted.

Total

Files count	Total bundle size	% Changed
9	5.34 MB	0%

Changeset

No files were changed

View detailed bundle breakdown

Added

No assets were added

Removed

No assets were removed

Bigger

No assets were bigger

Smaller

No assets were smaller

Unchanged

Asset	File Size	% Changed
static/js/resize-observer.js	18.37 kB	0%
static/js/BackgroundImage.js	122.29 kB	0%
static/js/indexeddb-main-thread-worker-e59fee74.js	13.5 kB	0%
static/js/usePreviewTransactions.js	1.64 kB	0%
static/js/AppliedFilters.js	21.31 kB	0%
static/js/narrow.js	82.58 kB	0%
static/js/wide.js	243.06 kB	0%
static/js/ReportRouter.js	1.51 MB	0%
static/js/index.js	3.34 MB	0%

[github-actions]

Bundle Stats — loot-core

Hey there, this message comes from a GitHub action that helps you and reviewers to understand how these changes affect the size of this project's bundle.

As this PR is updated, I'll keep you updated on how the bundle size is impacted.

Total

Files count	Total bundle size	% Changed
1	1.27 MB → 1.27 MB (+26 B)	+0.00%

Changeset

File	Δ	Size
packages/loot-core/src/server/aql/compiler.ts	📈 +29 B (+0.08%)	35.36 kB → 35.38 kB

View detailed bundle breakdown

Added

No assets were added

Removed

No assets were removed

Bigger

Asset	File Size	% Changed
kcab.worker.js	1.27 MB → 1.27 MB (+26 B)	+0.00%

Smaller

No assets were smaller

Unchanged

No assets were unchanged

[matt-fidd]

@coderabbitai full review

[coderabbitai]

Walkthrough

The pull request introduces significant updates to the compiler.ts and exec.test.ts files within the loot-core package. In compiler.ts, the compileOp function has been refined to improve handling of the $ne and $notlike operators, specifically addressing null value comparisons. This enhancement ensures that if the left-hand side of a comparison is null, the right-hand side is evaluated correctly, thereby increasing the robustness of query compilations.

In exec.test.ts, the test coverage for the runQuery function has been expanded. New test cases have been added to cover scenarios involving transactions with multiple categories, null parameter handling, and negative category filtering. These updates aim to ensure that the query logic can manage a broader range of inputs and conditions effectively. The function signatures for compileQuery and generateSQLWithState were updated contextually, although no changes in parameters were made.

Assessment against linked issues

Objective	Addressed	Explanation
Ensure net worth calculation reflects filtered categories (#3682)	❌	The changes do not directly address the net worth calculation issue described.

Suggested labels

:sparkles: Merged

---

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL

📥 Commits

Files that changed from the base of the PR and between 02824ad and c336e9a.

⛔ Files ignored due to path filters (1)

• upcoming-release-notes/3686.md is excluded by !**/*.md

📒 Files selected for processing (2)

• packages/loot-core/src/server/aql/compiler.ts (1 hunks)
• packages/loot-core/src/server/aql/exec.test.ts (3 hunks)

🧰 Additional context used

🔇 Additional comments (6)

packages/loot-core/src/server/aql/exec.test.ts (4)

170-170: LGTM: Additional category for comprehensive testing

Adding a new category 'cat2' enhances the test coverage by allowing scenarios with multiple categories. This aligns well with the PR objective of improving filtering for transactions with multiple categories.

---

183-188: LGTM: New transaction for multi-category testing

The addition of a transaction with 'cat2' category complements the new category created earlier. This provides a crucial test case for transactions with different categories, enhancing the coverage of the filtering logic.

---

200-236: Excellent addition: Comprehensive tests for $ne operator

These new test cases directly address the core issue of "$ne filters incorrectly excluding null values". They verify that:

1. When filtering out 'cat2', both null category and 'cat' category transactions are correctly included.
2. When filtering out null categories, both 'cat' and 'cat2' category transactions are properly included.

This comprehensive testing ensures that the $ne operator now correctly handles null values, which is crucial for resolving the reported issue with net worth calculations.

---

Line range hint 1-283: Overall assessment: Well-implemented test expansion

The changes in this file significantly enhance the test coverage for the runQuery function, particularly focusing on the handling of null values and the $ne operator. These additions directly address the PR objectives and provide a robust verification of the fix for the reported issue with net worth calculations.

The new test cases are well-structured and cover various scenarios, ensuring that the filtering logic now correctly handles transactions with null categories and multiple category types. This comprehensive approach will help prevent regression and ensure the stability of the fix.

packages/loot-core/src/server/aql/compiler.ts (2)

714-714: Confirm desired behavior for NULL handling in $ne operator

The SQL expression (${left} != ${right} OR ${left} IS NULL) will evaluate to TRUE when ${left} is NULL, which means rows where the left operand is NULL will be included in the result set when using the $ne operator. Please confirm if this behavior aligns with the intended semantics of the $ne operator in your application, as it differs from standard SQL behavior where comparisons with NULL return UNKNOWN.

---

709-710: ⚠️ Potential issue

Verify SQL syntax and null handling in $ne operator implementation

In the CASE statement for the $ne operator, the expression ${left} IS NOT ${right} may not be valid SQL syntax. Typically, IS NOT is used with NULL (e.g., IS NOT NULL), and inequality between two values is expressed using <> or !=. Please ensure that the SQL syntax is correct and that the logic accurately handles NULL values as intended.

Here's a suggested correction:

 return `CASE
   WHEN ${left} IS NULL THEN ${right} IS NOT NULL
   ELSE ${left} <> ${right}
 END`;

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[youngcw]

is this fixing the same issue as #3669?

[joel-rich]

I can take a deeper look this weekend, but quick glance is that they are similar but complementary changes.

#3669 appears to change the behaviour of "category is/not (empty)" to properly handle transfers, whereas mine changes the behaviour of "category isnot (real_category)" to properly not exclude transactions where category = null (transfers/off-budget).

[qedi-r]

I've also looked, and they don't conflict.

Mine are on queries where the null is passed in the filter directly, ie .filter({ payee: {'$ne': null}})
Whereas joel-rich's is in the .filter({payee: {'$ne': ':namedParam'}}) case. Where the named parameter is also passed in the query like in his test.

I'm not sure where the named parameters are in use by the app though, aside from the invalid-schema error in loot-core/src/server/sync/index.ts? My testing with category filters didn't use the param option.

[joel-rich]

I'm not sure where the named parameters are in use by the app though, aside from the invalid-schema error in loot-core/src/server/sync/index.ts? My testing with category filters didn't use the param option.

Yeah, I did the change for regular non-null $ne filters, and noticed the namedParameter portion. I couldn't find where in the app namedParams were actually used so I wrote the tests to make sure they worked in a way consistent with my initial change.

I've double checked the differences between this change and #3669 - they don't overlap.

[joel-jeremy]

LGTM!