Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add pytorch tabular benchmark #398

Merged
merged 18 commits into from
Sep 6, 2024
Merged

add pytorch tabular benchmark #398

merged 18 commits into from
Sep 6, 2024

Conversation

yiweny
Copy link
Contributor

@yiweny yiweny commented May 13, 2024 •
edited by wsad1
Loading

This PR adds a scirpt to compare TabTransformer across pytorch_tabular and pytroch_frame.
Results:

CPU:

  • Frame: 11.9 it/s
  • Tabular: 9.1 it/s

CUDA:

  • Frame: 45.06 it/s
  • Tabular: 41.76 it/s

In a follow up PR add comparison for other models.

@wsad1 wsad1 self-requested a review September 2, 2024 01:01
@wsad1 wsad1 requested review from akihironitta and removed request for wsad1 September 2, 2024 01:08
@wsad1 wsad1 self-assigned this Sep 2, 2024
@wsad1 wsad1 mentioned this pull request Sep 2, 2024
Merged
@wsad1 wsad1 self-requested a review September 3, 2024 02:44
@codecov Codecov
Copy link

codecov bot commented Sep 3, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 93.59%. Comparing base (ee98b87) to head (35f891a).
Report is 35 commits behind head on master.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master     #398      +/-   ##
==========================================
+ Coverage   93.52%   93.59%   +0.07%     
==========================================
  Files         124      126       +2     
  Lines        6456     6576     +120     
==========================================
+ Hits         6038     6155     +117     
- Misses        418      421       +3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

akihironitta
akihironitta reviewed Sep 6, 2024
View reviewed changes
Copy link
Member

@akihironitta akihironitta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing to note is that since PyTorch Tabular uses PyTorch Lightning, I assume extra features are turned on by default, e.g., logging, checkpointing, sanity val steps, etc., making it unfair for pure comparison.

@akihironitta akihironitta enabled auto-merge (squash) September 6, 2024 18:55
@akihironitta akihironitta merged commit 475105e into master Sep 6, 2024
13 checks passed
@akihironitta akihironitta deleted the yyuan/add-pytorch-tabular-comparison-script branch September 6, 2024 19:02
wsad1 added a commit that referenced this pull request Sep 6, 2024
@wsad1 @yiweny
@pre-commit-ci @akihironitta
Compare more models across frame and tabular (#444)
b28e9a3 
Follow up to #398. 
Adds an option to compare `FTTransformer` accross pytorch tabular and
frame.

CUDA: 
   Frame: 43.7 it/s
   Tabular: 40.15 it/s

---------

Co-authored-by: yiweny <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Akihiro Nitta <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@akihironitta akihironitta akihironitta approved these changes

@wsad1 wsad1 wsad1 approved these changes

Assignees

@wsad1 wsad1

Labels
benchmark
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@yiweny @wsad1 @akihironitta