Skip to content

Commit

Permalink
Add results from large-scale multiclass classification benchmarks (#436)
Browse files Browse the repository at this point in the history
No need for a changelog entry as this change impacts no users.
  • Loading branch information
akihironitta authored Aug 15, 2024
1 parent 4a9d91b commit 59994ec
Showing 1 changed file with 15 additions and 1 deletion.
16 changes: 15 additions & 1 deletion benchmark/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -194,4 +194,18 @@ Experimental setting: 20 Optuna search trials for XGBoost, CatBoost and LightGBM

#### `scale: large`

TODO
Experimental setting: 20 Optuna search trials for XGBoost, CatBoost and LightGBM. 3 Optuna search trials and 10 epochs training for deep learning models.

| | dataset_0 | dataset_1 | dataset_2 |
| :------------------ | :--------------------- | :--------------------- | :----------------------- |
| XGBoost | Too slow\* | Too slow\* | Too slow\* |
| CatBoost | Too slow\* | Too slow\* | Too slow\* |
| LightGBM | Too slow\* | Too slow\* | Too slow\* |
| Trompt | OOM | 0.889±0.063 (55428s) | 0.804±0.013 (23304s) |
| ResNet | 0.892±0.002 (417s) | **0.999±0.001 (396s)** | 0.915±0.001 (405s) |
| MLP | 0.770±0.001 (170s) | 0.549±0.000 (223s) | 0.895±0.001 (192s) |
| FTTransformerBucket | 0.897±0.004 (4436s) | 0.502±0.000 (1892s) | 0.888±0.009 (4414s) |
| ExcelFormer | OOM | TODO (code error) | **0.951±0.002 (10236s)** |
| FTTransformer | 0.872±0.005 (7004s) | 0.540±0.068 (3355s) | 0.908±0.004 (7514s) |
| TabNet | **0.912±0.004 (219s)** | 0.995±0.001 (301s) | 0.919±0.003 (187s) |
| TabTransformer | 0.843±0.003 (2810s) | 0.657±0.187 (2843s) | 0.854±0.001 (284s) |

0 comments on commit 59994ec

Please sign in to comment.