Merge pull request #292 from kozistr/refactor/utils

[Refactor] Remove direct import of many methods from `utils`, etc
kozistr · Nov 24, 2024 · 131314b · 131314b
2 parents 7868588 + ceb592a
commit 131314b
Show file tree

Hide file tree

Showing 128 changed files with 851 additions and 814 deletions.
diff --git a/README.md b/README.md
@@ -49,10 +49,7 @@ optimizer = load_optimizer(optimizer='adamp')(model.parameters())
 
 # if you install `bitsandbytes` optimizer, you can use `8-bit` optimizers from `pytorch-optimizer`.
 
-from pytorch_optimizer import load_optimizer
-
-opt = load_optimizer(optimizer='bnb_adamw8bit')
-optimizer = opt(model.parameters())
+optimizer = load_optimizer(optimizer='bnb_adamw8bit')(model.parameters())
 ```
 
 Also, you can load the optimizer via `torch.hub`.
@@ -61,6 +58,7 @@ Also, you can load the optimizer via `torch.hub`.
 import torch
 
 model = YourModel()
+
 opt = torch.hub.load('kozistr/pytorch_optimizer', 'adamp')
 optimizer = opt(model.parameters())
 ```
@@ -93,11 +91,13 @@ supported_optimizers = get_supported_optimizers()
 or you can also search them with the filter(s).
 
 ```python
->>> get_supported_optimizers('adam*')
-['adamax', 'adamg', 'adammini', 'adamod', 'adamp', 'adams', 'adamw']
+from pytorch_optimizer import get_supported_optimizers
 
->>> get_supported_optimizers(['adam*', 'ranger*'])
-['adamax', 'adamg', 'adammini', 'adamod', 'adamp', 'adams', 'adamw', 'ranger', 'ranger21']
+get_supported_optimizers('adam*')
+# ['adamax', 'adamg', 'adammini', 'adamod', 'adamp', 'adams', 'adamw']
+
+get_supported_optimizers(['adam*', 'ranger*'])
+# ['adamax', 'adamg', 'adammini', 'adamod', 'adamp', 'adams', 'adamw', 'ranger', 'ranger21']
 ```
 
 | Optimizer     | Description                                                                                       | Official Code                                                                                                  | Paper                                                                                       | Citation                                                                                                                            |
@@ -197,11 +197,13 @@ supported_lr_schedulers = get_supported_lr_schedulers()
 or you can also search them with the filter(s).
 
 ```python
->>> get_supported_lr_schedulers('cosine*')
-['cosine', 'cosine_annealing', 'cosine_annealing_with_warm_restart', 'cosine_annealing_with_warmup']
+from pytorch_optimizer import get_supported_lr_schedulers
+
+get_supported_lr_schedulers('cosine*')
+# ['cosine', 'cosine_annealing', 'cosine_annealing_with_warm_restart', 'cosine_annealing_with_warmup']
 
->>> get_supported_lr_schedulers(['cosine*', '*warm*'])
-['cosine', 'cosine_annealing', 'cosine_annealing_with_warm_restart', 'cosine_annealing_with_warmup', 'warmup_stable_decay']
+get_supported_lr_schedulers(['cosine*', '*warm*'])
+# ['cosine', 'cosine_annealing', 'cosine_annealing_with_warm_restart', 'cosine_annealing_with_warmup', 'warmup_stable_decay']
 ```
 
 | LR Scheduler    | Description                                                                     | Official Code                                                                                                                       | Paper                              | Citation                                                                                           |
@@ -224,11 +226,13 @@ supported_loss_functions = get_supported_loss_functions()
 or you can also search them with the filter(s).
 
 ```python
->>> get_supported_loss_functions('*focal*')
-['bcefocalloss', 'focalcosineloss', 'focalloss', 'focaltverskyloss']
+from pytorch_optimizer import get_supported_loss_functions
+
+get_supported_loss_functions('*focal*')
+# ['bcefocalloss', 'focalcosineloss', 'focalloss', 'focaltverskyloss']
 
->>> get_supported_loss_functions(['*focal*', 'bce*'])
-['bcefocalloss', 'bceloss', 'focalcosineloss', 'focalloss', 'focaltverskyloss']
+get_supported_loss_functions(['*focal*', 'bce*'])
+# ['bcefocalloss', 'bceloss', 'focalcosineloss', 'focalloss', 'focaltverskyloss']
 ```
 
 | Loss Functions  | Description                                                                                                             | Official Code                                          | Paper                              | Citation                                                                     |

diff --git a/docs/changelogs/v3.3.0.md b/docs/changelogs/v3.3.0.md
@@ -8,3 +8,19 @@
     * [Modified Adam Can Converge with Any β2 with the Optimal Rate](https://arxiv.org/abs/2411.02853)
 * Implement `FTRL` optimizer. (#291)
     * [Follow The Regularized Leader](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41159.pdf)
+
+### Refactor
+
+* Big refactoring, removing direct import from `pytorch_optimizer.*`.
+    * I removed some methods not to directly import from it from `pytorch_optimzier.*` because they're probably not used frequently and actually not an optimizer rather utils only used for specific optimizers.
+    * `pytorch_optimizer.[Shampoo stuff]` -> `pytorch_optimizer.optimizers.shampoo_utils.[Shampoo stuff]`.
+        * `shampoo_utils` like `Graft`, `BlockPartitioner`, `PreConditioner`, etc. You can check the details [here](https://github.com/kozistr/pytorch_optimizer/blob/main/pytorch_optimizer/optimizer/shampoo_utils.py).
+    * `pytorch_optimizer.GaLoreProjector` -> `pytorch_optimizer.optimizers.galore.GaLoreProjector`.
+    * `pytorch_optimizer.gradfilter_ema` -> `pytorch_optimizer.optimizers.grokfast.gradfilter_ema`.
+    * `pytorch_optimizer.gradfilter_ma` -> `pytorch_optimizer.optimizers.grokfast.gradfilter_ma`.
+    * `pytorch_optimizer.l2_projection` -> `pytorch_optimizer.optimizers.alig.l2_projection`.
+    * `pytorch_optimizer.flatten_grad` -> `pytorch_optimizer.optimizers.pcgrad.flatten_grad`.
+    * `pytorch_optimizer.un_flatten_grad` -> `pytorch_optimizer.optimizers.pcgrad.un_flatten_grad`.
+    * `pytorch_optimizer.reduce_max_except_dim` -> `pytorch_optimizer.optimizers.sm3.reduce_max_except_dim`.
+    * `pytorch_optimizer.neuron_norm` -> `pytorch_optimizer.optimizers.nero.neuron_norm`.
+    * `pytorch_optimizer.neuron_mean` -> `pytorch_optimizer.optimizers.nero.neuron_mean`.
diff --git a/docs/index.md b/docs/index.md
@@ -49,10 +49,7 @@ optimizer = load_optimizer(optimizer='adamp')(model.parameters())
 
 # if you install `bitsandbytes` optimizer, you can use `8-bit` optimizers from `pytorch-optimizer`.
 
-from pytorch_optimizer import load_optimizer
-
-opt = load_optimizer(optimizer='bnb_adamw8bit')
-optimizer = opt(model.parameters())
+optimizer = load_optimizer(optimizer='bnb_adamw8bit')(model.parameters())
 ```
 
 Also, you can load the optimizer via `torch.hub`.
@@ -61,6 +58,7 @@ Also, you can load the optimizer via `torch.hub`.
 import torch
 
 model = YourModel()
+
 opt = torch.hub.load('kozistr/pytorch_optimizer', 'adamp')
 optimizer = opt(model.parameters())
 ```
@@ -93,11 +91,13 @@ supported_optimizers = get_supported_optimizers()
 or you can also search them with the filter(s).
 
 ```python
->>> get_supported_optimizers('adam*')
-['adamax', 'adamg', 'adammini', 'adamod', 'adamp', 'adams', 'adamw']
+from pytorch_optimizer import get_supported_optimizers
 
->>> get_supported_optimizers(['adam*', 'ranger*'])
-['adamax', 'adamg', 'adammini', 'adamod', 'adamp', 'adams', 'adamw', 'ranger', 'ranger21']
+get_supported_optimizers('adam*')
+# ['adamax', 'adamg', 'adammini', 'adamod', 'adamp', 'adams', 'adamw']
+
+get_supported_optimizers(['adam*', 'ranger*'])
+# ['adamax', 'adamg', 'adammini', 'adamod', 'adamp', 'adams', 'adamw', 'ranger', 'ranger21']
 ```
 
 | Optimizer     | Description                                                                                       | Official Code                                                                                                  | Paper                                                                                       | Citation                                                                                                                            |
@@ -197,11 +197,13 @@ supported_lr_schedulers = get_supported_lr_schedulers()
 or you can also search them with the filter(s).
 
 ```python
->>> get_supported_lr_schedulers('cosine*')
-['cosine', 'cosine_annealing', 'cosine_annealing_with_warm_restart', 'cosine_annealing_with_warmup']
+from pytorch_optimizer import get_supported_lr_schedulers
+
+get_supported_lr_schedulers('cosine*')
+# ['cosine', 'cosine_annealing', 'cosine_annealing_with_warm_restart', 'cosine_annealing_with_warmup']
 
->>> get_supported_lr_schedulers(['cosine*', '*warm*'])
-['cosine', 'cosine_annealing', 'cosine_annealing_with_warm_restart', 'cosine_annealing_with_warmup', 'warmup_stable_decay']
+get_supported_lr_schedulers(['cosine*', '*warm*'])
+# ['cosine', 'cosine_annealing', 'cosine_annealing_with_warm_restart', 'cosine_annealing_with_warmup', 'warmup_stable_decay']
 ```
 
 | LR Scheduler    | Description                                                                     | Official Code                                                                                                                       | Paper                              | Citation                                                                                           |
@@ -224,11 +226,13 @@ supported_loss_functions = get_supported_loss_functions()
 or you can also search them with the filter(s).
 
 ```python
->>> get_supported_loss_functions('*focal*')
-['bcefocalloss', 'focalcosineloss', 'focalloss', 'focaltverskyloss']
+from pytorch_optimizer import get_supported_loss_functions
+
+get_supported_loss_functions('*focal*')
+# ['bcefocalloss', 'focalcosineloss', 'focalloss', 'focaltverskyloss']
 
->>> get_supported_loss_functions(['*focal*', 'bce*'])
-['bcefocalloss', 'bceloss', 'focalcosineloss', 'focalloss', 'focaltverskyloss']
+get_supported_loss_functions(['*focal*', 'bce*'])
+# ['bcefocalloss', 'bceloss', 'focalcosineloss', 'focalloss', 'focaltverskyloss']
 ```
 
 | Loss Functions  | Description                                                                                                             | Official Code                                          | Paper                              | Citation                                                                     |

diff --git a/docs/loss.md b/docs/loss.md
@@ -1,6 +1,6 @@
 # Loss Function
 
-::: pytorch_optimizer.loss.bi_tempered.bi_tempered_logistic_loss
+::: pytorch_optimizer.bi_tempered_logistic_loss
     :docstring:
 
 ::: pytorch_optimizer.BiTemperedLogisticLoss
@@ -35,7 +35,7 @@
     :docstring:
     :members:
 
-::: pytorch_optimizer.loss.jaccard.soft_jaccard_score
+::: pytorch_optimizer.soft_jaccard_score
     :docstring:
 
 ::: pytorch_optimizer.JaccardLoss

diff --git a/docs/optimizer.md b/docs/optimizer.md
@@ -1,5 +1,13 @@
 # Optimizers
 
+::: pytorch_optimizer.optimizer.create_optimizer
+    :docstring:
+    :members:
+
+::: pytorch_optimizer.optimizer.get_optimizer_parameters
+    :docstring:
+    :members:
+
 ::: pytorch_optimizer.A2Grad
     :docstring:
     :members:
@@ -168,10 +176,6 @@
     :docstring:
     :members:
 
-::: pytorch_optimizer.GaLoreProjector
-    :docstring:
-    :members:
-
 ::: pytorch_optimizer.centralize_gradient
     :docstring:
     :members:
@@ -180,14 +184,6 @@
     :docstring:
     :members:
 
-::: pytorch_optimizer.gradfilter_ema
-    :docstring:
-    :members:
-
-::: pytorch_optimizer.gradfilter_ma
-    :docstring:
-    :members:
-
 ::: pytorch_optimizer.GrokFastAdamW
     :docstring:
     :members:

diff --git a/docs/util.md b/docs/util.md
@@ -16,10 +16,6 @@
     :docstring:
     :members:
 
-::: pytorch_optimizer.optimizer.utils.get_optimizer_parameters
-    :docstring:
-    :members:
-
 ::: pytorch_optimizer.optimizer.utils.is_valid_parameters
     :docstring:
     :members:
@@ -36,14 +32,6 @@
     :docstring:
     :members:
 
-::: pytorch_optimizer.optimizer.utils.flatten_grad
-    :docstring:
-    :members:
-
-::: pytorch_optimizer.optimizer.utils.un_flatten_grad
-    :docstring:
-    :members:
-
 ::: pytorch_optimizer.optimizer.utils.channel_view
     :docstring:
     :members:
@@ -68,14 +56,6 @@
     :docstring:
     :members:
 
-::: pytorch_optimizer.optimizer.utils.neuron_norm
-    :docstring:
-    :members:
-
-::: pytorch_optimizer.optimizer.utils.neuron_mean
-    :docstring:
-    :members:
-
 ::: pytorch_optimizer.optimizer.utils.disable_running_stats
     :docstring:
     :members:
@@ -84,82 +64,10 @@
     :docstring:
     :members:
 
-::: pytorch_optimizer.optimizer.utils.l2_projection
-    :docstring:
-    :members:
-
 ::: pytorch_optimizer.optimizer.utils.get_global_gradient_norm
     :docstring:
     :members:
 
-::: pytorch_optimizer.optimizer.utils.reduce_max_except_dim
-    :docstring:
-    :members:
-
-::: pytorch_optimizer.optimizer.shampoo_utils.merge_small_dims
-    :docstring:
-    :members:
-
 ::: pytorch_optimizer.optimizer.utils.reg_noise
     :docstring:
     :members:
-
-## Newton methods
-
-::: pytorch_optimizer.optimizer.shampoo_utils.power_iteration
-    :docstring:
-    :members:
-
-::: pytorch_optimizer.optimizer.shampoo_utils.compute_power_schur_newton
-    :docstring:
-    :members:
-
-::: pytorch_optimizer.optimizer.shampoo_utils.compute_power_svd
-    :docstring:
-    :members:
-
-## Grafting
-
-::: pytorch_optimizer.optimizer.shampoo_utils.Graft
-    :docstring:
-    :members:
-
-::: pytorch_optimizer.optimizer.shampoo_utils.LayerWiseGrafting
-    :docstring:
-    :members:
-
-::: pytorch_optimizer.optimizer.shampoo_utils.SGDGraft
-    :docstring:
-    :members:
-
-::: pytorch_optimizer.optimizer.shampoo_utils.SQRTNGraft
-    :docstring:
-    :members:
-
-::: pytorch_optimizer.optimizer.shampoo_utils.AdaGradGraft
-    :docstring:
-    :members:
-
-::: pytorch_optimizer.optimizer.shampoo_utils.RMSPropGraft
-    :docstring:
-    :members:
-
-::: pytorch_optimizer.optimizer.shampoo_utils.build_graft
-    :docstring:
-    :members:
-
-## Block Partitioner
-
-::: pytorch_optimizer.optimizer.shampoo_utils.BlockPartitioner
-    :docstring:
-    :members:
-
-## Pre-Conditioner
-
-::: pytorch_optimizer.optimizer.shampoo_utils.PreConditionerType
-    :docstring:
-    :members:
-
-::: pytorch_optimizer.optimizer.shampoo_utils.PreConditioner
-    :docstring:
-    :members:
diff --git a/docs/visualizations/rastrigin_AdaBelief.png b/docs/visualizations/rastrigin_AdaBelief.png
diff --git a/docs/visualizations/rastrigin_AdaBound.png b/docs/visualizations/rastrigin_AdaBound.png
diff --git a/docs/visualizations/rastrigin_AdaMod.png b/docs/visualizations/rastrigin_AdaMod.png
diff --git a/docs/visualizations/rastrigin_AdaPNM.png b/docs/visualizations/rastrigin_AdaPNM.png
diff --git a/docs/visualizations/rastrigin_Adai.png b/docs/visualizations/rastrigin_Adai.png
diff --git a/docs/visualizations/rastrigin_AdamP.png b/docs/visualizations/rastrigin_AdamP.png
diff --git a/docs/visualizations/rastrigin_AdamW.png b/docs/visualizations/rastrigin_AdamW.png
diff --git a/docs/visualizations/rastrigin_Adan.png b/docs/visualizations/rastrigin_Adan.png
diff --git a/docs/visualizations/rastrigin_AggMo.png b/docs/visualizations/rastrigin_AggMo.png
diff --git a/docs/visualizations/rastrigin_DAdaptAdaGrad.png b/docs/visualizations/rastrigin_DAdaptAdaGrad.png
diff --git a/docs/visualizations/rastrigin_DAdaptAdam.png b/docs/visualizations/rastrigin_DAdaptAdam.png
diff --git a/docs/visualizations/rastrigin_DAdaptSGD.png b/docs/visualizations/rastrigin_DAdaptSGD.png
diff --git a/docs/visualizations/rastrigin_DiffGrad.png b/docs/visualizations/rastrigin_DiffGrad.png
diff --git a/docs/visualizations/rastrigin_Fromage.png b/docs/visualizations/rastrigin_Fromage.png
diff --git a/docs/visualizations/rastrigin_LARS.png b/docs/visualizations/rastrigin_LARS.png
diff --git a/docs/visualizations/rastrigin_Lamb.png b/docs/visualizations/rastrigin_Lamb.png
diff --git a/docs/visualizations/rastrigin_MADGRAD.png b/docs/visualizations/rastrigin_MADGRAD.png
diff --git a/docs/visualizations/rastrigin_MSVAG.png b/docs/visualizations/rastrigin_MSVAG.png
diff --git a/docs/visualizations/rastrigin_Nero.png b/docs/visualizations/rastrigin_Nero.png
diff --git a/docs/visualizations/rastrigin_PID.png b/docs/visualizations/rastrigin_PID.png
diff --git a/docs/visualizations/rastrigin_PNM.png b/docs/visualizations/rastrigin_PNM.png
diff --git a/docs/visualizations/rastrigin_QHAdam.png b/docs/visualizations/rastrigin_QHAdam.png
diff --git a/docs/visualizations/rastrigin_QHM.png b/docs/visualizations/rastrigin_QHM.png
diff --git a/docs/visualizations/rastrigin_RAdam.png b/docs/visualizations/rastrigin_RAdam.png
diff --git a/docs/visualizations/rastrigin_Ranger.png b/docs/visualizations/rastrigin_Ranger.png
diff --git a/docs/visualizations/rastrigin_Ranger21.png b/docs/visualizations/rastrigin_Ranger21.png
diff --git a/docs/visualizations/rastrigin_SGDP.png b/docs/visualizations/rastrigin_SGDP.png
diff --git a/docs/visualizations/rastrigin_ScalableShampoo.png b/docs/visualizations/rastrigin_ScalableShampoo.png
diff --git a/docs/visualizations/rastrigin_Shampoo.png b/docs/visualizations/rastrigin_Shampoo.png
diff --git a/docs/visualizations/rosenbrock_ASGD.png b/docs/visualizations/rosenbrock_ASGD.png
diff --git a/docs/visualizations/rosenbrock_AccSGD.png b/docs/visualizations/rosenbrock_AccSGD.png
diff --git a/docs/visualizations/rosenbrock_AdaBelief.png b/docs/visualizations/rosenbrock_AdaBelief.png
diff --git a/docs/visualizations/rosenbrock_AdaBound.png b/docs/visualizations/rosenbrock_AdaBound.png
diff --git a/docs/visualizations/rosenbrock_AdaDelta.png b/docs/visualizations/rosenbrock_AdaDelta.png
diff --git a/docs/visualizations/rosenbrock_AdaFactor.png b/docs/visualizations/rosenbrock_AdaFactor.png
diff --git a/docs/visualizations/rosenbrock_AdaHessian.png b/docs/visualizations/rosenbrock_AdaHessian.png
diff --git a/docs/visualizations/rosenbrock_AdaMax.png b/docs/visualizations/rosenbrock_AdaMax.png
diff --git a/docs/visualizations/rosenbrock_AdaMod.png b/docs/visualizations/rosenbrock_AdaMod.png
diff --git a/docs/visualizations/rosenbrock_AdaNorm.png b/docs/visualizations/rosenbrock_AdaNorm.png
diff --git a/docs/visualizations/rosenbrock_AdaPNM.png b/docs/visualizations/rosenbrock_AdaPNM.png
diff --git a/docs/visualizations/rosenbrock_AdaSmooth.png b/docs/visualizations/rosenbrock_AdaSmooth.png
diff --git a/docs/visualizations/rosenbrock_Adai.png b/docs/visualizations/rosenbrock_Adai.png
diff --git a/docs/visualizations/rosenbrock_Adalite.png b/docs/visualizations/rosenbrock_Adalite.png
diff --git a/docs/visualizations/rosenbrock_Adam.png b/docs/visualizations/rosenbrock_Adam.png
diff --git a/docs/visualizations/rosenbrock_AdamP.png b/docs/visualizations/rosenbrock_AdamP.png
diff --git a/docs/visualizations/rosenbrock_AdamS.png b/docs/visualizations/rosenbrock_AdamS.png
diff --git a/docs/visualizations/rosenbrock_AdamW.png b/docs/visualizations/rosenbrock_AdamW.png
diff --git a/docs/visualizations/rosenbrock_Adan.png b/docs/visualizations/rosenbrock_Adan.png
diff --git a/docs/visualizations/rosenbrock_AggMo.png b/docs/visualizations/rosenbrock_AggMo.png
diff --git a/docs/visualizations/rosenbrock_Aida.png b/docs/visualizations/rosenbrock_Aida.png
diff --git a/docs/visualizations/rosenbrock_Amos.png b/docs/visualizations/rosenbrock_Amos.png
diff --git a/docs/visualizations/rosenbrock_Apollo.png b/docs/visualizations/rosenbrock_Apollo.png
diff --git a/docs/visualizations/rosenbrock_AvaGrad.png b/docs/visualizations/rosenbrock_AvaGrad.png
diff --git a/docs/visualizations/rosenbrock_CAME.png b/docs/visualizations/rosenbrock_CAME.png
diff --git a/docs/visualizations/rosenbrock_DAdaptAdaGrad.png b/docs/visualizations/rosenbrock_DAdaptAdaGrad.png
diff --git a/docs/visualizations/rosenbrock_DAdaptAdam.png b/docs/visualizations/rosenbrock_DAdaptAdam.png
diff --git a/docs/visualizations/rosenbrock_DAdaptAdan.png b/docs/visualizations/rosenbrock_DAdaptAdan.png
diff --git a/docs/visualizations/rosenbrock_DAdaptLion.png b/docs/visualizations/rosenbrock_DAdaptLion.png
diff --git a/docs/visualizations/rosenbrock_DAdaptSGD.png b/docs/visualizations/rosenbrock_DAdaptSGD.png
diff --git a/docs/visualizations/rosenbrock_DiffGrad.png b/docs/visualizations/rosenbrock_DiffGrad.png
diff --git a/docs/visualizations/rosenbrock_FAdam.png b/docs/visualizations/rosenbrock_FAdam.png
diff --git a/docs/visualizations/rosenbrock_Fromage.png b/docs/visualizations/rosenbrock_Fromage.png
diff --git a/docs/visualizations/rosenbrock_GaLore.png b/docs/visualizations/rosenbrock_GaLore.png
diff --git a/docs/visualizations/rosenbrock_Gravity.png b/docs/visualizations/rosenbrock_Gravity.png
diff --git a/docs/visualizations/rosenbrock_GrokFastAdamW.png b/docs/visualizations/rosenbrock_GrokFastAdamW.png
diff --git a/docs/visualizations/rosenbrock_Kate.png b/docs/visualizations/rosenbrock_Kate.png
diff --git a/docs/visualizations/rosenbrock_LARS.png b/docs/visualizations/rosenbrock_LARS.png
diff --git a/docs/visualizations/rosenbrock_Lamb.png b/docs/visualizations/rosenbrock_Lamb.png
diff --git a/docs/visualizations/rosenbrock_Lion.png b/docs/visualizations/rosenbrock_Lion.png
diff --git a/docs/visualizations/rosenbrock_MADGRAD.png b/docs/visualizations/rosenbrock_MADGRAD.png
diff --git a/docs/visualizations/rosenbrock_MSVAG.png b/docs/visualizations/rosenbrock_MSVAG.png
diff --git a/docs/visualizations/rosenbrock_Nero.png b/docs/visualizations/rosenbrock_Nero.png
diff --git a/docs/visualizations/rosenbrock_NovoGrad.png b/docs/visualizations/rosenbrock_NovoGrad.png
diff --git a/docs/visualizations/rosenbrock_PAdam.png b/docs/visualizations/rosenbrock_PAdam.png
diff --git a/docs/visualizations/rosenbrock_PID.png b/docs/visualizations/rosenbrock_PID.png
diff --git a/docs/visualizations/rosenbrock_PNM.png b/docs/visualizations/rosenbrock_PNM.png
diff --git a/docs/visualizations/rosenbrock_Prodigy.png b/docs/visualizations/rosenbrock_Prodigy.png
diff --git a/docs/visualizations/rosenbrock_QHAdam.png b/docs/visualizations/rosenbrock_QHAdam.png
diff --git a/docs/visualizations/rosenbrock_QHM.png b/docs/visualizations/rosenbrock_QHM.png
diff --git a/docs/visualizations/rosenbrock_RAdam.png b/docs/visualizations/rosenbrock_RAdam.png
diff --git a/docs/visualizations/rosenbrock_Ranger.png b/docs/visualizations/rosenbrock_Ranger.png
diff --git a/docs/visualizations/rosenbrock_Ranger21.png b/docs/visualizations/rosenbrock_Ranger21.png
diff --git a/docs/visualizations/rosenbrock_SGD.png b/docs/visualizations/rosenbrock_SGD.png
diff --git a/docs/visualizations/rosenbrock_SGDP.png b/docs/visualizations/rosenbrock_SGDP.png
diff --git a/docs/visualizations/rosenbrock_SGDW.png b/docs/visualizations/rosenbrock_SGDW.png
diff --git a/docs/visualizations/rosenbrock_SM3.png b/docs/visualizations/rosenbrock_SM3.png
diff --git a/docs/visualizations/rosenbrock_SRMM.png b/docs/visualizations/rosenbrock_SRMM.png
diff --git a/docs/visualizations/rosenbrock_SWATS.png b/docs/visualizations/rosenbrock_SWATS.png
diff --git a/docs/visualizations/rosenbrock_ScalableShampoo.png b/docs/visualizations/rosenbrock_ScalableShampoo.png
diff --git a/docs/visualizations/rosenbrock_ScheduleFreeAdamW.png b/docs/visualizations/rosenbrock_ScheduleFreeAdamW.png
diff --git a/docs/visualizations/rosenbrock_ScheduleFreeSGD.png b/docs/visualizations/rosenbrock_ScheduleFreeSGD.png
diff --git a/docs/visualizations/rosenbrock_Shampoo.png b/docs/visualizations/rosenbrock_Shampoo.png
diff --git a/docs/visualizations/rosenbrock_SignSGD.png b/docs/visualizations/rosenbrock_SignSGD.png
diff --git a/docs/visualizations/rosenbrock_SophiaH.png b/docs/visualizations/rosenbrock_SophiaH.png
diff --git a/docs/visualizations/rosenbrock_StableAdamW.png b/docs/visualizations/rosenbrock_StableAdamW.png
diff --git a/docs/visualizations/rosenbrock_Tiger.png b/docs/visualizations/rosenbrock_Tiger.png
diff --git a/docs/visualizations/rosenbrock_Yogi.png b/docs/visualizations/rosenbrock_Yogi.png