change federeated learning paradigm into planb mode

Signed-off-by: Marchons <[email protected]>
Signed-off-by: Marchons <[email protected]>

EGG-INFO/PKG-INFO

@@ -0,0 +1,144 @@
+Metadata-Version: 2.1
+Name: ianvs
+Version: 0.1.0
+Summary: The ianvs package is designed to help algorithm developers                 better do algorithm test.
+Home-page: https://github.com/kubeedge/ianvs
+Author: MooreZheng
+Maintainer: MooreZheng
+Maintainer-email: 
+License: Apache License 2.0
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Operating System :: POSIX :: Linux
+Requires-Python: >=3.6
+Description-Content-Type: text/markdown
+License-File: LICENSE
+
+# Ianvs
+
+[![CI](https://github.com/kubeedge/ianvs/workflows/CI/badge.svg?branch=main)](https://github.com/sedna/ianvs/actions)
+[![LICENSE SCAN](https://app.fossa.com/api/projects/custom%2B32178%2Fgithub.com%2Fkubeedge%2Fianvs.svg?type=shield)](https://app.fossa.com/projects/custom%2B32178%2Fgithub.com%2Fkubeedge%2Fianvs?ref=badge_shield)
+[![LICENSE](https://img.shields.io/github/license/kubeedge-sedna/ianvs.svg)](/LICENSE)
+
+Ianvs is a distributed synergy AI benchmarking project incubated in KubeEdge SIG AI. Ianvs aims to test the performance of distributed synergy AI solutions following recognized standards, in order to facilitate more efficient and effective development. More detailedly, Ianvs prepares not only test cases with datasets and corresponding algorithms, but also benchmarking tools including simulation and hyper-parameter searching. Ianvs also reveals best practices for developers and end users with presentation tools including leaderboards and test reports.
+
+## Scope
+
+The distributed synergy AI benchmarking Ianvs aims to test the performance of distributed synergy AI solutions following recognized standards, in order to facilitate more efficient and effective development.
+
+The scope of Ianvs includes
+
+- Providing end-to-end benchmark toolkits across devices, edge nodes, and cloud nodes based on typical distributed-synergy AI paradigms and applications.
+  - Tools to manage test environment. For example, it would be necessary to support the CRUD (Create, Read, Update, and Delete) actions in test environments. Elements of such test environments include algorithm-wise and system-wise configuration.
+  - Tools to control test cases. Typical examples include paradigm templates, simulation tools, and hyper-parameter-based assistant tools.
+  - Tools to manage benchmark presentation, e.g., leaderboard and test report generation.
+- Cooperation with other organizations or communities, e.g., in KubeEdge SIG AI, to establish comprehensive benchmarks and developed related applications, which can include but are not limited to
+  - Dataset collection, re-organization, and publication
+  - Formalized specifications, e.g., standards
+  - Holding competitions or coding events, e.g., open source promotion plan
+  - Maintaining solution leaderboards or certifications for commercial usage
+
+## Architecture
+
+The architectures and related concepts are shown in the below figure. The ianvs is designed to run **within a single node**. Critical components include
+
+- Test Environment Manager: the CRUD of test environments serving for global usage
+- Test Case Controller: control the runtime behavior of test cases like instance generation and vanish
+  - Generation Assistant: assist users to generate test cases based on certain rules or constraints, e.g., the range of parameters
+  - Simulation Controller: control the simulation process of edge-cloud synergy AI, including the instance generation and vanishment of simulation containers
+- Story Manager: the output management and presentation of the test case, e.g., leaderboards
+
+![](docs/guides/images/ianvs_arch.png)
+
+More details on Ianvs components:
+
+1. Test-Environment Manager supports the CRUD of Test environments, which basically includes
+   - Algorithm-wise configuration
+     - Public datasets
+     - Pre-processing algorithms
+     - Feature engineering algorithms
+     - Post-processing algorithms like metric computation
+   - System-wise configuration
+     - Overall architecture
+     - System constraints or budgets
+       - End-to-end cross-node
+       - Per node
+2. Test-case Controller, which includes but is not limited to the following components
+   - Templates of common distributed-synergy-AI paradigms, which can help the developer to prepare their test case without too much effort. Such paradigms include edge-cloud synergy joint inference, incremental learning, federated learning, and lifelong learning.
+   - Simulation tools. Develop simulated test environments for test cases
+   - Other tools to assist test-case generation. For instance, prepare test cases based on a given range of hyper-parameters.
+3. Story Manager, which includes but is not limited to the following components
+   - Leaderboard generation
+   - Test report generation
+
+## Guides
+
+### Documents
+
+Documentation is located on [readthedoc.io](https://ianvs.readthedocs.io/). The documents include the quick start, guides, dataset descriptions, algorithms, user interfaces, stories, and roadmap.
+
+### Installation
+
+Follow the [Ianvs installation document](docs/guides/how-to-install-ianvs.md) to install Ianvs.
+
+### Examples
+
+- Scenario PCB-AoI：[Industrial Defect Detection on the PCB-AoI Dataset](docs/proposals/scenarios/industrial-defect-detection/pcb-aoi.md).
+
+  - Example PCB-AoI-1：[Testing single task learning in industrial defect detection](docs/proposals/test-reports/testing-single-task-learning-in-industrial-defect-detection-with-pcb-aoi.md).
+
+  - Example PCB-AoI-2：[Testing incremental learning in industrial defect detection](docs/proposals/test-reports/testing-incremental-learning-in-industrial-defect-detection-with-pcb-aoi.md).
+
+- Scenario Cityscapes-Synthia: [Curb Detetion on Cityscapes-Synthia Dataset](docs/proposals/algorithms/lifelong-learning/Additional-documentation/curb_detetion_datasets.md)
+
+  - Example Cityscapes-Synthia-1: [Lifelong learning in semantic segmentation](examples/cityscapes-synthia/lifelong_learning_bench/semantic-segmentation/README.md)
+
+  - Example Cityscapes-Synthia-2: [Lifelong learning in curb detetion](examples/cityscapes-synthia/lifelong_learning_bench/curb-detection/README.md)
+
+  - Example Cityscapes-Synthia-3: [Scene based unknown task recognition in curb detetion](examples/cityscapes-synthia/scene-based-unknown-task-recognition/curb-detection/README.md)
+
+  - Example Cityscapes-Synthia-4: [Integrating GAN and Self-taught Learning into Ianvs Lifelong Learning](examples/cityscapes/lifelong_learning_bench/unseen_task_processing-GANwithSelfTaughtLearning/README.md)
+
+- Scenario Cloud-Robotics: [Semantic Segmentation on Cloud-Robotics Dataset](docs/proposals/scenarios/Cloud-Robotics/Cloud-Robotics_zh.md)
+
+  - Example Cloud-Robotics-1: [Lifelong learning in semantic segmentation](examples/robot/lifelong_learning_bench/semantic-segmentation/README.md)
+
+  - Example Cloud-Robotics-2: [Class increment learning in semantic segmentation](examples/robot-cityscapes-synthia/lifelong_learning_bench/semantic-segmentation/README.md)
+
+  - Example Cloud-Robotics-3: [Lifelong learning in sam annotation](examples/robot/lifelong_learning_bench/sam_annotation/tutorial.md)
+
+## Roadmap
+
+- [2022 H2 Roadmap](docs/roadmap.md)
+
+## Meeting
+
+Routine Community Meeting for KubeEdge SIG AI runs weekly:
+
+- Europe Time: **Thursdays at 16:30-17:30 Beijing Time**.
+  ([Convert to your timezone.](https://www.thetimezoneconverter.com/?t=16%3A30&tz=GMT%2B8&))
+
+Resources:
+
+- [Meeting notes and agenda](https://docs.google.com/document/d/12n3kGUWTkAH4q2Wv5iCVGPTA_KRWav_eakbFrF9iAww/edit)
+- [Meeting recordings](https://www.youtube.com/playlist?list=PLQtlO1kVWGXkRGkjSrLGEPJODoPb8s5FM)
+- [Meeting link](https://zoom.us/j/4167237304)
+- [Meeting Calendar](https://calendar.google.com/calendar/u/0/r?cid=Y19nODluOXAwOG05MzFiYWM3NmZsajgwZzEwOEBncm91cC5jYWxlbmRhci5nb29nbGUuY29t) | [Subscribe](https://calendar.google.com/calendar/u/0/r?cid=OHJqazhvNTE2dmZ0ZTIxcWlidmxhZTNsajRAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ)
+
+## Contact
+
+<!--
+If you need support, start with the [troubleshooting guide](./docs/troubleshooting.md), and work your way through the process that we've outlined.
+-->
+
+If you have questions, feel free to reach out to us in the following ways:
+
+- [slack channel](https://kubeedge.io/docs/community/slack/)
+
+## Contributing
+
+If you're interested in being a contributor and want to get involved in developing the Ianvs code, please see [CONTRIBUTING](CONTRIBUTING.md) for details on submitting patches and the contribution workflow.
+
+## License
+
+Ianvs is under the Apache 2.0 license. See the [LICENSE](LICENSE) file for details.

EGG-INFO/SOURCES.txt

@@ -0,0 +1,61 @@
+LICENSE
+README.md
+setup.py
+core/__init__.py
+core/__version__.py
+core/cmd/__init__.py
+core/cmd/benchmarking.py
+core/cmd/obj/__init__.py
+core/cmd/obj/benchmarkingjob.py
+core/common/__init__.py
+core/common/constant.py
+core/common/log.py
+core/common/utils.py
+core/storymanager/__init__.py
+core/storymanager/rank/__init__.py
+core/storymanager/rank/rank.py
+core/storymanager/visualization/__init__.py
+core/storymanager/visualization/visualization.py
+core/testcasecontroller/__init__.py
+core/testcasecontroller/testcasecontroller.py
+core/testcasecontroller/algorithm/__init__.py
+core/testcasecontroller/algorithm/algorithm.py
+core/testcasecontroller/algorithm/module/__init__.py
+core/testcasecontroller/algorithm/module/module.py
+core/testcasecontroller/algorithm/paradigm/__init__.py
+core/testcasecontroller/algorithm/paradigm/base.py
+core/testcasecontroller/algorithm/paradigm/sedna_federated_learning.py
+core/testcasecontroller/algorithm/paradigm/federated_learning/__init__.py
+core/testcasecontroller/algorithm/paradigm/federated_learning/federated_class_incremental_learning.py
+core/testcasecontroller/algorithm/paradigm/federated_learning/federated_learning.py
+core/testcasecontroller/algorithm/paradigm/incremental_learning/__init__.py
+core/testcasecontroller/algorithm/paradigm/incremental_learning/incremental_learning.py
+core/testcasecontroller/algorithm/paradigm/lifelong_learning/__init__.py
+core/testcasecontroller/algorithm/paradigm/lifelong_learning/lifelong_learning.py
+core/testcasecontroller/algorithm/paradigm/multiedge_inference/__init__.py
+core/testcasecontroller/algorithm/paradigm/multiedge_inference/multiedge_inference.py
+core/testcasecontroller/algorithm/paradigm/singletask_learning/__init__.py
+core/testcasecontroller/algorithm/paradigm/singletask_learning/singletask_learning.py
+core/testcasecontroller/algorithm/paradigm/singletask_learning/singletask_learning_active_boost.py
+core/testcasecontroller/algorithm/paradigm/singletask_learning/singletask_learning_tta.py
+core/testcasecontroller/generation_assistant/__init__.py
+core/testcasecontroller/generation_assistant/generation_assistant.py
+core/testcasecontroller/metrics/__init__.py
+core/testcasecontroller/metrics/metrics.py
+core/testcasecontroller/simulation/__init__.py
+core/testcasecontroller/simulation/simulation.py
+core/testcasecontroller/simulation_system_admin/__init__.py
+core/testcasecontroller/simulation_system_admin/simulation_system_admin.py
+core/testcasecontroller/testcase/__init__.py
+core/testcasecontroller/testcase/testcase.py
+core/testenvmanager/__init__.py
+core/testenvmanager/dataset/__init__.py
+core/testenvmanager/dataset/dataset.py
+core/testenvmanager/dataset/utils.py
+core/testenvmanager/testenv/__init__.py
+core/testenvmanager/testenv/testenv.py
+ianvs.egg-info/PKG-INFO
+ianvs.egg-info/SOURCES.txt
+ianvs.egg-info/dependency_links.txt
+ianvs.egg-info/entry_points.txt
+ianvs.egg-info/top_level.txt

EGG-INFO/dependency_links.txt

@@ -0,0 +1 @@
+

EGG-INFO/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+ianvs = core.cmd.benchmarking:main

EGG-INFO/top_level.txt

@@ -0,0 +1 @@
+core

EGG-INFO/zip-safe

@@ -0,0 +1 @@
+

core/common/constant.py

@@ -22,6 +22,7 @@ class DatasetFormat(Enum):
     File format of inputting dataset.
     Currently, file formats are as follows: txt, csv.
     """
+
     CSV = "csv"
     TXT = "txt"
     JSON = "json"
@@ -31,6 +32,7 @@ class ParadigmType(Enum):
     """
     Algorithm paradigm type.
     """
+
     SINGLE_TASK_LEARNING = "singletasklearning"
     INCREMENTAL_LEARNING = "incrementallearning"
     MULTIEDGE_INFERENCE = "multiedgeinference"
@@ -43,6 +45,7 @@ class ModuleType(Enum):
     """
     Algorithm module type.
     """
+
     BASEMODEL = "basemodel"
 
     # HEM
@@ -68,19 +71,23 @@ class ModuleType(Enum):
     # FL_AGG
     AGGREGATION = "aggregation"
 
+
 class SystemMetricType(Enum):
     """
     System metric type of ianvs.
     """
+
     SAMPLES_TRANSFER_RATIO = "samples_transfer_ratio"
     FWT = "FWT"
     BWT = "BWT"
     TASK_AVG_ACC = "task_avg_acc"
     MATRIX = "MATRIX"
+    FORGET_RATE = "forget_rate"
 
 
 class TestObjectType(Enum):
     """
     Test object type of ianvs.
     """
+
     ALGORITHMS = "algorithms"

core/storymanager/rank/rank.py

@@ -35,15 +35,12 @@ class Rank:
 
     def __init__(self, config):
         self.sort_by: list = []
-        self.visualization: dict = {
-            "mode": "selected_only",
-            "method": "print_table"
-        }
+        self.visualization: dict = {"mode": "selected_only", "method": "print_table"}
         self.selected_dataitem: dict = {
             "paradigms": ["all"],
             "modules": ["all"],
             "hyperparameters": ["all"],
-            "metrics": ["all"]
+            "metrics": ["all"],
         }
         self.save_mode: str = "selected_and_all"
 
@@ -62,15 +59,21 @@ def _parse_config(self, config):
 
     def _check_fields(self):
         if not self.sort_by and not isinstance(self.sort_by, list):
-            raise ValueError(f"rank's sort_by({self.sort_by}) must be provided and be list type.")
+            raise ValueError(
+                f"rank's sort_by({self.sort_by}) must be provided and be list type."
+            )
 
         if not self.visualization and not isinstance(self.visualization, dict):
-            raise ValueError(f"rank's visualization({self.visualization}) "
-                             f"must be provided and be dict type.")
+            raise ValueError(
+                f"rank's visualization({self.visualization}) "
+                f"must be provided and be dict type."
+            )
 
         if not self.selected_dataitem and not isinstance(self.selected_dataitem, dict):
-            raise ValueError(f"rank's selected_dataitem({self.selected_dataitem}) "
-                             f"must be provided and be dict type.")
+            raise ValueError(
+                f"rank's selected_dataitem({self.selected_dataitem}) "
+                f"must be provided and be dict type."
+            )
 
         if not self.selected_dataitem.get("paradigms"):
             raise ValueError("not found paradigms of selected_dataitem in rank.")
@@ -82,8 +85,10 @@ def _check_fields(self):
             raise ValueError("not found metrics of selected_dataitem in rank.")
 
         if not self.save_mode and not isinstance(self.save_mode, list):
-            raise ValueError(f"rank's save_mode({self.save_mode}) "
-                             f"must be provided and be list type.")
+            raise ValueError(
+                f"rank's save_mode({self.save_mode}) "
+                f"must be provided and be list type."
+            )
 
     @classmethod
     def _get_all_metric_names(cls, test_results) -> list:
@@ -133,7 +138,7 @@ def _sort_all_df(self, all_df, all_metric_names):
 
             if metric_name not in all_metric_names:
                 continue
-
+            print(metric_name)
             sort_metric_list.append(metric_name)
             is_ascend_list.append(ele.get(metric_name) == "ascend")
 
@@ -198,7 +203,15 @@ def _get_selected(self, test_cases, test_results) -> pd.DataFrame:
         if metric_names == ["all"]:
             metric_names = self._get_all_metric_names(test_results)
 
-        header = ["algorithm", *metric_names, "paradigm", *module_types, *hps_names, "time", "url"]
+        header = [
+            "algorithm",
+            *metric_names,
+            "paradigm",
+            *module_types,
+            *hps_names,
+            "time",
+            "url",
+        ]
 
         all_df = copy.deepcopy(self.all_df)
         selected_df = pd.DataFrame(all_df, columns=header)
@@ -220,8 +233,8 @@ def _draw_pictures(self, test_cases, test_results):
         for test_case in test_cases:
             out_put = test_case.output_dir
             test_result = test_results[test_case.id][0]
-            matrix = test_result.get('Matrix')
-            #print(out_put)
+            matrix = test_result.get("Matrix")
+            # print(out_put)
             for key in matrix.keys():
                 draw_heatmap_picture(out_put, key, matrix[key])
 
@@ -287,4 +300,5 @@ def plot(self):
             except Exception as err:
                 raise RuntimeError(
                     f"process visualization(method={method}) of "
-                    f"rank file({self.selected_rank_file}) failed, error: {err}.") from err
+                    f"rank file({self.selected_rank_file}) failed, error: {err}."
+                ) from err

core/testcasecontroller/algorithm/algorithm.py

@@ -73,12 +73,14 @@ def __init__(self, name, config):
             'non_iid_ratio': 0.6,
             "label_data_ratio": 1.0
         }
+
         self.initial_model_url: str = ""
         self.modules: list = []
         self.modules_list = None
         self._parse_config(config)
         self._load_third_party_packages()
 
+    # pylint: disable=R0911
     def paradigm(self, workspace: str, **kwargs):
         """
         get test process of AI algorithm paradigm.
@@ -100,7 +102,6 @@ def paradigm(self, workspace: str, **kwargs):
         # pylint: disable=C0103
         for k, v in self.__dict__.items():
             config.update({k: v})
-
         if self.paradigm_type == ParadigmType.SINGLE_TASK_LEARNING.value:
             return SingleTaskLearning(workspace, **config)