Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update step 5 (evaluation) #64

Open
Mandresy-code opened this issue Sep 4, 2024 · 3 comments · May be fixed by #54
Open

Update step 5 (evaluation) #64

Mandresy-code opened this issue Sep 4, 2024 · 3 comments · May be fixed by #54
Assignees
@tztsai @Mandresy-code
Milestone
Generalisation of...

Comments

@Mandresy-code
Copy link
Collaborator

Mandresy-code commented Sep 4, 2024 •
edited by ma595
Loading

With the MLacc_results.csv, there are no .txt files anymore (e.g. som_R2.txt), which are needed in the plot_metric function (in eval_plot_un.py). This new issue aim is to update this function.

  • Add more information to the MLacc_results.csv i.e. indicating exactly what we're looping over.
@Mandresy-code Mandresy-code self-assigned this Sep 4, 2024
@Mandresy-code Mandresy-code linked a pull request Sep 18, 2024 that will close this issue
64 update step 5 evaluation #68
Closed
@ma595 ma595 mentioned this issue Oct 15, 2024
Closed
@ma595 ma595 linked a pull request Oct 16, 2024 that will close this issue
Add new models (Neural Networks, Ridge Regression, Gradient Boosting) + Ensemble method #54
Open
4 tasks
@ma595 ma595 removed a link to a pull request Oct 16, 2024
64 update step 5 evaluation #68
Closed
@ma595
Copy link
Collaborator

ma595 commented Oct 16, 2024

Currently being addressed in #54

@ma595 ma595 mentioned this issue Oct 16, 2024
Open
@ma595
Copy link
Collaborator

ma595 commented Oct 29, 2024

When running with CNP2 the following error is obtained:

Counter(over_samples_y):Counter({2: 25, 0: 25, 1: 25})
processing npp...
processing npp, variable npp_longterm, index (2,) (dim: ['pft'])...
Counter(lab):Counter({1: 54, 2: 13, 0: 9})
Counter(over_samples_y):Counter({0: 54, 1: 54, 2: 54})
processing npp, variable npp_longterm, index (3,) (dim: ['pft'])...
Counter(lab):Counter({2: 45, 0: 20, 1: 11})
Counter(over_samples_y):Counter({0: 45, 1: 45, 2: 45})
processing npp, variable npp_longterm, index (4,) (dim: ['pft'])...
Counter(lab):Counter({2: 64, 0: 37, 1: 30})
Counter(over_samples_y):Counter({1: 64, 0: 64, 2: 64})
processing npp, variable npp_longterm, index (5,) (dim: ['pft'])...
Counter(lab):Counter({1: 43, 0: 37, 2: 21})
Counter(over_samples_y):Counter({1: 43, 2: 43, 0: 43})
processing npp, variable npp_longterm, index (6,) (dim: ['pft'])...
Counter(lab):Counter({2: 49, 1: 28, 0: 12})
Counter(over_samples_y):Counter({0: 49, 1: 49, 2: 49})
processing npp, variable npp_longterm, index (7,) (dim: ['pft'])...
Counter(lab):Counter({0: 22, 1: 16, 2: 14})
Counter(over_samples_y):Counter({1: 22, 0: 22, 2: 22})
processing npp, variable npp_longterm, index (8,) (dim: ['pft'])...
Counter(lab):Counter({1: 24, 0: 20, 2: 9})
Counter(over_samples_y):Counter({1: 24, 2: 24, 0: 24})
processing npp, variable npp_longterm, index (9,) (dim: ['pft'])...
Counter(lab):Counter({0: 52, 2: 12, 1: 7})
Counter(over_samples_y):Counter({0: 52, 1: 52, 2: 52})
processing npp, variable npp_longterm, index (10,) (dim: ['pft'])...
Counter(lab):Counter({2: 36, 1: 23, 0: 19})
Counter(over_samples_y):Counter({0: 36, 2: 36, 1: 36})
processing npp, variable npp_longterm, index (11,) (dim: ['pft'])...
Counter(lab):Counter({1: 46, 0: 23, 2: 14})
Counter(over_samples_y):Counter({0: 46, 2: 46, 1: 46})
processing npp, variable npp_longterm, index (12,) (dim: ['pft'])...
Counter(lab):Counter({0: 71, 1: 19, 2: 14})
Counter(over_samples_y):Counter({0: 71, 1: 71, 2: 71})
processing npp, variable npp_longterm, index (13,) (dim: ['pft'])...
Counter(lab):Counter({0: 49, 1: 19, 2: 7})
Counter(over_samples_y):Counter({0: 49, 2: 49, 1: 49})
processing npp, variable npp_longterm, index (14,) (dim: ['pft'])...
Counter(lab):Counter({0: 57, 1: 10, 2: 5})
Counter(lab):Counter({0: 61, 1: 11})
Counter(lab):Counter({0: 61, 1: 11})
Counter(over_samples_y):Counter({0: 61, 1: 61})
processing npp, variable npp_longterm, index (15,) (dim: ['pft'])...
Counter(lab):Counter({1: 20, 2: 19, 0: 18})
Counter(over_samples_y):Counter({2: 20, 1: 20, 0: 20})
Traceback (most recent call last):
  File "/rds/project/rds-5mCMIDBOkPU/rse/spin/SPINacc/main.py", line 217, in <module>
    res_df = ML.MLloop(
  File "/rds/project/rds-5mCMIDBOkPU/rse/spin/SPINacc/Tools/ML.py", line 353, in MLloop
    result = list(filter(None, pool.map(MLmap_multidim, *zip(*result))))
  File "/usr/local/software/spack/spack-views/rocky8-icelake-20220710/python-3.9.12/gcc-11.3.0/pdcqf4o5kyppnbgu6s4xkujbzvxmezu5/lib/python3.9/concurrent/futures/_base.py", line 609, in result_iterator
    yield fs.pop().result()
  File "/usr/local/software/spack/spack-views/rocky8-icelake-20220710/python-3.9.12/gcc-11.3.0/pdcqf4o5kyppnbgu6s4xkujbzvxmezu5/lib/python3.9/concurrent/futures/_base.py", line 446, in result
    return self.__get_result()
  File "/usr/local/software/spack/spack-views/rocky8-icelake-20220710/python-3.9.12/gcc-11.3.0/pdcqf4o5kyppnbgu6s4xkujbzvxmezu5/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/usr/local/software/spack/spack-views/rocky8-icelake-20220710/python-3.9.12/gcc-11.3.0/pdcqf4o5kyppnbgu6s4xkujbzvxmezu5/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/rds/project/rds-5mCMIDBOkPU/rse/spin/SPINacc/Tools/ML.py", line 161, in MLmap_multidim
    ipft = int(varname.split("_")[1])
ValueError: invalid literal for int() with base 10: 'longterm'

@tztsai
Copy link
Collaborator

tztsai commented Nov 1, 2024

When running with CNP2 the following error is obtained:

Counter(over_samples_y):Counter({2: 25, 0: 25, 1: 25})
processing npp...
processing npp, variable npp_longterm, index (2,) (dim: ['pft'])...
Counter(lab):Counter({1: 54, 2: 13, 0: 9})
Counter(over_samples_y):Counter({0: 54, 1: 54, 2: 54})
processing npp, variable npp_longterm, index (3,) (dim: ['pft'])...
Counter(lab):Counter({2: 45, 0: 20, 1: 11})
Counter(over_samples_y):Counter({0: 45, 1: 45, 2: 45})
processing npp, variable npp_longterm, index (4,) (dim: ['pft'])...
Counter(lab):Counter({2: 64, 0: 37, 1: 30})
Counter(over_samples_y):Counter({1: 64, 0: 64, 2: 64})
processing npp, variable npp_longterm, index (5,) (dim: ['pft'])...
Counter(lab):Counter({1: 43, 0: 37, 2: 21})
Counter(over_samples_y):Counter({1: 43, 2: 43, 0: 43})
processing npp, variable npp_longterm, index (6,) (dim: ['pft'])...
Counter(lab):Counter({2: 49, 1: 28, 0: 12})
Counter(over_samples_y):Counter({0: 49, 1: 49, 2: 49})
processing npp, variable npp_longterm, index (7,) (dim: ['pft'])...
Counter(lab):Counter({0: 22, 1: 16, 2: 14})
Counter(over_samples_y):Counter({1: 22, 0: 22, 2: 22})
processing npp, variable npp_longterm, index (8,) (dim: ['pft'])...
Counter(lab):Counter({1: 24, 0: 20, 2: 9})
Counter(over_samples_y):Counter({1: 24, 2: 24, 0: 24})
processing npp, variable npp_longterm, index (9,) (dim: ['pft'])...
Counter(lab):Counter({0: 52, 2: 12, 1: 7})
Counter(over_samples_y):Counter({0: 52, 1: 52, 2: 52})
processing npp, variable npp_longterm, index (10,) (dim: ['pft'])...
Counter(lab):Counter({2: 36, 1: 23, 0: 19})
Counter(over_samples_y):Counter({0: 36, 2: 36, 1: 36})
processing npp, variable npp_longterm, index (11,) (dim: ['pft'])...
Counter(lab):Counter({1: 46, 0: 23, 2: 14})
Counter(over_samples_y):Counter({0: 46, 2: 46, 1: 46})
processing npp, variable npp_longterm, index (12,) (dim: ['pft'])...
Counter(lab):Counter({0: 71, 1: 19, 2: 14})
Counter(over_samples_y):Counter({0: 71, 1: 71, 2: 71})
processing npp, variable npp_longterm, index (13,) (dim: ['pft'])...
Counter(lab):Counter({0: 49, 1: 19, 2: 7})
Counter(over_samples_y):Counter({0: 49, 2: 49, 1: 49})
processing npp, variable npp_longterm, index (14,) (dim: ['pft'])...
Counter(lab):Counter({0: 57, 1: 10, 2: 5})
Counter(lab):Counter({0: 61, 1: 11})
Counter(lab):Counter({0: 61, 1: 11})
Counter(over_samples_y):Counter({0: 61, 1: 61})
processing npp, variable npp_longterm, index (15,) (dim: ['pft'])...
Counter(lab):Counter({1: 20, 2: 19, 0: 18})
Counter(over_samples_y):Counter({2: 20, 1: 20, 0: 20})
Traceback (most recent call last):
  File "/rds/project/rds-5mCMIDBOkPU/rse/spin/SPINacc/main.py", line 217, in <module>
    res_df = ML.MLloop(
  File "/rds/project/rds-5mCMIDBOkPU/rse/spin/SPINacc/Tools/ML.py", line 353, in MLloop
    result = list(filter(None, pool.map(MLmap_multidim, *zip(*result))))
  File "/usr/local/software/spack/spack-views/rocky8-icelake-20220710/python-3.9.12/gcc-11.3.0/pdcqf4o5kyppnbgu6s4xkujbzvxmezu5/lib/python3.9/concurrent/futures/_base.py", line 609, in result_iterator
    yield fs.pop().result()
  File "/usr/local/software/spack/spack-views/rocky8-icelake-20220710/python-3.9.12/gcc-11.3.0/pdcqf4o5kyppnbgu6s4xkujbzvxmezu5/lib/python3.9/concurrent/futures/_base.py", line 446, in result
    return self.__get_result()
  File "/usr/local/software/spack/spack-views/rocky8-icelake-20220710/python-3.9.12/gcc-11.3.0/pdcqf4o5kyppnbgu6s4xkujbzvxmezu5/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/usr/local/software/spack/spack-views/rocky8-icelake-20220710/python-3.9.12/gcc-11.3.0/pdcqf4o5kyppnbgu6s4xkujbzvxmezu5/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/rds/project/rds-5mCMIDBOkPU/rse/spin/SPINacc/Tools/ML.py", line 161, in MLmap_multidim
    ipft = int(varname.split("_")[1])
ValueError: invalid literal for int() with base 10: 'longterm'

It's because this part of code assumes the naming of variables follows the format in the current dataset. We need to modify it for CNP2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tztsai tztsai

@Mandresy-code Mandresy-code

Labels
None yet
Projects
None yet
Milestone
Generalisation of SPINacc to other ve...
3 participants
@tztsai @ma595 @Mandresy-code