Longitudinal preprocessor parallelization #246
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
MaryanMorel
force-pushed
the
longitudinal-preprocessor-parallelization
branch
2 times, most recently
from
2a5945a to
ab57615
Compare
MaryanMorel
force-pushed
the
longitudinal-preprocessor-parallelization
branch
2 times, most recently
from
94e0ff2 to
3d2275a
Compare
MaryanMorel
force-pushed
the
longitudinal-preprocessor-parallelization
branch
from
3d2275a to
04f35d6
Compare
PhilipDeegan
reviewed
Jun 8, 2018
| @@ -355,7 +355,8 @@ def fit_kfold_cv(self, features, labels, censoring, C_tv_range: tuple = (), | |||
| features, labels, censoring) | |||
| # split the data with stratified KFold | |||
| kf = StratifiedKFold(n_folds, shuffle, self.random_state) | |||
| labels_interval = np.nonzero(p_labels)[1] | |||
| # labels_interval = np.nonzero(p_labels)[1] | |||
There was a problem hiding this comment.
Is there a reason to keep this?
Mbompr
reviewed
Jun 8, 2018
| @@ -68,14 +73,15 @@ void LongitudinalFeaturesLagger::sparse_lag_preprocessor(ArrayULong &row, | |||
| ArrayULong &out_col, | |||
| ArrayDouble &out_data, | |||
| ulong censoring) const { | |||
| // TODO: add checks here ? Or do them in Python ? | |||
There was a problem hiding this comment.
Checks must be done in C++ to avoid segfault possibilities !
| @@ -14,9 +15,14 @@ class LongitudinalPreprocessor(ABC, Base): | |||
| set to the number of cores. | |||
| """ | |||
|
|
|||
| def __init__(self, n_jobs=-1): | |||
| _attrinfos = {'n_jobs': {'writable': True}} | |||
There was a problem hiding this comment.
This _attrinfos is not necessary
| caution. | ||
| """ | ||
| global _cpp_preprocessor | ||
| _cpp_preprocessor = _LongitudinalFeaturesLagger(n_intervals, n_lags) |
There was a problem hiding this comment.
Here is a clever way (using __call__ in a class) to avoid global variable, did you give it a try?
|
has this been obsoleted by #373 ? |
|
Yep
Le ven. 12 juil. 2019 à 22:46, ♦♣♠♥ <[email protected]> a écrit :
… has this been obsoleted by #373
<#373> ?
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#246?email_source=notifications&email_token=ABIEHAWIQCH76PKR6PQTAX3P7DNR5A5CNFSM4FCGDNJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ2WSDA#issuecomment-511011084>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABIEHAWPMZ2NPSYZYDV4M7LP7DNR5ANCNFSM4FCGDNJA>
.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Make the longitudinal preprocessors serializable and make them parallel using python.multiprocessing.
To avoid sending the whole instance of each class to spawned or forked processes, I used an initializer in the process Pool to create a cpp object instance per process. I also used higher-order functions for the same purpose.
The drawback of doing so is having the pool initializers (
_inject_cpp_objectmethods) usingglobalvariables to store the cpp objects. This should not have dire consequences in the multiprocessing context, as each process possesses its own namespace. However, such methods could cause some trouble if called outside of this context (rogue/monkey user).