part of/ fraction of dataset/benchmark

[DraBard]

This seems trivial, but I couldnt find appropriate method for taking only a fraction of dataset/benchmark.
Let's say I don't want to train on whole benchmark and want to take only 20% of examples, obviously evenly taken
from all experiences.

How can I do it on this exemplary code where I use both, benchmark and dataset?

class DataLoader:
    def __init__(self, config):
        self.config = config
        self.logger = logging.getLogger(__name__)
        self.dataset = None
        self.prepare_dataset()

    def prepare_core50(self):
        val_fraction = self.config.get("dataset").get("validation_fraction")
        scenario = self.config.get("scenario").get("type")
        if scenario == "ni":
            core50 = CORe50(scenario="ni", mini=True, object_lvl=True)
            f = lambda exp: random_validation_split_strategy(val_fraction, False, exp)
        elif scenario == "nc":
            # Load the CORe50 dataset
            core50_train = CORe50Dataset(train=True, mini=True)
            core50_test = CORe50Dataset(train=False, mini=True)
            # Create different split
            core50 = nc_benchmark(core50_train, core50_test, n_experiences=10, shuffle=False, task_labels=False, train_transform=transform(), eval_transform=transform())
            f = lambda exp: class_balanced_split_strategy(val_fraction, exp)
        else:
            raise NameError("Scenario name unknown")
        
        # num_samples = int(len(core50) * 0.2)
        core50 = benchmark_with_validation_stream(core50, custom_split_strategy=f)
        return core50

    def prepare_dataset(self):
        dataset_name = self.config.get("dataset").get("name")
        if dataset_name == "core50":
            self.dataset = self.prepare_core50()
        else:
            # Handle other datasets or raise an error
            raise ValueError(f"Dataset '{dataset_name}' not supported")
        
        if self.dataset:
            self.logger.info(f'--- Task labels: {self.dataset.task_labels}')
        else:
            self.logger.warning('Dataset is not properly initialized')

    def get_train_stream(self):
        if self.dataset:
            return self.dataset.train_stream
        raise RuntimeError('Dataset is not initialized')

    def get_val_stream(self):
        if self.dataset:
            return self.dataset.valid_stream
        raise RuntimeError('Dataset is not initialized')

    def get_test_stream(self):
        if self.dataset:
            return self.dataset.test_stream
        raise RuntimeError('Dataset is not initialized')

[vlomonaco]

Hi @Bard2803! You can subset an AvalancheDataset: https://avalanche-api.continualai.org/en/v0.4.0/generated/avalanche.benchmarks.utils.AvalancheDataset.html#avalanche.benchmarks.utils.AvalancheDataset

As for the benchmarks we have a benchmark generator to create a validation stream: https://avalanche-api.continualai.org/en/v0.4.0/generated/avalanche.benchmarks.generators.benchmark_with_validation_stream.html

[DraBard]

@vlomonaco thank you very much for the answer. Probably I haven't explained it sufficiently. I constructed the benchamark for validation stream, without a problem. Now I have train, val and test streams. I want to take only part of the data samples from each of them. Let's say 20%, in order to increase the training speed. How can I do it to be sure that I have taken 20% evenly spread over let's say classes in class incremental scenario or instances in instance incremental scenario?

So, if first experience has classes 1,2,3,4,5 I want to take 20% of each of them. So I have 20% of class 1, 20% of class 2 etc... . Not 20% from whole stream, being 100% of class 1 and none of the rest (2, 3, 4, 5).

[vlomonaco]

The subset method of the AvalancheDataset allows you to specify the sequence of index you want to consider. So you can look at their labels and randomly select 20% for each class. Once you have the list of indices you can call the subset method and use the resulting dataset, no?

[DraBard]

Exactly, this is the approach I implemented. I just thought that maybe there is some specific method for this in avalanche. Thanks a lot for quick answers.

[BenCrulis]

Hello @vlomonaco ,

The subset method is really useful, but then is there a better way to do this? I am trying to sub-sample the evaluation data.

for exp_id, experience in enumerate(benchmark.train_stream):
        print("Start of experience ", experience.current_experience)

        cl_strategy.train(experience, shuffle=True)
        print('Training completed')

        print('Computing accuracy on the current test set')
        if fast_eval:
            old_dataset = benchmark.test_stream[0].dataset
            new_ds = benchmark.test_stream[0].dataset.subset(range(0, len(old_dataset), 20))
            cl_stream = CLStream("test", [DatasetExperience(dataset=new_ds, current_experience=experience.current_experience)])
            eval_stream = ClassificationStream("eval", benchmark)
            cl_strategy.eval(cl_stream)
        else:
            cl_strategy.eval(benchmark.test_stream)

It is a bit cumbersome and I had a hard time figuring this hack. It also somewhat messes up the logging, it creates new charts in Weights & Biases.
Note that directly modifying benchmark.test_stream[0].dataset or benchmark.test_stream[0]._dataset once before the loop doesn't work, for some reason (using the Core50 benchmark).