Skip to content

Creating Experiment and Runner

Jump to bottom
chaofengwu edited this page Jun 28, 2020 · 8 revisions

An experiment contains the scripts to run, the information about search space, and the resource to perform the experiment. A runner contains the experiment to run, the optimizer to use, the objective to optimize, the storage configuration, and other information to help the runner.

Here are several part to create an Experiment:

  • command template string(optional: setup template string and finish template string)
  • compute
  • parameters

Here are several parts to create a Runner:

  • experiment
  • storage
  • optimizer
  • objective function

To make this process clear, we optimize running time for an example.

command_template_string

This is the command template for the experiment. By substituting the value of parameter, we create a script for one trial. A simple example for changing running time is

command_template_string = """
#! /bin/bash
sleep ${myParam}
sleep ${myParamB}
sleep ${myParamC}
"""

Here we have three parameters to change, which corresponds to the parameters part. We could also run some other things such as a python script. For the optional part, setup_template_string and finish_template_string will run before and after command_template_string separately. An example for setup_template_string:

setup_template_string = """
conda activate paropt
"""

and for finish_template_string:

finish_template_string = """
conda deactivate
"""

compute

We have three different types of computing resource. Here are examples for each of the three types:

PBS_compute = PBSProCompute(cpus_per_node=1, walltime='1:00:00', scheduler_options='#PBS -P 12345678\n#PBS -l select=1:mem=1G\n#PBS -N PBSPro_paraopt_test', worker_init='module load openmpi\nsource activate paropt')

# run on AWS (need a server to run paraopt_service, and submit experiment via paropat_sdk)
AWS_compute = EC2Compute(instance_family='c5', instance_model='c5.2xlarge', ami='xxxxxxxx')

# run locally
LOCAL_compute = LocalCompute(max_threads=8)

parameters

Since we use three parameters in the command_template_string, we need to specify three parameters and their attributes:

parameters = [
        Parameter(name="myParam", type=PARAMETER_TYPE_INT, minimum=5, maximum=10),
        Parameter(name="myParamB", type=PARAMETER_TYPE_FLOAT, minimum=3, maximum=5),
        Parameter(name="myParamC", type=PARAMETER_TYPE_INT, minimum=3, maximum=5)
        ]

Here the name should correspond to the name in command_template_String. The type of parameter is specified by type, and the bound of parameters are set by minimum and maximum.

experiment

Given these the components, we could create an experiment.

experiment_inst = Experiment(
    tool_name='test_script',
    parameters=parameters,
    command_template_string=command_template_string,
    setup_template_string=setup_template_string,
    finish_template_string=finish_template_string,
    compute=LOCAL_compute
)

Here we need to specify tool_name for helping identify the experiment. We use LOCAL_compute here to run locally.

storage

For storage, we provides examples for local storage and AWS_RDS storage:

LOCAL_storage = RelationalDB(
    'sqlite',
    '',
    '',
    '',
    './liteTest',
)
AWSRDS_storage = RelationalDB(
    'postgresql',
    DB_USER,
    DB_PASSWORD,
    DB_HOST,
    DB_NAME
)

For LOCAL_storage, the ./liteTest is the file to store the result. For AWSRDS_storage, we need to set the more variable to access the database.

optimizer

We need to create a optimizer for this experiment. Here are some example for creating optimizers:

bayesian_optimizer = BayesianOptimizer(
    n_init=2,
    n_iter=1,
    alpha=1e-3,
    kappa=2.5, 
    utility='ucb', 
    budget=None, 
    converge_thres=None, 
    converge_steps=None
)

# search on 2*2*2 grid
grid_optimizer = GridSearch([2,2,2])

random_optimizer = RandomSearch(n_iter=10, 
    random_seed=None, 
    budget=None, 
    converge_thres=None, 
    converge_steps=None
)

coordinate_optimizer = CoordinateSearch(n_init=1, 
    n_iter=20, 
    random_seed=None, 
    budget=None, 
    converge_thres=None, 
    converge_steps=None
)

Objective function and Runner

Now we have all the components. We could create a runner:

po = ParslRunner(
    obj_func=getattr(paropt.runner.parsl, "timeCmd"),
    obj_func_params={'timeout': 15},
    optimizer=grid_optimizer,
    storage=LOCAL_storage,
    experiment=experiment_inst,
    logs_root_dir='./myTestLogs')

We need to specify the objective function to use by this line obj_func=getattr(paropt.runner.parsl, "timeCmd"). Here we select timeCmd, which count the time used to finish the command. We could pass parameters to objective function by `obj_func_params={'timeout': 15}.

Clone this wiki locally