vrx_2022 testing

Testing your solution

This tutorial will allow you to verify that your submission is working as expected.

Dependencies

The steps below assume you have Docker installed and configured to work with your Nvidia graphics card.

• If you have not yet set up Docker on your system, complete steps 1 and 2 of this tutorial before proceeding. Note that step 3 (installing Rocker) is needed for building the vrx-server docker image.
• If you do not have an Nvidia card, you can still validate your submission by leaving out the -n option wherever it appears below. Note, however, that without an Nvidia card, the simulation is likely to run slower than real-time.
• In addition, currently you must also have followed the host-based install instructions to install VRX on your host system.
  • Note: we are working to eliminate this dependency.

Prerequisites

To complete this tutorial, you will need to have already prepared a solution to one or more VRX tasks. This means:

• You have created a Docker image containing your solution to one or more tasks and uploaded it to Docker hub.
• You have created the required files for a phase 2 or phase 3 submission.

Step 1: Prepare a local testing environment

This step installs the necessary testing tools on your system. You should only need to do this once.

1. Change into the vrx_ws/src directory you created when installing vrx.

   cd ~/vrx_ws/src

2. Clone the vrx-docker repository.

   git clone https://github.com/osrf/vrx-docker

   This should create a new vrx-docker directory alongside the original vrx repository directory.

3. Source your bash.setup file, change into the vrx-docker directory, and set the variable TEAM for later use:

   source ~/vrx_ws/devel/setup.bash
   cd vrx-docker
   TEAM=<your_team_name>

   Replace <your_team_name> with the team name you will use for your submission.

4. Build the vrx-server docker image (may take 30-60 minutes the first time):

   ./vrx_server/build_image.bash -n

Step 2: Configure your test

In this step you will set up your platform and specify which task you want to evaluate.

1. Copy your submission files (yaml config files and docker hub image name) to the vrx-docker/team_config folder.

2. Prepare your team's vehicle according to the configuration you provided:

   ./prepare_team_wamv.bash "$TEAM"

   Note that this will produce a REQUIRED process [wamv_config/wamv_generator-2] has died! message, which is expected.

3. Define a TASK variable to indicate the task you wish to test against.

   TASK=stationkeeping

   Other valid options include wayfinding, perception, wildlife, gymkhana, and scan_dock_deliver.

4. Get the practice worlds provided for your task:

   cd ~/vrx_ws/src/vrx-docker
   mkdir -p generated/task_generated/$TASK/worlds
   cp ~/vrx_ws/src/vrx/vrx_gazebo/worlds/2022_practice/$TASK*.world generated/task_generated/$TASK/worlds/
   

5. Define a TRIAL variable to indicate which of the task worlds you would like to run (options are 0,1,2).

   TRIAL=0

Step 3: Run and score your solution

This step will test your submission on the task and trial specified above:

1. Execute the run_trial.bash script with your team, task and trial variables as arguments:

   ./run_trial.bash -n $TEAM $TASK $TRIAL

   This command will run your submission image and the vrx-server image at the same time and generate multiple log files which are saved in the vrx-docker/generated/logs directory.

2. View your score for the task:

   cat generated/logs/$TEAM/$TASK/$TRIAL/trial_score.txt

3. See a replay of your system's performance on the task and verify that it behaved as you expected:

   ./replay_trial.bash -n ghostship wayfinding 0

Troubleshooting

If you have completed the steps above and playback of your trial shows that:

• either your WAMV does not move at all or
• the behavior of your WAMV in the container does not match its behavior when run directly on a host system

then there may be a problem with your Docker image setup. In this case, refer to our Docker troubleshooting tutorials for some suggestions for detecting and correcting docker-related problems.