Merge pull request #97 from jackyk02/main

[Python Target] CARLA Demo

examples/Python/src/CARLA/README.md

@@ -0,0 +1,74 @@
+# CARLA
+
+This guide provides detailed instructions for running the CARLA simulator with LF. Ensure the following prerequisites are met before proceeding with the installation:
+
+- **System requirements.** CARLA is built for Windows and Linux systems.
+- **An adequate GPU.** CARLA aims for realistic simulations, so the server needs at least a 6 GB GPU although we would recommend 8 GB. A dedicated GPU is highly recommended for machine learning.
+- **Disk space.** CARLA will use about 20 GB of space.
+- **Two TCP ports and good internet connection.** 2000 and 2001 by default. Make sure that these ports are not blocked by firewalls or any other applications.
+- **Other requirements.** CARLA requires some Python dependencies. Install the dependencies with the following command:
+
+```bash
+pip3 install --user pygame numpy carla
+```
+
+### **Downloading CARLA**
+
+Download CARLA version 0.9.15 from [the official repository](https://github.com/carla-simulator/carla/releases/tag/0.9.15/). Extract the release file, which contains a precompiled version of the simulator.
+
+### **Running the CARLA Server**
+
+1. **On Windows:** Navigate to the extracted CARLA folder and double-click the `CarlaUE4.exe` file to start the server. A window will appear indicating the server is active.
+2. **On Linux:** In the terminal, navigate to the CARLA folder and run `./CarlaUE4.sh` to initiate the server.
+
+Note: Please restart the CARLA Server before running each of the examples below.
+
+### **Compiling LF**
+
+Compile the `carla_sync.lf` and `carla_async.lf` using the `lfc` command. Make sure that `carla_client.py` is in the include folder.
+
+### Synchronous Example - Manual Control
+
+Run the generated script `/.carla_sync` which connects to the CARLA server in synchronous mode and initiates a driving simulator in pygame. Use ARROWS or WASD keys for control. Image and IMU data will be displayed on the console.
+
+<img src="img/carla_manual.png" alt="drawing" width="800"/>
+
+### Asynchronous Example - Driving in Circles
+
+Run the generated script `/.carla_async` which connects to the CARLA server in asynchronous mode and initiates a driving simulator in pygame. The vehicle will drive in circles. Image and IMU data will be displayed on the console.
+
+<img src="img/carla_circles.gif" alt="drawing" width="800"/>
+
+### Possible Configurations of CARLA Server
+
+The configuration of time-step and synchrony, leads for different settings. Here is a brief summary on the possibilities.
+
+|                   | Fixed time-step                                                        | Variable time-step                 |
+| ----------------- | ---------------------------------------------------------------------- | ---------------------------------- |
+| Synchronous mode  | Client is in total control over the simulation and its information.    | Risk of non reliable simulations.  |
+| Asynchronous mode | Good time references for information. Server runs as fast as possible. | Non easily repeatable simulations. |
+
+- **Synchronous mode + variable time-step.** This is almost for sure a non-desirable state. Physics cannot run properly when the time-step is bigger than 0.1s, and if the server has to wait for the client to compute the steps, this is likely to happen. Simulation time and physics will not be in synchrony. The simulation will not be reliable.
+- **Asynchronous mode + variable time-step.** This is the default CARLA state. Client and server are asynchronous. The simulation time flows according to the real time. Reenacting the simulation needs to take into account float-arithmetic error, and possible differences in time steps between servers.
+- **Asynchronous mode + fixed time-step.** The server will run as fast as possible. The information retrieved will be easily related with an exact moment in the simulation. This configuration makes possible to simulate long periods of time in much less real time, if the server is fast enough.
+- **Synchronous mode + fixed time-step.** The client will rule the simulation. The time step will be fixed. The server will not compute the following step until the client sends a tick. This is the best mode when synchrony and precision is relevant, especially when dealing with slow clients or different elements retrieving information.
+
+**Fixed time-step**
+
+Fixed delta seconds can be set in the world settings.
+
+```python
+settings = world.get_settings()
+settings.fixed_delta_seconds = 0.05
+world.apply_settings(settings)
+```
+
+**Variable time-step**
+
+The default mode in CARLA. The simulation time that goes by between steps will be the time that the server takes to compute these.
+
+```python
+settings = world.get_settings()
+settings.fixed_delta_seconds = None # Set a variable time-step
+world.apply_settings(settings)
+```

examples/Python/src/CARLA/include/carla_client.py

@@ -0,0 +1,229 @@
+import carla
+import weakref
+import random
+import pygame
+from pygame.locals import K_ESCAPE
+from pygame.locals import K_SPACE
+from pygame.locals import K_a
+from pygame.locals import K_d
+from pygame.locals import K_s
+from pygame.locals import K_w
+import numpy as np
+from abc import ABC, abstractmethod
+
+VIEW_WIDTH = 1920//2
+VIEW_HEIGHT = 1080//2
+VIEW_FOV = 90
+BB_COLOR = (248, 64, 24)
+
+
+def process_image_data(image):
+    """
+    Processes the raw image data from the camera sensor.
+    """
+    array = np.frombuffer(image.raw_data, dtype=np.dtype("uint8"))
+    array = np.reshape(array, (image.height, image.width, 4))
+    array = array[:, :, :3]
+    array = array[:, :, ::-1]
+    return array
+
+
+class BasicClient(ABC):
+    """
+    Basic implementation of a synchronous client.
+    """
+    @abstractmethod
+    def game_start(self):
+        pass
+
+    @abstractmethod
+    def control(self, car):
+        pass
+
+    def __init__(self):
+        self.client = None
+        self.world = None
+        self.camera = None
+        self.car = None
+
+        self.display = None
+        self.image = None
+        self.capture = True
+
+    def camera_blueprint(self):
+        """
+        Returns camera blueprint.
+        """
+        camera_bp = self.world.get_blueprint_library().find("sensor.camera.rgb")
+        camera_bp.set_attribute("image_size_x", str(VIEW_WIDTH))
+        camera_bp.set_attribute("image_size_y", str(VIEW_HEIGHT))
+        camera_bp.set_attribute("fov", str(VIEW_FOV))
+        return camera_bp
+
+    def set_synchronous_mode(self, synchronous_mode):
+        """
+        Sets synchronous mode.
+        """
+        settings = self.world.get_settings()
+        settings.fixed_delta_seconds = 0.05
+        settings.synchronous_mode = synchronous_mode
+        self.world.apply_settings(settings)
+
+    def setup_car(self):
+        """
+        Spawns actor-vehicle to be controlled.
+
+        """
+        car_bp = self.world.get_blueprint_library().filter("vehicle.*")[0]
+        location = random.choice(self.world.get_map().get_spawn_points())
+        self.car = self.world.spawn_actor(car_bp, location)
+
+    def setup_imu(self):
+        """
+        Spawns actor-IMU sensor to be used to get IMU data.
+        """
+        imu_bp = self.world.get_blueprint_library().find("sensor.other.imu")
+        imu_transform = carla.Transform(carla.Location(x=0.5, z=2.8))
+        self.imu = self.world.spawn_actor(
+            imu_bp, imu_transform, attach_to=self.car)
+        weak_self = weakref.ref(self)
+        self.imu.listen(lambda data: weak_self().process_imu_data(data))
+
+    def setup_camera(self):
+        """
+        Spawns actor-camera to be used to render view.
+        """
+        camera_transform = carla.Transform(carla.Location(
+            x=-5.5, z=2.8), carla.Rotation(pitch=-15))
+        self.camera = self.world.spawn_actor(
+            self.camera_blueprint(), camera_transform, attach_to=self.car)
+        weak_self = weakref.ref(self)
+        self.camera.listen(
+            lambda image: weak_self().set_image(weak_self, image))
+
+        calibration = np.identity(3)
+        calibration[0, 2] = VIEW_WIDTH / 2.0
+        calibration[1, 2] = VIEW_HEIGHT / 2.0
+        calibration[0, 0] = calibration[1, 1] = VIEW_WIDTH / \
+            (2.0 * np.tan(VIEW_FOV * np.pi / 360.0))
+        self.camera.calibration = calibration
+
+    def process_imu_data(self, data):
+        """
+        Processes and stores IMU data.
+        """
+        self.imu_data = data
+
+    @staticmethod
+    def set_image(weak_self, img):
+        self = weak_self()
+        if self.capture:
+            self.image = img
+            self.capture = False
+
+    def render(self, display):
+        """
+        Renders the image on the display.
+        """
+        if self.image is not None:
+            array = process_image_data(self.image)
+            surface = pygame.surfarray.make_surface(array.swapaxes(0, 1))
+            display.blit(surface, (0, 0))
+            return array
+
+    def game_step(self):
+        """
+        Simulates one step in the game, processing inputs and rendering the image.
+        """
+        self.world.tick()
+        self.capture = True
+        self.pygame_clock.tick_busy_loop(20)
+        image = self.render(self.display)
+
+        pygame.display.flip()
+        pygame.event.pump()
+
+        sensor_data = None
+        if not self.image or hasattr(self, "imu_data"):
+            sensor_data = [self.image, self.imu_data.accelerometer.x]
+
+        self.control(self.car)
+
+        return sensor_data
+
+
+class SyncClient(BasicClient):
+    def control(self, car):
+        keys = pygame.key.get_pressed()
+        if keys[K_ESCAPE]:
+            return True
+
+        control = car.get_control()
+        control.throttle = 0
+        if keys[K_w]:
+            control.throttle = 1
+            control.reverse = False
+        elif keys[K_s]:
+            control.throttle = 1
+            control.reverse = True
+        if keys[K_a]:
+            control.steer = max(-1., min(control.steer - 0.05, 0))
+        elif keys[K_d]:
+            control.steer = min(1., max(control.steer + 0.05, 0))
+        else:
+            control.steer = 0
+        control.hand_brake = keys[K_SPACE]
+
+        car.apply_control(control)
+        return False
+
+    def game_start(self):
+        """
+        Initializes the game, setting up the client, car, camera, IMU, and display.
+        """
+        pygame.init()
+
+        self.client = carla.Client("localhost", 2000)
+        self.client.set_timeout(10.0)
+        self.world = self.client.get_world()
+
+        self.setup_car()
+        self.setup_camera()
+        self.setup_imu()
+
+        self.display = pygame.display.set_mode(
+            (VIEW_WIDTH, VIEW_HEIGHT), pygame.HWSURFACE | pygame.DOUBLEBUF)
+        self.pygame_clock = pygame.time.Clock()
+
+        self.set_synchronous_mode(True)
+        vehicles = self.world.get_actors().filter("vehicle.*")
+
+
+class AsyncClient(BasicClient):
+    def control(self, car):
+        control = car.get_control()
+        control.throttle = 1
+        control.steer = 1
+        car.apply_control(control)
+        return False
+
+    def game_start(self):
+        """
+        Initializes the game, setting up the client, car, camera, IMU, and display.
+        """
+        pygame.init()
+
+        self.client = carla.Client("172.23.112.1", 2000)
+        self.client.set_timeout(10.0)
+        self.world = self.client.get_world()
+
+        self.setup_car()
+        self.setup_camera()
+        self.setup_imu()
+
+        self.display = pygame.display.set_mode(
+            (VIEW_WIDTH, VIEW_HEIGHT), pygame.HWSURFACE | pygame.DOUBLEBUF)
+        self.pygame_clock = pygame.time.Clock()
+
+        self.set_synchronous_mode(False)
+        vehicles = self.world.get_actors().filter("vehicle.*")

examples/Python/src/CARLA/src/carla_async.lf

@@ -0,0 +1,71 @@
+target Python {
+  files: include/carla_client.py
+}
+
+preamble {=
+  from carla_client import AsyncClient
+  from carla_client import process_image_data
+=}
+
+reactor Carla {
+  input actions
+  output raw_image
+  output imu
+  state client
+
+  reaction(startup) -> raw_image, imu {=
+    self.client = AsyncClient()
+    self.client.game_start()
+    sensor_data = self.client.game_step()
+    raw_image.set(sensor_data[0])
+    imu.set(sensor_data[1])
+  =}
+
+  reaction(actions) -> raw_image, imu {=
+    sensor_data = self.client.game_step()
+    raw_image.set(sensor_data[0])
+    imu.set(sensor_data[1])
+  =}
+}
+
+reactor Image {
+  input raw_image
+  output processed_image
+
+  reaction(startup) {=  =}
+
+  reaction(raw_image) -> processed_image {=
+    if raw_image.value:
+        array = process_image_data(raw_image.value)
+        processed_image.set(array)
+    else:
+        processed_image.set(None)
+  =}
+}
+
+reactor Fusion {
+  input imu
+  input processed_image
+  output actions
+
+  reaction(startup) {=  =}
+
+  reaction(imu, processed_image) -> actions {=
+    if imu is not None:
+        print("IMU Data: ", imu.value)
+    if processed_image.value is not None:
+        print("Image Data: ", processed_image.value[0][0])
+    actions.set(0)
+  =}
+}
+
+main reactor {
+  carla = new Carla()
+  image = new Image()
+  fusion = new Fusion()
+
+  carla.raw_image -> image.raw_image
+  carla.imu -> fusion.imu
+  image.processed_image -> fusion.processed_image
+  fusion.actions -> carla.actions after 50 ms
+}