How about add the real car four-domain communication to metadrive?

[CristinaKepner]

The real car four-domain communication refers to the integration of the communication information of the intelligent driving domain, intelligent cabin domain, vehicle body domain, and vehicle networking gateway domain. The integration of communication data with simulation training scenarios is an issue that is currently being explored.
Due to insufficient information, I cannot determine whether Metadrive can perform such processing. Can you provide more information about metadrive?

For example about the four-domain communication service：

1. Digital key service in the body domain and vehicle status mode setting service in the gateway domain of the Internet of Vehicles. These two services combine the analysis of the digital key signal and the vehicle status mode setting signal to confirm which vehicles are about to be started or locked in the cloud map. At the same time, through the vehicle data reporting service in the body domain, the data can be reported to the road network information system of the cloud. The combination of this system with the simulation scene generated by the automatic driving simulator is beneficial for the road network system to confirm whether to continue to track the vehicle information, and is conducive to the construction of diversified simulation scene information.
2. Cellular network management module, responsible for load dialing and cellular network status monitoring. By combining the information of the vehicle networking gateway domain with the simulation scene, the area range of each mobile communication supplier can be distinguished, and the signal range and status of vehicles in the simulation scene can be confirmed.

In short, I want to create a real-time autonomous driving simulation for vehicle-road coordination or dataset construction,but I am limited by my shallow knowledge and experience, as well as my understanding of the simulation.

[QuanyiLi]

Hi Cristina,

Vehicle-road coordination sounds interesting. But at this stage, we don't model any communication process between vehicles. Let alone vehicles and roads.

But I believe that MetaDrive is able to support this, as the biggest hurdle is not on the MetaDrive side but on how to model the communication process. Thus I think we should figure out some details like what kind of information to store in each entity and how to model the information exchange/update process for all entities. In this case, we can give accurate suggestions on how to adapt MetaDrive for your purpose.

[CristinaKepner]

Thank you for your answer, but there may be some misunderstandings in this.
What I want to say may be that in the real car environment, the data about the body itself, such as the percentage of window down, such as the front and rear positions of the seat, will be communicated through the domain controller. In addition, regarding the communication model, the DDS protocol can be used. Now it is true that due to the influence of various factors, vehicle-road coordination is difficult to achieve, but if it is possible to conduct joint simulation with metadrive, the simulation data of autonomous driving does not need to come from open-source data sets, and the pressure on the road-collected data sets will be much smaller. I want to use metadrive because of its offline and lightweight advantages. I just don't know how to access the communication data and perform operations such as simulation and rendering. And the computing resources needed to do these plans...

[QuanyiLi]

I see. So basically, MetaDrive control the whole simulation process via a set of manager like which maps to load, when to spawn an object, how to actuate the object, and so on. Let's take this ScenarioEnv as an example:

metadrive/metadrive/envs/scenario_env.py

Line 91 in d37e299

class ScenarioEnv(BaseEnv):

From the def setup_engine function:

metadrive/metadrive/envs/scenario_env.py

Line 146 in d37e299

def setup_engine(self):

You can find that the following managers are initialized and their functions are explained:

1. data_manager: loading/processing data from real-world dataset
2. traffic_manager: reading current scenario data and spawning traffic
3. map_manager: constructing real-world map of current scenario
4. light_manager: building traffic light
5. curriculum_manager: for controlling ML-planner curriculum training
6. agent_manager (initialized in the super().stepup_engine): managing agents in the world
   By having your own managers, you can compose your own simulation easily. Other trivial stuff will be finished by BaseEngine automatically.

Let's dive into the BaseEngine implementation and how it works. First, all simulations including MetaDrive are about how to move from one state to the next state and reset to the initial state. Thus the two functions BaseEngine.reset() and BaseEngine.step() is designed for controlling the forward/reset process. In these two functions, all managers will be called sequentially according to their priority. For example, data_manager has to be called before all other managers, as data has to bed loaded before constructing the scene and hence it has the highest priority. The calling processes of the two function are as follows:

def reset(self):
  # some trivial stuff ...
  for manager_name, manager in self._managers.items():
      # func 1
      manager.before_reset()

  for manager_name, manager in self.managers.items():
      # func 2
      manager.reset()

  for manager_name, manager in self.managers.items():
      # func 3
      manager.after_reset()
  # some trivial stuff ...

def step(self):
  # some trivial stuff ...
  for manager_name, manager in self._managers.items():
      # func 4
      manager.before_step()

  for manager_name, manager in self.managers.items():
      # func 5
      manager.step()

  for manager_name, manager in self.managers.items():
      # func 6
      manager.after_step()
  # some trivial stuff ...

Therefore, your simulation depends on which manager you are using and what will be executed in the six functions for each manager.

Here I will show you an example of ScenarioLightManager()https://github.com/metadriverse/metadrive/blob/d37e29956b9d55c02abbcb7fcb8c9169e1671b42/metadrive/manager/scenario_light_manager.py#L10 The basic idea of it is to get the scenario traffic light data inbefore_resetand spawn all traffic lights inafter_reset(). Then in every step when manager.after_step``` is called, it will update the traffic light status according to the real-world traffic light data.
Thus when to spawn objects, how to change their states in each step, and when to recycle/remove can be customized according to your needs.

You can find a lot of manager examples here: https://github.com/metadriverse/metadrive/tree/d37e29956b9d55c02abbcb7fcb8c9169e1671b42/metadrive/manager
Make sure you are familiar with some of them and some basic functions before starting. For example, each manager can access the internal engine state, which means you are allowed to access all other managers and all engine variables. I believe this system design is powerful and can help you with your project. One straightforward idea about your project is to write a new manager, which can access all vehicles' and roads via self.engine.traffic_manager.spawned_objects and self.engine.map_manager.current_map. After that, you can decide what information to retrieve from these objects and how to process this information. If you wanna the processed results, furthermore, help determine how to actuate the vehicle, you can still access the processed info in traffic_manager.engine.your_manager.

As for efficiency, we support disabling all rendering features by setting use_render=False, and it can reach 300-1000 FPS depending on whether your computation is heavy or not. It can be parallelized via some common libraries like Python's multiprocessing and ray. So we believe the efficiency is satisfying for most applications. For rendering efficiency, we now provide three rendering frontends arranged according to the rendering efficiency: 2D-pygame, 3D-panda3d-native, and 3D renderpipeline. Even for the most heavy 3D renderpipeline, it can run up to 30 FPS with an advanced deferred rendering pipeline to show vivid light/shadow effects.

Actually, this discussion reminds me that we should update our documentation @pengzhenghao. Now we have too many new functions, even a new system architecture, making developing extensions possible, but people can not find them anywhere.

[CristinaKepner]

I feel so sorry that I haven't responded to your comment for such a long time. Too much dirty work has washed me away to verify this matter; I am very grateful to Shady you for providing me with such a detailed verification method.
At present, we are working on the cloud backhaul data standard. All types of backhaul data can be analyzed in the cloud, search for visual solution review, and simulation into process solution discussion, Fusion recharge efficiency confirmation. I have been busy for this period, hope after the time when we ready for the data, I will continue to advance this matter.
Still very grateful to Shady for replying me so carefully! I believe that with the current development pace of MetaDrive, the future will be real good～

[QuanyiLi]

Super. And MetaDrive is undergoing some changes. We will make more tutorials, documents and more clear APIs in the near future for making it easy to extend. So feel free to ask for support. We would be happy to collect more user feedbacks. 发自我的iPhone

…

------------------ Original ------------------ From: Cristina Kepner ***@***.***> Date: Fri, Jun 2, 2023 9:06 AM To: metadriverse/metadrive ***@***.***> Cc: Quanyi Li ***@***.***>, Comment ***@***.***> Subject: Re: [metadriverse/metadrive] How about add the real car four-domaincommunication to metadrive? (Discussion #442) I feel so sorry that I haven't responded to your comment for such a long time. Too much dirty work has washed me away to verify this matter; I am very grateful to Shady you for providing me with such a detailed verification method. At present, we are working on the cloud backhaul data standard. All types of backhaul data can be analyzed in the cloud, search for visual solution review, and simulation into process solution discussion, Fusion recharge efficiency confirmation. I have been busy for this period, hope after the time when we ready for the data, I will continue to advance this matter. Still very grateful to Shady for replying me so carefully! I believe that with the current development pace of MetaDrive, the future will be real good～ — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>