[Epic] Loaded Resource Tracking

[dinomut1]

We need a way for the engine to introspect resources on any browser and then budget those resources. This means CPU, GPU, system, memory, and graphics memory. This should influence what is pulled off the CMS, turn on/off shaders, priority queues, change levels of detail, and possibly tweak physics engine.

We need to define our limits for different devices. We should define run-time limits for:

• maximum CPU and GPU memory (this is the most important, and we should start with this)
• minimum frame-rate
• maximum poly count
• maximum bitrate
• maximum number of workers
• etc

We may want to construct several dedicated benchmarks to find out these limits (filling a scene with physics objects / balls, filling our scene with textures, etc.) to determine what our theoretical capacities are on different devices. In some cases, we can use built-in browser apis to determine these limits (e.g. Network Information API, navigator.hardwareConcurrency). Finally, we need realtime runtime benchmarks that are tracking the resources we care about.

An asset in the scene can be judged by several metrics. We need to think about this thoroughly. An asset will have many levels of detail. Each level of detail is probably judged on several properties:

• poly count
• vertex count
• texture memory
• shader complexity score (we need to find a metric for this)
• collider complexity score (convex hull vs trimesh vs box, etc)
• download size (megabyes)
• etc

We should assume all assets are optimized (e.g., via MeshOpt)

---

1.) Define a ResourceManagerState which contains a list of references for:

• textures
• geometry
• ECS data

Each resource entry contains:

• loading status of that resource

• what is referencing that resource

• a direct reference to that resource

• metadata about resource size / budget

• For textures, we should track whether a texture has been uploaded to the GPU so that we can dispose of it in the heap, and know when we may need to fetch it again for reupload

When a resource is no longer referenced by anything in the scene, we dispose of it and clear it from the registry:

• Removing the key from the THREE.Cache
• Calling the dispose method on the actual resource itself
• Deleting the entry from ResourceManagerState

We will need to add custom logic to our AssetLoader loaders, and a plugin to our GLTFLoader, which create or check for entries in ResourceManagerState:

• Create hooks for object types:
• useGLTF
• useTexture
• useCompressedTexture
• useAudio
• useVideo
  These hooks handle the bookkeeping for registering when a load has begun, what is loading it, and when the load has completed

2.) Leverage the THREE.Cache object to actually store the URLs and resources in its built in map

• All threejs loaders use this cache by default, so we will only need to manage reference counting, progress and cleanup

3.) For ECS data, we need to register which components are associated with each entity, and whether they have been loaded yet or not

• Interfaces with the SceneLoadingSystem and ModelComponent to allow us to know when the ECS data from a given .gltf has fully deserialized into the ECS

Issues

• Create ResourceManagerState #9629
• Track aggregated collider complexity for a scene #9896
• Track whether a texture has been uploaded to the GPU so that we can dispose of it in the heap #9897
• Add additional resource hooks for audio and video #9898
• Track when the ECS data from a GLTF has been deserialized into the ECS #9899
• Collect metrics to determine a shader's runtime complexity #9902