Approaches for Loading and Unloading Game Objects

[JSideris]

Background

In plain English, the challenge is one of figuring out what objects the server should force clients to load. And in the broader sense, which objects are important enough to save (on the server or on disk) but no longer required to be loaded in memory (either on the client or server). It has been one I've had to face since the very beginning of this project. It's also been one of the biggest source for bugs.

The earliest need to solve this was in serving game objects from server to client game instances. The simplest solution to this is to have every client load a copy of every object on the sever. This may work for some smaller games with very few objects. But for large games with vast maps, this is not practical. Objects should only be loaded as they are needed by the client.

The solution implemented for Saucers.Space was to have a rectangular sensor follow the player robot around, and as objects collided with the sensor, they'd be loaded by the client. This became super buggy once the tractor beam was introduced, since objects could affect (or depend upon) other objects. As a result, the tractor beam had to be made super robust as to handle possible cases where it's source or target were not on-screen. Even then, this resulted in broken physics when something was being dragged from off screen. A whole bunch of code was then written to build a rudimentary dependency tree for objects so that all linked objects could be loaded at once, and even instantiated client-side in the correct order.

A lot of the learnings from this will undoubtable carry over to a more general form within the engine, but there are also some challenges that need to be addressed. For one, the collision sensors themselves use up a decent amount of processing power and RAM. In fact, during some of my tests, increasing the size of the sensors resulted in a notable increase in server lag, particularly when the sensors themselves were in close proximity, because the result was many overlapping collisions. It may be possible to lessen the effect of lag by using collision masks, but still, there is a potential for many, many overlapping collisions per second that would have to be processed by the sever. We can set up some benchmarks to estimate this later on.

Approach

With the plugin-oriented engine, physics-based loading alone doesn't even make sense, since some games may be entirely non-physical. So we must first strip the problem down to it's very core.

At it's core, the must-have functionality that needs to be a part of the main engine should solve the following two needs:

• The ability for some other logic (controlled by a game, or by a plugin) to decide when specific objects should be loaded or unloaded.
• Tracking of which objects are loaded by which controllers.
• Ability to double down on objects being associated with controllers, so, for example, that on-screen objects that are loaded again due to a dependency by another object don't simply get erased when they go offscreen, breaking the object that depends on them.
• Dependency trees so that objects that depend on each other will be loaded and unloaded in the correct order.
• An event to be triggered when an object is to be loaded or unloaded, which can by captured by a plugin or game and handled.

With this, a basic game (even with no physics) will be able to load and unload objects from a server authority. Plugins will make use of this, and can even work side-by-side if needed.

Always Load All Approach

The ability to just enable object loading should be available. The application of this will be smaller games where it makes sense to load all game objects for each user, or let clients handle culling (to offset processing power from the server). This might be particularly useful in games where objects need to be unloaded and reloaded many times in short succession, and a benefit to this is less network traffic, since the create messages are always the heaviest.

The downside is heavier up-front traffic, and possibly more updates. It may be possible to mitigate the frequent updates if proximity-based resolution is used, but this has its own set of challenges to navigate.

In benchmarking, this approach is the "control" experiment to measure performance without any specialized loading approaches.

Physical Sensor-Based Approach

2D and 3D physics plugins will still have their sensor approach, which is somewhat of a no-brainer to include. But it should be enabled manually by the developer of a game rather than automatically by the plugin.

The approximate cost is probably something around O(log(n+m)) where m is the number of sensors, and n is the number of physical objects. Note that in order to achieve this level of performance, it may be necessary to keep objects sorted at an additional cost, however, a lot of that cost is already sunk by the engine anyway - the only additional cost comes from having to sort the sensors.

Chunk-Based Approach

Another physics-based approach to include in the physics plugins will be chunk-based. This would still be a part of the physics engine because that's were position is stored, but the idea would be that every game object is associated with a chunk in 2D or 3D space (by the physics plugin). Each controller will be watching certain chunks, and as it's robot moves, the chunks it watches will change based on its zoom level. Whenever an object moves into a chunk (computed instantly based on position and size), it will be "watched" by the chunk. Leaving a chunk will cause the chunk to un-watch the object. When an object enters a chunk that's being watched by a controller (or when the controller starts watching a chunk that's watching that object), it will be loaded by the controller, and the reverse process for unloading objects also applies.

The main cost of this approach comes in two forms. One is that each moving object will need to be placed into one or more chunks at the cost of O(n). Second, as controllers move about, they will need to constantly be watching and un-watching the chunks around them. This does come with a fixed cost based on the size of the viewable area, but it's not zero. If there is a lot of teleporting, then an entire area of chunks will have to be unloaded and loaded. But again we're talking about something like 16 chunks consistently, and teleporting sensors everywhere also has a high cost.

Another unfortunate (but unavoidable) downside to chunks is that it introduces quantization error. We want to err on the side of loading more, not less, so that things don't look like they're flickering in and out of existence at the edges of the screen.

On the up side, chunks may be the best way to save unloaded data to disk in order to save memory. This really isn't possible with sensors since an unloaded object will have no way of being reloaded (since the physics body will be removed from the world and no longer collide with the sensor).

View-Based Chunks

To expand on physics-based chunks, the canvas 2D and 3D plugins could use their own chunk-based loading system to cull visual objects. This would not be used for syncing over the network. It's entirely client-side.

It then may make sense to create some kind of common conceptual interface for dealing with chunks in 2D and 3D.

Comparison of Physics-Based Approaches

TODO: Need to create several scenarios to benchmark the two approaches (for both 2D and 3D) and get some data on what works best given different .

Saving to Disk

One way to reduce wasted RAM and processing power on the server is to unload objects that are far away from any controllers for some defined amount of time. This is a must-have for single-player games, but should also be possible on the server for multi-player games (e.g. mmorpgs).

The actual mechanism for loading or unloading should probably be handled by the server and client plugins, since each has it's own libraries or mechanisms for saving data. In fact, the client function will vary greatly depending on the released platform. Mobile may end up relying on a wrapper like Cordova, which has it's own API for saving data. Desktop apps might function very similarly to the server. Browser-based apps might be forced to use local storage, or upload save data to the cloud. So there may be a motivation to add additional plugins for this purpose.

In all cases, unloaded data will be serialized and an interface will be provided (perhaps through GI events) to save the object to disk and destroy it. When the object is ready to be reloaded (because a user enters a nearby chunk, etc), the engine should find the desired file and load it into memory.

Serialization can use the existing COM serialization that's used to sync client and server. This is currently part of the core engine (which may change if it gets any bigger or more complicated). That means that persistent fields will need to be marked on each object.