Asynchronous Route Streaming

Asynchronous Route Streaming is a pre-release feature of Trainz. It has been discussed publicly in the TrainzDev forums and is summarised on this page, however users should understand that no release schedule has been provided for this feature. All discussions and details are speculative in nature and may not accurately reflect the final form of this feature.

Broad Overview

Asynchronous Route Streaming is a new technique within the Trainz environment that permits the majority of a route to remain on the local disk (or on a network server) while only the active portions of the route are loaded into the client computer's RAM. This offers a number of substantial advantages over keeping the entire route memory-resident:

• Local loading times are massively reduced. With Asynchronous Route Streaming enabled, even the very largest Trainz routes take only around five seconds to load on a mid-spec desktop.
• Memory usage is substantially reduced. This allows lower-end desktops to work with large routes without encountering memory-related performance or stability issues.
• Many runtime costs are substantially reduced. Even when highly-efficient data structures are used, looking up objects from a pool of several million active objects is mildly expensive. Unloading the majority of these objects reduces the active set down to tens of thousands, improving overall performance. In cases where inefficient data structures or techniques are used, performance can be improved by hundreds of times over, meaning that certain long-running operations automatically become cheap.
• Save times are substantially reduced, and autosave-style operations can occur in the background.
• Many causes of potential frame drops are removed, as code is rewritten to assume asynchronous asset loading.
• Many cases where meshes and other dependencies were forced to stay memory-resident or GPU-resident have been removed, as the code is rewritten to perform asynchronous loading for all asset types.
• Multiplayer Surveyor is made possible, as a client can operate while streaming data across the network.

To be clear, Trainz has never kept the entirety of a route's data in memory, and even data that was kept in memory was not necessarily kept permanently in "runtime ready" format, but previous Trainz versions did not perform comprehensive streaming of all asset instances and all asset types.

Route Format Changes

Starting a trainz-build 4.5, the route and session file formats have been rebuilt to permit Asynchronous Route Streaming. As always, routes and sessions built in prior versions of Trainz will be automatically upgraded to the latest trainz-build format when edited in Surveyor. To be clear, this does not mean that Asynchronous Route Streaming will be present in Trainz v4.5, but rather that the file formats will support it.

The most significant change is that route and session data is segmented into baseboard-sized tiles, with each tile being written out to a separate file.

Users accessing the file system directly (rather than through Trainz) will note two other key points:

1. As with T:ANE, the submitted copy of each asset is stored in a ".tzarc" file. 2. Unlike T:ANE, the open-for-edit copy of a route or session asset does not have the many individual files copied into the asset's editing folder. Instead, only modified files are stored in the editing folder.

Script Changes

A large number of script API changes are necessary in order to support Asynchronous Route Streaming. Script authors will be provided with full details of these changes well in advance of the changes being activated in a public build, to ensure that existing scripts can be updated to ensure compatibility with Asynchronous Route Streaming. Various diagnostic tools will be made available to test compatibility. Initial details are available on the TrainzDev forums, and interested script authors are encouraged to following along so that they are aware of the upcoming changes.

Not all scripts require updating for Asynchronous Route Streaming. API changes and behavioural changes are specific to functions which may access objects that could potentially be unloaded. If a given script does not access other objects on the route or is not meaningful for objects that are outside the user's view, it likely does not require any modification. Any APIs which may require loading data from outside the user's immediate locale is being reworked to allow asynchronous operation; scripts which use these functions will need to be reworked to allow asynchronous searches and to minimise the number and scope of searching performed. Any script functionality which uses the temporary script router IDs to refer to a placeable object will need to be reworked to use a new GameObjectID to refer to objects instead.