Resource streaming, bundles and the file system

How a path becomes loaded data: the file system's mount vector, the resource table, the open / load / parse stages and their request object, bundles and their reference contract, and the creature-bundle manager that decides which creature models exist in a region.

The file system

Every asset path is resolved through one priority vector on the file system object, a static at 0x2A54600 (a function-local static returned by 0x3DDD34). Entry 0 is consulted first.

Kind 0 is an archive tree (CArchiveTree, vt_0x22C21B8); kind 1 a directory source (DirectorySource, vt_0x22C21C8, with its directory string at +0x10). A directory source answers with GetFileAttributesW(directory + path); a hit stops the archive search, and the loader opens the loose file itself (0x529BF8 -> GetLooseFilePath 0x52A2F8). The retail game already mounts its Game\ folder as the last entry; that is how the cinematic .vib movies ship as loose files.

Mount(fs, Source** out, path, unused, index) is 0xC28560, the function behind the -arch command line option and the run-time DLC mounter. There is no unmount method. A mount affects the next load of a path only; an asset already resident keeps its data until it is streamed again.

Hashing names

Paths and names are hashed with StringHash 0x423E70: h = h * 919 + LUT[c] over the bytes, with the 256-byte table at 0x1DB50D0 (it folds case, so the hash is case-insensitive). The same hash names GameDB records, fields, animation clips, bones and world objects throughout the engine.

The resource table

Every resource the engine knows is a node in a hash table owned by the resource system [base+0x2A86758]: 0x800 buckets of 0x18 bytes, bucket = hash & 0x7FF, list head at +0x08. A node is resource + 0x10 (+0x08 next, +0x10 hash).

A resource whose count reaches zero is not freed: 0x1BEAAC unlinks it from an unused list (+0x10) and keeps it for reuse. "Released" therefore does not mean "gone"; residency is decided by the data pointer.

The I/O pipeline

[base+0x2A86750] holds the I/O core (core, the bundle owner + 0x30; the owner, vt_0x22C0FF8, is built by 0x119E5D4 and its stages by 0x1183E8C). Three stages run on three threads, each thread receiving its own stage as its argument:

The open and load stages each hold two queues of 0x38 bytes at +0x00 and +0x38, the thread object at +0x70, a CRITICAL_SECTION at +0xA0, a work event at +0xC8 and a stop event at +0xD0. A queue is a double-buffered vector of 16-byte {object, control block} handles:

The consumer swaps the two vectors when the output is empty and reverses the run, so service is first-in first-out, and the stage runner always takes the first non-empty queue: queue 0 drains before queue 1 is touched. The parse stage has a request vector at +0x00, a second work vector at +0x18 (capacity 128, consumed by 0x3DC654), its thread object at +0x30, locks at +0x60 and +0x88, and events at +0xB0 / +0xB8. Bytes [base+0x262B520 / 521 / 522] switch the open, load and parse threads on; when one is 0 that stage's work runs inline in the caller. All three are on in the shipped game.

The request object

A request (SResourceLoadRequest, the name in its destructor 0x3DD77C) is allocated together with its reference count (a make_shared layout) and has no vtable:

The flow

1. Submit 0x3DDDEC queues the request on the open stage (or straight on the load stage when the open thread is off). Producers include the texture streamer 0x3DD45C and the bundle content job 0x3DB5E4.
2. Open 0x3DD588: picks the path (resource +0x28, or the bundle's .embb), waits on a semaphore at core+0x7B8 that bounds how many files are open at once (0x1184864 creates it with an initial and maximum count of 32, the u32 at 0x262AAA0), opens the file through the file system and passes the request to the load stage.
3. Load 0x2D3444: a resource is read into new resource data (0x3DCFDC, 0x3DCF80) and passed to the parse stage (0x3DD248); a bundle content request goes to the bundle loader 0x3DBE50 with the path table below.
4. Parse 0x3DD2A8: releases the semaphore slot (0x3DD424), parses the data (0x3DD3C4) and hands it to its resource type (0x3DD310).

Texture tiers wait. The texture streamer puts any request whose path contains tex0 or texuhd (the higher-resolution texture tiers) in queue 1 and everything else in queue 0. Because the runners drain queue 0 first, base textures, meshes and bundle content always open before the high-resolution tiers.

In a long play period the open queues' high-water marks reached 1,812 (queue 0) and 1,567 (queue 1) against 33 and 32 for the load queues: the backlog builds in front of the file open.

The path table

The load stage carries the engine's global name table at +0xD8 (core+0x1D0): a vector of {u32 hash, u32 0, char* path} sorted by hash, 30,639 rows in a loaded region. 503 of 503 sampled keys equal the hash of their path, and every row is in order. It names every kind of streamed file (.fxp, .tex, .bnk, .bndl, .anix, .vol.inst, presentation bundles in every language). The load stage passes it to the bundle content loader. It is a row-for-row copy of the file system's hash -> path table (fs+0x80), which Mount builds from the path index file each archive may carry (0x1182428 merges every index into it, sorted); see Archives and file formats.

Bundles

A bundle is a manifest (.bndl) plus a container of its contents (.embb); see Archives and file formats. The bundle registry belongs to the owner (vt_0x22C0FF8): vtable slot 24 0x3DC538 gets a bundle forcing the .bndl extension, slot 25 0x119F1E8 takes a raw path, and the registry at owner+0x718 keeps 0x18-byte rows {u64 hash, CResourceBundle*, u64} (get-or-create 0x1B5DC0).

CResourceBundle (vt_0x1E6E678, 0x178 bytes):

The reference contract. A registry row holds no reference. Prefetch (0 -> 1) references the manifest, requests it and, once its data is present, queues the .embb content load as a job on the bundle system (0x3DB574 -> 0x3DB5E4). Discard is the exact reverse and asserts it is never the final reference. Release at zero unregisters and destroys the bundle (0x1B5C9C), detouring to 0x10DD00A if the prefetch count is still non-zero. A holder therefore keeps a handle and a prefetch, and lets go by discarding first and releasing second. A missing path is benign: the registry keeps an entry whose manifest has no data (ten such rows exist in a normal game).

Loading a region's layer bundle does not place it in the world: its resources become resident, but world contexts and placed objects are unchanged. In a loaded region the registry holds about 544 bundles, and every character bundle the region uses is already resident.

Gear needs two bundles. An armour piece's geometry lives in <piece>_g.bndl (.mesh, .skel, .anix) and its material and textures in a set-wide <set>_<slot>_m.bndl named by the design record (Inventory/ArmorData field DynamicBundle). The engine loads both through a gear load request (vt_0x1DB4F78, constructor 0x400AB8) that is created only for an equipped item (give path 0x355474 -> 0x61B868 -> 0x4B4C7C, item equip 0x40136C -> 0x401680, re-skin 0x645890). A piece placed into an appearance without an item gets no request: its mesh is in memory but its .mat is a registered resource with no data. The save loader 0x1C112F8 restores non-item pieces (smc_nonequipment_record_list -> 0x1162D40) without a request as well. A bundle that is resident only because another character's item requested it is released when that item goes away.

The gear budget. These requests go through NemesisBundleMgr ([GameManager+0x6C88], named by its console commands NemesisBundleMgr.PrintMemory and PrintDynamicBundleData). Its initialiser 0x1B42728, run once from the database setup, sets the budget +0x00 to Faction/Global.NemsisBundleMemoryLimit x 10 (320 x 10 = 3,200) and builds one object (vt_0x1FF6B90, constructor 0x1C52B10) for each of the 1,484 DynamicResource/BundleDefinition records, kept in a table at +0x08 of 16-byte {record, object} entries. An object's cost +0x3C starts as an estimate, its bundle type's Bundles/Types.MaxMemoryPC (10 for Flair_1MB, 20 for Flair_2MB, 30 for Flair_3MB, the names being the console figures). When the load finishes, 0x3FFA24 replaces it once (latch +0x58) with the real size, bytes / 2^20, sets state +0x48 to 2 and tells the requesters. The estimates are generous: resident Flair_1MB bundles held a median of 0.40 MB (124 bundles, largest 1.12) and Flair_2MB a median of 0.018 MB (20). In one loaded Lithlad 157 bundles were resident at a total cost of 66.3 of the 3,200, so the budget was far from limiting.

Each object's priority +0x38 is rebuilt from its requesters by 0x3FFB10. Each request carries five bytes at +0x50: bytes 0, 1, 2 and 4 are flags, merged by OR, and byte 3 is a number, merged by maximum (0x3FFD44). The score is the highest DynamicBundleLoadPriority among the requesters plus that number, +1,000 for byte 0 (when the bundle has a handle), +10,000 for byte 1, +500 for byte 2 and +1,000 for byte 4. It is then scaled by the requester count: x1.25 at exactly 4, x1.5 at 6 or 7, x1.75 above 8, and unchanged at 5 and 8. Recomputed from the live requesters, this formula gives the value in +0x38 for 157 of 157 resident bundles (requester counts 1 to 49).

The constructor 0x400AB8 copies the five bytes from a struct its caller passes. On the equip and re-skin paths 0x400084 fills that struct: byte 0 is always 1, byte 1 is set when the owner reference has a +0x40, byte 2 when the owning character has +0x2700, the follower byte +0x23B8 or +0x2C04 set (0x366B88), byte 3 is the member's Player Interaction Score at that moment (0x595364, row 2 column 2 of the score object FM+0x150, the dword at +0x38) and byte 4 comes from 0x77F804. The byte is a snapshot: on 34 requests of this kind held by spawned captains it was at or below the current score every time, and equal on 16. Requests made on the give path (0x4B4C7C) instead take their bytes from a struct at +0xF8 of the 25-slot item table, and those carry byte 0 clear.

Each frame 0x614438 re-evaluates the requests when they have changed (+0x50): it sorts them by +0x38, highest first, and adds up their costs. Requests within the budget are loaded (0x614CB0); the rest go on a local list, and for each of those 0x1B427E4 walks the armour's and then the weapon's OutOfMemoryBackups and takes the first backup whose bundles are all resident already (0x1B424BC: for each bundle with an object, 0x614D3C requires a handle, its counters +0x4C and +0x50 equal and state +0x48 2). The backup goes to the request's owner (0x1C11CD8, 0x1C11CFC). So a piece over the budget is replaced by something already in memory, and nothing new is loaded for it. The budget applies to this gear path only. Bundles/Policies carries budgets of its own (Character_Nemesis 1,600 on PC), which this manager does not read.

Effects only load through bundles. An effect reference (for example an attack's AOEFX path) never loads anything; the effect system finds resident effects by name, and only a bundle (or start-up) makes one resident.

A malformed effect stalls the loader. Every .fxp ends with a u32: the count of the database's (u32, u32) pair table (db+0x68; 0 in 968 of 968 shipped effects). A file without it makes the database body reader 0x4B9B50 read 4 bytes past the end of the file (0x4B9C60); on the raw asynchronous stream (vt_0x1E71010) that read does not return, and every later load waits behind it until the game restarts. A missing file, by contrast, is harmless. The effect database's categories are sorted by the hash of their path (371 of 371), and the loader binary-searches them (0x115B4D4).

Creature bundles

CharacterBundleMgr (CharacterBundleMgr.cpp) is MainScene+0x868 ([base+0x26FFAB0]+0x868). It decides which creature models are in memory.

The update 0x4B4FFC runs when the dirty byte is set: wanted = the first override list (or the default list) plus the always and extra sets; world set = the provider's list. The reconcile 0xE9F360 requests what is new (RequestBundleLoad 0x4B6204), queues what dropped out for unloading, then drains the queue, firing loaded / unloaded events through the global script agent (GM+0x6CF0).

A list's length is its field descriptor's element count (read by 0xE9EA30): every Server/NPCBundle/Lists record has two fields (the bundle array, hash 0x7AB8B862, and an empty second array). World_Island_Day holds six bundles (Goblin, Spider, Warg_Base, Fellbeast, Troll_Base, Troll_Poison) and World_MinasIthil_Day nine. A region keeps its whole creature set loaded for the visit; there is no per-encounter picker.

A region change resets everything: every set and every loaded creature bundle is cleared, including loads requested on demand, and the region's sets are rebuilt (skins first, then the world set, about 3 seconds to resident). Loads requested directly with RequestBundleLoad persist within a region, because no set owns them.

The spawn gate 0x16652C (template -> model -> NPCBundle -> IsBundleLoaded 0x1665B4) requests a missing bundle only when the region's WorldInfo.IsTestWorld is set (0x5B85B4); retail regions have it clear, so a spawn whose bundle is not loaded is rejected ([AI|SPAWNAMBIENT|REJECT] ... doesn't have character bundle loaded). The descriptor 0x166878 reads only the template and the spawn manager (SM+0x78 the per-template override table, +0x80 its count, +0x88 the AI/Templates category); the model and bundle are plain field reads (0x16680C, 0x166680). A null bundle allows the spawn: 94 of 177 templates have none (every orc and olog class, tamed wargs, the fellbeast), because their data is always resident. The chain from a spawn channel to its bundle is plain data: AITypes.Entries[] -> _StructuresAISpawnAmbientTypeEntry.AITemplate -> AI/Templates.CharacterModel -> Character/Models.Bundle.

Scripts and the console reach the manager through System:LoadOverrideNPCBundleList(list, slot) / LoadDefaultNPCBundleList and the console commands CharacterBundleMgr.Dump, LoadOverrideBundle and LoadDefaultBundleList (command table at 0x2674408); quests unload their overrides when they end.

Memory

The engine's own byte count understates the real cost of a single bundle and overstates a burst: Nazgul_base (36.0 MB by the engine's count) added 120 MB of committed memory and 120 MB of video memory, while 129 bundles requested at once (622 -> 4,337 MB by the engine's count) added only 571 MB of video memory and 665 MB of commit, because they share resources. Bundle loads do not consume the scarce low (below 4 GB) address space. A fast travel releases everything a region loaded; nothing accumulates across regions. WorldInfo.MaxBundleSizePC (1096 on PC, 548 on consoles) is checked only when content is packed; no run-time code reads it.

Open questions

• What the give path's flags struct at +0xF8 of the 25-slot item table holds. Its requests carry byte 0 clear and byte 3 values up to 41, and they are held at +0x108 of an object that is not a slot item, so the check that settled the equip path does not reach them.
• The meaning of the resource-data states other than 3 (finalized), and what the parse stage's second work vector carries.
• Why the global path table's second field is 0 while the per-bundle name arrays carry 1.
• Which other code submits I/O requests: only two callers of the submit function exist, so the resource system may queue some loads elsewhere.

Shadow of War (Steam), game version 1.21. An unofficial fan research project, not affiliated with or endorsed by Warner Bros. Games, Monolith Productions or Middle-earth Enterprises. Middle-earth: Shadow of War belongs to its owners.