Archives and file formats

How to open the shipped data: the LTAR archive family and its byte layout, name resolution and its traps, Oodle chunking, the archive config list and how the engine decides which copy of a file wins, the BNDL / EMBB bundle pair, textures and their tiers, meshes, skeletons and string tables, with the coverage measured on every shipped file.

The game ships its data as 76 .arch06 archives in the install root plus a small loose Game\ folder; every asset path the engine opens is resolved through those archives in a fixed search order. This chapter gives the archive layout, compression, the search order and the rules that pick one copy of a file, then the container and payload formats inside, each checked against the shipped files.

The shipped files

The archives fall into families:

Three archives are empty (Patch_00_LC, Patch_12, Patch_12_LC: a 65-byte header and one empty root folder). Blade.arch06 and Lithlad.arch06 each hold one file, dlc\blade.txt / dlc\lithlad.txt, whose eight bytes read Unlocked; neither is in the config list and neither is mounted in a running game.

HotChunk.arch06 holds the GameDB and the string tables: entry 0 is its path index (2,136 bytes, below), entry 1 database\game\game.gamedb (56,759,100 bytes), entries 2-29 the 14 language folders' string.strdb and string_uwp.strdb, entry 30 globalbundles\globaldatabase.bndl and entries 31-47 the 17 globalbundles\globalresources_<lang>.bndl bundles.

The archive config list and the search order

At start-up 0x119717C mounts any archive named with -arch on the command line, then reads the config list: 0x1196FE0 takes default.archcfg (string at 0x22C0B20) unless -archcfg (0x22C0B08) names another file. The list is text, one path per line, ; starting a comment; it opens with the Game directory lines and states "file search order is from the bottom up".

Every line goes through Mount 0xC28560 (see Resource streaming for the file system object and its mount vector). A directory becomes a directory source, an existing file an archive source (CArchiveTree, vt_0x22C21B8), and a line that names nothing is skipped: the shipped list names five archives that do not ship (Great_Hall, Great_Hall_Lighting, TextureArrayPatch, HighMipA, uhdmipA) and, for each high-resolution archive, a second spelling without ..\ that resolves to nothing in the Steam layout. The three empty archives are mounted like any other.

The live order. Read from a running game, the mount vector is the list reversed, so the bottom line is consulted first:

The DLC patches are interleaved with the numbered ones, so no "DLC above numbered above base" rule holds; HotChunk and Global sit above every patch, and the high-resolution texture archives above those. The resolver 0x3DD958 walks the sources from index 0 up to the archive-scan bound [fs+0xE0] and returns the first archive that holds the path. A path that ships in several archives (26,122 of the 431,226 distinct paths, up to 37 copies) is therefore served from the highest one: the engine's byte count for the loaded English string table is 7,797,636, which fits HotChunk's 7,797,584-byte string.strdb and none of the patch copies (7,110,464 and 7,143,600).

LTAR v4: the .arch06 layout

All 76 archives are LTAR version 4, little-endian. The engine's reader is 0x11AE284: it checks LTAR and version 4, reads the rest of the header (0x11AE20C), the name pool, the file records and the folder records (0x11AE154), then copies the archive's own path to archive+0x50.

File record (30 bytes):

The two bytes trail every record, the last one included. Opening an entry (0x11AE510): when the byte at +0x1D is 0 the entry is read as a plain byte range (0x11DC110); otherwise +0x1C selects the reader, 1 the chunked stream (0x11DC4B8) and 0 a whole-file read into fresh memory (0x11DCC64).

Folder record (16 bytes): {u32 name, u32 first child, u32 next sibling, u32 file count}, with 0xFFFFFFFF for "none". The name is the folder's full path (database\string\enus), the root folder (record 0) has an empty name, and the reader turns the two links into pointers to 0x30-byte folder objects. Files belong to folders in record order: folder 0 owns the first file count records, the next folder the following ones, and so on, so a file's path is folder + "\" + leaf. Across all 76 archives, 16,226 of 16,226 non-root folders extend their parent's path and the counts add up to the file count.

Lookup. 0x3DDAD8 splits a path at its last \ or /, finds the folder by binary search over the folder records (0x3DDB90, case-insensitive), then binary-searches that folder's leaf names with _stricmp and returns the file record. Both tables are kept sorted for this: the leaf names inside every folder are in case-insensitive order (16,299 of 16,299 folders), and the folder records are in case-insensitive order of their full paths compared as if each ended in a backslash (76 of 76 archives; plain order differs in 7 archives, only where one folder name is a prefix of another, such as bundles\dlc and bundles\dlc4).

Names: what is a file fact and what is a reader trap

• A record's name offset points at its own leaf, relative to 0x30. Read that way, with folders taken from the folder table, every typed entry's content matches its extension: 482,736 of 482,736 entries of 16 typed extensions (.bndl, .embb, .tex, .mesh, .anix, .bnk ...), and each of the 39 extensions in the install maps to one format (catalogue below). No per-archive naming rule is needed.
• The pool is not a list of entries in order. It also holds the folder paths, and it stores each leaf string once: 56,018 records share a leaf string with another record (string.strdb in 14 language folders), while full paths are unique within each archive (0 duplicates). A reader that takes "the string after the previous name", or resolves offsets against a wrong base (0x18 instead of 0x30), names a neighbouring file: on the same records those rules give a name whose extension contradicts the content for 396,483 and 388,454 records. The result looks plausible because a neighbour's name is still a real path.
• A path names a file only within one archive; the same path in another archive is another copy, chosen by the search order above.

Compression and chunks

File data is a chain of chunks, each {u32 stored length, u32 length, bytes}, with every chunk header 4-byte aligned relative to the start of the entry and a record's stored size equal to the chain's length rounded up to 4 (19,993 of 19,993 sampled entries, up to 400 per archive). The chunk reader 0x11DC664:

1. reads the two lengths and asserts "Encountered a corrupted compressed chunk header." when either exceeds 0x10000, so a chunk holds at most 64 KB;
2. copies the bytes when the stored length is at least the length (a stored chunk);
3. otherwise checks that OodleLZDecoder_MemorySizeNeeded(8 = Kraken, length) fits its 0x20800-byte work buffer and calls OodleLZ_Decompress (fuzz-safe) from oo2core_5_win64.dll, asserting "Decompression failure." unless exactly length bytes come out;
4. skips to the next 4-byte boundary.

In the sample, 638,807 chunks were packed and 31,098 stored. The first chunk of every one of the 488,752 entries decodes with the shipped DLL. Over the whole install the stored data is 62.8% of its unpacked size.

Two compressors. A packed chunk starts with an Oodle block header whose second byte names the compressor. The shipped DLL's own OodleLZ_GetChunkCompressor (which this build exports with two arguments: the buffer and an int* for the independent flag) and OodleLZ_Compress agree on the mapping: 8C 06 is Kraken and 8C 02 is LZB16. Of 543,319 packed chunks in 15,519 entries (up to 300 evenly spaced entries in each of the 76 archives), 541,955 are Kraken and 1,364 LZB16, and the header names the compressor on every one. The split is by size: every LZB16 chunk unpacks to 257 - 2,044 bytes, every Kraken chunk to 2,050 bytes or more (98.4% of them to the full 0x10000). LZB16 therefore appears only as an entry's last chunk (316) or only chunk (1,048), never as its first or a middle one, and in every file type whose length leaves a short tail (.bndl, .embb, .texuhd, .mat, .scripts and so on). The reader passes every chunk to OodleLZ_Decompress, which dispatches on the header, so a rebuilt chunk may use any compressor the DLL decodes.

The path index (ArchivePatchLooseFiles.txt)

37 archives (Patch_01 - Patch_11, Patch_13 and Patch_DLC1 - Patch_DLC5 with their _LC twins, Global, HotChunk, soundbundles) carry a root file archivepatchloosefiles.txt as record 0:

u32 1 | u32 string bytes | strings (NUL-terminated full paths) | u32 count | count x {u32 hash, u32 string offset}

The rows are sorted by hash and every hash is StringHash of its path (73,038 of 73,038). The index lists an archive's loose files: every loose .anix, .bnk, .objres, .scripts, .mesh, .skel in those archives, and a .bndl exactly when its container is not in the same archive (34,920 of 34,926; 10,849 of 10,883 manifests that sit beside their container are left out); no .embb is ever listed. It also names 495 paths that its archive does not hold.

When Mount adds an archive it looks the file up (0x3DDAD8) and passes it to 0x1182428, which checks the leading 1, keeps the string block, builds {u32 hash, char* path} rows and merges them, sorted, into the file system's hash -> path table at fs+0x80 under the lock at fs+0x98, incrementing the generation at fs+0xC0. In a running game that table holds 30,639 rows: exactly the 30,637 distinct hashes of the 37 shipped indexes, plus two rows from a directory mounted at run time. The load stage's name table (core+0x1D0, see Resource streaming) is a row-for-row copy of it (same 30,639 hashes, same string pointers). Archives without an index (regions, UI, presentations, high-resolution textures) are mounted and searched like the others.

Hashing. StringHash 0x423E70 computes h = h * 919 + LUT[c] with the table at 0x1DB50D0. The table folds case but not separators: \ is 42 and / is 52, so a path hashes as written with backslashes. Other values: . 55, - 39, _ 38, space 37, a-z 1-26, 0-9 27-36.

Bundles: the BNDL manifest and the EMBB container

A bundle is a manifest (.bndl, magic BNDL) naming its members, usually paired with a container (.embb, magic EMBB) of the same path stem holding their bytes. The bundle loader 0x3DBE50 reads the container through 0x11B5E68 (it checks EMBB and version 5); the manifest's loader 0x504C70, a slot of the manifest resource handler, checks BNDL and version 6. How bundles are requested, referenced and released is in Resource streaming.

EMBB v5

There is no padding in the section chain: a payload is followed at once by the next size word, and payload starts are spread evenly over all 16 residues mod 16 (643 - 1,008 sections each, 12,031 sections). All 190,407 containers are version 5 and all pass the header rules (offsets 4-aligned and chained by padded length, size exact, padding all NUL); 2,441 are empty, the largest holds 11,178 members, 428,440 members in all. Walking the chain of 780 containers (up to 30 per archive, each at most 64 MB) ends exactly at the end of the file in 780 of 780, with no zero-size or overrunning section. A member's size word is the only thing that locates the next one, so a payload whose size word disagrees with its length makes the loader read the next size from inside the payload.

BNDL v6

All 225,338 manifests are version 6 and parse to their last byte; the 0x0C rule holds on 225,338 (a constant +28 would be right on only 217,350). The type string names a record of the GameDB category Bundles/Types; 130 distinct strings are used, led by Presentation (208,584), WorldQuest (4,111), Sound_1 (1,656), FlairWorld (1,312) and Flair_1MB (982); 11 manifests have no members and an empty type.

How a manifest and its container relate

Every container has a manifest with the same path stem in its own archive (190,407 of 190,407). In 186,478 of those pairs the manifest mirrors the container exactly: identical path-table bytes and offsets, and sizes that predict the container's length,

16 + path table size + 4n + sum(4 + size[i]) = container size

In the other 3,929 pairs the manifest declares members the container does not hold: the Desert region's worlds\retail\desert\desert.bndl lists 3,805 members while desert.embb (1.43 GB, in Patch_DLC4) holds 3,763 (the other 42 are 41 .cres prefab resources and one .record06, none of them a loose entry). Taking the winning copy of each side by the search order (190,367 bundles):

Of the 13,695 paths that winning manifests list beyond their container, 8,521 ship as loose entries. Patches use this to update part of a bundle: 32,903 manifests ship in an archive above their container's best copy (27 below).

Container-less bundles. 2,001 manifests have no container anywhere: sound bundles (Sound_1 1,654, Sound_5 187 ...), the language string bundles (StringDB, 73), GlobalDatabase (7), GlobalRenderingResources and a few others. Their 3,398 members resolve as loose entries (3,257) or container members (34); 3,016 of the 3,257 declared sizes equal the winning entry's size and 206 more equal another copy's. globalbundles\globaldatabase.bndl declares the one member database\game\game.gamedb.

Which copy wins

• A path: the first archive in the search order that holds it.
• A bundle: its manifest is opened by path like any file, so the winning manifest gives the member list. A member can be in the container, a loose entry of the same path, or both; when both exist, the measured case below shows the loose copy in use.
• Standalone against embedded: 14,172 asset paths ship both as a loose entry and inside some container (.bnk 3,658, .tex 1,804, .anix 1,612, .fxp 1,165, .mat 900, .scripts 835, .objres 680 ...). For the Desert main part, the live context-action data equals the standalone desert.objres of Patch_DLC5, not the section embedded in Patch_DLC4's desert.embb nor the older standalone copies of Patch_09 and Patch_10; the winning desert.bndl (Patch_DLC5) declares exactly that copy's size. The same holds for a layer's .scripts (see Scripting); a path can also ship as different loose and embedded clips (see Animation).
• The GameDB: database\game\game.gamedb ships in nine archives (HotChunk 56,759,100 bytes; Patch_01 - Patch_04 and their _LC twins 58,084,930 - 58,099,992). HotChunk is above all of them, so a path lookup returns its copy (see GameDB).

Distinct asset paths by where they ship (manifests and containers excluded): 238,588 only inside containers, 34,409 only loose (mostly texture tiers), 14,172 both.

Format catalogue

Every file starts with a 4-character tag, and the extension decides it: across all 488,752 records each extension maps to one format (.txt aside). Section tags inside containers follow the same table (12,031 of 12,031 walked sections), with one exception: a .tex section is TEXR or SLVT (9 of 3,181). Each format is loaded by a slot of its resource handler (vf3, .vid vf4) whose code checks the tag; the loader column gives the function holding that check and, where it differs, the handler's vtable.

Textures (TEXR v8)

A texture file is a TEXR header followed by a standard DDS header and the pixel levels it carries. The loader 0x5030EC accepts TEXR version 8 and SLVT version 4 (a small header that wraps a TEXR).

The stored DXGI format is always the UNORM one; the sRGB flag marks colour textures (set on 5,128 of 5,203 _d diffuse maps, 1 of 4,712 _n normal maps, 17 of 3,485 _s maps, 95 of 2,511 _m masks). Texture creation (0x1E71F0) maps each format to a typeless / UNORM / sRGB triplet from the table at 0x2777E20. Normal maps store X in green and Y in alpha (the shaders decode (G * 2 - 1, A * 2 - 1)); _s maps store reflectance in red, a mask in green and roughness in blue.

Tiers

A texture can ship in up to three files with the same stem:

The tiers of one texture do not always describe the same size: the .tex0 and .texuhd of a stem agree in 5,873 of 6,165 stems, a .tex and its .tex0 in 497 of 1,037, and the level ranges of neighbouring tiers can overlap. The name field of a .tex0 or .texuhd names its own .tex (21,928 of 21,928); a .tex names its .tex0 (2,697) or itself (1,083); 12,673 world textures under worlds\ leave it empty. Only .tex and .texa files are bundle members (61,622 and 7,949 container members, no higher tier in any container or manifest); the higher tiers are loose entries streamed after base data (see Resource streaming). For the Ranger's starting sword the diffuse ships at 512 x 1024 (.tex, 11 levels), 1024 x 2048 (.tex0, 3 levels) and 2048 x 4096 (.texuhd, 1 level).

Texture arrays (TEXA v1, 1,946 files: .texa, .texa0, .texauhd) carry a header, their own path and the source path of each slice, then a DX10 DDS whose array size is the slice count, slices stored one after another with full mip chains. In a measured area of Cirith Ungol they textured about 90% of the static world's triangles (see World).

Meshes (MMSH v17)

All 711 loose meshes follow this exactly: the arena sizes add up to the rest of the file and every arena index is in range or -1. The descriptor block holds, at +0x2C, the cull distance (500 on 376 meshes, 400 on 153, 140 on 126) and at +0x30 a bounding sphere (centre, radius); one 48-byte record per arena {u32 vertex bytes, u32 index bytes, u32 palette bytes, u32 group count, u16[16] LOD triangle ladder}, the ladder rising to the arena's triangle count (index bytes / 6: 16-bit indices), found for 4,377 of 4,377 arenas; and the object records again as 16-byte {flags, object id, arena, material slot} entries, where an object's consecutive records are its LOD chain.

The groups are listed in one table of 40-byte records inside the descriptor block, arena after arena: +0x00 vertex count, +0x04 first index, +0x08 index count, +0x10 palette length and +0x1C a vertex-format id whose stride is fixed per file (+0x0C, +0x14 .. +0x1B and +0x20 .. +0x27 are not covered by the sums below). For every arena the vertex counts times their format's stride add up to its vertex bytes, the index counts times 2 to its index bytes and the palette lengths to its palette bytes, and the index ranges follow on from 0; searching the block for a table that closes like this finds it in all 348 distinct loose meshes (the 711 loose entries without copies; 2,124 arenas, 2,569 groups, and 93 meshes with an arena drawn as more than one group).

An arena is drawn as one or more groups that share its buffers; a group's vertices are two streams:

• stream A, 16 bytes per vertex: half4 position with w = 1, ubyte4 weights summing to 255, ubyte4 indices into the group's palette;
• stream B, the rest of the stride (60, 64, 68, 72 ... bytes in all): the UV area, then normal, tangent and binormal as float3 at stride - 40, - 28, - 16, and a 4-byte colour. UV0 is a float32 pair at +0 on stride 68 and +8 on stride 72; one LOD chain can mix strides.

Weights are stored in BGRA order against RGBA indices: the weight of index slot k is byte (2, 1, 0, 3)[k]. A palette byte indexes the skeleton's name-table order, not its bind order.

Shipped weapons and armour. The Numenorean sword (player_numenoreanswordt1.mesh) has its grip at the origin, the blade along +Y, width along Z and thickness along X (bounds X +-4.35, Y -34.9 to 102.1, Z +-7.74 cm), 810 vertices and 1,320 triangles on one bone Twitch, stride 64, cull distance 140; blades are about four times thicker than real ones (9.23 x 2.30 cm at y = 16). Tangents run along +dP/du and binormals along +dP/dv (median 0.996 and 0.997 over 16 weapon and armour meshes). Player weapons mostly ship one LOD (57 of 93) with 1:2 textures (81 of 93) and 1,500 - 2,348 LOD0 triangles; orc and olog weapons 1,334 - 1,996 LOD0 triangles with 1:1 textures (56 of 77). LOD chains follow two presets, 1 : 0.50 : 0.125 (a falchion: 2,000 / 1,000 / 250) and 1 : 0.7 : 0.4 : 0.2. Armour LOD0: helmets 1,661 - 2,857, chests 3,741 - 6,275, shoulder pairs 2,551 - 4,158 (both shoulders in one skinned mesh, vertex alpha 16 on the left and 64 on the right), legs 1,996 - 2,573.

Skeletons (SKEL v17)

0x00  'SKEL'
0x04  u32 x 16: [0] version 17, [1] header size 64, [2] string blob size, [3] string count
      (= [6] + [7]), [4] bone count, [5] bone bytes (= 37 x [4]), [6] .anix library count,
      [7] mesh-part count, [8] socket count, [9] bone count again
0x44  bounds: f32 radius, f32 min[3], f32 max[3]
0x60  mesh parts, each u32 LOD count + 14-byte LOD entries
      0xDCDCFF00, string blob (offset 0 = empty string)
      bones, 37 bytes each, depth-first: {u32 name offset, u8 deform, f32 position[3],
      f32 rotation[4] (x, y, z, w), u32 child count}
      masks {u32 name, u32 bone count, u32, u8 value per bone}, sockets {u32 bone, u32 name,
      f32 rotation[4], f32 position[3], f32 scale}
      0xDCDCFF01 string offsets (.anix libraries, then part names), 0xDCDCFF02 one flag per
      library, 0xDCDCFF03 one byte per bone, 0xDCDCFF04 tail (0xDCDCFF99 skeleton physics,
      optional Havok cloth)

All 973 loose skeletons satisfy the header identities and carry the five markers in order; their 73,958 bones all hold unit quaternions. The largest has 241 bones; the orc base rig (orcs\base\orc_base.skel) has 100, the player's 128, a Gondorian's 95 and the olog's 89. Vertex palettes and the live pose arrays index bones in the skeleton's name-table order, not in the depth-first record order.

String tables (SKDB v9, .strdb)

0x00  'SKDB' | u32 9 | u32 header size 0x1C | u32 key count | u32 0 | u32 text units | u32 0
0x1C  key count x {u32 key hash, u32 text offset}, strictly ascending by hash
then  UTF-16LE text, NUL-terminated strings, exactly `text units` code units to the end of the file

140 of 140 shipped tables follow this (version, header size, strictly ascending keys, offsets in range, text ending at the file's end). Each language folder ships string.strdb and string_uwp.strdb; the English table has 68,547 keys. A key is StringHash of the key name and an offset counts UTF-16 units: UI_1Sword_Starting_N (0xDCDB4713) reads "Urfael". Several keys can share one string.

Coverage on the shipped files

Open questions

• What the record bytes 1 and 9 encode beyond selecting the chunked reader, and what the archive header flags at 0x14 / 0x18 / 0x1C switch: every shipped archive has the same values, so only the code that reads archive+0x28..+0x2A and the size words' top bytes can tell.
• What the engine looks up in the hash -> path table besides copying it to the load stage; the callers of the merged table at fs+0x80 are not traced.
• Whether a loose member always replaces a container section, or only when its archive is above the container's: the measured Desert case has the loose copy higher. A bundle whose only loose member sits below its container would settle it.
• Whether the engine reads more than one copy of database\game\game.gamedb; the GameDB loader's use of the archives would settle it.
• Whether anything outside the executable reads Blade.arch06 and Lithlad.arch06. Inside it, nothing does: neither archive is in the config list or the live mount vector, no string in the executable names either archive or dlc\blade.txt / dlc\lithlad.txt, and the third caller of Mount, vt_0x22BF9C8 vf2 0x11ACD04, is a generic "mount this path and report success" method.
• Whether the files of the folders the folder search cannot find are reached by another path (the merged hash table at fs+0x80, or the next archive in search order). The search can miss folders: 0x3DDB90 binary-searches the archive's folder array (+0x40, 0x30-byte entries, count +0x48) with _strnicmp, whose length its main caller 0x3DDAD8 passes as 0x105, and a mounted archive keeps its folders in the on-disk order (79 of 79 mounted archives compared, the seven below included). Replaying that search over every archive, 15 of 16,302 folders cannot be found, all names that are a prefix of a neighbour: bundles\dlc4 in Patch_09, Patch_10 and Patch_DLC4; in HighMip the prefab folders fort\stairsfortwallb and stairsfortwallc, wilderness\human\mdbuildingb, flair\mystic\mystwalllow, volcano\architecture\bridge\walkway and balrogpit\balrogpltfrm_05bot; in uhdmip mystwalllow03 and balrogpltfrm_05bot2; in uhdmip_DLC4 stairsfortwallb, mdbuildingb and mystwalllow; and stairsfortwallb2 in Benchmark. The case that decides it: 11 of the 20 files in Patch_DLC4's bundles\dlc4 (the fourth story DLC's cinematic render bundles cin_render_d4can_* and cin_render_d4fin_*) exist in no other archive, its _LC twin included, so if those cinematics load, the engine reaches the files without this search.

Shadow of War (Steam), game version 1.21. An unofficial fan research project, not affiliated with or endorsed by Warner Bros. Games, Monolith Productions or Middle-earth Enterprises. Middle-earth: Shadow of War belongs to its owners.