GameDB: schema, records and field bindings

How the game's data database is stored and used: the GADB file, the loaded tables of categories, records and field descriptors, the slot rule that places every value, the value types and RecordLinks, name hashing and lookups, and the code side - the type table, the schema slots, the field getters and the compiled copies that decide whether a changed value is seen.

The database file

The game's data lives in one database, database\game\game.gamedb, stored in hotchunk.arch06 (56,759,100 bytes; see Archives and file formats). The running game's database is this copy: its 2,742 categories, 278,865 named (secondary) records and 415,767 inline (primary) records match the file one for one, with the same names in the same order. The same container, GADB version 17, also carries the behaviour graphs (.bvr, root type BlendDB; see Animation) and the database inside every effect file (see Visual effects).

The string block follows. Its offset 0 is the root type name (GameDatabase\0), and every offset in the file is measured from there: names, string values, and the binary values the file stores once and shares (float pairs, vectors, 64-bit integers). Names are stored whole, never as the tail of a longer string: of the 181,736 distinct record-name offsets, 179,567 follow a NUL and the other 2,169 follow binary values of the block. Then come the field descriptors, 15 bytes each:

The value section starts with u32 0 and the category count. Each category is a 24-byte entry followed by its primaries, then its secondaries:

Categories are stored in ascending sort-key order, and all 2,742 sort keys equal P919 of the name. Inside a category the secondaries are stored in ascending P919 of their names (1,371 of 1,371 categories with two or more; no two names in one category share a hash). Both orders exist for the binary searches below. The file ends with a u32 count of a (u32, u32) pair table (the loaded database's +0x68); the game database's is 0, and the walk over the value section lands exactly on it.

A file cell is 4 bytes: a word of Bools, a float, an integer, a string-block offset (string, enum name, 64-bit integer, float pair, vector) or an encoded RecordLink.

Loading

The loader 0x503694 (called from 0x74CF10) reads the 0x2C-byte header, refuses anything but GADB version 17 and sizes two allocations from the counts: an arena of categories x 0x58 + (descriptors + 2 x primaries) x 16 + secondaries x 0x28 + 0x88 bytes for the tables (30,491,600 for the game database), and an instance block of cells x 8 + 0x88 bytes for the values (68,357,496). Descriptors widen from 15 to 16 bytes.

The record reader 0x4BA1F0 reads each 4-byte cell into an 8-byte cell. For every field of type 4 to 9 (strings, enums, 64-bit integers, float pairs, vectors; the type test 0x4BA364 also accepts 12 to 16, which the game database does not use) it adds the string block's address to the cells, turning offsets into pointers into the block, which stays resident. A second pass, 0x4BAF6C, finishes two types:

• StringEnum (5): the cell is replaced by StringHash of the string it points at.
• RecordLink (10 and 11): the descriptor's type byte is rewritten to 10 (and 0xCA; or 0x0A, flag bits kept), and each element is resolved. Category = value >> 20; a category past the table gives null. Index = value & 0xFFFFF: an index below the category's primary count selects that primary, a larger one the secondary index - primary count, anything beyond both gives null.

In the running game the string, 64-bit, float-pair and vector cells equal the block's address plus the file's offset (721,156 of 721,164 strings, 210 of 210, 166,606 of 166,606 and 50,734 of 50,734).

The database in memory

A static object of class vt_0x1FBD460 at 0x2A68350 (constructed by 0x1B06B6C) owns the game database: +0x18 the database object ([base+0x2A68368]), +0x20 the path database\game\game.gamedb. Its vf3 0x1B09BBC fills the category cache (below). Every category header points back to the database at +0x18.

Other copies and the patch database. Four patch archives, Patch_01 to Patch_04 and their _LC twins, also ship a complete database\game\game.gamedb (58.08-58.10 MB against the HotChunk copy's 56.76 MB). Their headers name older builds: 2,435 to 2,438 categories against 2,742, and different build tags. The running game has 2,742 categories, so it loads the HotChunk copy and the older copies are shadowed by the archive search order, not merged. The loaders do merge one kind of database: unless the database being loaded is itself named gamedb_patch, they look for a patch database (0x1C3FF20 tries the resource system, then a loose file named after the database with gamedb_patch appended), keep it only when its build tag (+0x2C) equals the base's, and add its record and data sizes to the base allocation so its records load with the base's. The switch byte 0x266CD70 that gates this is 1 in the shipped executable and nothing writes it. Of the 488,679 entries in the shipped archives, the only game databases are the nine copies above, and none is a patch database.

The descriptor array ends exactly where the database object begins; the category array follows the object, and after it come each category's primaries and secondaries, in file order. Two rules place everything else, and both hold for the running game:

• a record's descriptor run = descriptor array + 16 x first-descriptor index (686,399 of 686,401 records with fields);
• a record's instance = instance block + 8 x (cells of all earlier records in file order) (694,628 of 694,632).

Instances are therefore packed back to back: a record owns exactly cell count x 8 bytes, and reading past them reads the next record. In the running game the whole database lies below 4 GB, so the pointers in its cells have a zero high dword (166,606 of 166,606 float-pair cells).

Categories

FindCategory 0x45A0E0(database, &hash) tail-jumps to 0x115B4D4, a binary search over database+0x50 (count +0x48, stride 0x58) on the key at +0x50.

The 2,742 categories fall under 134 top-level names (counting the _Structures* family as one):

The _Structures* categories hold the inline structure arrays: every primary lives in one of 736 of them, and no other category has primaries. 777 of the 799 also carry exactly one secondary, a placeholder named DEFAULT_RECORD_NAME<Type>; 22 carry none. Over the whole file 921 categories carry such a placeholder, always as their record 0 (921 of 921).

The category cache. At start-up 0x1B09BBC (vt_0x1FBD460 vf3) fills the static slots 0x26FFCE8..0x2701FD8, and 0x1BFDC4C 34 more past them, 1,260 slots in all: for each category name, StringHash(name) -> FindCategory -> mov [slot], rax (its first two stores go to 0x27019B8 AnimCurve and 0x27019C0 AnimatedExtras; 0x2701BC8 holds Client/PlayerMovement). 1,027 slots are read by other code, so a function that reads slot X works with category X. The most read are Interface/Shared (80 functions), Interface/ElementSetup 53, Faction/Definition 51, Inventory/Items 41, Quest/Quests 39 and Quest/Objectives 32. Script record types and native record arguments go through these slots (see Scripting).

Records

A primary is 0x20 bytes: +0x00 the descriptor count (the rest of the first qword is 0, so it has no 0x0A byte), +0x08 the run, +0x10 the instance, +0x18 its category (415,767 of 415,767).

FindRecord 0x278510(category, &hash) passes [cat+0x38] and [cat+0x28] to the binary search 0x278540, which compares the dword at record+0x04 with stride 0x28 (20 callers; 0x4594AC, for example, finds the category BehaviorSource and then a record by its name hash). 0x618F7C(category, name) hashes the name first, then runs the same search. Script lookups use the same binary search (see Scripting).

A name identifies a record only together with its category. The 278,865 secondaries carry 181,736 distinct names; 50,607 of them (27.85%) exist in more than one category, and 147,736 records carry such a name. The most shared are Default (95 categories), Player 45, True 35, Balrog 35, Warg 25, Troll 25, FellBeast 24 and False 22.

Zero-field records. 8,231 records have a descriptor count of 0. All of them point at the same run, the first descriptor of the array, and own no cells; a walk of "their" run reads another type's fields.

Descriptors and the slot rule

A loaded descriptor is 16 bytes, the file's 15 plus a pad:

A record's run lists its fields in declaration order. In all 13,612 distinct runs of the running game the field index ascends, the +0x04 column ascends and the +0x09 column is an exact permutation. FindDescriptor 0x25180C(record, &hash) (693 callers) uses them: with record flag 0x01 clear it calls FindDescriptorPermuted 0x251BD0, which binary-searches the +0x04 column and returns run + 16 x byte [entry+0x09].

Every record of a category declares the same fields in the same order (2,713 of 2,713 categories that hold records). What can differ per record is the element count of an array, and with it the links: 876 categories hold records with more than one layout (Combat/Attack has 1,898 layouts over 16,976 records, Faction/Roles/Definitions 578 over 914). A cell is therefore found through the record's own run ([rec+0x08]), never through another record's. The links are part of the file: over 11,413,092 descriptor uses compared, the running game's descriptors equal the file's in hash, sorted column, field index, run position and type (11 read as 10) every time, in element count in all but 2 and in link in all but 1.

The slot rule. Element i of a field is the 8-byte cell [rec+0x10] + (link + i) x 8, for i below the element count; for any other i every getter returns the caller's default. A field with count 0 therefore has no value in that record, a scalar has count 1 whether it is set or not (a null RecordLink still has count 1), and an array of n elements occupies n consecutive cells. GetIntFast 0x278758 is the rule:

movzx ecx, dl                 ; cached descriptor index
shl   rcx, 4
add   rcx, [r10+8]            ; descriptor = run + index * 16
movzx eax, word [rcx+0Ch]     ; element count
cmp   r8d, eax
jae   default
movzx ecx, word [rcx+0Eh]     ; link
mov   rax, [r10+10h]          ; instance
add   ecx, r8d
mov   eax, [rax+rcx*8]        ; cell link + element

Bools are bits. A Bool's link is a bit index: the value is bit link & 31 of the 32-bit word in cell link >> 5 (GetBoolFast 0x274AE8; MakeFieldRef 0x2775C0 builds the same cell pointer and bit). In the 13,612 distinct loaded runs every Bool has count 1, and read this way none of them shares a cell with a non-Bool field of its run (52,003 of the 60,879 would if the link were a cell number). Up to 32 Bools share one word, so a Bool is changed as a single bit.

Live cell i is file cell i. Comparing every cell of every record with the file: integers equal in 640,915 of 641,086 cells, floats in 900,372 of 900,436, Bools in 2,943,742 of 2,943,938 (the same bit rule applies to the file's 4-byte cells) and string values in 721,086 of 721,164.

Example: an AI/Templates record

The type AI.Template has six fields. The first record's run:

A lookup of CombatProp (0x9CE11687) finds it at position 3 of the +0x04 column; that entry's +0x09 is 2, the descriptor of CombatProp. Its value is the RecordLink in cell 3 of the instance.

Value types

The low six bits of the type byte are the type; by-name getters compare type & 0x3F. Flag 0x40 appears only on Strings (code 0x44) and marks a resource path: of 196,950 such cells, 120,185 are empty and 73,591 of the other 76,765 name a file (.bndl 29,795, .fxp 16,814, .mat 14,642, .tex 7,788, .skel 3,197, .vid 940). Flag 0x80 is unused.

A StringEnum holds a hash, not a pointer. All 336,338 cells have a zero high dword, and 336,337 equal P919 of the name the file stores. The hash folds case, so HIGH and High are one value. Read as an address, such a hash can land inside the database's own memory, which lies below 4 GB.

The executable names its enums. Initialised data from 0x2A42590 to 0x2A60340 holds 16-byte entries {char* name, u32 P919(name), u32 value}: 3,654 of them in 330 runs of two or more (a whole-image scan that accepts an entry only when its hash is P919 of the string it points at). Most runs start at 0 (314) and count up (266); an empty entry usually separates two enums, but not always (one run of 156 goes from a quality list into the PC button codes, ending at Mouse_Wheel_Down 1008). For example ECommandType reads COM_MOVE 0, COM_INV 1, COM_VIEW 2, COM_MISC 3, COM_WEAP 4 and kNumCommandTypes 5. Since a StringEnum cell holds the same P919, these tables turn a cell into its integer; which code does that has not been traced.

A float pair is a pointer. GetFloatRangeFast 0x311320 loads the qword in the cell and copies the 8 bytes it points at; the pair is not stored in the cell.

Shared values

The file stores each distinct string, float pair and vector once, and every cell that holds that value points at the same bytes; loading keeps the sharing. The 166,606 float-pair cells point at 3,071 distinct pairs: (0, 0) is shared by 72,774 cells across the database, (1, 1) by 17,492, (100, 200) by 13,591, (0.25, 0.25) by 13,257 and (0, 200) by 10,743. The 50,734 vector cells point at 2,596 distinct vectors, one all-zero vector serving 25,616 cells. Writing the floats behind such a pointer changes every field that shares them; storing a different pointer in one cell changes that field alone.

RecordLinks

In the file a link is (category ordinal << 20) | index, where the ordinal is the category's position in the hash-sorted table (the loaded header's +0x2C) and the index counts the category's primaries first, then its secondaries. 0xFFFFFFFF is null; 0 is a real link (category 0, primary 0). In memory the cell holds the target's header address.

In the running game 5,204,006 link cells hold 3,471,923 nulls (0xFFFFFFFF in the file, every one), 1,316,316 pointers to secondaries and 415,767 to primaries, and none point anywhere else. Every link to a primary targets a _Structures* category and every link to a secondary targets another category, so once both file types read 10, the target category tells which side a link addresses. 1,732,055 of the 1,732,083 non-null cells hold exactly the record the file's link names.

Links carry no type. GetLinkCellFast 0x30E7B4 returns the cell as it is; GetLinkCellByName 0x68BB08 checks only that the field is a RecordLink (type & 0x3F == 10) and that the element exists. Which category a field should point into is recorded only in its registration (a public <Type> annotation, Inventory.Item for an item link). The code that follows a link treats the target as the declared type, so a link to a record of another category is read with the wrong layout.

Native-structure records

A record with flag 0x10 is not addressed by the slot rule: its instance is a native structure laid out by its type's slots (SlotOffset 0x251C24, size step 0x251CAC), the by-name getters divert to the reflected path (GetLinkCellByName -> ReadLinkReflected 0x16CE2E8 -> ReadReflectedField 0x5DA158) and MakeFieldRef hands such records to 0x60B5FC. Every record of the loaded behaviour graphs carries 0x1A (see Animation); no record of the game database sets the bit.

Names and hashing

StringHash 0x423E70 is the engine's name hash, written P919 in this reference:

h = seed                    ; r8, 0 for names (wrapper 0x115B52C)
for each byte c:  h = h * 919 + (int8) LUT[c]     ; 32-bit wrap-around

The 256-byte table is at 0x1DB50D0 (95 non-zero entries):

Letters fold, so every name hash is case-insensitive. Inlined copies of the loop exist, among them PropRegister 0xC6DA00. The hash keys categories (AI/Templates, slash included), records, fields, enum values and resource paths (see Resource streaming). The database holds 11,756 distinct field names over 18,554 (category, field) pairs, and a field name can start with a digit (3DRotation, 0xA37788CC).

The schema in code

The type table

A category's type has a static 0x40-byte entry in a reflection type table (the file-backed part of .idata), and the category's +0x20 points at it (2,713 of 2,742 categories; 29 have none). The same table describes the ClientFX node types.

AI/Templates points at 0x266CB70: schema object 0x2955C80, 6 slots, Template, AI.Template, no parent. The table holds 3,099 entries; 2,686 are linked by categories, 517 of those derive from another entry, and every parent pointer lands on an entry. The 2,742 categories carry 2,686 distinct type names; 42 names serve several categories (Combat.Int 7, Combat.FloatRange 7, Profile.Analog 3 for the PC, Orbis and Durango input categories).

Schema objects and field registration

A schema object is one 0x48-byte slot per field, zero at rest. A static initializer clears it and tail-jumps into the type's builder (0xA7EB0: memset(0x2955C80, 0, 0x1B0), then a jump to the AI/Templates builder), so a builder has no direct caller. A builder registers each field in declaration order, usually by calling one small stub per field with the slot in rcx. Client/PlayerMovement's builder 0x843EDC calls the stubs of FallDamageMinHeight, FallDamageMaxHeight, FallDamageMin ... on slots 0x48 apart from 0x2871AA8; the stub 0x163580C registers BaseSpeedMultiplier, type 2, annotation min=0.0, default read from 0x202E5A0, and 0x1631698 registers AimingRadiusScale with min=1.0. Some builders call PropRegister inline, and some registration code inlines PropRegister itself: 0xC75150 hashes and stores a group marker (BeginThermalVent, annotation GROUP_BEGIN=ThermalVent) without calling it. Range and type annotations are rare: the executable holds 325 strings that begin with type=, 161 that begin with min= and 5 that begin with max=.

PropRegister 0xC6DA00(slot, name, type, &default, annotation, arg6, flags, accessor) fills the slot and calls the accessor:

Slot k is field k (the descriptor's +0x08), except that a group marker takes a slot and no field index (Combat/Node.KBMNode is slot 16 and field 14, after two markers). The same PropRegister and slot size build the ClientFX node properties and the BlendDB node types: the three are one reflection system. Read in a running game, the schema objects hold 26,935 slots of 2,951 types, every name equal to the hash stored beside it, 2,463 of them group markers; two launches gave identical slots and the same cached index in all 22,630 slots cached in both. For every cached slot, the descriptor at that index in a record of the slot's category carries the slot's hash (14,558 of 14,558; the neighbouring slot's hash, as a control, 0 of 14,310).

The six AI.Template slots in a running game:

Reading a field

A compiled field accessor names its field only through the slot. AI/Templates.CharacterModel (0x16680C) is the template every RecordLink accessor follows (5,215 of them):

if ([slot+0x30] >= 2)   GetLinkCellFast 0x30E7B4(record, byte [slot+0x40], element, &default)
else                    GetLinkCellSlow 0x115AE24 -> GetLinkCellByName 0x68BB08(record, &[slot+0x20], ...)

The slot is reached through the type entry: 0x166858 tests [[0x266CB70]+0x78] >= 2 (slot 1's state), 0x166868 reads [[0x266CB70]+0x88] (its index), 0x1822494 returns [0x266CB70]+0x68 (its hash). The fast path applies the cached index to any record of the type, which is sound because the field order is fixed per category. Every type has a fast and a by-name getter (callers are distinct functions):

Writers and array walkers use field references: MakeFieldRef 0x2775C0(out, record, descriptor) (2,085 callers) builds {+0x00 record, +0x08 cell pointer, +0x18 u16 type, +0x1A u16 count, +0x1C u8 Bool bit, +0x20 hash}; MakeFieldRefByName 0x1957C4 adds FindDescriptor. FieldIterBegin 0x465A78 adds an index (+0x28) and the count (+0x2C), and FieldIterAtEnd 0x2774A8 compares them.

Who reads a field, and when

Static analysis of the slots names the code that reads each field: 39,960 functions touch the slot of a GameDB field, and 13,425 call a one-field accessor. Following the value out of every read classifies 12,789 fields:

A copied pointer value (a float pair, a vector, a RecordLink) stays live for the values behind it: changing the pair's floats is seen through the copy, while pointing the cell elsewhere is seen only after the next copy. When a copy is refreshed depends on what runs the copier:

• Movement values. 0x187F94C(object, movement record) copies Client/PlayerMovement BaseSpeedMultiplier to +0x550, FallMinHeight to +0x558 and PlayerYawInterpolateTime (an integer, converted to float and scaled) to +0x554, and keeps the record at +0x548; given the record it already holds, it returns without copying. 0x187F71C passes the record named PlayerMovement, found by name in the cached Client/PlayerMovement category. A change to that record's cells is therefore not seen by this consumer until a different movement record is applied and the original is applied again.
• Per-event copies refresh at every event: _StructuresCombatAttackData.AOE and AOEOnStart are copied by 0x2DED44, which runs under every attack start, so a change applies to the next attack.
• Start-up and load compilers do not refresh on their own: the targeting cache 0x1C0B634 and the attack cache (see Combat), the material-set cache 0x1B6190C (see Characters), item prompt links (see Items) and the AudioHooks cache, which the audio thread compiles at load (see Audio and video).

Changing values in a running game

• A cell is plain memory: the getters read it on every call, with no lock, checksum or cache in between (the fast getters above are the whole read path). Whether a change is seen is decided by the readers' copies above.
• Only the cells a record's own run claims belong to it: a field with count 0 has no cell, and an element written past a field's count lands in another field of the same record, or past the instance in the next record.
• A Bool shares its word with up to 31 others; a whole-cell write changes all of them.
• A StringEnum cell takes P919 of the enum name; a String, Int64, float-pair or vector cell takes a pointer, and the bytes it points at are shared by every cell with the same value.
• A RecordLink cell accepts any record: neither the getters nor the cell know the declared target, so nothing stops a link to a record of the wrong category, and its consumer reads that record with the declared type's layout.

Open questions

• Apart from the AudioHooks records, which the audio thread compiles at load, which categories are read off the main thread (render, audio, input polling, animation jobs), so that a change can race its reader? Camera, interface world-UI, audio control values, input profiles, native wildlife, AI sensors and the weather and lighting structures are the candidates; sampling the field readers per thread, or tracing their callers to the thread routines, would settle it.
• Do loads rebuild record instances from the file? A changed cell can return to the file's value after a loading screen, yet changed cells can also survive a region change; whether a load re-reads the instance block, and on which path, is not traced.
• How a merged patch database's records combine with the base's when both define the same record (replace, or append under the same name); the merge sizes are traced, the per-record rule is not.
• Which container sets record flag 0x01 (runs sorted by their own hashes): no game-database record sets it, and the byte scan that settled flag 0x08 finds 352 tests of some object's byte +0x01 against 1, which it cannot tell apart from record tests.
• The meaning of the database's +0x58 / +0x60, the type entry's +0x38 flags, and whether anything checks the build tag at +0x28.
• What the loader's rebasing test for types 12 to 16 is for. No shipped container uses them: the 314 GameDB containers in the archives (the game databases and every behaviour graph) use only types 1-11 and 68, and the 1,310 world parts' property databases only types 1, 2, 3, 4 and 8.

Shadow of War (Steam), game version 1.21. An unofficial fan research project, not affiliated with or endorsed by Warner Bros. Games, Monolith Productions or Middle-earth Enterprises. Middle-earth: Shadow of War belongs to its owners.