Scripting: HavokScript, world scripts and natives

The game's scripting layer: the HavokScript virtual machine and its bytecode, where every world layer's script lives, how layer scripts are compiled and cached, how scripts reach engine records and natives, the event dispatcher and the running-script stack that some natives require, and the shipped script corpus.

The virtual machine

The game's scripts run on HavokScript, Havok's fork of Lua 5.1 with typed structures (hstructure declarations, hmake constructors) and light-userdata literals (0x572b96b9hi); its standard library table is at 0x21DAEB0 and the source compiler ships in the executable. The main VM is [GM+0x6CE8] -> a CScriptVM (vt_0x230EF00) whose +0x58 is the lua_State: L+0x10 the global state (+0x560 compiler settings, +0x588 the panic handler 0x1CFEEB4), L+0x48 / +0x50 the stack top and base; stack slots are 16 bytes {u32 tag, value}.

• CreateVM 0x1CFE890 and SetupVM 0x1CFEBCC build a VM; RegisterAllScriptBindings 0x1C01778 registers every native, the per-type binder 0x1B9F69C registers the DB* record types (see GameDB), and a helper 0x1D87644 registers into the main VM through the global script agent [GM+0x6CF0].
• LoadBuffer 0x55B920(L, compiler settings, ...) and Pcall 0x406344 run a chunk.
• Natives do not validate their arguments, and engine asserts write to a null address, so a native called with an argument shape the shipped scripts never use can fault.

Records and natives from script

A script names a record as DB<Kind>.Name; the lookup (WBDBRecordLookup) takes the kind's live category and the name's hash and binary-searches the category (0x278540). The Lua names differ from the category paths: DBFX / DBFXSeq (Client/ClientFXList / ClientFXSequence), DBCombatBuff, DBMaterialState (Combat/FX/MaterialController/States, 168 records), DBSlowMo (332), DBAlignments (52), DBBandType, DBCombatAppliedState, DBCharacterModels. A native's argument types are visible in its stub: each record argument is fetched as mov rdx, [rip+X]; call 0x407BAC, where [X] holds the live category of the expected kind (checked on 27 of 27 natives). Most DEBUG_* bindings are stripped (no implementation, or the empty stub 0x5009B0).

Script calls into the engine fall in two groups: calls that set engine state or queue work (weather, material states, AI requests: AISpawnManager:SummonGrunts only queues a spawn request) and calls that dispatch to script listeners - System:ActivateRemoteScript is the script bus's Notify 0x605990, which calls the global script agent per listener and so belongs on the main thread.

The running-script stack

The engine keeps a stack of the scripts it is running: depth u32 [base+0x2702540], entries 0x277C1B0 + i * 0x80 (at most 16; depth 0 is the static 0x2680000), entry +0x28 = the running script's name. 0x503EF8 returns the top entry's name. Only the engine pushes: the __eventmap dispatcher 0x406538 (push 0x4066B8 -> 0x5029D0 -> 0x502A00 -> 0x502A44, which always writes a name, "Invalid" at worst; pop 0x406744 -> 0x115F8E0) and the resource loaders 0x4FD9C0, 0x4FEFA8 and 0x4A6088. A few natives require it: NPC::SetAlignment 0x7D2ED4 asserts "This function should only be called from script code" when the name is empty; the other direct users of the name are CPlayerObj::SetAISpectatorInvisible, the player's SetHidden (vf32, logging only), SetRetreatingEnabled 0x1BAE1F8, 0x1BA7C8C and 0x1D86F28. Such natives work only from inside an engine-dispatched script handler; LoadBuffer + Pcall push nothing.

World scripts

Every world layer's script is one SCRI payload:

"SCRI" | u32 version (1) | u8 flag | u32 length | body
flag 1 = Lua source text, flag 0 = HavokScript bytecode ("\x1bLua" + header)

It ships as a standalone <layer>.scripts archive entry (the patch archives ship these, and the standalone copy wins) and / or as a SCRI section inside the layer's container. A layer's chunk ends with _G.loadedresource = section_<hash>; its entries are script threads function(self, threadId, scriptId) that may LithThreadSleep(threadId, seconds). Objects named *.SC are StartupCommands whose _<prefab>_SC_Script_<hash> entry the engine starts as a thread. Script threads run on simulation time, so they wait while the game is paused or unfocused.

Compiled layer scripts are cached for the whole process in the main VM's global reinstall (0x6A60EC: reinstall[id] unset -> compile the buffer and set it; set -> reuse it), so a layer's script is compiled once per process, the first time its region loads.

Which object runs which script is world data: the layer's ScriptBind block maps objects and events to functions of the layer's own script (see World).

The shipped corpus

The sources are the authored scripts, comments included. A layer's script object (_<layer>_SO_Script_<hash>_meta / _inst) declares its methods (Init, Enable ...) and the instance fields it drives (dynamic object groups, relays, flags). Script_SetState appears in 1,370 of the 2,562 scripts. The largest bytecode program, worlds\retail\desert\desert.scripts, has 2,227 functions, 184,681 instructions, 56,969 constants and 204 structure declarations, with no debug information. Function names carry a content hash and prefab scripts compile into every region that uses them: 1,167 of the 1,939 distinct functions the desert's objects bind (60.2%) are defined by name in another region's source, and the same name means the same code (616 of 617 compared functions match completely; a random other function matches 33 of 617).

The bytecode

The interpreter. The main loop 0x1D88000 fetches a u32 instruction, takes op = insn >> 25, refuses op > 0x5B and jumps through a 92-entry table at 0x1D8B208. Two further interpreter functions start at 0x1D8B378 and 0x1D8E188, each with a 61-case dispatch of its own. The callers (0x6C8E38, 0x406C94) pick one by the mode word of the VM's global state [[L+0x10]+0x1C8]: 0 runs the main loop, 1 runs 0x1D8B378 and 2 runs 0x1D8E188. The word is written once, when the global state is built (0x1D0A294), from +0x80 of the VM-creation settings. The engine creates exactly one VM: CreateVM 0x1CFE890 is the only caller of both the settings initialiser 0x1D0173C (which writes 0 there, names the state "Main State" and leaves the field untouched afterwards) and the state constructor 0x1D098C0. Modes 1 and 2 are therefore never selected in the shipped game, and the main VM runs in mode 0.

The two other loops are generic builds of the same interpreter. They dispatch in two steps, a 92-byte opcode-to-handler table (0x1D8E12C, 0x1D91308) and then a 61-entry handler table (0x1D8E038, 0x1D91214), with identical groupings in both: one handler serves the five call kinds (CALL_I, CALL_C, CALL_M, CALL, CALL_I_R1), one the five tail calls, one the three GETTABLE forms, one the six SETTABLE forms, and each _BK arithmetic or comparison opcode shares its plain twin's handler, as do EQ / EQ_BK and CHECKTYPE_D / GETSLOT_D. The main loop's table has 74 distinct targets instead. Mode 2 also checks every fetch: the new instruction pointer must lie inside the function's code ([proto+0x20] to + [proto+0x18] x 4) and be 4-byte aligned, otherwise it raises "VM Kernel error: invalid instruction pointer." (0x1D911D7). In all three loops the INTRINSIC_* opcodes (31-36) and DATA (76) return straight to the fetch, so no interpreter executes an intrinsic. Fields: A = bits 0-7 (a register, or a flag for EQ / LT / LE / TEST); C = bits 8-16 (RK: 256 and above = constant K[C - 256]); B = bits 17-24, with bit 25 set by the *_BK opcode of a pair (B is then a constant index); Bx = bits 8-24; sBx = Bx - 0xFFFF, relative to the next instruction. DATA (op 76) is a word the VM skips; several opcodes own a fixed number of DATA words after them (inline caches, structure ids, SETLIST counts). NEWSTRUCT's DATA word holds the structure id. return f(x) compiles to TAILCALL followed by an unreachable RETURN A, 0, and jumps are threaded (an inner else skip or a break lands directly on the enclosing construct's destination).

The opcodes. The executable carries HavokScript's opcode-name table (records of 0x30 bytes from 0x26809C8, names HKS_OPCODE_*, in opcode order; entry 92 is MAX):

In the main loop's table, opcodes 31-36, the INTRINSIC_* family, and 76 (DATA) point at the dispatch loop itself: the main interpreter treats them as no-ops. The shipped bytecode never uses the intrinsics (0 of 2,040,690 instructions in 18,994 functions). It uses 44 of the 92 opcodes; the 48 that never occur are 3, 5, 6, 11, 12, 18-21, 23, 24, 27, 29-39, 43, 48-51, 55, 61-63, 65, 67, 70, 77 and 79-90 (among them the plain GETGLOBAL, SETGLOBAL, CALL, GETUPVAL, the numeric for pair FORPREP / FORLOOP, and every slot and type-check opcode from 77 to 90 except SETSLOTI).

The chunk format (the engine's loader: LoadHeader 0x1D1A234, LoadFunction 0x1D1AFDC ("code too deep" past depth 128), LoadConstants 0x1D1AB48, LoadDebug 0x1D1AD68, LoadStructures 0x1D1F048, LoadString 0x1D1B548, Align4 0x1D1A114 (to a multiple of 4 of the bytes read so far); the writer's DumpHeader is 0x1D06A14). Little endian; int 4 bytes, size_t 8, instruction 4, lua_Number 4 (float32).

header    "\x1bLua" | 0x51 | 0x0E format | 1 endian | 4 int | 8 size_t | 4 instruction | 4 number
          | 0 integral | 3 build flags | 0 referenced
types     int 13, then 13 x {int id, int length, "TNIL\0" ... "TSTRUCT\0"}
function  int upvalues | int params | u8 flags | int max stack | size_t code count | pad to 4
          | code | int constant count, constants {u8 type: 0 nil, 1 u8 bool, 2 size_t light userdata,
          3 f32 number, 4 string} | int has debug [debug] | int child count, children
debug     int lines | int locals | int upvalue names | int line defined | int last line defined
          | string source (null = the parent's) | string name | lines | locals {string, start pc,
          end pc} | upvalue names
structs   int 1, then {string name (null ends) | size_t id | int | int | size_t slot count
          | slot count x {string name, size_t, int, int, size_t}}
strings   size_t length (0 = null), then length bytes including the terminating NUL

Checks on the shipped corpus: all 117 bytecode chunks reassemble byte for byte from a lossless listing; an operand-range check finds 0 violations over 2,040,690 instructions in 18,994 functions, while the same check with the A and B fields swapped finds 9,935 and with the opcode read one bit low 141,077. Decompiling the desert chunks back to Lua and comparing the 617 functions that also exist as source: 617 of 617 have the same calls and string literals.

The main thread's loop and the window pump

The main loop is 0xC25568: it calls the application object [base+0x26FEFD8] (vt_0x1FFE760) slot 33 (+0x108) and slot 29 (+0xE8), then pumps window messages (PeekMessageW / DispatchMessageW at 0xC2589A) and runs a sleep limiter. The game window belongs to the main thread. The main thread's frame wait (slot 28 0x380FAC -> 0x38105C -> 0x1CF430) is back-pressure for space in the render command ring and pumps messages too (0x6EE5D0, dispatch 0x6EE670); both pumps share the re-entrancy byte 0x277DBB0. Posting to the render ring (0x381FCC posts {0x1EE088, context, function}) uses an unlocked bump allocation, so only the main thread may post.

Shadow of War (Steam), game version 1.21. An unofficial fan research project, not affiliated with or endorsed by Warner Bros. Games, Monolith Productions or Middle-earth Enterprises. Middle-earth: Shadow of War belongs to its owners.