From WerWolv · WerWolv · · 3 min
Reverse Engineering File Formats with ImHex
Learn how to analyze custom binary files by decompiling game code and writing declarative pattern definitions in ImHex.
In brief
Learn how to analyze custom binary files by decompiling game code and writing declarative pattern definitions in ImHex. Reverse engineering unknown binary files relies on finding the parsing logic in decompiled code and incrementally building declarative pattern definitions to decode every byte. Originally reported by…
Reverse engineering custom binary files
Demystifying raw binary formats requires pairing decompiled program logic with declarative pattern definitions. The save format of the game FEZ serves as an ideal subject.
“We’ll go from a completely custom binary save file for the game FEZ to a full definition written in the Pattern Language...”
Inspecting raw uncompressed bytes
Opening a file in a hex editor like ImHex reveals immediate clues. Readable ASCII strings and repeating byte patterns indicate whether data is compressed, encrypted, or missing magic headers.
“The file seems to be uncompressed and unencrypted, as seen by the plain-text strings and other patterns in the file...”
Decompiling the application binary
Managed languages like C# (.NET) decompile back to clean source code using tools like JetBrains Rider. Navigating assembly references quickly locates file I/O handlers.
“The game is written in the C# programming language, which is generally really easy to reverse engineer.”
Locating the serialization logic
Searching for save file naming strings leads directly to the core write methods. In FEZ, PCSaveDevice.cs uses BinaryWriter to serialize game data into byte streams.
“In the constructor of that class, we can also immediately see string str = "SaveSlot" + (object) index;...”
Starting the ImHex pattern
Pattern files map binary structures declaratively onto raw offset addresses. Defining a top-level struct with the @ operator anchors data decoding at offset zero.
“We can start simply by creating a struct FezSaveFile and placing it at the start of the file using the @ placement operator.”
Decoding standard Windows timestamps
Binary headers frequently embed standard system timestamps. Using ImHex's imported standard time library parses 64-bit Windows FILETIME numbers into readable dates.
“This simple change now turns that unreadable number from before into a nice, human readable representation of the actual time value...”
Enforcing fixed buffer bounds
Many binary writers enforce fixed byte sizes and zero-fill remaining space. Attaching [[fixed_size]] attributes documents expected memory bounds directly in the pattern code.
“This maps incredibly well to the [[fixed_size(0xA000)]] attribute that can be attached to FezSaveFile to ensure that.”
Mapping primitive struct fields
Sequential scalar types in C# serializers map directly to primitive types in pattern code. Version integers, timestamps, and boolean flags form the foundational layout.
“Here we can see aaaaaaalll the different fields that are being written out to the binary.”
Validating versions with assertions
Inline pattern logic allows format verification during parsing. Adding std::assert guarantees that incompatible save versions trigger clean errors before parsing corrupts.
“...if we want to be extra fancy and make sure that we only load files that are actually compatible with our pattern, we can easily assert on this field.”
Modeling optional nullable objects
Custom serializers often write a boolean presence flag prior to optional values. Generic template structs evaluate these flags dynamically to conditionally parse payload fields.
“First, a bool is written to the file that represents whether or not the object is null.”
Decoding variable integer encodings
.NET string lengths use 7-bit encoded integers where the most significant bit signals whether additional bytes follow. Reading loop patterns decode these compressed sizes.
“...all this does is use the MSB of each byte as a flag to tell the parser if there’s another byte still coming.”
Transforming custom bitwise data
ImHex attributes [[format]] and [[transform]] execute custom bitwise shift algorithms during parsing, presenting clean mathematical values in UI tree views.
“Additionally, to make this type a bit easier to work with, we can use the [[format]] attribute to display the decoded integer value...”
Parsing dynamic string structures
Combining 7-bit length decoding with dynamic character array fields creates reusable string components that mirror native framework string writers.
“Now that all of this is done, we can finally define our String type.”
Expressing generic list collections
Serialized dictionaries write an integer count followed by paired items. Generic template definitions easily handle repetitive key-value parsing across entire files.
“All of this together now lets us finally decode the list.”
Replacing integers with enums
Enumeration types substitute raw integer codes with human-readable textual labels in hex editors, making parsed binary properties instantly recognizable.
“We could just treat it as an int like the serializer code does, but it would be nicer to keep the names available in ImHex as well.”
Handling nested complex structs
Hierarchical game data translates into nested pattern structs. Serializers invoking sub-functions map cleanly to distinct sub-struct definitions.
“This maps really nicely to a new struct that we can call LevelSaveData and just keep going in there as before...”
Modifying decoded binary data
Completing a pattern highlights every byte in the hex viewer. Users can visually navigate nested tree structures and edit binary values directly in place.
“You can now browse through the Pattern Data View and inspect what all these different values mean and even modify them by double-clicking the value!”
Step 1: Detect known formats
Begin reverse engineering by checking magic bytes and signatures. Tools like binwalk and ImHex auto-detection identify standardized compression or container wrappers.
“This can be done in various ways, ImHex magic detection and tools like binwalk can help a lot.”
Step 2: Locate target program logic
Use decompilers tailored to the binary language, Rider for .NET, Ghidra or Binary Ninja for native executables, and search for file I/O strings to locate serialization code.
“Rider works great for .NET, Ghidra, IDA or Binary Ninja for native-compiled programs, Recaf for JVM languages.”
Step 3: Identify structural primitives
Map high-level language constructs to raw byte representations by cataloging how the target program writes primitives, strings, arrays, and flags.
“Identifying them is the first step to understanding the file step by step”
Step 4: Build declarative patterns
Write pattern files iteratively to verify assumptions and document binary schemas. Patterns serve as live documentation and interactive parsers for binary reverse engineering.
“Patterns are great not only for decoding the file once you know how it works but also for documenting and verifying your findings along the way.”
The headline
Reverse engineering unknown binary files relies on finding the parsing logic in decompiled code and incrementally building declarative pattern definitions to decode every byte.
