NET-301 Week 11 -- Reverse Engineering Cross-Cut: Protocol Analysis and Wireshark Dissectors

"Protocol RE is not guessing. It is observation, then hypothesis, then systematic testing until the specification emerges from the traffic." -- Davidoff & Ham, Network Forensics: Tracking Hackers Through Cyberspace, Ch 3

Lecture (90 min)

11.1 Protocol RE Methodology

The Davidoff and Ham approach to network protocol reverse engineering follows a disciplined five-phase cycle. This is not a linear process -- you iterate through phases as evidence accumulates.

Phase 1: Observation. Capture traffic of the unknown protocol in normal operation. Generate varied behavior: connection establishment, data exchange, idle periods, graceful termination, error conditions if observable. The more behavioral variety in the capture, the more states and transitions you can identify. Minimum useful capture: 10 minutes of normal operation generating all message types you know exist.

Phase 2: Hypothesis. Examine the byte stream and form hypotheses about the framing structure:

• Where do messages begin and end? Look for length-prefix fields, delimiter bytes (0x00, 0x0A), or fixed-size records.
• Is there a magic number (fixed byte sequence that appears at the start of every frame)?
• Is there a message-type field? If so, how many distinct values appear?
• Are fields fixed-width or variable-width?

Phase 3: Field annotation. In Wireshark, annotate individual byte ranges with their hypothesized meaning using the "Add Column" feature or the tshark output with specific byte offsets. For each annotated field, track: offset from frame start, length, value in all captured frames, and the range of values seen.

Phase 4: State machine extraction. Group messages by type. Order them by capture time within each connection. Draw the state transitions: what message type follows what, under what conditions, from which direction? A state machine has: named states, named events (message types), guards (conditions under which a transition fires), and actions (what the receiving end sends in response).

Phase 5: Verification. Test your hypotheses by crafting packets with Scapy and sending them to the server. Does the server respond as your state machine predicts? Does it reject malformed packets in the way you predict? Confirmed behaviors earn "CONFIRMED" confidence; untested inferences earn "INFERRED"; purely observed-but-not-tested patterns earn "HYPOTHESIZED."

Confidence vocabulary (from Davidoff & Ham):

• CONFIRMED: behavior verified by active testing with crafted packets
• INFERRED: behavior consistent with all observations but not directly tested
• HYPOTHESIZED: behavior consistent with some observations; alternatives not ruled out

11.2 Wireshark Lua Dissector Development

Wireshark's Lua scripting API allows writing custom protocol dissectors that parse unknown protocols directly in the Wireshark packet view. A dissector registers for a transport-layer protocol and port, then parses the payload tree.

Minimal Lua dissector scaffold:

-- dissect_kvstore.lua
-- Dissector for a simple key-value store protocol:
--   2-byte magic (0xAB 0xCD)
--   1-byte message type (0x01=CONNECT, 0x02=AUTH, 0x03=GET, 0x04=SET)
--   2-byte payload length (big-endian)
--   N-byte payload

local kvstore_proto = Proto("kvstore", "Key-Value Store Protocol")

-- Define fields
local f_magic   = ProtoField.uint16("kvstore.magic",   "Magic",        base.HEX)
local f_type    = ProtoField.uint8( "kvstore.type",    "Message Type", base.HEX,
                                    {[0x01]="CONNECT", [0x02]="AUTH",
                                     [0x03]="GET",     [0x04]="SET"})
local f_length  = ProtoField.uint16("kvstore.length",  "Payload Length", base.DEC)
local f_payload = ProtoField.bytes( "kvstore.payload", "Payload")

kvstore_proto.fields = { f_magic, f_type, f_length, f_payload }

-- Dissection function
function kvstore_proto.dissector(buffer, pinfo, tree)
    if buffer:len() < 5 then return end  -- minimum frame: magic(2) + type(1) + length(2)

    local magic = buffer(0, 2):uint()
    if magic ~= 0xABCD then return end   -- not our protocol

    pinfo.cols.protocol = "KVSTORE"

    local subtree = tree:add(kvstore_proto, buffer(), "Key-Value Store Protocol")
    subtree:add(f_magic,   buffer(0, 2))
    subtree:add(f_type,    buffer(2, 1))

    local payload_len = buffer(3, 2):uint()
    subtree:add(f_length,  buffer(3, 2))

    if buffer:len() >= 5 + payload_len then
        subtree:add(f_payload, buffer(5, payload_len))
    end
end

-- Register on TCP port 9876
local tcp_port_table = DissectorTable.get("tcp.port")
tcp_port_table:add(9876, kvstore_proto)

Loading the dissector:

# Copy to Wireshark's personal Lua directory
cp dissect_kvstore.lua ~/.local/lib/wireshark/plugins/

# Or load interactively: Tools > Lua > Reload Lua Plugins
# Or: wireshark -X lua_script:dissect_kvstore.lua capture.pcap

Using tshark with a custom dissector:

tshark -r unknown_protocol.pcap -X lua_script:dissect_kvstore.lua \
    -T fields -e kvstore.type -e kvstore.length -e frame.number

11.3 Scapy for Protocol Fuzzing

Once you have a field hypothesis, use Scapy to test boundary conditions:

from scapy.all import *

TARGET_IP   = "10.0.1.10"
TARGET_PORT = 9876

def make_kvstore_frame(msg_type: int, payload: bytes) -> bytes:
    magic   = b'\xAB\xCD'
    mtype   = bytes([msg_type])
    length  = len(payload).to_bytes(2, 'big')
    return magic + mtype + length + payload

# Test: valid CONNECT message
valid_connect = make_kvstore_frame(0x01, b'client_id=test')
pkt = IP(dst=TARGET_IP) / TCP(dport=TARGET_PORT, flags="PA") / Raw(load=valid_connect)
resp = sr1(pkt, timeout=2)

# Test: oversized length field (potential buffer overflow probe)
overflow = make_kvstore_frame(0x03, b'A' * 65535)
pkt_overflow = IP(dst=TARGET_IP) / TCP(dport=TARGET_PORT, flags="PA") / Raw(load=overflow)
resp_overflow = sr1(pkt_overflow, timeout=2)
print("Server response to overflow:", resp_overflow.summary() if resp_overflow else "No response")

# Test: invalid magic
bad_magic = b'\xFF\xFF\x01\x00\x04test'
pkt_bad = IP(dst=TARGET_IP) / TCP(dport=TARGET_PORT, flags="PA") / Raw(load=bad_magic)
resp_bad = sr1(pkt_bad, timeout=2)

What to look for:

• Does the server close the connection on bad magic? (CONFIRMED: frame validation at layer 1)
• Does the server accept an oversized payload silently? (potential HYPOTHESIZED: buffer vulnerability)
• Does the server send an error message for invalid message types? (can reveal error-message format)

11.4 State Machine Extraction in Practice

A state machine for the key-value store protocol example:

States:     INIT, ESTABLISHED, AUTHENTICATED, CLOSING

Transitions:
  INIT         --[CONNECT (0x01) from client]--→  ESTABLISHED
  ESTABLISHED  --[AUTH (0x02) from client]------→  AUTHENTICATED  (if credentials valid)
  ESTABLISHED  --[AUTH (0x02) fails]------------→  CLOSING        (server sends RST)
  AUTHENTICATED --[GET (0x03) from client]------→  AUTHENTICATED  (server replies GET_RESP)
  AUTHENTICATED --[SET (0x04) from client]------→  AUTHENTICATED  (server replies SET_RESP)
  any state    --[TCP FIN]--------------------→  INIT

Confidence notes for the example:

• INIT → ESTABLISHED: CONFIRMED (directly tested with CONNECT frame; server responds with 0x01 ACK)
• ESTABLISHED → CLOSING on bad AUTH: INFERRED (AUTH failure observed in capture; TCP RST follows; not tested with all failure cases)
• AUTHENTICATED loops: INFERRED (multiple GET/SET cycles observed; no state-change behavior observed after valid auth)

11.5 Capstone Gate 5 Walkthrough

Gate 5 requires a protocol RE write-up with: 3 named states + transitions, 3 message types with field tables, confidence levels for each claim.

What earns full credit:

Common Gate 5 failure modes:

1. No active testing: the write-up is pure observation with no Scapy verification. All claims are INFERRED. This caps the score because CONFIRMED claims are required.
2. Undifferentiated fields: the write-up says "bytes 0-3 = header" without identifying the sub-fields within the header. Field-level granularity is required.
3. State machine too simple: only 2 states (CONNECTED / CLOSED). The unknown protocol has 3+ observable states; a 2-state machine fails to account for the authentication or error states.

Architecture Comparison Sidebar: Protocol RE Toolchain

RE-101 graduates: If you have completed RE-101, the Ghidra column is in scope for Lab 13's extra credit path. Analyze the server binary to confirm field formats via static analysis. The CONFIRMED confidence level can be achieved via Scapy testing or via binary analysis -- both count.

Reflection Prompts

1. The confidence vocabulary (CONFIRMED/INFERRED/HYPOTHESIZED) is borrowed from intelligence analysis methodology, not from software engineering. Why is this vocabulary appropriate for protocol RE? What would a software engineer use instead (e.g., "test coverage"), and why does that vocabulary not translate cleanly to the RE context?
2. A Lua dissector can be registered on a TCP port. What happens when the unknown protocol uses dynamic ports (ephemeral port negotiated in an HELLO message)? How would you modify the dissector framework to handle dynamic port assignment?
3. Scapy's sr1() function sends a packet and waits for the first response. In a state-machine with 3+ states, you may need to maintain connection state across multiple Scapy sends. Write pseudocode for a Scapy session that: establishes a connection, authenticates, sends one GET, and then gracefully closes -- tracking which state you are in at each step.