Week 7: Tool Safety Engineering II (Tool v0.3, logging + idempotency)

Structured logging; auditable artifacts; idempotent semantics; rollback. Tool v0.3 promotes Tool v0.2 from "safe enough for the cohort" to "auditable in front of a defense panel."

Theme

Tool v0.2 had the safety controls (authz; dry-run; confirmation). Tool v0.3 adds the audit trail and the post-action discipline that turn a defensible tool into a portfolio-quality artifact.

Three new properties arrive in v0.3:

• Structured logging. Every run writes a JSON log to ~/.sb6141-csrf/runs/<ISO8601>.json AND a human-readable summary to stderr. The JSON entry carries every relevant field; the stderr summary is the operator's at-the-moment view. JSON for machines; prose for humans; both at once.

• Idempotent semantics. Running the tool twice in a row has the same effect as running it once. For the SB6141 CSRF, idempotency means: the second run detects that the modem is ALREADY at factory defaults (the action's effect) and exits with "already done" rather than re-applying. The detection uses the fingerprint extended to capture configuration state.

• Rollback path. For state-changing actions, the tool documents what would undo the action and (where possible) implements it. For SB6141 factory-reset, the rollback would require capturing the pre-reset configuration before issuing the reset; the tool's --rollback mode replays the captured configuration. For actions that are genuinely irreversible (e.g., a one-shot exploit that triggers a one-way state transition), the tool documents the irreversibility AND refuses to run unless --accept-irreversible is also passed.

The Schneier framing for the week is accountability: a tool whose every run is logged is a tool whose history is examinable. Examinability is the property that lets the operator defend the tool months or years after the run.

By the end of Week 7 you have Tool v0.3: it inherits all v0.2 safety controls; it writes structured logs to disk per run; it implements the SB6141 idempotency check; it documents the rollback path. v0.3 is what becomes Tool v1.0 in the capstone (Week 12-13) with packaging, pytest coverage, and README polish added.

Schneier weave (~280 words, A Hacker's Mind Ch 12)

Schneier devotes Chapter 12 of A Hacker's Mind to the question of what makes a system auditable. His worked examples include central-bank reserve audits (where every gold bar's chain-of-custody is documented across decades), aircraft maintenance logs (where every part replacement is signed by a named mechanic), and OSS supply-chain SBOMs (where every dependency version is recorded for vulnerability cross-referencing).

The pattern across these examples: auditability is not the same as logging. A log that records "something happened" is not an audit trail; an audit trail records WHAT happened, WHO did it, UNDER WHAT AUTHORITY, and WITH WHAT OUTCOME, in a format another reviewer can mechanically verify. Schneier's argument is that auditability requires structure: human-readable prose is for the at-the-moment operator; machine-readable structure is for the post-hoc reviewer.

Tool v0.3 embodies the distinction. The stderr human-readable summary is the operator's at-the-moment view ("Action complete; HTTP 200; modem will be reachable in ~60 seconds"). The JSON run log is the reviewer's structured artifact: {"timestamp": "2026-MM-DDTHH:MM:SSZ", "tool_version": "0.3", "target": "192.168.100.1", "action": "factory-reset", "authorized_by": "Alice (cohort 2026-A)", "dry_run": false, "fingerprint": {"match": true, "firmware": "7.5.0"}, "result": {"success": true, "status_code": 200}, "exit_code": 0}. Both formats; same event.

The idempotency check is auditability extended into time: a second run, looking at the first run's log, can decide "this action has already been performed" without re-applying. The rollback path is auditability extended into reversibility: if the action is reversible, the tool records WHAT it changed so the reversal is mechanical.

Schneier's broader argument is that auditable systems are the systems whose actions can be defended. Tool v0.3 is the smallest expression of that discipline in an adversarial-research tool.

Reading list (~1 hour)

1. Schneier, A Hacker's Mind, Ch 12 ("Hacking the Defenses"). Academy library; calibre id 677.
2. The OSS SBOM specification (CycloneDX) at https://cyclonedx.org/specification/overview/. The reference structured-format for software-supply-chain auditability; the format ideas apply directly to per-run audit logs.
3. Python logging cookbook at https://docs.python.org/3/howto/logging-cookbook.html. The structured-logging patterns section is the relevant part.
4. structlog library at https://www.structlog.org/. Optional; v0.3 uses stdlib logging with custom JSON formatter; structlog is the third-party alternative students may prefer in capstone.
5. The Twelve-Factor App methodology, section XI (Logs) at https://12factor.net/logs. Industry reference for log structure as a stream; informs the JSON-per-run pattern.

Lecture outline (~50 min)

Part 1: Structured logging at the per-run level (15 min)

• One JSON object per run. Not one log line per HTTP request; one JSON object per tool invocation. The object captures the run's start, intent, and outcome as a self-contained record.
• The schema. Required fields: timestamp, tool_version, target, action, authorized_by, dry_run, fingerprint, result, exit_code. Optional fields: git_sha, operator_hostname, os_info. The schema lives in your tool's source; treat it as part of the contract.
• Where the JSON goes. ~/.sb6141-csrf/runs/<ISO8601>.json is the academy default. Per-run files (not appended to a single log) so each run is atomic; cumulative log construction is a downstream concern.
• What the JSON does NOT include. Full HTTP response bodies (too large; potentially sensitive); cookies; PII. The JSON is a record OF the action, not a forensic dump.
• Why JSON not YAML or TOML. JSON is parseable by every language without extra dependencies; YAML has gotchas (string-vs-number disambiguation); TOML is for configuration, not event logs. JSON is the lingua franca.

Part 2: Human-readable stderr summary (10 min)

• The pair pattern. JSON to file (auditability); prose to stderr (operator awareness). Both at once.
• What the stderr summary includes. One line per logical step: "Probing 192.168.100.1...", "Fingerprint OK (firmware 7.5.0)", "Dry-run mode (use --no-dry-run to execute)" or "Confirming destructive action...", "Action complete (HTTP 200)". Each line is a sentence.
• Why both formats. A reviewer asking "what happened" wants the JSON. The operator asking "is this working" wants the prose. Producing both at run-time is cheap; producing one after the fact is expensive (or impossible if the run already ended).

Part 3: Idempotent semantics (15 min)

• Definition. A tool is idempotent when running it N times has the same effect as running it once. For a state-changing tool, idempotency requires the tool to DETECT the prior state before re-applying.
• The SB6141 case. After a factory-reset, the modem's configuration is "default." The fingerprint at v0.1 detects "this is an SB6141"; an extended fingerprint at v0.3 detects "this is an SB6141 AT DEFAULT CONFIGURATION." A second run, finding the modem already at default, exits with "already done" rather than re-issuing the reset.
• Why idempotency matters here. Repeated factory-resets on a lab modem are not catastrophic (the modem survives) but they ARE wasted time and they degrade the lab's audit trail (the log shows N redundant actions). Idempotency makes the tool defensible at the run-frequency level.
• What the extended fingerprint checks. Specific configuration-state markers in the SB6141 admin page (default Wi-Fi name; default password prompt visibility; default firmware-version banner). The implementation: an extended fingerprint_with_state() function that returns both "is this an SB6141" and "is it at default state."
• Idempotency vs --force. A --force flag could override the idempotency check; v0.3 does NOT include it. Force is operationally tempting and discipline-degrading; if the operator needs to re-run after the first run, they can clear the prior-state record or document the override in the lab notebook.

Part 4: Rollback paths (10 min)

• When rollback is possible. State-changing tools that record their pre-action state can reverse the change. The SB6141 case: BEFORE issuing the factory-reset, capture the current configuration (admin password if set; custom DNS; static IPs; firewall rules). The captured configuration becomes the rollback target.
• When rollback is impossible. Some actions are one-way: a one-shot exploit that triggers a non-recoverable state transition; an action that depends on session-state the tool does not own. For these, the tool documents the irreversibility AND requires --accept-irreversible to proceed.
• The rollback mode. --rollback FILE where FILE is a prior-run's captured-state JSON. The tool reads the captured state, computes the diff between current and captured, and re-applies the captured values. For the SB6141 case, this means re-uploading the captured Wi-Fi settings, custom DNS, etc., via the admin interface (which itself uses the same CSRF-vulnerable endpoints; the rollback is the legitimate inverse of the destructive action).
• What the rollback does NOT do. Restore data that was lost (factory-reset wipes some data; rollback restores configuration but not user-modified content). The documentation makes the limit explicit.

Disclosure-Ethics Sidebar

The audit trail does double duty: it satisfies the cohort's discipline AND it builds the evidence base for the capstone disclosure report. The capstone report can cite specific run-log entries ("see runs/2026-05-30T14-22-00Z.json for the canonical reproduction transcript") rather than narrative-only descriptions.

Labs (~3 hr)

Lab 7: Tool v0.3, Logging + Idempotency (labs/lab-7-tool-v03.md)

• Goal: extend Tool v0.2 to Tool v0.3: structured JSON run logs + human-readable stderr + idempotency check + rollback path documentation
• Time: ~3 hr
• Artifact: lab-7/sb6141_csrf/v03.py + expanded tests + updated README in ~/adv-101/lab-7/

Independent practice (~5 hr)

1. Read three production-grade tool source codes (1.5 hr). ansible's --check mode handler; terraform's apply log handlers; kubectl apply --dry-run=server output. Notice the JSON-or-yaml structured output pattern; the human-readable summary pattern; the idempotency assumptions.
2. Build a small custom JSON formatter (1 hr). Write a logging.Formatter subclass that emits one JSON object per record. Use it in v0.3.
3. Idempotency exercise (1 hr). For an existing CLI tool you use (e.g., a deploy script; a backup script), describe in 200 words: is it idempotent? If yes, what's the detection mechanism? If not, what would it take to make it idempotent?
4. Schneier A Hacker's Mind Ch 13 (45 min). Continues the auditability framing into political-system examples.
5. Cross-tool comparison (45 min). Compare the run-log format of three CLI tools (e.g., apt, pip, git, docker). What do they capture; what do they omit; are they structured or freeform? Write a 200-word note for the Adversary Diary.

Reflection prompts (~30 min)

1. The v0.3 JSON log records authorized_by. Could an operator post-hoc edit the log file to change the value? What does that say about JSON-on-disk as evidence?
2. Idempotency in v0.3 means "the second run detects prior state." Suppose the modem has been factory-reset by HAND between two tool runs. Does the tool's idempotency check correctly identify this? Why or why not?
3. The rollback for SB6141 factory-reset requires capturing pre-action state. For a hypothetical CVE where the action is "drop all user accounts," what would the rollback look like? Could you implement it?
4. The tool emits stderr (prose) AND JSON (file). What if the operator pipes stderr to a file (tool ... 2> capture.log)? Does the JSON path break? How would you defend against this?
5. One thing from this week you want to know more about?

Adversary Diary (Week 7)

New entries:

• Python logging.Formatter subclassing for custom JSON output.
• structlog library at https://www.structlog.org/ (third-party; not used in v0.3 but worth knowing).
• CycloneDX SBOM spec at https://cyclonedx.org/specification/overview/. The structured-format reference.
• Idempotency as a design property; cite https://en.wikipedia.org/wiki/Idempotence for the mathematical foundation.
• Twelve-Factor App methodology at https://12factor.net/. Industry reference for cloud-application discipline; the logging section (XI) informs v0.3's log structure.

What would a reviewer ask?

1. "Show me a JSON log entry from your tool. Walk me through each field; explain why it is there."
2. "Run your tool twice in a row. Does the second run behave differently from the first? How?"
3. "Your tool documents a rollback path. Walk me through one. What does the rollback NOT restore?"

What comes next

Week 8 introduces CVSS v3.1 + impact scoring. You score the SB6141 CSRF formally; produce a vector string + per-metric justification; defend the score against adversarial review. Week 8 is the first week the documentation work (Weeks 8-9-10) begins; the tool engineering is structurally complete after v0.3.