All projects & work

PWN-AI

Inside the agent roles, session memory and evidence records behind PWN-AI.

Team project · contributor

Agent roles
Head, implementor, feedback
Session runtime
Python and LangGraph
Record validation
Pydantic
Review surfaces
Live stream and Markdown
A PwnAI test session against a local DVWA target with streamed output
A local DVWA test session with its goal, execution controls and streamed findings.

LangGraph · RAG · Pydantic · FAISS · Express

View repository

The idea

I contributed to PWN-AI as part of a team. When I open a session, I want to be able to answer three questions. What did we ask the system to investigate? What did it actually observe? Which parts of its conclusion still need a person to check?

Our browser workspace gives me a live view. Underneath it, a Python graph keeps the planning trace, typed run records and feedback for the current session. The Markdown report draws from those records, so I can revisit a claim after the live output has scrolled away.

The agent system and its session record

The agent system and its session record
The head, implementor and feedback roles contribute different records to the Python session. Express streams process output to the browser. The report writer reads the structured session history.

One workspace, two kinds of conversation

I can use ordinary chat to ask a question or open a testing session with its own scope, goal and session ID. The Express controller gives the client two ways to follow the session. A streaming request forwards chunks from the Python process. A buffered request returns an ID, output and completion status.

The streaming controller sends plain UTF-8 text, flushes the headers and forwards both standard output and standard error. I can watch a failure appear in the same view as other progress. Closing the request invokes cancellation. When I need to verify a finding later, I use the typed records and the saved report.

Read the source · chat.controller.ts

What each agent contributes

I read the agent history as three different accounts of the same work. The head records the question being pursued. The implementor records what happened within an objective. The feedback role reviews that account against the associated evidence.

  • The planning trace preserves the head’s stated reasoning about the objective.
  • The implementor’s assessment carries a status, summary and remaining uncertainty.
  • Feedback records attach lessons to attempt IDs. A session reflection collects those lessons and any questions still open.

Read the source · contracts.py

Memory has a session boundary

The LangGraph state keeps the current task and result alongside appendable histories of runs, findings, notes and reflections. Working memory is rebuilt from these session records. Looking back through a session therefore has a specific meaning: I can follow the records that went into its summary.

When I read the exported report, I also look for the stop reason. A session can end with questions still open. Its end marks a boundary in the work, and the last objective assessment tells me how much evidence we had at that point.

There is a second store with a different lifetime. The FAISS index holds reference documentation embedded with Ollama. I treat that material as background about the software. The run records and evidence references tell me what happened in this particular session.

Read the source · state.py

An execution record and a finding answer different questions

A TaskRunRecord groups the request, decision, result and status for one task. Its ExecutionResult carries timestamps, return code, output streams, artifact references and an optional error. Those fields let me reconstruct what the runtime recorded.

A Finding is a separate Pydantic model with its own ID, title, evidence and references. Severity describes impact. Confidence is a bounded number attached to the assessment. A successful process exit alone gives me neither of those judgments.

The finding contract

Pydantic rejects extra fields, restricts severity to five values and bounds confidence between zero and one. These checks constrain the record shape. A reviewer still has to assess the claim.

class Finding(BaseModel):
    model_config = ConfigDict(extra="forbid")

    finding_id: str
    tool_name: ToolName
    title: str
    severity: Literal["info", "low", "medium", "high", "critical"] = "info"
    confidence: float = Field(default=0.5, ge=0.0, le=1.0)
    evidence: str = ""
    references: List[str] = Field(default_factory=list)
    created_at: str = Field(default_factory=utc_now_iso)

Read the source · schemas.py

How a timeout stays a timeout

Imagine a run reaches its deadline and a later summary describes the objective as resolved. I want that contradiction to be visible before it reaches the report. The feedback validator derives required outcomes from the recorded attempt and checks the proposed reflection against them.

  • The reflection must preserve the attempt ID and reference that exact attempt.
  • A runtime timeout constrains the allowed outcome. Incomplete attempts cannot claim verified progress.
  • Closing an open question requires a conclusive outcome and the exact question already present in the session.

Checking a reflection against its source

The validator checks identity, evidence references and the recorded outcome before accepting a reflection.

attempt_id = attempt["attempt_id"]
required = required_outcome(attempt)
if record.attempt_id != attempt_id:
    raise ValueError("Preserve the supplied attempt_id.")
if not record.evidence_refs or set(record.evidence_refs) != {f"attempt:{attempt_id}"}:
    raise ValueError("Evidence references must identify this exact supplied attempt.")
if required and record.outcome != required:
    raise ValueError(f"Recorded execution status requires outcome={required}.")

Read the source · validation.py

Reading the final report

I start with scope and the planner trace, then read the execution log beside the findings. The Markdown writer includes the decision reason, return code and available output for each run. The findings list prints severity, confidence to two decimal places and up to 300 characters of evidence.

The last sections are useful when a conclusion feels too tidy. Objective feedback preserves remaining uncertainty. Session reflection includes references for individual lessons and lists unresolved questions. I can take one of those questions back to the team with its original context intact.

Read the source · report_generator.py

Scope & practical notes

  • I traced our local AI-development working tree. The linked branch can differ from local changes.
  • The session screenshot illustrates the interface. It supplies no independent measure of detection accuracy.