Security by Design for Generative AI Applications

A conventional upload only has to be validated as a file: is it the type it claims to be, is it a safe size, does it contain anything malicious. A generative AI feature adds a second layer of scrutiny, because the same file also becomes model input, and model input can be adversarial in ways a file scanner will never catch — an image or a piece of text engineered to manipulate what the model does with it. Treating a user upload as safe once it passes a conventional file check is the mistake I have seen cost the most: the file check and the model-input check are different problems, and skipping the second because the first passed is how a generative feature ends up doing something nobody intended.

It is tempting to treat a model’s output as a finished artifact once it is produced — store it, display it, pass it downstream — the same way an application trusts a value it read from its own database. A generated output deserves none of that trust by default. It should be validated against the constraints the feature actually needs — is this a plausible result, does it stay within the boundaries the product promised — before it reaches storage or a user, and it should be treated as untrusted content if it is ever rendered back into an interface, with the same caution given to any other unverified user-facing content.

A well-designed feature gives a user a narrow, well-understood way to influence a model — a description, a style choice, an uploaded reference image. It does not give the user’s input a path to influence the system beyond that: which internal instructions the model follows, what data outside the current request it can draw on, what it is permitted to do next. Every generative feature should be able to answer plainly what part of the system a user’s input can touch, and everything outside that boundary should behave as if the input did not exist.

A guardrail I have come to rely on, learned first in an applied-AI product and later in a much more consequence-heavy compliance context: the model proposes, and where a decision genuinely matters, a human still disposes. In a consumer image product that might mean a clear boundary that generated output is inspiration, not a professional recommendation, communicated honestly rather than implied away. In a higher-stakes setting it might mean specific categories of judgment a model is never allowed to finalize on its own.

The design question worth asking of every generative feature is the same either way: if the model is confidently wrong here, what stands between that output and a consequence a user or the organization has to live with? If the answer is nothing, the feature is not finished.