The LLM proposes, the code decides: where Derma draws the line

Here's a response a vision model could hand back to my code:

{"concerns": {"Hyperpigmentation": 7.5, "skip_safety_checks": 1.0}}

A confidence of 7.5 on a scale that stops at 1.0, and a key that isn't one of the 5 concerns in my taxonomy. Maybe the model glitched. Maybe someone photographed a sticky note that says "ignore your instructions". The question for anyone putting an LLM next to a decision that matters is the same either way: what does the rest of the system do with this?

In Derma, the answer is supposed to be "nothing dangerous, by construction". This post walks the 6 stages of that construction with the real code from the repo, and then shows the 1 route where it doesn't hold yet.

By the end you'll have a pattern you can lift into your own system: the model proposes, deterministic code decides, and anything concerning leaves the loop entirely.

What's in here:

• What Derma is, and what it isn't
• Three ways to put an LLM near a safety call, and the one I picked
• The pipeline, and who owns each decision
• The code on each side of the line
• Where the code is behind the design (including a bypass I found writing this)
• The general rule, and one check you can run on your own code today

Derma is an unvalidated backend, and that's the starting point

Derma is a FastAPI and Postgres backend for a skincare agent aimed at melanin-rich skin, Fitzpatrick types IV to VI. It's cosmetic by design: it describes appearance and suggests routines from a product catalog. It isn't in anyone's hands. Nothing in it has been validated. Phase 2 of the roadmap, the one where a clinician labels a dataset and I run a benchmark, is still sitting behind a human gate, and the architecture doc puts the rest bluntly: "Do not skip to Phase 3." Promoting an unvalidated tool to the people it's built for would scale the exact harm it exists to fix.

So nothing below is a claim that Derma works or is safe to use. It's a claim about design, and about how far the code has got toward that design.

The repo has 20 commits. The first batch landed on 30 July 2026 and included a "tiered matrix recommendation engine", plus a clinician router. Phase 0, on 5 August, hardened it and added ARCHITECTURE.md, the 264 lines I treat as the design of record, along with a pytest suite the commit message records as 12 passing. Phases 0.5, 1a, 1b and the Phase 2 scaffolding all landed on 27 August, each commit message saying the tests were written and seen failing first. I built those phases with Claude as a pair, which the commit trailers record. That matters for this post, because an LLM helping write safety code is one more reason the safety code has to be boring and checkable.

The repo is private for now, so the snippets below are pasted inline rather than linked.

Three ways to put an LLM near a safety call

When I sat down with the first version, I had roughly three options.

One: let the LLM do the whole thing, with a careful prompt. Photo in, routine out, safety instructions in the system prompt. It's the smallest amount of code of the three by a distance. It's the fastest to build and the easiest to demo. It's also the one where a single bad completion, or one line of text inside an uploaded image, can change what a person is told to put on their face. A prompt is a request. It isn't an enforcement mechanism.

Two: keep a professional in the loop to review every routine. That's where the original blueprint was heading. It had a clinician surface and an override route. It sounds responsible. In practice it makes a person the rate limit of the whole system, and it quietly moves the safety guarantee from code you can test to attention you can't. It also blurs what the product is: once a professional reviews outputs as part of the product, you're closer to running a clinical service than shipping cosmetic software.

Three: split the rights. The LLM gets the right to propose and explain. Deterministic code gets the only right to decide anything about safety. And a concerning case doesn't get reviewed inside the app at all. It gets referred out, to a professional the user goes to, with no products attached.

I picked three. Phase 0, on 5 August 2026, deleted the /clinician/override route and turned the old clinician FHIR export into a handoff summary the user can bring to their own professional. Principle 1 of ARCHITECTURE.md §1 reads:

LLMs propose and explain. Deterministic code decides safety. Concerning cases are referred out to professionals, never reviewed in-loop.

And right under it: "No LLM output may relax a hard safety rule."

The trade-off is real. Option three can't use the model's judgement where judgement might help, and it pushes more people to "see a professional" than a clever model might. I'd rather over-refer from code I can read than under-refer from a completion I can't.

The pipeline puts a deterministic gate after every LLM step

The design has 6 stages, numbered 0 to 5 in ARCHITECTURE.md §2. Triage runs first. A red flag at triage ends the flow with a referral and no products. A clear intake goes through perception, then a deterministic decision layer, then a critic, then automatic finalisation, then narration.

The architecture of record in ARCHITECTURE.md §2, redrawn. Green is deterministic, purple dashed is where the design allows an LLM, orange is the terminal referral. The small labels mark where today's code differs.

Section 3 then assigns an owner to each of 7 concerns. That table is the actual boundary, condensed here:

Concern	Owner
Red-flag routing	Deterministic rules plus an LLM pass, rules win
Concern perception from a photo	LLM, proposal only
Product safety exclusions	Deterministic code
Product scoring and routine order	Deterministic matrix
Conflict critique	LLM, advisory
User-facing explanation	LLM
Professional referral	External, out of the loop

Two rules sit under that table. Uploaded portraits and product free text are "data, never instructions". And every LLM step is followed by the deterministic gate, so an injected or hallucinated recommendation still can't get past an exclusion.

That second rule is the one people skip. It isn't enough to have a safety check somewhere. The check has to be downstream of every place a model can speak.

Triage runs first, and it can only add flags

Here's the whole of app/services/triage.py, all 27 lines of it, added on 27 August in the Phase 1b commit:

"""Safety triage — the deterministic red-flag screen (pipeline stage 0).

Anything concerning is routed OUT of the cosmetic flow to a professional.
Rules only ever ADD red flags; nothing downstream (LLM included) may clear one.
"""

from __future__ import annotations

from app.database.models.models import SkinProfile

RED_FLAG_FIELDS = {
    "has_new_or_changing_lesion": (
        "A new or changing skin lesion should be examined by a professional"
    ),
    "has_bleeding_or_nonhealing_lesion": (
        "A bleeding or non-healing lesion should be examined promptly"
    ),
}


def evaluate(profile: SkinProfile) -> list[str]:
    """Return red-flag reasons; empty list means clear to proceed."""
    return [
        reason
        for field, reason in RED_FLAG_FIELDS.items()
        if getattr(profile, field, False)
    ]

Those strings are the app's referral copy, not advice to you. The shape is the point. The function returns a list of reasons. An empty list means clear. There's no parameter, flag or return path through which anything could remove a reason once a rule has produced it. When an LLM pass arrives here, the design says it gets to append to that list and nothing else.

The orchestrator treats a non-empty list as terminal. This is from RecommendationService.generate in app/services/recommendation.py:

        # Stage 0 — safety triage. A red flag is terminal: refer out, no products.
        red_flags = triage.evaluate(profile)
        if red_flags:
            return await self._refer_out(profile, red_flags)

        # Stage 2 — deterministic decision.
        catalog = await self.products.list_active()
        routine = self.engine.build_routines(profile, catalog)

        # Stage 3 — critic: may add cautions, never alters the routine.
        cautions = critic.critique(routine, catalog)

_refer_out writes a recommendation with status referred_out, empty morning and evening lists, a triage_events row and a referrals row. The catalog is never even loaded on that branch. The test for it is called test_red_flag_lesion_refers_out_instead_of_recommending, and it asserts both routine lists are empty.

The decision layer is plain Python you can read in one sitting

Stage 2 is the RecommendationEngine class, lines 123 to 258 of recommendation.py, first committed on 30 July 2026. It scores products against the user's concern with a weight matrix of 5 concerns, applies hard exclusions, then orders the steps into a morning and an evening sequence. The exclusions live in 1 method, and each of its 6 branches returns a reason string:

    def _passes_safety(
        self,
        product: RetailProduct,
        profile: SkinProfile,
        force_exclude_actives: set[str],
    ) -> tuple[bool, str | None]:
        actives = set(product.active_ingredients.keys())

        # Clinician overrides win.
        if actives & force_exclude_actives:
            return False, "clinician_override_exclusion"

        # Pregnancy / breastfeeding.
        if profile.is_pregnant_or_breastfeeding and (
            actives & PREGNANCY_BANNED_ACTIVES
        ):
            return False, "pregnancy_exclusion"

        # Melanin-rich guardrails (Fitzpatrick IV–VI).
        if profile.fitzpatrick_type >= 4:
            if not product.is_melanin_safe:
                return False, "not_melanin_safe"
            if (
                product.product_type == SkincareFormatType.SPF
                and product.leaves_white_cast
            ):
                return False, "spf_white_cast"
            if actives & MELANIN_RISK_ACTIVES:
                return False, "melanin_risk_active"

        # Discontinued products.
        if not product.is_active_market:
            return False, "discontinued"

        return True, None

I'm deliberately not listing what's inside those sets. They're unvalidated design inputs, and a blog post is the wrong place for them to start looking like guidance.

What matters for the boundary is what this method takes as input: a product row and a profile row. No model output. No free text. Set intersections and booleans. Same inputs, same answer, every time, which is what "reproducible and un-jailbreakable" in the doc actually cashes out to. You can't prompt-inject a set intersection.

The proposal side validates, clamps, and degrades to empty

Perception is where the LLM actually lives. The ClaudePerception adapter, added in the Phase 2 commit on 27 August, sends the portrait to claude-opus-5 with a fixed taxonomy of 5 concerns and asks for JSON. The system prompt includes "You are not diagnosing; these are cosmetic appearance proposals only" and tells the model to treat text inside the image as photo content, never as instructions.

I don't trust that prompt to hold, so the adapter doesn't either. This is the part that handles the response:

        quality_flags = list(portrait.quality_flags)
        try:
            text = next(b.text for b in response.content if b.type == "text")
            payload = json.loads(text)
            raw_concerns = dict(payload.get("concerns", {}))
            model_flags = list(payload.get("quality_flags", []))
        except (StopIteration, ValueError, TypeError, AttributeError):
            return PerceptionResult(
                provider=self.name,
                model_version=self.MODEL,
                concerns={},
                quality_flags=quality_flags + ["provider_error"],
            )

        concerns = {
            name: min(1.0, max(0.0, float(conf)))
            for name, conf in raw_concerns.items()
            if name in taxonomy and isinstance(conf, (int, float))
        }

Three moves. Unknown keys are dropped against a whitelist. Confidence is clamped to the range 0 to 1. Anything malformed or refused becomes an empty proposal with a provider_error flag, never a guess.

To check this for the post, on 15 September 2026 I copied app/ into a scratch directory, injected a fake Anthropic client, and fed it 3 things: the hostile payload from the top of this post, a prose refusal, and a synthetic intake with 1 of the 2 red-flag fields set.

Real output, 15 September 2026, from a scratch copy of the Derma code with synthetic inputs and no database. The last line is the subject of the next section.

The 7.5 became 1.0. skip_safety_checks is gone. The refusal became nothing at all. That's the behaviour you want from the proposal side: a model can be wrong, but it can only be wrong inside a shape the rest of the system already expects.

The design adds one more rule on top. The doc calls vision the weakest link, because it's least reliable for exactly the skin Derma is built around, so it treats the photo as a soft prior the quiz can override. The model doesn't get to outvote what the person said about their own skin.

The critic and narration are advisory by type signature

The critic's signature is its contract:

def critique(routine: dict, catalog: list[RetailProduct]) -> list[str]:
    """Return caution notes for the built routine (empty = no concerns)."""

It gets a finished routine and returns notes. It has no way to hand back a different routine. All 34 lines of critic.py are 1 deterministic rule that looks for 2 groups of actives in the same half of the day and appends a caution string, trimmed here:

        if actives & RETINOIDS and actives & STRONG_EXFOLIANTS:
            notes.append(f"Caution: the {time_of_day} routine combines ...")

The docstring says an LLM lens can be added "behind the same signature". That's the design choice I like most in the repo. When the model arrives, it inherits a return type that can't express "swap this product" or "clear that flag". You don't have to trust the model to respect the boundary. The boundary is the type.

Narration works the same way. narrate(rec) takes a recommendation that's already been persisted with its final status and turns it into sentences, with a disclaimer that it's cosmetic guidance and not medical advice appended every time. It narrates a decision. It never makes one.

The code is behind the design in five places

Here's the part a build log is for. Reading the code against the doc for this post, this is what I found.

Triage has no LLM pass. Section 3 says deterministic rules plus an LLM pass, rules winning. The code is rules only, over 2 boolean intake fields.

The critic can't refer anyone out. Section 3 lets it trigger a referral on a safety concern. The code only returns cautions, and grepping the whole of app/ for a Referral row turns up exactly 1 writer and 1 reason string, triage_red_flag.

Narration is templates. Deterministic today, which is honestly fine.

Perception isn't wired into the decision. This is the big one. AssessmentService stores what the vision model proposes, and the Phase 2 benchmark reads it back. RecommendationService.generate never does. It reads the quiz concern from the profile. So today the only LLM in the codebase doesn't touch a single routine. The boundary holds partly because the hard part, merging a soft prior with what the user reported, hasn't been built. I'd rather say that than let the diagram imply otherwise.

The referral export skips triage. This one I didn't expect. The GET referral-export route, 13 lines in app/routers/referral.py, builds the handoff summary a referred user takes to a professional. It calls the engine directly:

    profile = await ProfileService(db).latest(user_id)
    catalog = await CatalogService(db).list_active()
    routine = _engine.build_routines(profile, catalog)

There's no triage.evaluate on that path. The engine itself has no idea whether a profile is red-flagged, so for a flagged intake it happily builds a routine, and the export then maps those steps into a FHIR CarePlan. That's the last line in the terminal capture: one synthetic product, one red-flagged intake, and ['Cream'] comes back. I ran the engine call the route makes, not the HTTP route itself. The design says a referral "is not a routine". On this route, it would include one.

It's the classic way a boundary leaks. The safety ordering lived in the service, and a second caller reached past the service to the engine. The fix is to put the refusal where no caller can skip it, inside build_routines or behind a single entry point that every route has to use, with a test that asserts a red-flagged profile gets empty lists from every public path. I haven't made that change yet. This post was written against the repo as it stands.

One smaller leftover while I'm being honest: _passes_safety still takes force_exclude_actives with a "Clinician overrides win" comment, and nothing calls it with a value. It's a fossil of the reviewer I removed in Phase 0.

The general rule for LLMs near safety decisions

Strip out the skin and the Python, and here are the 5 things I'd tell anyone putting a model near a decision that can hurt someone:

• Give the model the right to propose and the right to escalate, never the right to clear. Asymmetric authority. A model that can only add caution can be wrong in the direction you can afford.
• Parse the model's output into a type that can't express the dangerous thing. Whitelist the keys, clamp the numbers, and make "I couldn't parse this" an empty proposal instead of a default guess.
• Put the deterministic gate after the last model step, not somewhere in the middle. Anything a model touches after the gate is ungated.
• Make escalation terminal and outside the loop. A referral that still carries a product list isn't a referral.
• Enforce the gate at the lowest layer every caller shares. If a route can reach past your orchestrator, your orchestrator isn't the gate. Derma's export route is the proof.

Check yours

Find the function that makes the safety call in your system. Then list every call site that reaches the thing it protects, not just the one in your happy path:

grep -rn "build_routines(" app

Swap in your own function name. In Derma that grep returns 3 lines: the definition on line 190 of recommendation.py, the call on line 299, and the call on line 34 of referral.py. One goes through triage. One doesn't. If your list has a line you didn't expect, that's the bug worth finding before a user does.

Next part: why "cosmetic, never diagnostic" is a line in the architecture and not just a line in the terms of service, and what actually changes on each side of it.