HIPAA-Compliant LLM: The 4 Ways to Run AI on Private Data (2026)

There is no HIPAA-compliant LLM. No model is HIPAA compliant on its own, and no HIPAA certification exists for any software — any vendor selling you a "certified" model is selling vocabulary. What exists are HIPAA-compliant LLM deployments: the model plus the endpoint plus your own HIPAA work around it. The same structure holds for accounting and legal workloads where the data is privileged and cannot leave your walls — the compliance question is not which model, but who can see the prompt, what covers them, and what you log.

The good news: building a compliant deployment is a solved problem in 2026, and there are exactly four architectures that work. We operate all four for clients. Much of our consulting lives in the two where you control the hardware — open-weights models on rented dedicated GPUs and on-prem GPUs — because that is where PHI, financial ledgers, and client matters stay out of any vendor's training or retention path. This is the build guide for all four, what each demands operationally, and how to pick.

The ground rules, whatever you pick

Three constraints apply to every architecture below. Architecture picks who you trust. These pick whether the deployment is actually defensible.

• The BAA chain has to be unbroken. Every party that creates, receives, maintains, or transmits PHI on your behalf needs a signed BAA — the model provider if one exists, the cloud provider if PHI touches its infrastructure, and any broker in between. One uncovered hop makes every request through it an impermissible disclosure. The same discipline applies to accounting and legal: if privileged data transits a vendor's infrastructure, that vendor is in your trust boundary whether you call it HIPAA or not.
• The Security Rule applies to the whole path. Encryption in transit and at rest, role-based access control, and audit controls that record who prompted what, when, and what came back. Your inference logs are now compliance artifacts. So are your retention settings for them.
• Compliance stays shared. A BAA covers the vendor's side. Risk analysis, workforce training, minimum necessary policies, and configuration discipline stay on you. AWS, Microsoft, and Google all state this explicitly — believe them.

Architecture 1: a cloud model endpoint under your cloud BAA

What it is. Frontier models served as a managed service inside your cloud account: Amazon Bedrock on AWS, Azure OpenAI under Microsoft's BAA, and Google's covered Gemini services under the Google Cloud BAA. Bedrock is on AWS's HIPAA-eligible list, Azure OpenAI is in scope through the Online Services DPA, and Google publishes a covered-services list — check it by name, Google renames its AI products often.

• Who holds the BAA: your cloud provider, under the agreement you almost certainly already have. No new business associate enters your risk register.
• What it requires operationally: region pinning for data residency, CloudTrail-equivalent logging wired into your audit story, confirming no-training defaults in writing, and IAM scoping so only the PHI-approved workload can invoke the endpoint. The honest caveat: prompts process on the provider's managed service — inside your cloud boundary, not on your silicon.
• Choose it if you want frontier quality with the least new paperwork. For most healthcare teams this is the default answer. For accounting and legal, it fits when outside counsel or your cloud BAA already covers the workload and you want speed over maximum isolation.

Architecture 2: dedicated capacity or a vendor API with a BAA

What it is. Going direct to the model vendor: OpenAI's zero-retention API under its Healthcare Addendum or Anthropic's HIPAA-ready first-party API. For throughput or isolation, both clouds also sell dedicated capacity — provisioned throughput on Bedrock, provisioned deployments on Azure — that keeps the same BAA story as Architecture 1 with single-tenant performance.

• Who holds the BAA: the model vendor — a new business associate and a vendor review.
• What it requires operationally: staying strictly inside the covered configuration, which is narrower than the full product. OpenAI's BAA covers only zero-retention-eligible endpoints on an approved org, with third-party GPTs and plugins excluded. Anthropic covers a defined feature subset of the Messages API. Feature drift is the failure mode: an engineer enables an excluded capability and your covered deployment quietly stops being covered. Pin the allowed surface in code review, not in a wiki.
• Choose it if you need a vendor-specific capability the cloud endpoints do not expose and you are staffed to manage another business associate.

Architecture 3: open-weights, self-hosted in your cloud or on rented dedicated GPUs

What it is. Serving open-weights models — Qwen, DeepSeek, Mistral, Llama and similar — yourself on GPU instances you control: in your VPC on AWS, GCP, or Azure, or on rented dedicated GPUs (CoreWeave, Lambda, RunPod and similar) behind vLLM or equivalent. No model provider exists in the chain; nobody outside your organization can see a prompt. This is where most of our PHI, accounting, and legal consulting lives.

• Who holds the BAA: only your infrastructure provider, because PHI still lives on its hardware. There is no model BAA because there is no model vendor. The same holds for accounting ledgers and legal matters — no model vendor ever touches privileged data.
• What it requires operationally: you own the Security Rule story end-to-end — TLS termination, authentication in front of the endpoint, audit logging of every request and response, patching the serving framework, and capacity planning. No-training guarantees become trivial, but retention becomes your code: if your gateway logs prompts in plaintext, you built the leak. Token costs flatten into a GPU bill, and you get portability across clouds. We stand this up for clients on rented dedicated GPUs when they want control without buying hardware, and migrate to owned hardware when volume justifies it.
• Choose it if policy forbids any model vendor from seeing the data, you want cost to be flat, or you need deployment portability. For accounting and legal teams, choose it when financials or attorney-client material must never transit a vendor API.

Architecture 4: on-prem GPUs — open-weights on hardware you own

What it is. Open-weights models on hardware you own, in your data center. The air-gap tier.

• Who holds the BAA: nobody. This is the only architecture with no third party in the inference path at all — the only true no-BAA-required answer. Any cloud variant of self-hosted still requires the cloud provider's BAA because the infrastructure holding PHI is theirs.
• What it requires operationally: Architecture 3's burden plus hardware — procurement, racking, power, failover, and physical security, which the Security Rule also cares about. Model updates arrive on your schedule, a feature for change control and a tax on capability. Egress can go to zero.
• Choose it if you are a hospital system, payer, large clinic, accounting firm, or law firm with an existing data-center practice and a policy that says this data does not leave the building. Defense and public-sector teams end up here for the same structural reasons. We rack and operate this tier for clients who require it.

The decision table — how the four compare

Architecture picks who you trust. Cost shape, ops burden, and air-gap decide which trust boundary you can actually operate.

• Cloud endpoint (Bedrock / Azure OpenAI / Google): BAA with your cloud provider · prompts in managed service inside your boundary · frontier quality · low ops burden · per-token on your cloud bill · not air-gapped.
• Vendor API with BAA: BAA with the model vendor (new associate) · prompts on vendor infrastructure · frontier quality · low plus vendor management · per-token new invoice · not air-gapped.
• Open weights in your VPC or on rented dedicated GPUs: BAA with your infra provider only · prompts in your VPC or rented dedicated host · strong, a step behind frontier · high ops burden · GPU hours, flat-ish · private network, not air gap.
• On-prem GPUs: no BAA — no third party exists · prompts in your building · strong, a step behind frontier · highest ops burden · capex then power · air-gapped, egress can go to zero.

The checklist every architecture still owes

Six items exist no matter what you picked. Skip one and the architecture choice does not save you.

1. BAAs signed before the first PHI request, with every third party in the path. For accounting and legal, name the boundary even when the BAA label does not apply — who is allowed to see privileged data, in writing.
2. No-training and retention terms in writing. Defaults are not commitments — get the clause.
3. Audit logging of every prompt and response, retained per your policy and queryable when OCR, a regulator, or a client asks.
4. Access controls that enforce minimum necessary. The LLM sees what the calling user is allowed to see, not what the service account can see.
5. Data residency you can state in one sentence, region by region.
6. An incident path that includes the model endpoint. If the endpoint misbehaves, who gets paged and who gets notified?

An LLM endpoint is not the stack

Here is the trap at the end of this build. You stand up a beautifully compliant model endpoint, then someone asks the question that motivated the whole thing: can it look at our patient data — our ledger, our matters? Now a database, a semantic layer, a file store, and an agent runtime all enter the PHI and privileged-data path, and most teams bolt those on as SaaS, reopening every boundary question the endpoint just closed. The model is one component. The system around it is the unit of compliance.

We deliver the whole system inside your boundary — connectors, data layer, and agent runtime calling whichever of the four endpoints you chose — so the unit of compliance is actually compliant. Whether the model runs on Bedrock, Azure, or Google under your cloud BAA, on rented dedicated GPUs, or on your own metal, the surrounding stack stays in the same trust boundary. See where we draw the approval boundary for how the operating model enforces that.