AI and Patient Data: What HIPAA Actually Requires

AI is being used in most healthcare practices, whether anyone signed off on it or not. A front desk employee drafting a patient message with ChatGPT. An assistant asking AI to find an open appointment slot. A practice owner testing a tool a vendor pitched last week. The tools arrive faster than any policy does, and in a lot of practices, there was never a policy to begin with.

That's a data governance problem even in places that would never use that phrase. Data governance is the set of rules an organization uses to control who can see data, where it can go, and what can be done with it. At a large enterprise, that usually lives with a dedicated data or IT team and a policy document, that honestly, nobody rereads. Most of the healthcare, aesthetics, and wellness practices we work with don't have that team, or that document. They're adopting AI because the benefits and pressure to be more efficient are real, however, the infrastructure to govern it was never ever built.

AI raises what's at stake regardless of whether a formal policy exists. AI is used to read data and answer questions and, increasingly, take action on what it finds: sending a reminder, flagging a schedule gap, drafting a message to a patient. Governance has to cover not just who can see the data, but what an AI system is allowed to do with it once it does.

In healthcare, that shift lands differently than it does anywhere else. Protected health information isn't a column you tag and wall off. It's the clinical note, the claim, the appointment history, the message a patient sent through the portal. Most of what a provider organization collects is PHI. So “govern the sensitive data” isn't a policy here, it's the entire system.

This piece is meant to help any healthcare or provider-based-care organization figure out what actually has to be true for AI to work safely and legally around PHI. We'll cover:

• What HIPAA requires
• What a Business Associate Agreement (BAA) is and why you need it when using AI
• How minimum necessary access is enforced with AI and in the database
• Vetting criteria you should have in mind when evaluating tools and vendors

What HIPAA requires

Three things HIPAA requires on top of general data governance, once AI is involved, and what each one actually means in practice.

1. A signed Business Associate Agreement (BAA) before any AI vendor touches PHI. A BAA is a contract required under HIPAA between a healthcare organization and any vendor that creates, receives, stores, or transmits PHI on its behalf. That applies to AI specifically: whether the tool is answering a question, drafting a message, or taking an action, it needs a signed BAA in place first, not a general privacy policy, and not a promise made on a sales call. Current guidance is specific about this too: the agreement has to state whether a vendor uses a customer's PHI to improve a model that later serves someone else's customers. CorralData's answer is in writing either way: HIPAA compliance with a BAA is included on every plan, at no extra cost, and PHI is never stored or used to train models.
2. Minimum necessary access, enforced at the AI layer, not just the database. HIPAA's minimum necessary standard says a person, or a system, should only access the PHI required for the task in front of it. An AI agent answering “which appointment slots are open next week” shouldn't have first read access to a patient's behavioral health notes on the way to figuring out a scheduling gap, even if the underlying database technically allows it. That scoping has to happen at the AI layer itself, it can't be assumed from broader database permissions.
3. An audit trail that separates what an AI recommended from what it executed. A recommendation a human reviewed and approved is a different event from an action an AI carried out on its own, and an organization has to be able to show that difference on request. Logging both the same way makes it harder to demonstrate, after the fact, exactly what a person decided versus what the AI did independently.

None of this is theoretical. Risk analysis failures, the kind that show up when no one has mapped where PHI actually flows before implementing a new tool, account for roughly 70% of the HIPAA settlements the U.S. Department of Health and Human Services (HHS) has announced over the past two years, more than any other single cause. An AI agent connected to scheduling, billing, and care records at once, without a clear map of who's allowed to see what, runs into that same failure mode, just faster.

Where horizontal tools fall short

These requirements don't automatically hold for most AI tools on the market today. That's because most of them are horizontal tools: built to work for retailers, law firms, marketing teams, and healthcare practices alike, not for any one of them specifically. A tool built for the widest possible range of businesses usually ships without a BAA or a PHI-aware governance layer, because most of its customers never touch PHI in the first place, and there was never a reason to build for it.

Generic AI copilots and general-purpose agent frameworks, Claude connected directly to an internal database, a BI copilot pointed at a warehouse, fall into that category. That's exactly why healthcare organizations need tools built specifically for regulated, PHI-heavy environments, not a horizontal platform retrofitted after the fact. What's actually at stake when that gap isn't closed: a scheduling agent or a note-summarizing copilot with unscoped access to PHI is a data breach waiting to happen. Once PHI is exposed to the wrong person or the wrong system, the organization is on the clock for HIPAA's 60-day breach notification requirement, and an OCR investigation into how it happened typically follows.

There isn't yet a landmark case of an AI agent specifically causing that kind of exposure. But the absence of a headline case isn't the same as the absence of risk. Doctors are already pasting patient notes into ChatGPT to draft correspondence and consolidate notes, and every time that happens, the data leaves for a server with no BAA behind it, which is, technically, a reportable breach. Enforcement catches up to new technology slowly. Being the first documented case is an expensive way to find out it caught up.

Row-level security has to bind the AI agent, not just the person

If a front desk employee's access to reporting only shows patients at their own location, an AI agent acting on that employee's behalf needs the identical scope. Not a broader one because it was easier to build that way, not an exception because the agent “needed more context to answer well.” An agent that can see across locations or across patients because nobody scoped it as tightly as the human it's standing in for isn't a hypothetical risk. It's a violation that hasn't happened yet.

Let's use an example: a regional director asking an AI analytics agent why rebooking is down at the west side location should get a response where the AI can see every location's rebooking data, because that's what the director's role allows. A front desk lead at that same location, asking the identical question through the identical platform, should get an answer for that one location's numbers only. Same question, same interface, different access, because the person asking is different. That distinction has to be enforced at the agent layer, every time, not assumed because it holds true for the dashboard sitting next to it.

This is what CorralData's own row-level security is built to do. When it's turned on, AskCorral AI is scoped by the same row-level security as the dashboard sitting next to it, and that boundary extends to CorralData's MCP layer too, which is what connects Claude or ChatGPT to a customer's data, managed by CorralData. It's worth asking any vendor, including us, whether that access control is standard or something you have to request. More on that below.

Questions to ask any AI analytics vendor

Whether you're evaluating CorralData or anyone else, these are worth asking before AI touches PHI. A vendor that can't answer them plainly is a red flag.

1. Will you sign a BAA, and does it cover the whole product? A passing answer names exactly what's covered. CorralData includes a BAA on every plan at no extra cost, applying across the platform rather than a subset of features.
2. Does minimum necessary access apply to your AI, or only to database logins? An agent should only be able to see what it needs for the question in front of it. Ask for a specific example of how that's enforced, not just a policy statement.
3. Is row-level security standard, or something we have to ask for and configure separately? This one matters more than it sounds like it should. At CorralData, row-level security is available and, once it's turned on, applies to AI access the same way it applies to a dashboard, including AskCorral and any AI connected through our MCP layer. It isn't on by default. If a vendor implies otherwise, or can't say what an AI agent can see without it, that's worth a second question.
4. If we connect our own AI, Claude, ChatGPT, or something else, to your platform, does your BAA extend to that AI provider too? Usually not, and CorralData is no exception. Our BAA protects PHI inside CorralData and AskCorral. Connecting your own AI through our MCP means you need your own BAA with that provider if PHI is going to be part of the conversation. More on this in the FAQ below.
5. Can you show an audit trail that separates what your AI recommended from what it executed? A single combined log isn't enough once an agent can act on its own. Ask to see the distinction, not just hear that it exists.

A vendor should be able to answer all five without hedging. That's the actual due diligence, not the sales page.

You can't govern AI without governing PHI

At CorralData, we believe in making data easy. That starts with the unglamorous work: ingesting data from all your source systems, cleaning it, building data models, and providing AI agents that can provide trustworthy answers to business questions.

In a regulated environment, easy can't mean ungoverned. For CorralData specifically, that means HIPAA-aligned safeguards are standard on every account, a BAA included on every plan at no extra cost, PHI breach notification inside 10 business days if something ever goes wrong, and row-level security available to extend that same protection to Claude and ChatGPT when they're connected through CorralData's own MCP layer. That's what governing PHI actually looks like in practice, not a policy binder, a property of how the system was built for healthcare and wellness practices.

We've covered what AI and data governance actually require around PHI here, and that's mostly a question of access and security: who's allowed to see PHI, what an AI agent can do with it, and how you prove that after the fact. Getting AI to answer with numbers you can actually trust is a separate problem, one of infrastructure rather than security. If you're interested in learning more about how CorralData tackles that, check out the related articles below, we cover how we go from raw data to trusted answers.