What are the biggest risks of LLM deployment in regulated industries?
In order of frequency in production incidents: data leakage through prompts and embeddings,
hallucinated answers presented as authoritative, prompt injection from untrusted document content,
unauthorized actions when LLMs are given tool access, and silent drift in model behavior across vendor updates.
Each requires a specific control rather than a single generic mitigation.
How do you prevent hallucinations in production AI assistants?
Hallucinations cannot be eliminated but they can be constrained. The effective controls in order of impact are:
grounding every answer in retrieved sources via RAG, requiring source citations in the response schema,
rejecting or escalating low-confidence answers, narrowing the assistant scope so that out-of-domain questions
are routed to a fallback, and adding a verification step (rule-based or second-model) for high-stakes outputs.
Temperature tuning and prompt engineering help marginally; architecture changes help structurally.
When does an AI deployment cross the EU AI Act high-risk threshold?
The Act lists eight categories where AI is high-risk by definition: biometrics, critical infrastructure, education,
employment and worker management, access to essential services, law enforcement, migration and border control,
and administration of justice. Any AI system that materially influences a decision in those domains triggers
high-risk obligations: risk management system, data governance, technical documentation, logging, human oversight,
accuracy and robustness, and post-market monitoring. Most enterprise AI assistants are not high-risk;
AI used in hiring, credit, insurance underwriting or medical triage typically is.