Gartner Predicts Explainable AI Will Drive LLM Observability Investments to 50% for Secure GenAI Deployment by 2028

31 March 2026 — Gartner, Inc., a business and technology insights company, predicts that the growing importance of explainable AI (XAI) will drive large language model (LLM) observability investments to 50% of GenAI deployments by 2028, , up from 15% today.

Gartner defines XAI as a set of capabilities that describes a model, highlights its strengths and weaknesses, predicts its likely behaviour and identifies any potential biases. It can clarify a model's functioning to a specific audience to enable accuracy, fairness, accountability, stability and transparency in algorithmic decision making.

LLM observability solutions monitor, analyse and provide actionable insights into the behaviour and performance of LLMs. They go beyond standard IT measurements, such as response times to look at specific LLM metrics such as hallucinations, bias and token utilisation. These tools are used by teams that develop and operationalise AI systems, and increasingly by IT operations and SREs responsible for the performance and resilience of these systems in production.

"As enterprises scale GenAI, the trust requirement grows faster than the technology itself," said Pankaj Prasad, Sr Principal Analyst at Gartner. "XAI provides visibility into why a model responded a certain way, while LLM observability validates how that response was generated and whether it can be relied on.

"Without robust XAI and observability foundations, GenAI initiatives will be restricted to low risk, internal, or noncritical tasks where output verification is easily managed or inconsequential, severely limiting the potential return on investment."

Growing Need for XAI and LLM Observability as Mandatory Trust Mechanisms
Gartner forecasts the global GenAI models market will exceed $25 billion in 2026 and reach $75 billion by 2029, driven by rapid adoption across industries. As usage increases, so does the need for mechanisms that verify AI-generated content and protect against hallucinations, factual inaccuracies and biased reasoning.

"Traditional observability is focused on speed and cost, but the priority is now moving toward deeper quality measures such as factual accuracy, logical correctness and sycophancy. This shift requires new governance-focused metrics and evaluation methods, such as human-in-the-loop validation of the generated content's narrative and citation accuracy," said Prasad.

"Explainability turns a GenAI output into a defensible, auditable insight. LLM observability ensures the model behaves as expected over time. Without both, GenAI cannot mature beyond controlled lab environments."

To improve the reliability, transparency and business value of GenAI use cases, organisations should prioritise the following steps:

• XAI Tracing for High Impact Use Cases: Mandate verifiable XAI tracing for all high impact GenAI use cases to document the model's reasoning steps and the source data behind each output.
• Multidimensional LLM Observability: Prioritise observability platforms that monitor latency, drift, token usage and cost, error rates, and output‑quality metrics to ensure reliable GenAI performance.
• Continuous LLM Evaluation in CI/CD Pipelines: Integrate LLM evaluation metrics, including factual‑accuracy benchmarks and safety checks, into continuous integration (CI)/continuous delivery (CD) pipelines for continuous validation before deployment.
• Stakeholder Education on Explainability Requirements: Educate legal, compliance, and other key stakeholders on explainability requirements to ensure alignment on risk, governance expectations, and implementation challenges.

Gartner clients can read more in Predicts: AI will unlock Observability at Scale.

About Gartner IT Infrastructure, Operations & Cloud Strategies Conference
Gartner analysts will provide additional insights on cloud strategies and infrastructure and operations trends at the Gartner IT Infrastructure, Operations & Cloud Strategies Conferences taking place May 11-12 in Sydney, June 1-2 in Mumbai, November 16-17 in London, December 1-3 in Tokyo and December 8-10 in Las Vegas. Follow news and updates from these conferences on X and LinkedIn using #GartnerIO.

About Gartner
Gartner (NYSE: IT) delivers actionable, objective business and technology insights that drive smarter decisions and stronger performance on an organisation's mission-critical priorities. To learn more visit gartner.com.