Explainable Artificial Intelligence

Explainable artificial intelligence (XAI) refers to methods that help people understand why an AI system produced a particular output. In consequential automated decisions, such as those concerning credit, employment, healthcare or public benefits, explanations connect the decision to relevant inputs, rules or patterns. They enable scrutiny without necessarily revealing every internal computation.

How explanations are produced

Some models are intrinsically interpretable, because their decision rules or relationships can be directly examined. Complex models often require post-hoc explanations, which describe their behaviour after training.

• A global explanation describes how the system generally reaches decisions, including influential variables and broad decision patterns.
• A local explanation addresses one particular outcome, such as why an application was rejected.
• Feature importance, simplified representations and counterfactual examples can show which factors mattered or what feasible change could alter an outcome.

Requirements of a meaningful explanation

The NIST Four Principles of Explainable AI state that a system should provide an explanation, make it meaningful to the intended user, ensure that it accurately reflects the process, and recognise its knowledge limits.

• The level and form of explanation should suit the audience, because a developer, regulator and affected individual require different information.
• Explanation accuracy is distinct from predictive accuracy: a model may predict well while its stated explanation poorly represents how it reached the result.
• Uncertainty, unsupported cases and conditions outside the system's intended scope should be communicated.

Role in responsible automated decision-making

XAI supports transparency, auditing, error detection and accountability. For high-impact uses, it should be combined with documentation, testing, human oversight and accessible review or grievance mechanisms.

• An explanation can help an affected person understand and contest a decision or correct inaccurate input data.
• Explainability does not itself guarantee fairness, privacy, safety or freedom from discrimination.
• Post-hoc explanations may be incomplete or misleading, so they must be tested for fidelity to the actual model.

How UPSC asks this