When AI gets it wrong: why governance can no longer be an afterthought

By Amritesh Anand, Vice President & MD – Technology Services Group at In2IT Technologies

Artificial Intelligence (AI) has quickly shifted from a futuristic idea to a useful business tool. Across South Africa, organisations are using AI to create content, analyse data, automate tasks, and improve decision-making. At the same time, the government is starting to look into how to regulate and manage this technology.

Recent events have made this discussion more urgent. South Africa’s decision to withdraw its draft National AI Policy Framework raised concerns about inaccurate, possibly AI-generated references. This situation has highlighted a larger issue for organisations everywhere: while AI can be incredibly powerful, it is not perfect.

The real question now is not if AI should be used, but how it should be managed. For leaders, this means taking practical steps such as developing clear policies for AI use, establishing oversight processes, and implementing robust risk assessment procedures. By focusing on these key management actions, organisations can harness the benefits of AI while ensuring responsible and safe option.

The confidence problem nobody talks about

One of the most concerning traits of generative AI is its ability to present false information with complete confidence.

Unlike traditional software, which usually produces predictable outputs, large language models generate responses based on probabilities. This means they can create information that sounds credible but is actually wrong. The industry calls these errors “hallucinations,” but this term can make the problem seem less serious than it really is.

The consequences can be significant. Imagine a financial services firm using AI to draft regulatory submissions. A healthcare provider depends on AI-generated research summaries for decision-making. A legal department relying on AI to prepare policy documents. In each case, inaccurate references, made-up statistics, or incorrect interpretations could lead to compliance risks, reputational damage, and costly operational mistakes.

The incident involving the South African policy shows that even important documents are vulnerable to these mistakes.

Why human oversight remains non-negotiable

As organisations adopt AI, many find that the gains in efficiency can create a false sense of security. When an AI-generated report looks professional, contains citations, and follows a logical format, it is easy to assume the information has been carefully checked. However, a polished appearance should never be confused with accuracy.

This is why human oversight is one of the most important principles in responsible AI use. Human reviewers offer something that AI cannot: contextual judgment. They can evaluate whether information makes sense in a specific business context, identify inconsistencies, validate sources, and question dubious outputs.

Consider a procurement team that uses AI to assess supplier proposals. The technology may efficiently summarise hundreds of pages in minutes, but a human expert is still needed to verify conclusions, evaluate risks, and ensure important details are not overlooked. Instead of replacing human expertise, effective AI use should enhance it.

Governance is moving from IT issue to boardroom priority

For many organisations, AI governance has traditionally been seen as a technology issue. However, it is increasingly becoming a business governance issue.

Boards, executives, and risk committees are beginning to understand that AI introduces new organisational risks. These include concerns about data privacy, intellectual property challenges, algorithmic bias, cybersecurity issues, and compliance responsibilities.

As regulators worldwide continue to develop AI frameworks, businesses will face rising expectations regarding transparency, accountability, and responsible use. The organisations that thrive will be those that set up governance structures before regulations mandate them.

This means creating clear policies for AI use, identifying approved use cases, documenting decision-making processes, and establishing accountability for AI-generated outputs. A robust AI use policy should address several key elements: data privacy and protection requirements, human review and approval protocols for AI-generated outputs, escalation procedures for identifying and responding to questionable or high-risk content, record-keeping standards, user training, and regular oversight to update the policy as technology and regulations evolve. Just as cybersecurity transformed from a technical concern into a strategic business priority, AI governance is following a similar trajectory.

Preparing for regulation before it arrives

While the future direction of South Africa’s AI regulatory landscape remains unclear, one thing is certain: greater oversight is coming. Proactive organisations are already taking steps to get ready. Preparation starts with understanding how AI is currently used in the organisation. Many businesses are surprised to learn how widely employees already use AI tools for tasks such as content creation, research, analysis, and operations.

From there, organisations can establish governance frameworks that define acceptable use, verification requirements, and risk management processes. Importantly, businesses should ensure that AI-generated content is properly reviewed before use in customer communications, compliance documents, strategic planning, or public-facing materials. One way to balance oversight and efficiency is to use tiered review processes, where routine or low-risk outputs are checked through quick workflows or assisted by automated validation tools, while high-value or high-risk content is escalated for in-depth expert review. Tools such as AI output detectors, workflow automation platforms, or collaborative document systems can help streamline this process, flagging potential errors for human attention and reducing manual workload. The aim is not to slow down innovation, but to ensure it happens responsibly.

Turning AI into a trusted business asset

The lesson from South Africa’s recent AI policy issue extends beyond government. AI is becoming an integral part of how organisations operate, communicate, and make decisions. Yet its value relies entirely on the quality of the governance around it.

Businesses that adopt AI without proper controls risk avoidable errors and compliance issues. Those that combine AI innovation with solid governance, human oversight, and clear accountability will be better equipped to enjoy its benefits while managing its risks.

Experienced IT governance and consulting partners can help organisations set AI policies, implement safeguards, audit AI-generated content, and establish human-in-the-loop verification processes so businesses can use AI confidently and responsibly.

The discussion about AI is no longer just about its technological capabilities. It increasingly centres on trust. In an era where a single AI-generated mistake can shake confidence, trust might prove to be the most valuable asset of all. The organisations that pair innovation with strong governance will be best positioned to earn and keep it.