Chris Hillman, global director of data science at data management company Teradata, has recently seen more attention directed at the cost of data science and AI teams, as companies look to demonstrate the value of their investments in emerging technology.
However, he believes that data scientists are capable of building AI models at a technical level and it is often business stakeholders who frustrate successful AI projects when they fail to understand how AI models work or fail to turn model recommendations into actions.
“In the data science world, everything is a technical problem and we solve it with technology,” Hillman explained. “But I firmly believe that a lot of the reason this isn’t built into business processes is basically a cultural, political or people problem, not a technical problem.”
Teradata's experience in creating models for a variety of international clients suggests:
- Business executives must understand AI to promote and achieve project success.
- Executives learn better from use case examples than from “data science 101” courses.
- Companies should conduct impact assessments before starting AI projects.
Culture, politics and people: obstacles to the success of AI projects
Hillman argues that the failure of AI projects can often be caused by business stakeholders:
- Do not trust the results of the AI model because they were not part of the process.
- Not taking the results of the model and turning them into real processes and actions.
Hillman explained that as long as the data is provided to a data science and AI team, the AI problem is not technical. Instead, it is more often a matter of difficulties with business stakeholders understanding this technology and turning AI results into business actions.
Business executives should be involved in the AI development process
As long as the data is there, Hillman’s team can successfully train, test and evaluate AI models.
“We write the output of that model somewhere and the work is done,” he said. “Production is that model that runs every month and puts something in a table somewhere.”
However, this is where it can fail.
“This fails because business owners have to be in the process,” Hillman added. “They have to take that score and decide, ‘what’s the signal?’ If I say something has a 90% probability of fraud, what does that actually mean?”
SEE: Evidence of Australian innovation in pursuit of generative AI at scale
“If the signal is to block payment and they decide to do so, someone has to do it. In many companies, that means having at least three or four teams involved: the engineers and data scientists, the business owners, and the app developers.”
This can become a dysfunctional process, where teams fail to communicate effectively, AI fails to influence business processes, and AI fails to create the desired value.
Business owners need to understand how AI models work
The rise of AI means every business executive needs to know how these models are created and how they work, Hillman said.
“They need to understand the outcome, because they need to guide the process,” he explained. “They need to ask themselves, ‘What does this mean for my customer or my business processes?’”
While technical knowledge of algorithms is not required, business executives should understand the basic mathematics involved in AI, such as the probabilistic nature of AI models. Business stakeholders should understand why the accuracy of AI models will be different than what is expected from traditional business intelligence reporting tools.
“If I went to the CFO with a report and they asked me, ‘How accurate is it?’ and I said, ‘About 78% accurate,’ I’d probably get fired,” Hillman said. “But if an AI model is 78% accurate, that’s good. When it’s over 50% accurate, you’re winning.”
“We have had clients ask us to say, ‘We want this model and we want 100% accuracy with no false positives.’ And we have to say, ‘Well, we can’t do that because it’s impossible.’ And if you get that kind of model, you’ve done something wrong.”
Use cases: Effective tools for training business executives on AI models
Hillman doesn’t believe business owners should be given “data science 101” courses, which could prove “useless” to them in practice. Instead, he said AI use cases can be leveraged to demonstrate how AI models work for entrepreneurs much more effectively.
“I think the use case approach is definitely better for people on the business side because they can relate to it and then they can get involved in the conversation,” he said.
Tips to ensure your AI project actually gets off the ground
Hillman offered several recommendations for business owners to ensure their AI projects move from idea and proof of concept to production:
Conduct an impact assessment
An impact assessment should be conducted beforehand. This assessment should include key considerations, such as why the company is pursuing the AI project and the concrete business benefits.
“I rarely see that in the original specs,” Hillman noted.
In contrast, impact evaluations are often initiated when a project is underway or after the technical work has been done, which can contribute to projects being shelved and not reaching production.
Choose the right use cases
Although transformer models were gaining popularity before ChatGPT, the hype caused by the launch of OpenAI’s chatbot led companies to launch generative AI projects to stay relevant. This has led to some use case selections that may be flawed.
SEE: 9 innovative AI use cases in Australian businesses in 2024
Hillman often asks companies if they can “do a report instead” as there are often simpler ways to achieve business goals than building an AI model. He says AI models often fail at launch due to a lack of impact assessment or because the use case was wrong.
Have a strong corporate sponsor
AI projects work best when they have a strong business sponsor driving them forward. A business leader can ensure that other teams in the company understand the potential impact of an AI project and work together to implement AI data into processes.
“IT can have the budget for technology, and someone else can have the data and security and privacy, but really, the drive always has to come from the business side,” Hillman said.
