Triathlon is a multi-stage competition that pushes the competitors’ endurance to their limits. Only few can make it but if you do, you can not only brag the fact you’ve done it, but you also know that you are able to stretch yourself. The same scenario is true for AI according to Tuomas Heroja and Olli Lindroos from Solita, a Finnish digital business consultancy and digital services company.
AGILE RACE TO UNKNOWN
So much about AI is just hype right now – but I believe at the grass-roots level, building such AI intensive solutions still requires plenty of raw human work. How would you summarise what it is really like to develop AI intensive solutions?
Developing AI (Artificial Intelligence) tests the whole organisation’s data drivability. It is like sports. You can’t just decide to start to do AI just like you simply start a triathlon. Instead it’s a kind of endgame for the highly dedicated people to test their data drivability. Understanding the end user needs, business value proposition and available data is essential in successful AI deployment. Agile AI Triathlon is a three-phase, almost never-ending race. It starts from the business case definition and ends when the whole business case becomes irrelevant.
The AI development journey is basically about growing understanding. Typically, it starts from understanding the problem at hand. After the business case is clear, the team needs to start to understand the solution’s end-users. When the team is familiar with the business case and the end users, the team then needs to understand the environment where the end-users and their solution operates in.
This journey can be divided into three phases. Like a triathlon consists of swimming, cycling and running, AI development can be divided into discovering, prototyping and fine-tuning. In an agile triathlon, the order of these phases is not a waterfall-like series of individual activities but instead it’s about iterating the solution step by step towards the goal and using the best proceeding method for the next obstacle. Every crossed obstacle grows the team’s understanding of the challenge at hand.
SWIMMING – DISCOVER
Before starting to actually build an AI solution, data scientists need to get to grip with huge amounts of time-series data etc. Is there any specific process for how to get started? It´s a kind of scientific riddle where the needle from the haystack needs to be found. How can you get through it without getting lost?
Discovery is about a deep dive into the data! Notice: No AI is developed at this phase! In order to start solving the selected business case, the team needs to know what they should try to discover next. Understanding the data and the business it’s related to is essential. If the development team doesn’t understand the data and the business environment inside out, there is no way that the AI can figure it out. Or at least no one ever finds out if and how it did.
1. Plan: In a workshop, create a plan where you are heading towards, formulate a question and make a hypothesis of a solution.
2. Do: With the help of a data scientist, the whole team takes a deep breath and dives into the question at hand. Test the planned hypothesis, gather information and form observations from the domain-specific business rules and discover how different algorithms fit to your findings. Summarise and communicate the results with the team and the end users interviewed.
3. Check: Was the selected hypothesis right? Are you swimming to the right direction? Do the findings answer the planned business challenge? What did you learn about the business case, the end-users and the business environment?
4. Act: Consider whether you are ready to start prototyping or whether do you still need to do some additional discovery activities? If you are ready to start planning the next potential prototype, move on to cycling. If not, continue the discovery process.
CYCLING – PROTOTYPE
Tailoring algorithms to the data driven problem – how do you check that actual perceived end users believe the predictions delivered by AI? An AI solution without active users due to, for example, lack of human trust in the predictions, would be a waste of money after all.
Full speed ahead! Thanks to the swimming phase,the team has an idea of a prototype for testing. It’s time to expose your idea to real end users. With quick-phased test-and-trial prototyping, it is possible to test different kinds of AI solutions in the real business environment.
1. Plan: Plan what data should be used to train an AI and what kind of answers it should give and in what format. Book interviews to test the prototype with the potential end users.
2. Do: Deliver a prototype with the whole development team as fast and as cheaply as possible. Interview and test the potential solution with end users and gather the results. Use paper sketches and calculate results manually. Do whatever you can to make it as fast as possible because most probably you’ll re-do it again many times.
3. Check: Did the expected results match real life? Do you need to re-do everything? Did your end users like the results and how could they be improved?
4. Act: Luckily re-work is easy if needed since you didn’t spend too much time prototyping a single solution. Based on the results, get back to planning a prototype-phase or deep dive back to swimming to gain a better understanding based on what you learned. If your prototype was a hit, move to running!
RUNNING – FINE-TUNE
Providing that during the previous phase end users explicitly showed a leap of trust on the predictions delivered by prototypes and there is mutual scientific evidence about algorithm suitability to the defined problem, how do you actually build the AI solution and embed it into easy-to-use user interfaces?
After cycling you probably have many battles behind you and with stamina it’s time to start integrating the existing, already loved prototype into the business environment.
1. Plan: How the existing prototype should be integrated to everyday business so that people will love to use it. Focus on integrating the solution to its surroundings and create a framework for the AI model where it runs automatic learning cycles which in turn leads to better prediction accuracies as time goes by.
2. Do: Automate the solution as much as possible, e.g. do some test data validations to ensure that suspected corrupted input data gets flagged and diverted from the self-learning cycle. Addressing the above “administration” type topics in AI code usu ally requires higher volumes of code than developing the actual prediction models.
3. Check: After running, check your position. Is the solution stable enough and how did it integrate into the surrounding business environment? Does the solution impact the end users’ work in a positive way?
4. Act: Based on findings do the needed actions. Get back to data discovering, prototype development or do even more fine-tuning. Is the solution ready now or could improving it bring more benefits than investment? Find where you are and what you’ve learned during the race. If you consider yourself ready, take a deep breath. Very few people take only one triathlon.
When you start to develop AI, you don’t know what will come up and what will be the actual landing point. Luckily with the agile delivery model you don’t have to know everything beforehand. Developing AI has similar rules to developing software: You have to decide which one is scaled: cost, time, quality or scope. Instead of traditional fixed time and scope, you can raise quality at the heart of the development work. It means that if and when something does not go to plan, either scope or resources have to stretch.
Based on experience, the effort split between different triathlon phases is usually around 30%-30%-40%. But the split can vary heavily. In some cases, a huge amount of data discovering is needed and for some solutions, integration into the existing business environment is very demanding. The only thing common with all the projects that we have completed is that there is always a unique path to the end solution. The AI development path is individual.
You’ve seen countless AI initiatives over the years. There must be some common challenges and remedies that are familiar to most AI initiatives. Can you list those?
Individuality doesn’t mean that every project equals to a different kind of chaos. Based on our experience we’ve identified some common attributes that can be found in all our cases.
TOP 5 CHALLENGES ON DEVELOPING AI
1. The solved case needs to be considered carefully – Focus on how to formulate the challenge: Most development projects go wrong in the very beginning. If the solved challenge is formulated poorly, the solution can’t be any better than that.
2. It is hard to find know-how – Technical and Domain understanding: Finding the right people to develop the AI can be challenging. It is hard to form a development team that can gain deep understanding about the business domain, the data itself and the potential end users.
3. Unpredictable benefits: Like in any research work, in AI development there is no guarantee of any results. Results might be what you expected or something you don’t like at all. Do you have the guts to walk the walk even when you might not like it?
4. Hard-to-estimate quality, scope & resources: Developing AI requires high quality and artificial help is useless if it does the wrong things. If the data discovering and prototyping phases are prolonged, you end up fine tuning a solution that no-one wants to use.
5. The solution lifecycle is hard to estimate: After the AI is ready and deployed to everyday business use, it is very fragile to business environment change.
What do you suggest is the winning formula?
From our experience the winning formula has five components:
1. Focus on formulating the challenge that needs to be solved: Describe the business case and hypothesis of the results so clearly that everyone in the team can stand behind them.
2. Right people with time to focus: The key point of bonding the right people to the project is to motivate them and give them
time to help the development team. An overly busy team or “we know what end users want” -attitude will not help in gaining the best possible understanding of the case.
3. Expectation management: From the outset, expectation management is one of the key parts in developing an AI project. The hype around the topic is so hot that expectations can be overwhelming even before any development has been done. AI development happens one step at a time, and no one knows where the path ends. The development team needs the mandate and the resources to do the necessary discovery and to test their findings, even when the results sound unpredicted.
4. Develop with agile methods: With agile development methods e.g. Scrum, you can lock in quality and resources based on short range best estimate. This approach works well with AI development in order to keep quality at the demanded level.
5. Continuous development: AI must adapt to Business environment changes. To keep the trained model valid, it must be trained whenever something changes. Some additional development work might therefore be necessary.
- Understanding the end user needs, the business value proposition and the available data are essential in a successful AI
- The AI journey is about iterating the solution step-by-step towards the goal. Every crossed obstacle grows the team’s understanding about the challenge at hand.
- Like in any research work, in AI development there is no guarantee of any results. The results might be what you expected or something you don’t like at all.
The author is Jarkko Pukkila.