More than data: How to understand and measure AI products

Reading Time: 7 minutes

Artificial intelligence (AI) is rapidly gaining ground in more than just the tech industry. From healthcare to pharmaceuticals, business, and beyond, more and more companies are exploring ways to develop products that leverage AI technology to solve consumer problems. When done successfully, AI product management has the potential to reimagine consumer needs, reshape marketplaces, and revolutionize entire industries. Whether it’s virtual assistants like Alexa, Tesla’s self-driving cars, or just a better queue of movies to choose from on Netflix, the possibilities of AI are endless.

But first, what is an AI product?

AI products are products designed intentionally to simulate human intelligence. By using machine learning and deep learning, AI products endeavor to use not just data – but the correct data – to create accurate predictions. For this reason, decision-makers across a wide-span of industries are looking for ways to utilize this technology to create transformative products.

What is AI products management?

For entrepreneurs, entering the world of AI can be enticing, but entry barriers can be precarious if navigated incorrectly. Before discussing how to embark on a journey to AI product management effectively, it’s crucial to understand how AI works.

When speaking about AI technology, there are two core pillars that make-up artificial intelligence. Machine learning uses algorithms trained with data sets to provide forecasts, while deep learning weighs inputs to train AI to predict outputs. These two subsets work together, enabling software to make human-like decisions.

However, the design of AI products involves more than just parsing data. For data scientists, data engineers, and agile product managers, the challenge is to create an intuitive environment. Often, this requires teams to recognize percept sequences in a product’s environment and perform defined actions based on those sequences. For example, a defined action might be to reduce speed to ensure a safe distance from other vehicles in slow-moving traffic for a self-driving car. Doing this requires the AI product manager to define the mapping from percept to action. Or, put more simply, consider all possible scenarios to ensure that a product performs the appropriate action based on a sequence of events in its environment.

Who manages AI products? 

In the world of AI technology, it’s normal for job titles and responsibilities to change with the wind. Although AI Product Manager is a common title for those working in AI product management, the skills required to develop an AI product successfully go far beyond what the title suggests. Agile product managers, data scientists, and agile product managers are all essential to AI product management. 

Agile product manager

As the name suggests, an agile product manager supports an agile project environment. The agile methodology relies on four main principles:

• Supporting individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

For agile product managers, this means managing an adaptive environment that enables teams to pivot approaches to product planning and implementation in response to user feedback.

Data scientist

In cross-functional AI product teams, data scientists are necessary for analyzing company data to derive application scenarios. The data scientist is responsible for designing algorithms and building models that create solutions for business problems. A successful data scientist needs to have a strong background in statistics, math, or another quantitative field.

Data engineer

Contrary to a data scientist, a data engineer manages the operational systems to store, process, and transfer data. Among these duties includes setting up and operating required data processing architecture. The data pipelines created by data engineers are critical to support machine learning model training.

Designing AI products

Creating a successful AI product is more than assembling the right team or securing investments; it has to be strategic in its intention. The primary goal should be to use AI technology to solve a customer problem, whether marketing the product for B2C or B2B consumers. Although it’s easy to recognize consumer applications of AI technology, B2B products account for nine out of every 10 AI products developed. 

How to formulate a hypothesis

Before formulating a hypothesis, it’s necessary to ask four essential questions:

1. For which industry is the AI product trying to solve a problem?
2. What specific problem does that industry face that needs to be solved?
3. How can AI be utilized to solve the problem?
4. What evidence is required to know that the problem is solved?

After thoughtful consideration, formulate a hypothesis that seeks to answer these questions. Remember, it’s crucial to design a hypothesis to test a theory that can be validated and proven using data.  

Here’s an example of how a hypothesis can be formulated and applied in AI development:

1. Define the null hypothesis. For example, machine learning algorithms’ usage improves the relevance of personalized recommendations for logged in users.
2. Determine the metrics. Precisely, how can “relevance” be quantified?
3. Define the test scope. How many test results are necessary to meet the significance level?
4. Define the threshold to accept or reject the hypothesis.
5. Perform samples and tests.
6. Accept or reject the hypothesis.

Once a hypothesis is accepted, measuring the algorithm for accuracy will determine whether the product has legs.

Why measure? 

Measurement is one of the most important aspects of AI product management. Without measuring results, there’s no certainty of market success. AI products are only as strong as their algorithm, which is why it’s vital to dial-in an algorithm to create accurate forecasts and make informed decisions. 

After why comes how

Operational KPIs

Key Performance Indicators (KPIs) are metrics used to understand how a product or service performs against predetermined business objectives. Because AI products tend to rely heavily on investment, operational KPIs are useful for measuring product performance and using data to determine product roadmaps.

Common operational product KPIs that measure AI compatibility include cost per acquisition, conversion rate, retention rate, new sessions, and the number of trial sign-ups. One might hypothesize that the higher a product’s AI compatibility, the greater its influence on operational KPIs. 

For example, a consumer who purchases a Google Home Device is likely to download Google Assistant, lending to the idea that the products’ high AI capabilities directly affect sales figures or the number of active users.

On the contrary, an e-mail client that uses machine learning to classify incoming e-mails in categories like “work,” “private,” “social,” or “marketing” has relatively low AI capabilities. Meaning that the operational KPIs are unlikely to be impacted by AI features alone.

AI products and machine learning proxy metrics

Although operational KPIs are great for marketers, they don’t measure the algorithm’s accuracy. Instead, AI proxy metrics such as objective function, classification accuracy, mean absolute errors (MAE), root mean square error (RMSE), and sensibleness and specificity average (SSA) are better suited to quantify the accuracy of AI product algorithms.

Here’s why:

Objective function

An objective function is a mathematical formula that needs to be minimized or maximized to solve a specific problem using AI. 

For example, if the algorithm’s goal is to optimize a travel connection, minimizing the objective function is necessary to achieve a short travel time. However, if the objective function quantifies accuracy, the goal is to maximize the function.

Classification accuracy

To better understand how accurately an algorithm makes a classification, a classification algorithm assigns a new data point to an existing category. One of the more simple ways to illustrate this is to use “binary classifiers” with two target categories. For example, by classifying e-mails into junk or relevant, there are four possible results:

• True positive: A correct positive prediction
• True negative: A correct negative prediction
• False-positive: An incorrect positive prediction
• False-negative: An incorrect negative prediction

The true positive rate (TPR) and false-positive rate (FPR) are key figures for determining the classification accuracy. True positive (TP) is the absolute number of positive results, whereas false positive (FP) is the absolute number of false-negative results.

Mean absolute error

The mean absolute error is a simple method to determine the accuracy of predictions from regressions. By assuming that an error is an absolute difference between the actual and predicted value, the MAE uses the average absolute difference between all errors to give a final result between zero and infinity. Using MAE is particularly helpful when the magnitude of error is irrelevant when errors are not weighed.

The MAE can apply currency or other units of measurement. For example, if a machine learning algorithm’s goal is to predict the development of property prices, one would use a unit of currency. If applied in an MAE calculation, the result might be that the forecasted property prices deviate by an average unit of currency – such as EUR 50,000 –from the actual value.

Root mean squared error

Similar to an MAE, the RMSE determines the accuracy of predictions. Since errors are squared before they are averaged, the RMSE gives weight to larger errors. Unlike an MAE where there is no magnitude of error, the RMSE determines the average number of prediction errors.

Sensibleness and specificity average

The sensibleness and specificity average is a metric used to quantify how natural a user perceives a chatbot’s dialogue. The SSA focuses on two core aspects of human communication: whether something makes sense (sensibleness) and specific (specificity). 

It’s the tester’s responsibility to judge whether the chatbot’s response to the use is reasonable within the context of the dialogue. Once established, the tester evaluates whether the answer is specific to the dialogue content. The SSA metric is produced by averaging the score of the two results.

AI products use cases

Currently, AI technology applies to industries of all shapes and sizes. However, areas seeing favorable adoption rates of AI technology include insurance, healthcare, banking and finance, and pharmaceuticals. Why do these industries have a high probability of expansion? Let’s take a look.

Insurance

The insurance industry is incredibly complex and one that has often been a pain point for consumers. The material is dense, the policies can be strict, and relying on an insurance agent can create a disconnect between consumers and service quality. 

However, insurance is lush with data, making it a primary candidate for an AI revolution. According to the Center for Insurance Policy and Research, AI technology is currently being used in customer service, claims processing, underwriting, and fraud detection. While chatbots can help to serve customer needs like billing, account management, and answer consumer questions, machine learning models “can help to quickly assess the severity of damages and predict the repair costs from historical data, sensors, and images.”

Healthcare

Like the insurance industry, healthcare is also a prime candidate for adopting AI technology but in significantly different ways. According to the Chicago-based Healthcare Information and Management Systems Society, Inc. (HIMSS), seven key areas are likely to see an AI transformation, including clinical decision support, cost savings, healthcare policy regulation, precision medicine, patient data, provider satisfaction, and security. 

Banking and finance

The banking and finance industries have been quick adopters of new technology, ushering in the era of ATMs, card-based payment systems, online banking, and now, AI technology. McKinsey, a premier thought leader in the global consulting space, says that AI technology has the potential to unlock value creation of upward of $1T in incremental value for banks annually. McKinsey forecasts that AI technology can be applied to boost revenue in the banking industry across 25 different use cases, including increased personalization of services to customers and employees, driving down costs through automation efficiencies, reduce error rates, and improved resource utilization.

Pharmaceuticals

The pharmaceutical industry has been on an upward AI trajectory for some time, but the COVID-19 pandemic has only caused to strengthen its use case. In an interview with Forbes, Subroto Mukherjee, the Head of Innovation and Emerging Technology, Americas at GlaxoSmithKline Consumer Healthcare, identified five primary use cases for AI in the pharmaceutical industry. Among them include identifying and diagnosing diseases, personal treatment using digital therapeutics, drug discovery and manufacturing, predictive forecasting of a pandemic or seasonal illnesses to strengthen global supply chains, and identifying candidates for clinical trials. 

AI products conclusion

In most industries, the far-reaching benefits of AI technology have yet to be fully realized. By designing and integrating AI to solve business or consumer problems, entrepreneurs can spur innovation within entire industries. But a strong product requires a talented team of data scientists, engineers, and agile product managers to turn an idea into a tangible working product. Explore how OUTVISE’s portfolio of specialized talent working in digital, business, and technology can help to get your product off the ground.