What is the current ranking of the NLU

Chatbot study in the financial sector 2020: NLU technology as a success factor

The Chatbot Study 2020 in the financial sector by Prof. Dr. Peter Gentsch, AI expert and institute director for conversational business at Aalen University, shows surprising results.

Savings banks and Volksbanks beat fintech: The results show that savings banks and Volksbanks, with an average hit rate of 70 percent, rank well ahead of direct banks and fintechs with an average of 41 percent, while insurance companies achieve an average hit rate of only 35 percent.

It is astonishing that big players like Ergo or Arag come off with a hit rate of less than 5 percent. It is just as astonishing that in some cases the off-topic and personal questions were answered better than the technical questions. The chatbot study in the financial sector can be downloaded free of charge: https://chatbots.conversationalbusiness.de

Chatbots: artificial intelligence for customer centricity

Against the background of the current discussion about customer centricity and artificial intelligence, the sometimes poor performance of the examined chatbots is surprising. It turns out that chatbots are not a plug & play solution, but rather require dialogues that have been conceived and trained by the customer.

Results of the chatbot study in the financial sector

Who is leading the study ranking? Sparkasse Nürnberg, Sparkasse Köln, Volksbank Freiburg and VR-Bank Westmünsterland lead the ranking. This is followed by insurance companies, direct banks and fintechs. In the last places, Ergo and Arag bring up the rear in the study.

Differentiation of industry comparisons

However, the comparison between the industries has to be put into perspective. The relatively poor performance of the insurance companies can partly be explained by the fact that they often specialize in a few products, but the questions were based on a broad range of products in order to be able to cover all products.

Chatbots with NLU technology do the best

If companies use modern conversational AI technologies such as NLP, NLU and deep learning, then chatbots can bring high economic benefits and thus competitive advantages such as positioning and reputation through their scalability and time and cost efficiency. It turns out that the companies that use NLU (Natural Language Understanding) technology do significantly better overall.

Reliable understanding of context only with NLU approach

Companies could benefit from this for general dialogues and larger industry topics. For company- and product-specific answers, however, the training volume is not comparable in terms of volume. Likewise, customer and company-specific contexts can only be reliably guaranteed with NLU (Natural Language Understanding) approaches, which require appropriate lexicons and extensive context modeling.

Research design of the chatbot study on the financial sector

In the study, 13 selected chatbots from the insurance and finance industries were analyzed at the Institute for Conversational Business at the Aalen University of Applied Sciences for Technology and Economics. On the basis of qualitative and quantitative questions, the respective chatbots were examined from the consumer's point of view with regard to their ability (intents), i.e. to correctly recognize the inquiries and needs and inquiries of customers and to deliver the correct answer.

Evaluation criteria

In order to evaluate the conversational ability of the chatbots accordingly, "off-topic" and personal questions were asked in addition to technical questions about products and conditions. Five criteria were used to operationalize the quality of conversations and responses:
1. Accuracy of the answers to the respective question
2. Response speed
3. Ease of use / user-friendliness
4. Informative character of the answers
5. Entertaining nature of the answers.

Confirmed study findings "Conversational Business 2020"

In the previous chatbot study “Conversational Business 2020 - Status Quo and Trends”, 600 consumers were asked about their acceptance of chatbots. The findings show that chatbots are often still developed too much from the perspective of cost savings and process automation and not from the customer's perspective. Customers generally welcome chatbots if the response quality is acceptable. However, many consumers are frustrated with their chatbot experiences and therefore avoid them. A bad chatbot is therefore counterproductive for the customer experience. This suggests the recommendation: Better not have a chatbot than a bad chatbot. On the other hand, companies that use chatbots with a high response rate report very positive experiences.

When do chatbots get intelligent?

The latest success reports from Google’s Meena, Facebook’s Blender or OpenAI’s GPT-3 proclaim the next big AI milestone after the hype about deep learning. Indeed, the AI-based dialogues are fascinatingly human-like and entertaining. All three AI systems are based on huge amounts of training on the Internet. However, there is no monitored quality control here, and the bias that exists on the Internet is also learned. The question is what benefits this new level of conversational AI means for companies in addition to the entertainment effect.

The silver bullet for the future is a hybrid approach

According to Prof. Dr. Gentsch a hybrid approach that combines knowledge concepts and control logics with the generative approach. This allows the power of implicit learning through deep learning to be used in a quality-assured manner and at the same time to bring in company and product-specific knowledge.

Potential of chatbots through customer centricity

This shows that chatbots in Germany are only gradually becoming more intelligent. However, the wide spread of the results also shows that it is of great importance which technology is used. If companies want to use the potential of chatbots in terms of their scalability as well as time and cost efficiency, they have to think and understand dialog lines from the customer's point of view on the one hand, and on the other hand they have to deal with conversational AI technologies such as NLP that have become massively better in recent years , NLU and deep learning.

About Prof. Dr. Peter Gentsch

Prof. Dr. Peter Gentsch is a speaker, entrepreneur and scientist in one person and has been one of the pioneers and top experts in the field of digital transformation, artificial intelligence (AI) and big data since the 1990s. While others only talk about digitization and technology trends, he lives them. With numerous company foundations and investments and five successful exits, he is one of the most successful internet entrepreneurs in Germany: https://chatbots.conversationalbusiness.de

If you are interested in a Kauz chatbot demo, just send us a message.

Subscribe to our newsletter right away so that you always stay up to date.