Comparison: Manual or automatic semantic analysis?

Published on January 21, 2021  - Updated on December 04, 2023

Comparison: Manual or automatic semantic analysis?

When conducting surveys or wanting to analyze customer feedback in-depth, it is very difficult to know how to go about it.

Indeed, when dealing with open responses or verbatim review sites, the task is much more difficult than taking an average with closed responses.

Yet this is the richest source of information about your customers. It would be a shame to do without it, wouldn't it?

This is why we are going to review in this article the two main methods used to perform a semantic analysis of your customer verbatim: manual analysis and automatic analysis.

This is why we are going to review in this article the two main methods used to perform a semantic analysis of your customer verbatim: manual analysis and automatic analysis.

1 / Manual analysis of client's verbatim

It is traditionally carried out with the technique of post-coding. This method consists of defining response themes and then assigning them a number. Thus, this makes it possible to process open responses in surveys with a level of precision approaching the processing of closed questions.

To perform this analysis, you can use spreadsheet software like Excel or equivalent. Then, you have to define the themes corresponding to your survey. You can also define sub-themes to gain precision in your results.

Once this step is complete, select all of the responses to your questionnaire (or a representative sample), then note the number of the corresponding theme or subtheme next to each response.

To better understand, we will see in more detail how to perform this analysis with a concrete example.

a) Define the themes of the survey

Manual verbatim analysis can be applied to any type of verbatim, whether from a survey but also to reviews on online platforms (TripAdvisor, TrustPilot, Google Reviews, Verified Reviews, Critizr, etc.).

For the purposes of this analysis, consider the example of a survey in the banking sector with an open-ended question such as "Are you satisfied with your bank branch?"

By discovering the answers to the questionnaire, define the themes that emerge and that seem most relevant to you.

For example :

Staff

1.1 Staff reception

1.2 Listening skills

1.3 Expertise & Skills

1.4 Availability

Agency

2.1 Opening hours

2.2 Access to the agency

2.3 Geographical proximity

2.4 Atmosphere

2.5 ATM

Customer journey

3.1 Making an appointment

3.2 Support

3.3 Termination

Product

4.1 Current Accounts

4.2 CB / CR

4.3 Pricing

The codification plan must however be fixed at the time of codification. For the sake of comparability, we will hesitate to change it too often. Any addition of a theme during analysis may require a new analysis of all data already classified as well as historical data. Unlike an automatic classification tool, the thematic plan is therefore not very scalable.

On an ad hoc basis and if comparability with your other surveys or sources of customer reviews is not useful, we recommend that you add current topics in your topics.

In addition to analyzing the themes raised, you can also classify the comments by sentiment (positive, negative, neutral) or primary emotion. In order to carry out a cross-analysis, it is, therefore, necessary to make the themes neutral and without valence.

Thus: The Personnel> Availability theme must include both times of availability and times of staff unavailability. It is the feeling/emotion dimension that will add the level of satisfaction or emotion of the subject.

To organize your time: Allow between 1 day and 3 days to establish a codification plan that can be used on the upcoming survey (s).

b) Carry out the codification

Now that our themes and sub-themes are defined, the real fun begins! You must now assign the corresponding number to each of the responses to the survey. To better visualize the task, here is an example with 10 customer comments:

To summarize, manual semantic analysis of verbatim with coding allows for thorough and precise results. On the other hand, the preparation and the analysis are very time consuming because it is necessary to go through all the verbatim. To limit the coding time, we can use a random extract (eg: each answer every 10 answers is coded) but therefore the analysis is no longer exhaustive and problems of statistical representativeness may arise.

It should also be noted that the longer the verbatim, the more tedious the classification.

On the other hand, if the verbatim are multilingual, it will be necessary to ensure that the analyst will master all the subtleties of the different languages ​​of expression in order to proceed with the proper consolidation of the results by country or by language.

Therefore, depending on the volume of the survey (or other sources of advice), the length of the comments, it is not advisable to use this time-consuming and resource-intensive method and instead opt for the analysis automated.

On the other hand, the efforts made are not capitalized for future studies or data. Any new analysis will require starting from scratch.

In addition, although the results are accurate, there may be a human bias in responses related to lack of objectivity depending on each analyst. For the same data, the results may indeed vary from one person to another depending on the perception and personal experience of the person who coded the survey or analysis. This bias can be a particularly important problem if there are prejudices or corporate culture that do not spontaneously encourage listening to the customer.

Finally, these manual analyzes are difficult to compare with each other.

To organize your time: Allow between 2 to 3 minutes per verbatim to classify, i.e. approximately 1 week for 1000 short verbatim and 1 week 3 days for 1000 long verbatim (greater than 144 characters). This time is in addition to the time dedicated to designing the codification plan.

Our recommendation: Using the analysis of customer comments by manual coding can be good for getting to know what customers say and their opinion, but it is always very time-consuming. If your survey or opinion source has a significant number of comments and you want to be thorough, you will quickly find yourself overwhelmed. This analysis is therefore particularly suitable for small amounts of data.

As briefly explained previously, there is another much faster and better way to analyze your customer feedback: automated verbatim analysis.

2 / Automatic verbatim analysis

Unlike manual analysis, a semantic tool is able to analyze a large number of verbatim in no time and identify the subjects and themes that emerge from them.

You have to understand that in the vast majority of projects, setup times are negligible and that this approach allows you to go more directly to analysis and research of insights. On the other hand, in addition to the automated analysis itself, the solution provides you with visualization tools that allow you to go directly to the data and easily discover the lines of analysis relevant to your survey. By choosing the analysis grids for your sector offered by the solution, you save configuration time and you can even benefit from a sector benchmarking vision.

In this article, we'll walk you through how to use an easy-to-learn tool like CXinsights.io, which can help you speed up your customer feedback analysis ... and find insights quickly and effortlessly.

First of all, you should know that the algorithm of the CXinsights.io solution is able to identify the topics discussed and associate a felt emotion (among the 6 primary emotions: joy, surprise, fear, sadness, anger, surprise) to reveal new insights at a glance.

If the verbatims are multilingual, the classification will allow the processing of each verbatim in a transparent way to easily give results by country or by language.

The size of the comments has a marginal impact on the processing times of the automatic classification tools.

To start your analysis, all you need to do is import your data file into the SaaS platform. You can also import verbatim directly from Google Reviews, Trustpilot, Tripadvisor and other review platforms if needed.

You just have to wait a few moments for the algorithm to automatically process your data and give you the first results.

On the CXinsights.io automatic analysis platform, you can choose a theme corresponding to your industry or even customize your entire project based on personalized themes and sub-themes that seem more relevant to you. This setup only takes a few minutes. Customization of themes, routes etc. is also possible on request (custom setting requiring a few hours of additional work).

Note: unlike manual analysis, any customization effort made during project configuration can be capitalized on all data.

For example, if you create a new topic even 2 months after the first treatment, you can update all the data effortlessly in minutes.

To organize your time: Once the file has been submitted, plan a processing time of 2 to 3 minutes per batch of 5000 verbatim to classify, or about 20 minutes for 50,000 verbatim. The tool will then automatically place all verbatim in the corresponding themes and you can immediately access data navigation.

You can compare at a glance the number of comments by theme as well as the main emotion that emerges (the E-index in the screenshot below corresponds to a temperature varying from -20 ° C to + 40 ° C defined by the main emotion detected in the speech, the higher the temperature, the more positive the emotion).

Monitoring is much more easily done than with manual analysis, the tool offers you a follow-up of the evolution of emotions in customer verbatim as well as of the E-Index.

In the screenshot above, for example, we see that there has been a drop in joy and an increase in sadness at the same time from the month of September 2018. It is therefore important to take note mainly of the subjects evoked over this period to understand where this change comes from.

To do so, we choose the corresponding period on the platform which will then automatically identify the 3 points of success and the 3 points of improvement mostly mentioned in the comments.

In this example, we see that there are 3 major risks that correspond to this fall in joy and increase in sadness: the churn risk, Current Operation and Ability to Solve Issues. By clicking on the “Ability to Solve Issues” button, we can quickly realize that poorly done customer support is the root of this problem and is mentioned in customer reviews. We can then prioritize the actions to be implemented to solve the problem.

As you understood, automatic semantic analysis is essential when you are conducting large-scale surveys or receiving numerous comments each month on review platforms or social networks.

Set up alerts by keyword or theme

One of the main strengths of the automation of this analysis is the possibility of creating alerts to be notified in real-time when a verbatim corresponding to the chosen criteria is online.

This makes it possible to switch from a passive approach to an active approach and to react very quickly in the event of risk of attrition, bad buzz, etc.

Indeed, when we receive an email that speaks of churn risk, it must be treated as a priority.

However, it can also be put in place positively to share with the different teams the points of enchantment evoked by the customers.

So how do you set up this alert on an automatic analysis tool?

To create an alert with Q°emotion, it's very simple, you have to go to the Email alerts tab.

Then, just follow these 4 steps: Choose or verify your project; Create the alert; Define the conditions, the recipients, and the sending frequency, Validate your choice.

The most important step is to properly define the conditions. If they are not specific enough, you will receive a large number of alerts that are not always relevant, while if the conditions are too restrictive you will probably never receive alerts!

How to properly define the conditions?

To do so, it is possible to use 3 different levels:

- The type of selector. On our platform, it is possible to use 4 different types of criteria: emotion, the themes mentioned, key criteria (such as the number of stars left on a review for example), keywords. This allows you to be very precise in targeting the alerts you want to receive.

- The equivalence criterion, "is equal" or "is not equal to" in order to define the trigger for the alert.

- The link between the conditions. This is what defines whether the alert should be sent when a condition is met or when all the conditions are met.

Thanks to alerts, you will no longer miss critical verbatim and this will greatly help you prioritize your actions and improve customer satisfaction.

A tool like Q°emotion (editor of the CXinsights.io platform) will allow you to save precious time and very quickly identify the subjects mentioned by your customers as well as the emotions they feel.

We were only able to review some of the features of our SaaS platform, if you want to know more, read our client case here. You can also book a live demo by clicking here.