Evolution of Performance Metrics for Accurate Evaluation of Speech-to-Speech Translation Models: A Literature Review

I. INTRODUCTION

The translation of speech from one language, the source, to another language, the target, demands an efficient evaluation metric for its evaluation. Researchers in the area of speech processing applications are considering objective evaluation metrics such as the Word Error Rate (WER), Bilingual Evaluation

Understudy (BLEU) scores, and BERTScore as well as subjective evaluation metrics such as the Mean Opinion Score (MOS) Naturalness and Similarity for evaluating the output of such models. Currently, there are no standard objective evaluation metrics applied directly to the generated output target speech [1]. This is because all objective evaluation demands speech output to be converted to texts. Hence, there are issues associated with such metrics. For instance, to utilise the BLEU score metric, the output speech needs to be transcribed to texts after which the texts are being compared or evaluated against the reference or ground truth texts. Researchers have observed overtime errors introduced in the obtained BLEU score due to the Automatic Speech Recognition (ASR) model utilised for such task. According to them, using an evaluation metric that takes the output target speech directly will be better than the ASR for computation of BLEU score. The ASR models have inherent errors that alter the expected metric result obtained using the BLEU score. To overcome this issue, some researchers have utilised large ASR models such as Whisper, based on large hours of data training to generate the transcripts to be compared with the ground truth [2 - 5]. Other researchers have also explored MOS Naturalness and MOS Similarity which are both subjective evaluation in conjunction with objective metrics. Here, raters are sourced to evaluate the performance of the generated output speech using their natural instinct to rate the speech model's output. Such metrics are being utilised to evaluate the fluency, accuracy, quality, and correctness of the generated speech, and the ratings are based on human judgments. To further enhance the objective evaluation metrics, it was confirmed in [1] that character-based performance metrics like character-based F1 score (chrF), and character-based BLEU score (chrBLEU) [6] are more robust metrics for speech-to-speech translation and speech synthesis tasks [1]. They were discovered to show a high correlation with human judgment than BLEU and MCD [1]. Some state-of-the-art speech-to-speech translation models such as Whisper [26], Translatotron, Translatotron2, SeamlessM4T and AudioPalm, and speech models developed by other researchers have utilised the aforementioned metrics to evaluate their models.

Metrics such as the Metric for the Evaluation of Translation with Explicit Ordering (METEOR) [1] that shows high correlation with human judgment [7], ROUGE – L, and BLASER have found low utilisation for computation of speech model evaluation. METEOR, ROUGE - L are mostly utilised for texts summarisations, and other Natural Language Processing (NLP) tasks such as machine translation and question answering [7]. Translation Error Rate (TER) is another metric for machine translation [7] which could be explored for speech-to-speech translation tasks [8] as it could be used to evaluate the texts equivalents of the target speech. As a result, it is suggested that researchers could explore ROUGE-L for the speech-to-speech translation tasks since it is evaluated on the texts obtained from generated target speech. The ROUGE – L is a metric that makes use of non-contiguous subsequence obtained from both the predicted texts and ground truth texts by comparing the two texts. It has been shown to have relationship with the well-known statistical metrics such as precision, recall, and f1-scores, by setting the beta parameter in its formula. Other variants of ROUGE such as the ROUGE-S, and ROUGE-W (ROUGE-Weighted) [9] have performed successfully in machine translation tasks [7, 9]. Other common ROUGE used in translation tasks are ROUGE-WE (ROUGE-Word Embedding), ROUGE-G (ROUGE-Graph based) and ROUGE-1, ROUGE-2 [7]. WER, an objective evaluation is mostly utilised for Automatic Speech Recognition (ASR) where the transcribed texts are compared with the reference texts. In addition to this is the BERTScore that is mostly utilized for machine translation models. This has been confirmed to perform better than ROUGE-L, METEOR, and BLEU score due to its high similarity measure between the candidate (machine translated output) and reference or ground truth samples using the cosine similarity procedure [7]. However, it lacks word ordering. It can also be utilized for speech-to-speech translation model by using the transcribed texts rather than the target speech obtained. There is also Cross Lingual Semantic Textual Similarity (XSTS) [10-11], a human metric that measures semantic similarity between a source speech and target translation. At present, more AI platforms and leaderboards are being engaged in the areas of the Natural Language Processing (NLP) to rate and compare AI models. Some are known to speech models to evaluate the performances of various SOTA models. Examples of such are the Hugging Face Leaderboards, IWSLT Challenge etc. [45-50] and some are specifically for speech-to-speech translation models. Each platform has its own metrics used to evaluate speech models. Going by the aforementioned, there are numerous metrics/models that have been used in the past and are currently being engaged with varying performance levels. Hence, there is a need to evaluate the performance of some of the models to ascertain their efficiency. This is the basis for this study. Various metrics that are used in speech-to-speech translation tasks and speech processing applications were reviewed with a view to comparatively determining their efficiencies.

II. PERFORMANCE METRICS

The evaluation of speech-to-speech translation models are either carried out on the target output speech (Subjective evaluation), transcribed texts of the target output speech using an ASR model (objective evaluation), or spectral representations of the target output speech. At other times, the evaluation could be carried out in subjective approach in terms of comparing the target output speech with the source speech as in the case of Cross Lingual Semantic Textual Similarity (XSTS). Others are statistical evaluation metrics such as Accuracy, Precision, Recall, and F1-Score.

2.1 Subjective Performance Metrics

In this evaluation metrics, raters are hired to judge the output speech obtained from the speech-to-speech model. In order words the evaluation is performed directly on the speech obtained. Examples of such metrics are MOS Naturalness, MOS Similarity, XSTS etc.

2.1.1 Mean Opinion Score (MOS)

This is a subjective performance metric that is based on the judgment made by the observer on the output translated speech of the target language. It is the most utilised for speech- to-speech evaluation metric $[3]$. It could be MOS Naturalness or MOS Similarity. In this type of subjective evaluation, raters are sought, who then score the output generated speech on a scale from 1 to 5 which could be 1 = Poor, 2 = Satisfactory, 3 = Good, 4 = Very good, and 5 = Excellent.

2.1.1.1 Mean Opinion Score (MOS) Naturalness

In MOS Naturalness, the raters judge the quality, naturalness, and appropriateness of pro nunciation of the speech output on a scale of 1 to 5. In this case, an incorrectly translated natural target output speech will be rated higher [2] when compared with a correctly unnatural target output speech.

2.1.1.2 Mean Opinion Score Similarity

In MOS Similarity, the raters score the output speech obtained by comparing it with a reference or ground truth output (which can be human-generated speech or synthesised) on a scale of 1 to 5 using quality such as the fluency (flow or correctness of grammar), and adequacy (how deviated speech is from its intended meaning or deviation from the ground or reference speech) of the generated output speech.

2.1.2 Cross Lingual Semantic Textual Similarity (XSTS)

This is a subjective evaluation that is carried out by human raters to assess the quality of the translated target speech. It is conducted by comparing the adequacy (how close it is to its intended meaning) of the generated speech to the source speech. This implies that the human rater or judge must be bilingual to be able to access such target speech for its meaning. The human annotator judges the semantic similarity rather than fluency between the source and target speech $[10-11]$. Using a score on a scale of 1 to 5, the annotator assigned each language pair (source-translated target speech) for semantic meaning where a score of 3 or more indicates the two speeches are close in terms of meaning being conveyed. XSTS is a subjective evaluation metric that also checks for the audio quality as it is utilised directly on the audio generated output target speech. It was originally developed for texts evaluation $[12]$. To obtain the final XSTS results, an average value is computed across selected XSTS computed scores by the human annotators.

2.1.3 Blaser

This is a modality agnostic evaluation metric that works on both speech and texts $[13-14]$. A version of BLASER, BLASER 2.0 utilised in $[14]$ was a modification on the first version $[13]$. For speech-to-speech translation tasks, it offers the advantage of being text-free unlike the ASR BLEU performance metric. For BLASER 2.0, the source speech, the translated target speech and the reference texts are converted into Sentence-level multimOdal and language-Agnostic Representations (SONAR) embedding vectors $(h_{src},h_{mt},\text{and }{h}_{ref})$. These vectors are then fed into a small dense neural network for prediction of XSTS scores for each output of the translation for the supervised version of BLASER 2.0. For the unsupervised version, the cosine similarities between the source and target translated output is obtained for BLASER computation.

2.2 Objective Performance Metric

In the objective, the target output speech of the speech-to-speech translation model is fed to an ASR model to obtain the texts or transcripts equivalent of the speech. Evaluation is then performed on these texts to assess the performance of the model. Examples are the WER, BLEU Score, ROUGE-L, BERTScore, etc.

2.2.1 Bilingual Evaluation Understudy (BLEU)

This is an objective n-gram evaluation that involves the comparison of the speech target output with that of the reference or ground truth. To compute the BLEU score, the output of the speech translation is fed to an ASR model to generate the text equivalent. The texts generated are then compared with the ground truth texts, and the BLEU score is computed. Mathematically, the BLEU score is computed using equation (1) [15]:

where:

which is also computed as:

r = length of machine translated output (text or speech) c = length of reference translation (text or speech)

$p_i=n-\text{gram modified precision score of order }i$, which is given in equation (4) as:

N = maximum number of n - gram order to consider (usually up to 4)

$w_i=$ weight for $n-$ gram precision of order

2.2.1.1 Evaluation of BLEU Score Computation

Given the information below for both the machine-translated output (obtained using ASR) and reference output text:

Machine Translation (MT): The picture the picture by me.

Reference (Ref) 1: The picture is clicked by me.

Where:

The n-gram modified precision score of order i, as depicted in equation (4) is computed using Table 1.

Hence, using equation (3), the Brevity Penalty is computed as:

Substituting BP with other parameters into equation (1) gives:

BLEU score = 0.718 = 71.8

2.2.1.2 Character-level BLEU (charBLEU)

This is a BLEU score metric that computes the BLEU score on the character level rather than on the sentence level $[6]$. It is a better evaluation metric than the ASR BLEU score.

2.2.1.3 Weaknesses of BLEU Score

1. It is an n-gram precision-based metric that does not take into consideration the recall, and its reliance on the exact matching of the n-gram [7].

2. It does not show correlation when compared with human judgment for speech-to-speech translation tasks [7].

2.2.2 Word Error Rate (WER)

For speech-to-speech translation tasks, the WER, an objective evaluation for comparing the ground truth word string to machine translated word string is obtained using equation (5) [16-17] as:

where:

$S=$ The number of substitutions

$D=$ The number of deletions

I = The number of insertions

N = The number of words in the reference

2.2.2.1 Computation of WER in Speech-Speech Translation

Given the reference text (ground truth) of the target language (Yorùbá) as GT and the Translated text equivalent of the translated speech of the target language (Yorùbá) of the output of the translator as MT as given below, the WER is computed using equation (13) as illustrated below:

GT: A bèrè ìmúlò àjęsára ibà pónjú — póntò ní ōdún 1938.

MT: A bèrè ìmúlò àwón àjesá ibà pónjú — póntò ōdúni.

The WER is computed using equation (13) as:

S = 2 [àjesá for àjesára and ódúni for ódún

D = 2 [ní and 1938 deleted in the MT obtained from ASR (input from target speech output)]

N = 9

2.2.3 Recall-Oriented Understudy Gisting Evaluation (ROUGE - L)

This is an evaluation metric that compares the machine translated text sequence (ASR output obtained from speech audio output) with that of the ground truth text sequence by finding the Longest Common Subsequence (LCS) of words. It is mostly used in texts summarisation models like the GPT-4 $[18]$ as well as machine translation $[7]$. According to findings it is much more efficient to compute the non-contiguous LCS of words than its contiguous counterpart to capture more flexible matches between the ground truth and machine translation texts as the order of words may be different. Statistical metrics such as precision, recall and f1 score can be computed using the LCS of both the ground truth word strings and machine translation word strings.

Given that:

X = Reference or ground truth word or sequence

Y = Machine Translation word or sequence output

$LCS(X,Y)=LongestCommonSubsequenceofX,\text{ and }Y(\text{non contiguous})$

The statistical metrics are computed as giving in equations (6) to (8) follows [9]:

where:

len(LCS(X, Y)) = Length of the LCS of X, and Y len(X) = Length of the ground truth word string or sequence

len(Y) = Length of the machine translated word string or sequence

$\beta =\text{Parameter that is chosen to compute the }f1-\text{score and controls the trade}-\text{off between}$

Note that setting $\beta =1$ makes equation (8) equals equation (9) (for computation of the statistical, f1) as illustrated below:

When the value of $\beta =1$, the weight of recall, r is equal to that of the precision p. The value $\beta$ can also be set below or above 1. When $\beta =\frac{1}{2}$, more weights are allocated to the precision, p and this applicable where precision, p is crucial and when $\beta =2$, more weights are allocated to the recall and it is utilised where recall, r is crucial.

Note that $\beta$ is also obtained using $\frac{p}{r}$ [9].

2.2.3.1 Computation of Statistical Metrics and F-Based ROUGE-L in Speech-Speech Translation

Given the reference text (ground truth) of the target language (Yorùbá) as X and the Translated text equivalent of the translated speech of the target language (Yorùbá) of the output of the translator as Y as given below, the precision, recall, and F1-Score are calculated for ROUGE-L using equations (6-8) as illustrated below:

X: A bèrèìmúlò àjęsára ibà pónjú — póntò ní ódún 1938.

Y: A bèrèìmúlò àjesá ibà pónjú — póntò ōdúni 1938.

$LCS(X,Y)=A\text{ bèrè‘imúlò àjesá ibà pónjú - póntò odún 1938. (non contiguous subsequence)}$ $len(X)=9$

len(Y) = 8 len(LCS(X, Y)) = 8

Note that in the first example given in this section using equations (6-8), the computation of precision, recall, and f1-score were carried out using word level counting and the parameter, $\beta =1$

Considering the character level ROUGE - L computation, the following is obtained:

len(Y) = 46

Note that all characters such as alphanumerical, and special characters as well as white spaces are counted as characters.

2.2.3.2 Advantages of using ROUGE-L for Speech-to-Speech Translation Model

The ROUGE-L score takes into consideration the longest common subsequence between the machine translated output and reference texts. This subsequence can either be contiguous or non-contiguous. For machine translation, utilising the contiguous nature of ROUGE-L ensures it avoids the consecutive matching of words for word level metric or consecutive matching of character for character level matching. This ensures it generalises across the whole texts capturing differences between the machine generated and ground truth texts. For speech-to-speech translation models, it could be used in integration with BLEU, and METEOR to enhance the translation model performance since it is possible for two different texts (sharing relationship to the ground truth text) to have same ROUGE-L score when compared to the ground truth texts $[9]$. In addition to that, the ROUGE-L score can also be utilised for assessing the quality of the translated texts which is essential in speech-to-speech translation tasks.

2.2.3.3 Limitation of using ROUGE-L for Speech-to-Speech Translation Model

ROUGE-L being a text-based metric, shows it has the capability to introduce errors typical of text-based speech processing metrics that could affect the performance of the model developed. Another limitation to utilising ROUGE-L is its usage in evaluating or comparing two similar machine translated texts obtained from two different models when compared to the ground truth texts. This is because, ROUGE-L cannot tell which one is close to the ground truth, rather it is an analytical approach to computing the performance metric. Such a limitation could be handled by the subjected evaluation using MOSN or MOSS where raters rate the speech output directly. In addition to that, ROUGE-L is not suitable for evaluating the naturalness, or quality of speech as this can be harnessed from the speech translated output. It is only used to evaluate the quality of the texts which may have deviated from the original speech due to ASR errors.

2.2.4 Mel-Cepstra Distortion (MCD)

This is a performance metric that compares the predicted target mel-cepstra with the reference mel-cepstra [19]. It is calculated as the difference between the MFCCs of the predicted target and reference audio. Mathematically, it is represented in equation (10) as proposed by [20]:

where:

$m_{tk}=kth$ MFCC of the $t-th$ frame from the reference audio

${\hat{m}}_{tk}=kth$ MFCC of the $t-th$ frame from the predicted audio

Then, $MC{D}_k$ is the sum of the squared differences over the first K MFCCs. When the length of the two MFCCs sequences are not equal, Dynamic Time Warping (DTW) is utilised to compute the minimum distance MCD obtainable [19-21]. This is used to evaluate speech synthesis models.

2.2.5 BERTScore

This is language generation [39] evaluation metric that is mostly utilized for machine translation. It can also be utilized for speech-to-speech translation models by feeding the output speech obtained to an ASR model and then compares the texts obtained with the reference texts. BERTScore is based on pretrained BERT contextual embeddings. Hence, it computes the cosine similarity of the machine translation texts and ground truth or reference texts—this it does by finding the sum of the cosine similarity between their respective token's embeddings [39]. BERTScore was known to perform better than n-gram metrics such as BLEU score, METEOR, as well as ROUGE-L in machine translation [7]. It addresses two major problems associated with n-gram metrics, which are penalizing semantical-ordering of words, inability to capture distant dependencies, and their inability to match paraphrases. It was confirmed to have evaluation performance close to human judgement [39]. Given:

The reference/ground truth parameters as:

Tokenized sentence: $y=(y_1,y_2,y_3,y_4,\dots ,y_k)$

Embedding vectors of y: $Y=(Y_1,Y_2,Y_3,Y_4,\dots ,Y_k)$

The machine translated output parameters as:

Tokenized sentence: $\hat{y}=({\hat{y}}_1,{\hat{y}}_2,{\hat{y}}_3,{\hat{y}}_4,\dots ,{\hat{y}}_m)$

Embedding vectors of y: $\hat{Y}=({\hat{Y}}_1,{\hat{Y}}_2,{\hat{Y}}_3,{\hat{Y}}_4,\dots ,{\hat{Y}}_m)$,

The recall, precision, and F1-score are computed for BERTScore using equations (11-13) as [39]:

The computed BERTScore of equations (11-13) are within the cosine similarity range of -1 to 1, which does not affect human correlation or ranking of BERTScore. To ensure human readability, the range is adjusted to fall within o and 1. This is carried using empirical lower bound, b that is computed using Common Crawl monolingual datasets. The rescaled BERTScores for equations (11-13) are computed using equations (14-16) as [39]:

It is to be noted that scaling carried out above is an optional step. Prior to this step, another optional step that involves weighting with Inverse Document Frequency (IDF) score can also be done to obtain the weighted BERTScores instead of equation $(11-13)$.

2.2.5.1 Pseudo Code for the Computation of BERTScore for Speech-to-Speech Translation Model

A. Pseudo code for BERTScore (Precision BERTScore)

1. Begin

1. Compute the tokenised sequences y, and $\hat{y}$ of both the reference texts and machine translation respectively.

2. Compute the embeddings $Y$ and $\hat{Y}$ of both the reference texts and machine translation respectively.

3. For the first word ${\hat{y}}_1$ in the machine translation, compute the cosine similarity between its embedding, ${\hat{Y}}_1$ and all embeddings $Y_1,Y_2,Y_3,Y_4,...,Y_k$ in the reference texts.

4. Compute the maximum value given as: $Y_i^T{\hat{Y}}_j$

5. Repeat step 5 for the remaining words ${\hat{y}}_2$, ${\hat{y}}_3$, ${\hat{y}}_4$,..., ${\hat{y}}_m$ in the machine translated output.

6. Compute the sum of the results of step 5 and 6 as: $\sum _{{\hat{y}}_{j\in \hat{y}}}{Y}_i^T{\hat{Y}}_j$

7. Divide the result of step 7 by the total number of tokens in the machine translation output (candidate output) given by \frac{1}{|\hat{y}|}\sum_{\substack{\hat{y}_{j\in \hat{y}}}}Y_i^T$\hat{Y}$j to obtain P{BERT}.

8. Compute the optional scaling precision score using: ${\hat{P}}_{BERT}=\frac{P_{BERT}-b}{1-b}$

10. End

B. Pseudo code for BERTScore (Recall BERTScore)

1. Begin

1. Compute the tokenised sequences y, and $\hat{y}$ of both the reference texts and machine translation respectively.

2. Compute the embeddings $Y$ and $\hat{Y}$ of both the reference texts and machine translation respectively.

3. For the first word $y_1$ in the reference texts, compute the cosine similarity between its embedding, $Y_1$ and all embeddings ${\hat{Y}}_1,{\hat{Y}}_2,{\hat{Y}}_3,{\hat{Y}}_4,\dots ,{\hat{Y}}_m$ in the machine translated output.

5. Compute the maximum value given as: $Y_i^T{\hat{Y}}_j$

1. Repeat step 5 for the remaining words $y_2,y_3,y_4,\dots ,y_k$ in the reference texts.

2. Compute the sum of the results of step 5 and 6 as: $\sum _{y_i\in y}{Y}_i^T{\hat{Y}}_j$

3. Divide the result of step 7 by the total number of tokens in the reference texts given by $\frac{1}{|y|}$$\sum _{y_i\in y}$Y_i^T$\hat{Y}$j to obtain R{BERT}.

4. Compute the optional scaling recall score using: ${\hat{R}}_{BERT}=\frac{R_{BERT}-b}{1-b}$

10. End

2.2.6 Real Time Factor (RTF)

This metric is utilised to calculate the performances of ASR models. It measures the processing speed of audio. A higher RTF value shows a faster processing of audio signals. Mathematically, it is computed as the ratio of processing time to audio duration given in equation (17) as $[61,62]$:

2.2.7 Sort-Time Objective Intelligibility (STOI)

This metric measures the intelligibility of the speech signals. It can be used to evaluate how clean a speech signal is from its degraded replica. It is mostly utilised for text-to-speech models. For speech-to-speech translation task, it can be utilised to find how intelligible the target speech is from its reference speech. It is measured on a scale of 0 to 1 where 0 denotes unintelligibility while 1 means perfect intelligibility. The steps to compute the STOI of a speech signals are as follows [60]:

2.2.7.1 Steps to Compute the STOI of Speech Signal

1. Split the speech signals into short-time frames which is typically between 32 - 64 ns (256 - 512 samples with $50\mathrm{\%}$ overlaps.

2. Calculate the Sort-Time Fourier Transform (STFT) for both the clean and degraded speech signals of each frame.

3. Compute the spectral magnitude of the STFT for each frame of both the clean and degraded speech signals.

4. Normalize the calculated spectral magnitude to have the same energy in each frame for both the clean and degraded speech signals.

5. Compute the correlation coefficients between the clean and degraded speech signals for each frame.

6. Calculate the STOI score by finding the average of step 5 across all frames.

2.2.8 Perceptual Evaluation of Speech Quality (PESQ)

This measures the quality of speech signals obtained from speech models. It is mostly used for speech synthesis models. It can also be used for speech-to-speech translation models. It measures speech quality on a scale of -0.5 to 4.5 where a higher score means better speech quality. A PESQ of -0.5 means bad speech quality while 4.5 PESQ denotes excellent speech $[59,63]$.

2.2.8.1 Steps to Compute the PESQ

1. Pre-process the clean and degraded speech signals (reference and target speech in the case of speech models) using pre-processing techniques such as filtering, and normalization.

2. Carry out time alignment of both the degraded and clean speech to check for any distortions or delays.

3. The disturbance or degradation which is the difference between the clean and target speech signals is computed.

4. Mapping of the disturbance to a PESQ score that ranges from -0.5 to 4.5 to calculate the score.

2.3 Statistical Evaluation

This refers to the utilisation of machine learning evaluation metrics. This involves Precision, Recall, F1-Score, etc.

2.3.1 Accuracy

This is the ratio of the sum of True Positive (TP) and True Negative (TN) to the sum of TP, TN, False Positive (FP), and False Negative (FN). It is obtained using equation (18) as [23]:

2.3.2 Precision

This is the ratio of TP to the sum of TP and FP. Mathematically, it is obtained as given in equation (19) as [22-24]:

Relating it to speech-to-speech translation tasks, it is computed using equation (20) as [25]:

2.3.3 Recall

This is the ratio of TP to the sum of TP and False Negative (FN). Mathematically, it is obtained as given in equation (21) [22-24]:

For speech-to-speech translation tasks, it is computed using equation (22) as [25]:

2.3.4 F1-Score

This is the ratio of the product of precision and recall to the sum of precision and recall. Mathematically, it is obtained using equation (23) [22-24], which is same as equation (9) as:

For speech-to-speech translation tasks, it is also computed using equation (9) which is same as equation (23).

2.3.5 Computation of Precision, Recall, and F1-Score in Speech-Speech Translation

Given the reference text (ground truth) of the target language (Yorùbá) as RT and the Translated text equivalent of the translated speech of the target language (Yorùbá) of the output of the translator as MT as given below, the precision, recall, and F1-Score are calculated using equations (19), (21), and (22) respectively as illustrated below [25].

RT/GT: A be`rè îmúlò àjesára ibà pó 'njú-póntò ní odún 1938.

MT: A be`rè îmúlò àjesá ibà pó 'njú-póntò odúni 1938.

Total number translated words in the output of the translator = 8