Evolution of Performance Metrics for Accurate Evaluation of Speech-to-Speech Translation Models: A Literature Review

I. INTRODUCTION

The translation of speech from one language, the source, to another language, the target, demands an efficient evaluation metric for its evaluation. Researchers in the area of speech processing applications are considering objective evaluation metrics such as the Word Error Rate (WER), Bilingual Evaluation

Understudy (BLEU) scores, and BERTScore as well as subjective evaluation metrics such as the Mean Opinion Score (MOS) Naturalness and Similarity for evaluating the output of such models. Currently, there are no standard objective evaluation metrics applied directly to the generated output target speech [1]. This is because all objective evaluation demands speech output to be converted to texts. Hence, there are issues associated with such metrics. For instance, to utilise the BLEU score metric, the output speech needs to be transcribed to texts after which the texts are being compared or evaluated against the reference or ground truth texts. Researchers have observed overtime errors introduced in the obtained BLEU score due to the Automatic Speech Recognition (ASR) model utilised for such task. According to them, using an evaluation metric that takes the output target speech directly will be better than the ASR for computation of BLEU score. The ASR models have inherent errors that alter the expected metric result obtained using the BLEU score. To overcome this issue, some researchers have utilised large ASR models such as Whisper, based on large hours of data training to generate the transcripts to be compared with the ground truth [2 - 5]. Other researchers have also explored MOS Naturalness and MOS Similarity which are both subjective evaluation in conjunction with objective metrics. Here, raters are sourced to evaluate the performance of the generated output speech using their natural instinct to rate the speech model's output. Such metrics are being utilised to evaluate the fluency, accuracy, quality, and correctness of the generated speech, and the ratings are based on human judgments. To further enhance the objective evaluation metrics, it was confirmed in [1] that character-based performance metrics like character-based F1 score (chrF), and character-based BLEU score (chrBLEU) [6] are more robust metrics for speech-to-speech translation and speech synthesis tasks [1]. They were discovered to show a high correlation with human judgment than BLEU and MCD [1]. Some state-of-the-art speech-to-speech translation models such as Whisper [26], Translatotron, Translatotron2, SeamlessM4T and AudioPalm, and speech models developed by other researchers have utilised the aforementioned metrics to evaluate their models.

Metrics such as the Metric for the Evaluation of Translation with Explicit Ordering (METEOR) [1] that shows high correlation with human judgment [7], ROUGE – L, and BLASER have found low utilisation for computation of speech model evaluation. METEOR, ROUGE - L are mostly utilised for texts summarisations, and other Natural Language Processing (NLP) tasks such as machine translation and question answering [7]. Translation Error Rate (TER) is another metric for machine translation [7] which could be explored for speech-to-speech translation tasks [8] as it could be used to evaluate the texts equivalents of the target speech. As a result, it is suggested that researchers could explore ROUGE-L for the speech-to-speech translation tasks since it is evaluated on the texts obtained from generated target speech. The ROUGE – L is a metric that makes use of non-contiguous subsequence obtained from both the predicted texts and ground truth texts by comparing the two texts. It has been shown to have relationship with the well-known statistical metrics such as precision, recall, and f1-scores, by setting the beta parameter in its formula. Other variants of ROUGE such as the ROUGE-S, and ROUGE-W (ROUGE-Weighted) [9] have performed successfully in machine translation tasks [7, 9]. Other common ROUGE used in translation tasks are ROUGE-WE (ROUGE-Word Embedding), ROUGE-G (ROUGE-Graph based) and ROUGE-1, ROUGE-2 [7]. WER, an objective evaluation is mostly utilised for Automatic Speech Recognition (ASR) where the transcribed texts are compared with the reference texts. In addition to this is the BERTScore that is mostly utilized for machine translation models. This has been confirmed to perform better than ROUGE-L, METEOR, and BLEU score due to its high similarity measure between the candidate (machine translated output) and reference or ground truth samples using the cosine similarity procedure [7]. However, it lacks word ordering. It can also be utilized for speech-to-speech translation model by using the transcribed texts rather than the target speech obtained. There is also Cross Lingual Semantic Textual Similarity (XSTS) [10-11], a human metric that measures semantic similarity between a source speech and target translation. At present, more AI platforms and leaderboards are being engaged in the areas of the Natural Language Processing (NLP) to rate and compare AI models. Some are known to speech models to evaluate the performances of various SOTA models. Examples of such are the Hugging Face Leaderboards, IWSLT Challenge etc. [45-50] and some are specifically for speech-to-speech translation models. Each platform has its own metrics used to evaluate speech models. Going by the aforementioned, there are numerous metrics/models that have been used in the past and are currently being engaged with varying performance levels. Hence, there is a need to evaluate the performance of some of the models to ascertain their efficiency. This is the basis for this study. Various metrics that are used in speech-to-speech translation tasks and speech processing applications were reviewed with a view to comparatively determining their efficiencies.

II. PERFORMANCE METRICS

The evaluation of speech-to-speech translation models are either carried out on the target output speech (Subjective evaluation), transcribed texts of the target output speech using an ASR model (objective evaluation), or spectral representations of the target output speech. At other times, the evaluation could be carried out in subjective approach in terms of comparing the target output speech with the source speech as in the case of Cross Lingual Semantic Textual Similarity (XSTS). Others are statistical evaluation metrics such as Accuracy, Precision, Recall, and F1-Score.

2.1 Subjective Performance Metrics

In this evaluation metrics, raters are hired to judge the output speech obtained from the speech-to-speech model. In order words the evaluation is performed directly on the speech obtained. Examples of such metrics are MOS Naturalness, MOS Similarity, XSTS etc.

2.1.1 Mean Opinion Score (MOS)

This is a subjective performance metric that is based on the judgment made by the observer on the output translated speech of the target language. It is the most utilised for speech- to-speech evaluation metric . It could be MOS Naturalness or MOS Similarity. In this type of subjective evaluation, raters are sought, who then score the output generated speech on a scale from 1 to 5 which could be 1 = Poor, 2 = Satisfactory, 3 = Good, 4 = Very good, and 5 = Excellent.

2.1.1.1 Mean Opinion Score (MOS) Naturalness

In MOS Naturalness, the raters judge the quality, naturalness, and appropriateness of pro nunciation of the speech output on a scale of 1 to 5. In this case, an incorrectly translated natural target output speech will be rated higher [2] when compared with a correctly unnatural target output speech.

2.1.1.2 Mean Opinion Score Similarity

In MOS Similarity, the raters score the output speech obtained by comparing it with a reference or ground truth output (which can be human-generated speech or synthesised) on a scale of 1 to 5 using quality such as the fluency (flow or correctness of grammar), and adequacy (how deviated speech is from its intended meaning or deviation from the ground or reference speech) of the generated output speech.

2.1.2 Cross Lingual Semantic Textual Similarity (XSTS)

This is a subjective evaluation that is carried out by human raters to assess the quality of the translated target speech. It is conducted by comparing the adequacy (how close it is to its intended meaning) of the generated speech to the source speech. This implies that the human rater or judge must be bilingual to be able to access such target speech for its meaning. The human annotator judges the semantic similarity rather than fluency between the source and target speech . Using a score on a scale of 1 to 5, the annotator assigned each language pair (source-translated target speech) for semantic meaning where a score of 3 or more indicates the two speeches are close in terms of meaning being conveyed. XSTS is a subjective evaluation metric that also checks for the audio quality as it is utilised directly on the audio generated output target speech. It was originally developed for texts evaluation . To obtain the final XSTS results, an average value is computed across selected XSTS computed scores by the human annotators.

2.1.3 Blaser

This is a modality agnostic evaluation metric that works on both speech and texts . A version of BLASER, BLASER 2.0 utilised in was a modification on the first version . For speech-to-speech translation tasks, it offers the advantage of being text-free unlike the ASR BLEU performance metric. For BLASER 2.0, the source speech, the translated target speech and the reference texts are converted into Sentence-level multimOdal and language-Agnostic Representations (SONAR) embedding vectors . These vectors are then fed into a small dense neural network for prediction of XSTS scores for each output of the translation for the supervised version of BLASER 2.0. For the unsupervised version, the cosine similarities between the source and target translated output is obtained for BLASER computation.

2.2 Objective Performance Metric

In the objective, the target output speech of the speech-to-speech translation model is fed to an ASR model to obtain the texts or transcripts equivalent of the speech. Evaluation is then performed on these texts to assess the performance of the model. Examples are the WER, BLEU Score, ROUGE-L, BERTScore, etc.

2.2.1 Bilingual Evaluation Understudy (BLEU)

This is an objective n-gram evaluation that involves the comparison of the speech target output with that of the reference or ground truth. To compute the BLEU score, the output of the speech translation is fed to an ASR model to generate the text equivalent. The texts generated are then compared with the ground truth texts, and the BLEU score is computed. Mathematically, the BLEU score is computed using equation (1) [15]:

where:

which is also computed as:

r = length of machine translated output (text or speech) c = length of reference translation (text or speech)

, which is given in equation (4) as:

N = maximum number of n - gram order to consider (usually up to 4)

weight for gram precision of order

2.2.1.1 Evaluation of BLEU Score Computation

Given the information below for both the machine-translated output (obtained using ASR) and reference output text:

Machine Translation (MT): The picture the picture by me.

Reference (Ref) 1: The picture is clicked by me.

Where:

The n-gram modified precision score of order i, as depicted in equation (4) is computed using Table 1.

Table 1: A summary results for the Computation of n-gram Modified Precision Score

Hence, using equation (3), the Brevity Penalty is computed as:

Substituting BP with other parameters into equation (1) gives:

BLEU score = 0.718 = 71.8

2.2.1.2 Character-level BLEU (charBLEU)

This is a BLEU score metric that computes the BLEU score on the character level rather than on the sentence level . It is a better evaluation metric than the ASR BLEU score.

2.2.1.3 Weaknesses of BLEU Score

1. It is an n-gram precision-based metric that does not take into consideration the recall, and its reliance on the exact matching of the n-gram [7].

2. It does not show correlation when compared with human judgment for speech-to-speech translation tasks [7].

2.2.2 Word Error Rate (WER)

For speech-to-speech translation tasks, the WER, an objective evaluation for comparing the ground truth word string to machine translated word string is obtained using equation (5) [16-17] as:

where:

The number of substitutions

The number of deletions

I = The number of insertions

N = The number of words in the reference

2.2.2.1 Computation of WER in Speech-Speech Translation

Given the reference text (ground truth) of the target language (Yorùbá) as GT and the Translated text equivalent of the translated speech of the target language (Yorùbá) of the output of the translator as MT as given below, the WER is computed using equation (13) as illustrated below:

GT: A bèrè ìmúlò àjęsára ibà pónjú — póntò ní ōdún 1938.

MT: A bèrè ìmúlò àwón àjesá ibà pónjú — póntò ōdúni.

The WER is computed using equation (13) as:

S = 2 [àjesá for àjesára and ódúni for ódún

D = 2 [ní and 1938 deleted in the MT obtained from ASR (input from target speech output)]

N = 9

2.2.3 Recall-Oriented Understudy Gisting Evaluation (ROUGE - L)

This is an evaluation metric that compares the machine translated text sequence (ASR output obtained from speech audio output) with that of the ground truth text sequence by finding the Longest Common Subsequence (LCS) of words. It is mostly used in texts summarisation models like the GPT-4 as well as machine translation . According to findings it is much more efficient to compute the non-contiguous LCS of words than its contiguous counterpart to capture more flexible matches between the ground truth and machine translation texts as the order of words may be different. Statistical metrics such as precision, recall and f1 score can be computed using the LCS of both the ground truth word strings and machine translation word strings.

Given that:

X = Reference or ground truth word or sequence

Y = Machine Translation word or sequence output

The statistical metrics are computed as giving in equations (6) to (8) follows [9]:

where:

len(LCS(X, Y)) = Length of the LCS of X, and Y len(X) = Length of the ground truth word string or sequence

len(Y) = Length of the machine translated word string or sequence

Note that setting makes equation (8) equals equation (9) (for computation of the statistical, f1) as illustrated below:

When the value of , the weight of recall, r is equal to that of the precision p. The value can also be set below or above 1. When , more weights are allocated to the precision, p and this applicable where precision, p is crucial and when , more weights are allocated to the recall and it is utilised where recall, r is crucial.

Note that is also obtained using [9].

2.2.3.1 Computation of Statistical Metrics and F-Based ROUGE-L in Speech-Speech Translation

Given the reference text (ground truth) of the target language (Yorùbá) as X and the Translated text equivalent of the translated speech of the target language (Yorùbá) of the output of the translator as Y as given below, the precision, recall, and F1-Score are calculated for ROUGE-L using equations (6-8) as illustrated below:

X: A bèrèìmúlò àjęsára ibà pónjú — póntò ní ódún 1938.

Y: A bèrèìmúlò àjesá ibà pónjú — póntò ōdúni 1938.

len(Y) = 8 len(LCS(X, Y)) = 8

Note that in the first example given in this section using equations (6-8), the computation of precision, recall, and f1-score were carried out using word level counting and the parameter,

Considering the character level ROUGE - L computation, the following is obtained:

len(Y) = 46

Note that all characters such as alphanumerical, and special characters as well as white spaces are counted as characters.

2.2.3.2 Advantages of using ROUGE-L for Speech-to-Speech Translation Model

The ROUGE-L score takes into consideration the longest common subsequence between the machine translated output and reference texts. This subsequence can either be contiguous or non-contiguous. For machine translation, utilising the contiguous nature of ROUGE-L ensures it avoids the consecutive matching of words for word level metric or consecutive matching of character for character level matching. This ensures it generalises across the whole texts capturing differences between the machine generated and ground truth texts. For speech-to-speech translation models, it could be used in integration with BLEU, and METEOR to enhance the translation model performance since it is possible for two different texts (sharing relationship to the ground truth text) to have same ROUGE-L score when compared to the ground truth texts . In addition to that, the ROUGE-L score can also be utilised for assessing the quality of the translated texts which is essential in speech-to-speech translation tasks.

2.2.3.3 Limitation of using ROUGE-L for Speech-to-Speech Translation Model

ROUGE-L being a text-based metric, shows it has the capability to introduce errors typical of text-based speech processing metrics that could affect the performance of the model developed. Another limitation to utilising ROUGE-L is its usage in evaluating or comparing two similar machine translated texts obtained from two different models when compared to the ground truth texts. This is because, ROUGE-L cannot tell which one is close to the ground truth, rather it is an analytical approach to computing the performance metric. Such a limitation could be handled by the subjected evaluation using MOSN or MOSS where raters rate the speech output directly. In addition to that, ROUGE-L is not suitable for evaluating the naturalness, or quality of speech as this can be harnessed from the speech translated output. It is only used to evaluate the quality of the texts which may have deviated from the original speech due to ASR errors.

2.2.4 Mel-Cepstra Distortion (MCD)

This is a performance metric that compares the predicted target mel-cepstra with the reference mel-cepstra [19]. It is calculated as the difference between the MFCCs of the predicted target and reference audio. Mathematically, it is represented in equation (10) as proposed by [20]:

where:

MFCC of the frame from the reference audio

MFCC of the frame from the predicted audio

Then, is the sum of the squared differences over the first K MFCCs. When the length of the two MFCCs sequences are not equal, Dynamic Time Warping (DTW) is utilised to compute the minimum distance MCD obtainable [19-21]. This is used to evaluate speech synthesis models.

2.2.5 BERTScore

This is language generation [39] evaluation metric that is mostly utilized for machine translation. It can also be utilized for speech-to-speech translation models by feeding the output speech obtained to an ASR model and then compares the texts obtained with the reference texts. BERTScore is based on pretrained BERT contextual embeddings. Hence, it computes the cosine similarity of the machine translation texts and ground truth or reference texts—this it does by finding the sum of the cosine similarity between their respective token's embeddings [39]. BERTScore was known to perform better than n-gram metrics such as BLEU score, METEOR, as well as ROUGE-L in machine translation [7]. It addresses two major problems associated with n-gram metrics, which are penalizing semantical-ordering of words, inability to capture distant dependencies, and their inability to match paraphrases. It was confirmed to have evaluation performance close to human judgement [39]. Given:

The reference/ground truth parameters as:

Tokenized sentence:

Embedding vectors of y:

The machine translated output parameters as:

Tokenized sentence:

Embedding vectors of y: ,

The recall, precision, and F1-score are computed for BERTScore using equations (11-13) as [39]:

The computed BERTScore of equations (11-13) are within the cosine similarity range of -1 to 1, which does not affect human correlation or ranking of BERTScore. To ensure human readability, the range is adjusted to fall within o and 1. This is carried using empirical lower bound, b that is computed using Common Crawl monolingual datasets. The rescaled BERTScores for equations (11-13) are computed using equations (14-16) as [39]:

It is to be noted that scaling carried out above is an optional step. Prior to this step, another optional step that involves weighting with Inverse Document Frequency (IDF) score can also be done to obtain the weighted BERTScores instead of equation .

2.2.5.1 Pseudo Code for the Computation of BERTScore for Speech-to-Speech Translation Model

A. Pseudo code for BERTScore (Precision BERTScore)

1. Begin

2. Compute the tokenised sequences y, and of both the reference texts and machine translation respectively.

3. Compute the embeddings and of both the reference texts and machine translation respectively.

4. For the first word in the machine translation, compute the cosine similarity between its embedding, and all embeddings in the reference texts.

5. Compute the maximum value given as:

6. Repeat step 5 for the remaining words , , ,..., in the machine translated output.

7. Compute the sum of the results of step 5 and 6 as:

8. Divide the result of step 7 by the total number of tokens in the machine translation output (candidate output) given by \frac{1}{|\hat{y}|}\sum{\substack{\hat{y}{j\in \hat{y}}}}Y_i^T\hat{Y}j to obtain P{BERT}.

9. Compute the optional scaling precision score using:

10. End

B. Pseudo code for BERTScore (Recall BERTScore)

1. Begin

2. Compute the tokenised sequences y, and of both the reference texts and machine translation respectively.

3. Compute the embeddings and of both the reference texts and machine translation respectively.

4. For the first word in the reference texts, compute the cosine similarity between its embedding, and all embeddings in the machine translated output.

5. Compute the maximum value given as:

6. Repeat step 5 for the remaining words in the reference texts.

7. Compute the sum of the results of step 5 and 6 as:

8. Divide the result of step 7 by the total number of tokens in the reference texts given by \frac{1}{|y|}\sum_{y_i\in y}Y_i^T\hat{Y}j to obtain R{BERT}.

9. Compute the optional scaling recall score using:

10. End

2.2.6 Real Time Factor (RTF)

This metric is utilised to calculate the performances of ASR models. It measures the processing speed of audio. A higher RTF value shows a faster processing of audio signals. Mathematically, it is computed as the ratio of processing time to audio duration given in equation (17) as :

2.2.7 Sort-Time Objective Intelligibility (STOI)

This metric measures the intelligibility of the speech signals. It can be used to evaluate how clean a speech signal is from its degraded replica. It is mostly utilised for text-to-speech models. For speech-to-speech translation task, it can be utilised to find how intelligible the target speech is from its reference speech. It is measured on a scale of 0 to 1 where 0 denotes unintelligibility while 1 means perfect intelligibility. The steps to compute the STOI of a speech signals are as follows [60]:

2.2.7.1 Steps to Compute the STOI of Speech Signal

1. Split the speech signals into short-time frames which is typically between 32 - 64 ns (256 - 512 samples with overlaps.

2. Calculate the Sort-Time Fourier Transform (STFT) for both the clean and degraded speech signals of each frame.

3. Compute the spectral magnitude of the STFT for each frame of both the clean and degraded speech signals.

4. Normalize the calculated spectral magnitude to have the same energy in each frame for both the clean and degraded speech signals.

5. Compute the correlation coefficients between the clean and degraded speech signals for each frame.

6. Calculate the STOI score by finding the average of step 5 across all frames.

2.2.8 Perceptual Evaluation of Speech Quality (PESQ)

This measures the quality of speech signals obtained from speech models. It is mostly used for speech synthesis models. It can also be used for speech-to-speech translation models. It measures speech quality on a scale of -0.5 to 4.5 where a higher score means better speech quality. A PESQ of -0.5 means bad speech quality while 4.5 PESQ denotes excellent speech .

2.2.8.1 Steps to Compute the PESQ

1. Pre-process the clean and degraded speech signals (reference and target speech in the case of speech models) using pre-processing techniques such as filtering, and normalization.

2. Carry out time alignment of both the degraded and clean speech to check for any distortions or delays.

3. The disturbance or degradation which is the difference between the clean and target speech signals is computed.

4. Mapping of the disturbance to a PESQ score that ranges from -0.5 to 4.5 to calculate the score.

2.3 Statistical Evaluation

This refers to the utilisation of machine learning evaluation metrics. This involves Precision, Recall, F1-Score, etc.

2.3.1 Accuracy

This is the ratio of the sum of True Positive (TP) and True Negative (TN) to the sum of TP, TN, False Positive (FP), and False Negative (FN). It is obtained using equation (18) as [23]:

2.3.2 Precision

This is the ratio of TP to the sum of TP and FP. Mathematically, it is obtained as given in equation (19) as [22-24]:

Relating it to speech-to-speech translation tasks, it is computed using equation (20) as [25]:

2.3.3 Recall

This is the ratio of TP to the sum of TP and False Negative (FN). Mathematically, it is obtained as given in equation (21) [22-24]:

For speech-to-speech translation tasks, it is computed using equation (22) as [25]:

2.3.4 F1-Score

This is the ratio of the product of precision and recall to the sum of precision and recall. Mathematically, it is obtained using equation (23) [22-24], which is same as equation (9) as:

For speech-to-speech translation tasks, it is also computed using equation (9) which is same as equation (23).

2.3.5 Computation of Precision, Recall, and F1-Score in Speech-Speech Translation

Given the reference text (ground truth) of the target language (Yorùbá) as RT and the Translated text equivalent of the translated speech of the target language (Yorùbá) of the output of the translator as MT as given below, the precision, recall, and F1-Score are calculated using equations (19), (21), and (22) respectively as illustrated below [25].

RT/GT: A be`rè îmúlò àjesára ibà pó 'njú-póntò ní odún 1938.

MT: A be`rè îmúlò àjesá ibà pó 'njú-póntò odúni 1938.

Total number translated words in the output of the translator = 8

III. COMPARATIVE ANALYSIS OF THE DIFFERENT EVALUATION METRICS

Table 2 shows the comparison of some of the speech-to-speech translation metrics so that researchers can make right choices for their speech-to-speech models. It details the subjective and objective evaluation metrics, such as MOS Naturalness, MOS Similarity, BLASER, & XSTS; and BLEU score, WER, METEOR, ROUGE-L, BERTScore, & MCD respectively. The table also highlights the statistical metric that could also be explored for speech-to-speech translation tasks . Table 3 gives highlights of when, application areas and how each of the discussed metrics are utilised.

Table 2: Comparison of Various Performance Metrics for Speech-to-Speech Translation Tasks

Table 3: When, where, and how to use the Speech Translation Metrics

IV. STATE OF THE ART (SOTA) SPEECH-TO-SPEECH MODELS' PERFORMANCE METRICS

The performance metrics utilised by some SOTA speech-to-speech translation models metrics utilised by researchers in this field are highlighted in Table 4. It shows that some researchers utilised the BLEU score along with MOS for speech-to-speech translation tasks. Some new metrics like the BLASER, XSTS, and ROUGE – L are beginning to be used by researchers in this area of research including speech-to-texts, and automatic speech-to-texts translation [27].

Table 4: Speech-to-Speech Translation Metrics Utilised for well-known SOTA models

V. APPLICATIONS OF SPEECH METRICS ON LEADERBOARDS AND MODERN AI APPLICATIONS

In this modern-day era, speech models are mostly compared with other models to know how they perform among themselves. Different models are evaluated or tested using the same dataset and ranked based on their performance. For instance, speech-to-speech translation models such as Whispers , Translatotron, SeamlessM4T may be benchmarked with same datasets of different varieties, across different databases such as FLUERS, LibriSpeech, MustC, Fisher datasets, and then the BLEU scores are computed for each model across these datasets and are ranked based on these metrics. When more than one source of datasets is utilized, the average across these sources is computed to rank each model.

At present, there are quite a few leaderboards and modern AI platforms that evaluate and rank speech models such as the Hugging Face, International Workshop on Spoken Language Translation (IWSLT), Real-World Speech-to-Text API Leaderboard, Open ASR leaderboard etc . Metrics such as BLEU scores, WER, METEORS, Real Time Factor (RTF), Short-Time Objective Intelligibility (STOI), Perceptual Evaluation of Speech Quality (PESQ), CER, are being utilized to evaluate and rank speech models. There is no known information about whether subjective metrics such as MOS Naturalness, MOS Similarity, BLASER, and MCD are being used on any of these platforms. BERTScore, METEOR, ROUGE-L, and BLEU scores are used on Hugging Face for texts generation models like texts summarization, and machine translation models. On the Open ASR Leaderboard on Hugging Face, ASR models are being compared and ranked, and metrics of choice are the WER, and Real Time Factor (RTF). Where the WER and RTF are utilized to evaluate the performance, the speech-to-texts models and are ranked based on these metrics. However, this review paper is suggesting the utilization of BERTScore for speech-to-speech translation models where it can be used on the transcribed texts of the target speech. Then the text is compared to the reference text by computing the cosine similarity between them. In such application, it is important to not rely on the BERTScore alone but to integrate it with other metrics such as the subjective evaluation metrics, and the BLEU score because the BERTScore cannot evaluate speech quality even though it evaluates the word order. Some metrics are peculiar to each leaderboard as well as speech models. For instance, on the Hugging Face leaderboard, A detailed overview of these leaderboards is given in the next sub section.

5.1 Hugging Face Leaderboard

Hugging Face leaderboard is an AI platform that evaluates the SOTA AI models such as the image processing models, text-based models, and speech processing models . The platform evaluates and compares ASR models on its Open ASR Leaderboard . ASR models are ranked using the WER, and RTF. TTS models are also being ranked via this platform .

5.2 ICASSP 2024 Speech Signal Improvement Challenge

This is a speech models competing leaderboard that aims to improve speech signals by evaluating speech models for their speech signals quality and intelligibility. Speech signals are evaluated using techniques such as filtering, noise reduction, and speech enhancement. The metrics utilized here are Short Time Objective Intelligibility (STOI), and Perceptual Evaluation of Speech Quality (PESQ), Word Accuracy (WAcc) [65].

5.3 IberSPEECH 2024 Challenge

This platform was developed to promote research and development in speech processing applications such as speech synthesis, ASR etc. It looks into the evaluation of speech recognition and speech synthesis models. WER, and Character Error Rate (CER) are used to evaluate the ASR models while text-to-speech models are evaluated using PESQ, and STOI .

5.4 International Workshop on Spoken Language Translation (IWSLT)

This platform promotes research in Spoken Language Translation (SLT) such as speech-to-text, speech-to-speech translation, automatic speech translation, Speech synthesis etc., [40-41]. Speech-to-texts and speech-to-speech translation models are evaluated and ranked on this platform where BLEU scores, and METEOR are utilized to evaluate the ASR models, while speech-to-speech translation models are evaluated using BLEU scores, METEOR, and WER.

5.5 Speech Generation Evaluation and Leaderboard

This platform evaluates and ranks speech generation models using metrics such as speech intelligibility, which is measured by speech recognition error rates; Naturalness, which is predicted utilising speech models trained on human naturalness ratings; and Similarity, which measures the cosine similarity speaker embeddings mostly used for voice cloning systems .

VI. TYPICAL LEADERBOARD RATINGS FOR DIRECT AND CASCADED SPEECH-TO-SPEECH TRANSLATION MODELS.

6.1 ASR BLEU Speech-to-Speech Translation on FLEURS X-ENG

For the ASR BLEU metric, the speech-to-speech translation tasks from other language to English on FLUERS corpus shows that GenTranslateV2, GenTranslateV1, SeamlessM4T LargeV2, SeamlessM4T Large, AudioPaLM2, WhisperV2, SeamlessM4T Medium have ASR BLEU scores of 32.3, 30.1, 29.4,

25.8, 24.0, 23.5, and 20.4 respectively, where the highest BLEU score of 32.3, was obtained for GenTranslateV2, with SeamlessM4T Medium having the lowest BLEU score. This leaderboard is available on this link: https://paperswithcode.com/sota/speech-to-speech-translation-on-fleurs-x-eng?utm_source=chatgpt.com. It should be noted that the authors of the GenTranslateV2 developed both end-to-end and cascaded system where the end-to-end models performed better than all the other models over 30 languages to English translation on both FLEURS X-Eng and CoVoST X-Eng datasets. For 15 languages to English on FLEURS X-Eng dataset, it achieved an average BLEU score of 32.3 while GenTranslateV1, SeamlessM4T LargeV2, SeamlessM4T Large, AudioPaLM2, and WhisperV2 have average BLEU scores of 30.1, 29.4, 27.1, 24.0, 23.5 respectively. For the cascaded system, GenTranslateV2, GenTranslateV1, SeamlessM4T V2, SeamlessM4T, and WhisperV2 have average BLEU scores of 34.2, 34.0, 32.3, 31.9, and 31.2 respectively. This shows GenTranslateV2 still performs better in the cascaded speech-to-speech translation task [52].

6.2 ASR BLEU For Speech-to-Speech Translation on CVSS dataset

On another rank, where both SeamlessM4T Large and SeamlessM4T Medium were ranked, results on the leaderboard shows that SeamlessM4T Large had the best ASR BLEU with a value of 36.5 in comparison with 28.1 for SeamlessM4T Medium when both were trained on CVSS Dataset. The link is available here: https://paperswithcode.com/sota/speech-to-speech-translation-on-cvsss

6.3 ASR WER Speech-to-Text Translation on Hugging Face Leaderboard

Using the 8 datasets used in the ESB paper [53], which consists of LibriSpeech clean, LibriSpeech other, VoxPopuli, TED-LIUM, GigaSpeech, SPGISpeech, Earnings-22, and AMI datasets as the benchmark datasets, Granite-speech-3.3-8b which was trained on public and synthetically generated datasets for ASR, and Automatic Speech Translation (AST) tasks achieved the best WER of 5.85 in comparison to Massively Multilingual Speech (MMS) - Finetuned ASR - FL102 with a wave2vec architecture which has the worst WER of 39.8. SOTA Whisper large, and Whisper medium have WERs of 7.94, and 8.09 respectively. Whisper-large-v3 has the best WER amongst all the Whisper models ranked with a value of 7.44. Facebook's Hubert-xlarge-ls960-ft has WER of 22.55. It should be noted that all models ranked on this leaderboard were trained with the same 8 training datasets aforementioned after which comparison were made. The average WER across all the 8 datasets were computed for each ASR model [54].

6.4 WER for ASR on TedLium Dataset

In another leaderboard for ASR WER on TedLium dataset (a dataset that contains English language TED Talks which spans from 118 to 452 hours sampled at 16 kHz with their respective transcripts), United-MedASR trained on 764 parameters, parakeet-rnnt-1.1b, Whispering-LLaMa-7b, and SpeechStew trained 100 M parameters yielded WERs of 0.29, 3.92, 4.6, and 5.3 respectively, which shows that United-MedASR has the best WER. This leaderboard is available at this link: https://paperswithcode.com/sota/speech-recognition-on-tedlium

VII. CASCADED & DIRECT SPEECH-TO-SPEECH PERFORMANCE METRICS

The conventional approach to speech-to-speech translation models before the exploration of machine/deep learning-based methods involves utilising cascading of traditional statistical approaches such as Hidden Markov Model (HMM), Forward Algorithms, Viterbi Algorithm etc. for ASR, statistical approaches such as HMM, analysis of transcribed words for machine translation and concatenative approach, rule-based approach, statistical approach for text-to-speech . At a later time when AI is evolving, the machine learning based approach involving deep learning such as neural networks, LSTM etc., [20, 57-58] are being applied in this cascaded approach to modeling speech-to-speech translation task. Due to the availability of high computing power [61, 62], coupled with big data, deep learning based a direct end-to-end approach begins to take over the training of speech-to-speech translation model, where texts representation as seen in the cascaded approach is not available [4]. Of course, one, would expect that metrics of evaluation for the cascaded approach will involve integration of metrics for ASR, MT, and SS models, whereas the metrics of the direct approach involves a single metrics per time to evaluate target speech output. Table 5 highlights some of the metrics utilised for cascaded and direct speech-to-speech translation tasks.

Table 5. Summary of Cascaded and Direct Speech-to-Speech Translation Metrics

VIII. DISCUSSION

The study highlights the various performance metrics utilised by researchers for speech processing applications, particularly speech-to-speech translation tasks. Based on the findings as reported in this work, most of the existing speech-to-speech translation models have utilised objective BLEU score performance metric for evaluation . MOS Naturalness, and MOS Similarity follow next as the performance metrics in this domain . Other metrics that could be utilised for speech-to-speech translation tasks are METEOR, and ROUGE-L [1], [27]. Other metrics such as the WER, CER, NIST, XSTS, SER, PER, and BLASER are also of interest in this field of speech processing applications.

A thorough analysis of the various performance metrics utilised for speech-to-speech translation tasks show that all the subjective evaluations are computed on the generated output speech while that of the objective evaluation metrics show they are computed on the transcribed text string using a STT model except the MCD which is mostly utilised for TTS . Hence, it is expected that the subjected evaluation metrics gives the best performance when compared to its objective counterpart due to no errors introduced as a result of the absence of ASR transformation . Findings show that when it comes to adequacy which shows how close the predicted target output speech is to the source speech, MOS Similarity as well as XSTS, BLEU score, and ROUGE-L could be utilised to perform evaluation. MOS Naturalness can be utilised to check the naturalness of the output speech. For audio translation quality, the MOS Naturalness, BLASER, XSTS, as well as BLEU score could be utilised to achieve that . The outcome of the study further suggests that ROUGE-L performance metric and its utilization in the NLP tasks such as machine translation and speech recognition , should be given priority as an evaluation metric for speech-to-speech translation tasks. This is due to the fact that it is to be applied on the transcribed texts obtained from the generated output target speech just like the case of the BLEU score metric. However, care must be taken due to its limitation of not being able to give good evaluations of two or more different translated texts that have same number of words (word-level) or characters (character-level) but different order - same ROUGE-L scores . It is therefore advisable that it is integrated with other metrics such as BLEU, and METEOR. Coupled with that, ASR chrF is known to perform better than ASR BLEU due to its ability to perform more matching between the translated output texts and ground truth texts at the character level. WERs are mostly utilised for ASR models, but they can also be used for speech-to-speech translation tasks. MCD has little application for speech translation tasks. They are mostly utilised for speech synthesis from texts (TTS model) .

The study went further to survey some of the present leaderboards and modern AI tools available for evaluating and comparing speech models . Some of the existing leaderboards such as Facebook's Hugging Face, IWSLT etc., evaluates speech models such as the TTS, and ASR. These evaluations give experts insight into how the speech models perform in comparison to one another. There are also leaderboards specifically for TTS and speech generations that compare TTS models with one another. The choice of metrics utilized in any of the leaderboards is dependent on the speech models being considered, as well as the leaderboards of interest. For instance, on Facebook's Open ASR leaderboards, the WER, and RTF are being used to compare and evaluate ASR models.

IX. CONCLUSION

The various metrics utilised for speech-to-speech translation models, their benefits and the comparison of the metrics were highlighted in this study. The results showed that among the various speech models metrics that have been employed, BLEU score is predominant, followed by MOS Naturalness and MOS Similarity. Metrics such as the BLASER, XSTS, ROUGE-L, BERTScores, and METEORS are beginning to gain recognition particularly for speech models. Leaderboards ratings of speech models show that existing BLEU scores, WER, and RTF are mostly utilized for speech models. This further suggests the importance of these metrics. RTF even though it is being utilized on the leaderboards, has the least utilization by researchers. Also, metrics such as ROUGE-L, METEORS, and BERTScore have been utilised successfully in machine translation, they could be explored for speech-to-speech translation particularly when they are used in conjunction with each other coupled with BLEU score. The findings show that the subjective metrics are computed directly on the target speech output, which depicts a good evaluation of the quality of speech model, unlike the objective metrics that are prone to errors introduced due to the ASR utilization. The study therefore serves as an eye opener to researchers or readers hoping to develop speech-to-speech translation models to make the right choices on the performance metrics of their models.

ACKNOWLEDGEMENTS

The authors wish to acknowledge the Covenant University Center for Research, Innovation and Discovery (CUCRID), and Google for providing funds towards the publication of this study