research & development Quality comparison of wideband coders including tandeming and transcoding Catherine Quinquis ([email protected]) France Telecom Research & Development, France ETSI Workshop on Speech and Noise In Wideband Communication, 22nd and 23rd May 2007 - Sophia Antipolis, France
22
Embed
Quality comparison of wideband coders including … · Quality comparison of wideband coders including tandeming and transcoding ... Quality comparison of wideband coders including
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
research & development
Quality comparison of wideband coders including tandeming and transcoding
Catherine Quinquis ([email protected])France Telecom Research & Development, France
ETSI Workshop on Speech and Noise In Wideband Communication,
22nd and 23rd May 2007 - Sophia Antipolis, France
Quality comparison of wideband coders including tandeming and transcoding – p 2 research & development France Telecom Group
Wideband codecs
Last year, wideband extensions of narrowband codecs have been standardised
G.729.1• providing high packetized wideband voice quality with scalability and
interoperability with existing G.729 based VoIP networks and terminalsEVRC-WB
• providing wideband in 3GPP2 networks using the same rate set as the current EVRC.
Other ITU-T wideband codecsG.722
• Mainly used in conference call, but introduced also in VoIP networks G.722.2
• providing wideband in 3GPP networks (also called AMR-WB)
Quality comparison of wideband coders including tandeming and transcoding – p 3 research & development France Telecom Group
Impact of transcoding and tandemmingSelf tandemingtranscoding
Comparison of wideband codecs with narrowband codecs
1
2
3
summary
Quality comparison of wideband coders including tandeming and transcoding – p 4 research & development France Telecom Group
Comparison of Wideband codecs
Quality comparison of wideband coders including tandeming and transcoding – p 5 research & development France Telecom Group
Compared Subjective WB quality of G.729.1, G.722 & G.722.2
Extract from the characterisation phase (step 2)Experiment 1
• narrowband
Experiment 2• Purpose : evaluate the performance of G729.1 algorithm with respect to well
known references, in wide band clean speech (free of background noise) conditions with a variety of input levels and frame error rates.
• Methodology : Absolute Category Rating (ACR) method with the Mean Opinion Score (MOS) rating scale for WB subjective tests (MOS-LQSW).
• Languages : French (Canada) & English (US)• Subjects : 32 naïve listeners
Experiment 3• Wideband Music
Quality comparison of wideband coders including tandeming and transcoding – p 6 research & development France Telecom Group
Compared Subjective WB quality of G.729.1, G.722 & G.722.2 (no FER)
Experiment 2 - clean speech
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
Direct G.72264k
G.729.132k
G.729.124k
G.729.114k
AMR-WB
23.85k
AMR-WB
12.65k
MO
S LQ
SW
Dynastat
VoiceAge
Quality comparison of wideband coders including tandeming and transcoding – p 7 research & development France Telecom Group
Compared Subjective WB quality of G.729.1, G.722 & G.722.2 (FER)
Exp 2 clean speech + FER - Lab E
1.00
1.50
2.00
2.50
3.00
3.50
4.00
4.50
5.00
FER = 0% FER = 3% FER = 6% FER = 10%
G.729.1 14k
G.729.1 24k
G.729.1 32k
AMR-WB 12.65k
AMR-WB 23.85k
Exp 2 clean speech + FER - Lab F
1
1.5
2
2.5
3
3.5
4
4.5
5
FER = 0% FER = 3% FER = 6% FER = 10%
G.729.1 14k
G.729.1 24k
G.729.1 32k
AMR-WB 12.65k
AMR-WB 23.85k
Quality comparison of wideband coders including tandeming and transcoding – p 8 research & development France Telecom Group
Compared Subjective EVRC-WB quality with AMR-WB and VMR Mode-0
Extract from EVRC-WB Characterization test Experiment 1
• Purpose : evaluate the performance of EVRC-WB algorithm with respect to well known references, in wide band clean speech (free of background noise) conditions with a variety of input levels and frame error rates.
• Methodology : Absolute Category Rating (ACR) method with the Mean Opinion Score (MOS) rating scale for WB subjective tests (MOS-LQSW).
• Languages : English (US)• Subjects : 32 naïve listeners
Experiment 2• Purpose : evaluate the performance of EVRC-WB algorithm with respect to
well known references, in wide band noisy speech, and VAD/DTX scheme• Methodology : P.835
Experiment 3 & 4• Narrowband
Quality comparison of wideband coders including tandeming and transcoding – p 9 research & development France Telecom Group
Compared Subjective EVRC-WB quality with AMR-WB and VMR Mode-0
CT1 Results
1
2
3
4
5
-22 dB level -32 dB level -12 dB level 1% FER withHRPD Rev A
mask
2% FER withHRPD Rev A
3% FER 6% FER 1% D&B+1%packet levelsignaling*
MO
S
EVRC-WB EVRC-WB with DTX AMR-WB @ 12.65 kbps with DTX VMR Mode 0
Quality comparison of wideband coders including tandeming and transcoding – p 10 research & development France Telecom Group
Performance of G.722 with packet loss concealment
Extract from G.722 PLC Selection test Experiment 1a &1b
• Purpose : evaluate the performance of PLC algorithm with respect to well known references, in wide band clean speech (free of background noise) conditions with a variety of frame error rates (random for exp1a, burst for exp 1b).
• Methodology : Absolute Category Rating (ACR) method with the Mean Opinion Score (MOS) rating scale for WB subjective tests (MOS-LQSW).
• Languages : Japanese, French & English (US)• Subjects : 32 naïve listeners
Experiment 2a &2b• Purpose : evaluate the performance of PLC algorithm with respect to well known
references, in noisy speech, (random for exp2a, burst for exp 2b).
Quality comparison of wideband coders including tandeming and transcoding – p 11 research & development France Telecom Group
Performance of G.722 with packet loss concealment
1
2
3
4
5
0% 1% 3% 6% 3%+0.1%RBER
PLC A PLC C PLC0 G.729.1 -32k
1
2
3
4
5
0% 1% 3% 6% 3%+0.1%RBER
PLC A PLC C PLC0 G.729.1 -32k
Clean peech, Random FER Clean speech, Bursty FER
Quality comparison of wideband coders including tandeming and transcoding – p 12 research & development France Telecom Group
Conclusion on Wideband Quality
All these codecs provide high wideband quality that can be roughly divided into 2 categories:
Maximum wideband quality for the most recent codecs at their maximum bit rates: very close to "direct" quality in the test conditions
Slightly lower quality for these codecs when operating at reduced bit rates around 12-14 kbit/s and for G.722 at 64 kbit/s but for much reduced complexity
Quality comparison of wideband coders including tandeming and transcoding – p 13 research & development France Telecom Group
Impact of transcoding and tandemming
Quality comparison of wideband coders including tandeming and transcoding – p 14 research & development France Telecom Group
Impact of transcoding/tandeming G.722.2 & G.722
Extract from the characterisation phase of AMR-WBExperiment 1
• Purpose : evaluate the performance of AMRWB algorithm, in wide band clean speech (free of background noise) tandeming conditions with a variety of input levels.
• Methodology : Absolute Category Rating (ACR) method with the Mean Opinion Score (MOS) rating scale for WB subjective tests (MOS-LQSW).
• Languages : Finnish & English • Subjects : 32 naïve listeners
Experiment 2• Purpose : evaluate the performance of AMRWB algorithm, in wide band
clean speech (free of background noise) conditions in transcoding with other wideband standards
• Methodology : Absolute Category Rating (ACR) method with the Mean Opinion Score (MOS) rating scale for WB subjective tests (MOS-LQSW).
• Languages : French & English (US) • Subjects : 32 naïve listeners
Quality comparison of wideband coders including tandeming and transcoding – p 15 research & development France Telecom Group
Compared Subjective WB quality of G.722 & G.722.2 in self tandeming
1
1.5
2
2.5
3
3.5
4
4.5
5
- singlecoding
self-tandem singlecoding
self-tandem singlecoding
self-tandem
Direct G.722-64kbit/s
G.722-64kbit/s
Mode 2(12.65kbit/s)
Mode 2(12.65kbit/s)
Mode 8(23.85kbit/s)
Mode 8(23.85kbit/s)
Lab = NokiaLab = BT
Quality comparison of wideband coders including tandeming and transcoding – p 16 research & development France Telecom Group
Compared Subjective WB quality of G.722 & G.722.2 in transcoding
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
- - - - G.722-64 - G.722-64
Direct G.722-64 G.722-48kbit/s
Mode 2(12.65kbit/s)
Mode 2(12.65kbit/s)
Mode 8(23.85kbit/s)
Mode 8(23.85kbit/s)
Lab = FTLab = LMGT
Quality comparison of wideband coders including tandeming and transcoding – p 17 research & development France Telecom Group
Conclusion on tandemming and transcoding
Codecs self tandemings produce quite limited quality degradations of around 0.2 MOS-LQSW.
Transcodings between different wideband formats produce more significant degradation :
G722↔AMR-WB transcoding quality score 0.2 to 0.4 MOS-LQSW below G.722 64 k quality.
Quality comparison of wideband coders including tandeming and transcoding – p 18 research & development France Telecom Group
Comparison od wideband codecs with narrowband codecs
Quality comparison of wideband coders including tandeming and transcoding – p 19 research & development France Telecom Group
Comparison of wideband codecs with narrowband codecs (1)
Extract from the G.729.1 characterisation phase (step1)Experiment 1a
• NarrowbandExperiment 1b
• Purpose : evaluate the performance of G.729.1 algorithm, in wide band clean speech (free of background noise) with a variety of input levels.
• Methodology : Absolute Category Rating (ACR) method with the Mean Opinion Score (MOS) rating scale for WB subjective tests (MOS-LQSW).
• Languages : French & English (US)• Subjects : 32 naïve listeners
Quality comparison of wideband coders including tandeming and transcoding – p 20 research & development France Telecom Group
Comparison of wideband codecs with narrowband codecs (2)
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
DirectNB
DirectWB
G.729A-8k (NB)
G.722 –48k
(WB)
G.722 –56k
(WB)
G.729.1-14 k(WB)
G.729.1-24 k(WB)
G.729.1-32 k(WB)
MO
S-LQ
SM
Lab: VoiceAgeLab : FT
Quality comparison of wideband coders including tandeming and transcoding – p 21 research & development France Telecom Group
Conclusion WB versus NB
Results show that wideband voice, even coded at the lowest bit rates of G.722 (48 kbit/s), gets better score than direct narrow band quality with a gap up to +0.5 MOS-LQSM
MOS-LQSM difference between narrow band and wideband direct speech is greater than 1 MOS-LQSM and remain between 0.5 MOS-LQSM and 1 MOS-LQSM between direct narrow band and high quality wideband coded speech.
Quality comparison of wideband coders including tandeming and transcoding – p 22 research & development France Telecom Group
ITU-T-SG12-TD42rev3(WP1/12), " G.729EV Characterization phase step 2 Quality Assessment Test Plan ", Source: Rapporteurfor Question 7/12, Geneva, 5-13 June 2006ITU-T-SG16-TD258(GEN/16), "LS on testing issues", Source: Rapporteurs Q7/12, Geneva, 14 - 24 November 2006ITU-T-SG16-TD258(GEN/16)-Attachment 2, " Executive summary of G729.1 Characterisation step 2– Experiments 1, 2 & 3. ", Source: France Télécom, Geneva, 14 - 24 November 2006
G.729.1 Characterization step 2 referencesITU-T-SG12-TD22rev2(WP1/12), " G729EV Characterisation/Optimisation step1 Test plan ", Source: Rapporteur for Question 7/12, Geneva, 17-21 October 2005ITU-T-SG16-TD202(GEN/16), " LS on audio issues ", Source: Rapporteurs Q7/12, Geneva, 3 - 13 April 2006ITU-T-SG16-TD202(GEN/16)-Attachment 1, " G729EV Characterisation/Optimisation step1: Summary of results ", Source: France Télécom, Geneva, 3 - 13 April 2006
AMR-WB Characterization referencesETSI TR 126 976 V6.0.0 (2004-12), " Performance characterization of the Adaptive Multi-Rate Wideband (AMR-WB) speech codec", Tdoc S4 (01)0351R1, " AMR Wideband Characterisation Phase 1 Listening Tests, Experiment 1 - BT Results", Source: BT, June 4-8, 2001, Naantali, FinlandTdoc. S4 (01)0326, " Executive Summary from France Télécom R&D for the ETSI AMR-WB Characterisation Phase – Results for Experiments 2 & 5", Source: France Telecom, June 4-8, 2001, Naantali, FinlandTdoc S4 (01)0321, " AMR WB Characterization Experiments 2A and 6A - LMGT Results ", Source: LMGT, June 4-8, 2001, Naantali, FinlandTdoc S4-010353, " Nokia report for AMR-WB characterisation experiments 1A & 6B ", Source: Nokia, June 4-8, 2001, Naantali, Finland
EVRC-WB Characterization referencesITU-T-SG16-TD291(GEN/16) "Updated LS reply on follow-up on embedded extension to G.722.2 and media coding summary database " Source: Chairman SG 16 (on behalf of 3GPP2 TSG-C) Geneva, 14 - 24 November 2006
G.722 Packet Loss Concealment referencesITU-T-SG16-TD217(WP3/16), " Report of Question 10/16 “Software tools for signal processing standardization activities and maintenance and extension of existing voice coding standards”", Source: Rapporteur for Question 10/16, Geneva, 14 - 24 November 2006Extract from the G.729.1 characterisation phase (step1)