Estimation of Quality Judgments

Auditory tests like the ones described in the previous sections are expensive and time-consuming; thus, we checked the available quality prediction models as to whether they provide valid estimations also for NGMN handover situations. Two different types of models have been checked: A parametric model which estimates quality on the basis of the parameters describing the network conditions, and a signal-based model which estimates quality as a perceptually-weighted distance between the input and the output signals of the network under test.

We used an extended version of the E-model as a parametric model. It is based on the original algorithm described in ITU-T Rec. G. 107 for NB networks and has been modified to take into account WB transmission (by linearly extending the underlying transmission rating scale from 100 to 129), WB speech codecs (by defining codec-specific equipment impairment factors), and packet loss. The necessary modifications are recommended in the most recent update of ITU-T Rec. G. 107, Amendment 1. As the E-model does not yet consider NB↔WB transitions, we decided to calculate two separate scores for the samples in which such NB↔WB transitions occur, then calculate an average of the underlying transmission ratings, and transform this back to the MOS scale. As a signal-based type of model, we used the WB extension of the PESQ model which is described in ITU-T Rec. P.862.2 .

It has to be emphasized that both types of prediction models have not yet been validated for the scenarios investigated here. In addition, they estimate instantaneous speech quality of short samples, and not of entire conversations. Thus, we applied the predictions only to Tests 1b and 2b. The parametric E-model uses packet loss as an input parameter, which has not been manipulated in a controlled way in Test 2b; thus, for this test, only the signal-based model can be used. We compare the model estimations to the auditory test results in terms of MOS, and calculated the Pearson correlation r and the root mean squared error σ for each test. The results are given in Table 1.

Table 1: Pearson Correlation and Root Mean Squared Error between auditory and estimated MOS 

Test	WB E-model		WB-PESQ
	r	σ	R	σ
1b	0.58	1.44	0.96	0.44
2b	n.a.	n.a.	0.89	0.70

The parametric E-model does only use general information on the network condition (average packet-loss percentage Ppl, codec type) as an input; consequently, the predictions are not accurate. In contrast to this, WB-PESQ is able to recognize speech signal degradations caused by the network handover, codec changeover and packet loss. For Test 1b, this leads to very good prediction accuracy, even better than the value of r = 0.93 which is obtained for in-scope data (Rix et al., 2006). The prediction accuracy is slightly lower for Test 2b which contains conditions with G.722.2 coding at 12.65 Kbit/s; WB-PESQ has been shown to have bigger problems in predicting the effects of this codec compared to the 23.05 Kbit/s bit-rate used in Test 1a/b.

Description of the Experiments | 4G Networks

In NGMN scenarios, speech quality is expected to vary during a call, as a result of changing conditions in the connection, network handover and/or codec changeover. Thus, in order to quantify quality, the entire length of a call has to be considered. Standard listening-only tests which make use of speech samples of 4-8 s length (ITU-T Rec. P.800, 1996) are not suitable for this purpose. On the other hand, conversational tests - despite being comparable to normal telephone usage and thus being ecologically valid - place a content-related focus on the user's attention; in such a situation, users are generally less analytic in their judgments, and it might happen that subtle perceptual differences get blurred.

As a compromise, we opted for a twofold test protocol: (a) We simulated conversations of 60s length by concatenating 5 meaningful speech segments alternating with pauses, playing them back to the test participants, asking them to answer content-related questions during the pauses, and asking for an overall quality judgment at the end of the simulated conversation; this approach has been developed by Berger and is now recommended for call-quality measurement in ETSI Technical Report 102 5 06 v. 1.1.1 (2006); (b) The composing segments of the simulated conversations and some additional segments of approx. 6 s length were presented to the participants in a standard listening-only context, asking for an overall quality rating after each sample. We carried out two tests of the first type (Tests la and 2a) and two corresponding tests of the second type (Tests lb and 2b). The following subsections describe the test conditions, set-up and participant group in more detail.

Test Conditions

In the experimental work, Test 1a concentrated on WB/NB transitions and the effects of packet loss on perceived quality. It contained two conditions with pure NB and WB calls, 4 conditions where packet loss continuously increases until the middle (3rd segment) of the call, and then switching occurs to a loss-free network with a different codec (or not), and 4 conditions where NB→WB or WB→NB transitions occur at the beginning (2nd segment), or at the end (4th segment) of a call. We consider packet loss rates of 10-20%to be realistic constraints when a handover should be executed at latest. Table 1 summarizes the conditions. The corresponding Test 1b contains all segments of the simulated conversations, plus additional samples with similar degradations, addressing also Flash-OFDM networks. The resulting list of 26 segments for this test is presented in Table 3.

Table 1: Test la conditions. H: HSDPA; W: WLAN; Ppl: packet loss in %; switching at the beginning (beg.), middle (mid.) or end of a simulated call 

No.	Network(s)	Codec(s)	Ppl per segment
1	H	G.711	0
2	W	G.722.2	0
3	H	G.711	0,10,20,10,10
4	W	G.722.2	0,10,20,10,10
5	H→W mid.	G.711→G.722.2	0,10,20,0,0
6	W→H mid.	G.722.2→G.711	0,10,20,0,0
7	H→W beg.	G.711→G.722.2	0
8	H→W end	G.711→G.722.2	0
9	W→H beg.	G.722.2→G.711	0
10	W→H end	G.722.2→G.711	0

Test 2a was designed to put a magnifier on the most interesting findings of the first test. It focused on the switching position within a simulated conversation, as well as on additional packet loss rates (see Table 2). The corresponding Test 2b with short samples also included different network load and high packet-loss-rate scenarios for limited WLAN networks (overall 27 conditions).

Table 2: Test 2a conditions. See Table 1 for explanations 

No.	Network(s)	Codec(s)	Ppl per segment
1	W	G.722.2	0
2	H	G.711	0
3	H→W beg.	G.711→G.722.2	0
4	H→W mid.	G.711→G.722.2	0
5	W→H mid.	G.722.2→G.711	0,3,3,0,0
6	W→H mid.	G.722.2→G.711	0,5,5,0,0
7	W→H mid.	G.722.2→G.711	0,10,10,0,0
8	W→H→W	G.722.2→G.711	0
9	H→W→H→W	G.711→G.722.2	0
10	W	G.722.2	0,0/5,5,5/0,0

Table 3: Test 1b conditions. H: HSDPA; W: WLAN; F: Flash-OFDM; Ppl: packet loss in %; codec switching directly before (b.) or after (a.) network handover 

No.	Network(s)	Codec(s)	Ppl per network
1	F->H	G.722.2	0
2	F->H	G.722.2	10,0
3	F->H(a.)	G.722.2->G.711	0
4	F->H(b.)	G.722.2->G.711	0
5	H	G.711	0
6	H	G.711	10,0
7	H	G.711	20,0
8	H	G.711->G.722.2	0
9	H	G.722.2->G.711	0
10	H->F	G.711	0
11	H->F	G.711	10,0
12	H->F(a.)	G.711->G.722.2	0
13	H->F(b.)	G.711->G.722.2	0
14	H->W	G.711	0
15	H->W	G.711	10,0
16	H->W(a.)	G.711->G.722.2	0
17	H->W(b.)	G.711->G.722.2	0
18	H->W(b.)	G.711->G.722.2	20,0
19	W	G.722.2	0
20	W	G.722.2	10,0
21	W	G.722.2	20,0
22	W->H	G.722.2	0
23	W->H	G.722.2	10,0
24	W->H(a.)	G.722.2->G.711	0
25	W->H(b.)	G.722.2->G.711	0
26	W->H(b.)	G.722.2->G.711	20,0

Test Setup

Tests 1a/b and 2a/b were carried out at distinct points in time, with different participant groups. Test participants were invited to a sound-insulated laboratory, were instructed about the purpose of the test, and listened to the samples in three sessions of approx. 25 min. each (2 sessions for parts a, 1 session for parts b). Speech samples were presented over a Sennheiser HMD 410 headset at a comfortable listening level, with a background level below 35 dB(A) (ITU-T Rec. P.800, 1996). At the end of each simulated conversation of part a, as well as after each sample of part b, participants had to rate the overall quality on a 5-point absolute category scale, with 5 corresponding to "excellent" and 1 to "bad" quality. The test set-up and scale followed mainly the requirements given in ITU-T Rec. P.800  and ETSI Technical Report 102 506 v. 1.1.1. 13 participants took part in Test 1a, 24 in Test 1b, 14 in Test 2a, and 17 in Test 2b. They were recruited from the normal telephone-user population, did not report any hearing impairment, and received a voucher in return for their effort.