Drive test campaign results – Second semester 2018

IBPT & Commsquare

Introduction

When considering the quality of mobile networks and communication services, we think in terms of network availability (coverage), phone calls quality and quality of data services over the Internet, such as streaming and web browsing.

Since 2015, BIPT has been publishing online maps on mobile coverage showing where services are available outdoors. Since 2019, these maps include several availability levels depending on whether you are indoors or outdoors.

A good coverage constitutes a prerequisite for good quality services. However, this is not always sufficient. Several factors may influence the quality of experience perceived by service users. The way the operator configures its network will thus play a significant role.

By enhancing the transparency on the quality of experience on the different networks, BIPT wishes to enable consumers to make better choices. The purpose is also to help operators identify the aspects related to the quality of experience requiring improvements to their networks and possible investments.

Methodology

To objectively compare the performance of the three mobile networks in Belgium in terms of quality of experience, BIPT launched a measurement campaign on the roads. Based on an objective and transparent method, BIPT’s study compares the operators according to 17 quality indicators.

Roads were selected to obtain results corresponding to the experience of an average user in Belgium. The routes are spread over the whole territory, with a higher number of measurements in the most densely populated areas. Rural areas are also considered, with a weighting factor proportional to their population.

The first set of indicators relates to voice quality. Customer experience during phone calls was tested: what is the risk of not being able to set up a call, what are the percentage of successful calls with no interruption, what is the sound quality perceived by the user, etc.? The second set of indicators relates to the data transmission quality. The purpose here is to compare download speeds, video streaming quality, browsing quality and the ease of use of social networks.

The approach is a valid approach but has some inherent known limitations. The results in this report are based on measurements in September 2018. Mobile networks are very dynamic and undergo constant changes and improvements. Therefore, the observations and conclusions are a valid snapshot of mobile network performance in Belgium during the measurement period.

Results concerning the voice telephony service

The performance of voice services is expressed in 3 categories of service indicators.

Results are summarised in the table below.

Voice Summary
Success Rates
Orange Proximus Telenet
Call setup success rate (%)
99.8% 99.7% 99.7%
Call completion rate (%)
99.4% 99.2% 99.5%
Uninterrupted call rate
98.2% 98.7% 99.4%
Call Setup Time (s)
Orange Proximus Telenet
Average
4.6 3.7 4.7
10% longest
5.8 4.7 5.4
Voice Quality
Orange Proximus Telenet
Average
3.7 3.6 3.0
10% lowest
2.9 2.8 2.5
Voice Summary Orange Proximus Telenet
Success Rates
Call setup success rate % 99.8% 99.7% 99.7%
Call completion rate % 99.4% 99.2% 99.5%
Uninterrupted call rate % 98.2% 98.7% 99.4%
Call Setup Time
Call Setup Time (s) Average 4.6 3.7 4.7
Call Setup Time (s) long samples 10% longest 5.8 4.7 5.4
Voice Quality
Voice Quality Average 3.7 3.6 3.0
Voice Quality low samples 10% lowest 2.9 2.8 2.5

Summary on the performance of voice telephony services

All operators have good performance for call setup success and completion rate indicators. There are no statistically significant differences between the networks.

Telenet has the lowest uninterrupted call rate, i.e. no gaps in the conversation. Orange has the highest number of interrupted calls.

Orange and Proximus offer high definition voice and hence have the highest voice speech quality. Orange has a higher voice quality score than Proximus. Telenet has the lowest voice quality score.

Proximus has the lowest call setup time, with Orange and Telenet roughly 1s slower.

Results concerning data transmission services

The performance of data transmission services is expressed in 5 categories of service indicators.

Results are summarised in the table below.

Data Summary
Orange Proximus Telenet
Throughput
HTTP DL fixed duration
DL throughput (Mbps, average)
58 74 93
Slowest 10% (Mbps)
9.6 10.1 14.9
HTTP UL fixed duration
UL throughput (Mbps, average)
18.9 20.2 22.0
Slowest 10% (Mbps)
2.4 3.1 2
File Transfer
HTTPS DL fixed size 10MB
DL time (seconds, median)
2.7 1.5 2.3
Success rate (%)
98.3% 98.0% 96.7%
HTTPS UL fixed size 5MB
UL time (seconds, median)
2.7 2.7 2.5
Success rate (%)
95.6% 96.7% 93.7%
Web Browsing
All web pages (blended KPI)
Web browsing time (seconds, median)
1.9 1.8 1.9
Success rate (%)
99.4% 99.3% 98.9%
YouTube Video (buffered streaming)
Success Rate (%)
99.5% 99.4% 99.1%
Time to 1st picture (seconds, average)
1.6 1.3 1.5
Video Quality (average)
4.4 4.4 4.4
Facebook
Success Rate (%)
99.2% 99.6% 99.4%
Total time (seconds, average)
2.1 2.1 2.2
Data Summary Orange Proximus Telenet
Throughput
HTTP DL fixed duration DL throughput (Mbps) Average 58 74 93
Slowest 10% 9.6 10.1 14.9
HTTP UL fixed duration UL throughput (Mbps) Average 18.9 20.2 22.0
Slowest 10% 2.4 3.1 2
File Transfer
HTTPS DL fixed size 10MB DL time Median (s) 2.7 1.5 2.3
Success rate (%) 98.3% 98.0% 96.7%
HTTPS UL fixed size 5MB UL time Median (s) 2.7 2.7 2.5
Success rate (%) 95.6% 96.7% 93.7%
Web Browsing
All web pages (blended KPI) Web browsing time Median (s) 1.9 1.8 1.9
Success rate (%) 99.4% 99.3% 98.9%
YouTube Video
YouTube – buffered streaming Success Rate  % 99.5% 99.4% 99.1%
Time to 1st picture Average (s) 1.6 1.3 1.5
Video Quality Average 4.4 4.4 4.4
Facebook
Facebook Success Rate % 99.2% 99.6% 99.4%
Total time Average (s) 2.1 2.1 2.2

Summary on the performance of data transmission services

All mobile networks offer very good to excellent data performance. There are small variations in performance (possibly the result of design choices made by the operators), with one network offering higher network throughput and the other networks with somewhat higher application success rates.

As to network throughput, Telenet has the highest average download and average upload throughput. The highest throughput is not entirely reflected in the time to download or upload files, where Proximus has the fastest download time and Telenet the fastest upload time. Proximus and Orange have higher success rates for file transfers than Telenet.

For typical applications, such as web browsing, YouTube video and Facebook, the success rate is high and is testament to reliable networks: for Orange and Proximus above 99% for all apps, for Telenet just below or above 99% as well. The various load times for different apps vary between 1.3s to 2.2s, with Proximus better by 0s to 0.3s maximum.

Conclusions

The performance of Belgian mobile networks was assessed, simulating customer experience. The measurements were conducted by driving 2 cars during 250 hours across the entire country in September 2018. Performance was measured in all major cities and highways, as well as in a selection of medium size cities, villages and roads. A total distance of approx. 12,000 km was driven; over 8,000 voice calls were made on each network; and roughly 60,000 data tests were conducted per network. The measurements were configured such that these are representative of customer experience in indoor conditions (i.e. meaning in houses and buildings, close to a window, or in-car).

For voice performance, call setup success and completion rates are close, with Telenet (99.5%) followed by Orange (99.4%) and Proximus (99.2%). As to uninterrupted calls, Telenet users experience very few interruptions, with an uninterrupted call rate for Telenet of 99.4%, followed by Proximus (98.7%) and Orange (98.2%). Proximus is leading in terms of the call setup time by roughly 1s ahead of the competition. For voice quality, on a scale from 5 (excellent) to 1 (poor), Orange (3.7) and Proximus (3.6) are followed by Telenet (3.0).

As to data performance, Telenet has the highest average throughput (93Mbps down, 22Mbps up), followed by Proximus (74Mbps down, 20.2Mbps up) and Orange (58Mpbs down, 18.9Mbps up). For the download of a 10MB file, Proximus (1.5s) is faster than Telenet (2.3s) and Orange (2.8s). For the download and upload of a file, Proximus has the highest success rates (98.0%/96.7%), followed by Orange (98.3%/95.6%) and Telenet (96.7%/93.7%). For other applications, such web browsing, YouTube video and Facebook, Proximus has a faster time than the other networks by 0 to 0.3s. For most data applications, Proximus and Orange present higher success rate, i.e. the probability to complete the intended action, than Telenet.

We observed performance differences between the different networks. However, there is not a single network outperforming the other networks on all service indicators we have measured. These conclusions are valid based on the measurements made in September 2018, simulating indoor coverage conditions. Performance deep into buildings will be different.

Based on our international experience and considering the combination of all service indicators we have measured, all mobile networks in Belgium demonstrate very good performance.

IBPT — Ellipse Building
Boulevard du Roi Albert II, 35
1030 Bruxelles


Tél. : 02 226 88 88
Fax : 02 226 88 77
info@ibpt.be