On 5/10/2016 we cut over to a new Internet access service, switching from Telstra 8Mb/s ADSL to iiNet NBN 12Mb/s.
Over some years, I have run an automated file transfer to measure the speed of our access service. The tests are done between 6:00 and 22:00, I am interested in performance during the times I want to use the service, and less interested in times when I would usually be sleeping.
One of the performance measures is speed, it seems simple enough especially when it is talked about by politicians and promoted by RSPs and NBN’s advertisements.
So, how does one characterise the speed. People like to think of simple concepts like ‘average’, but let us look at the distribution of speeds.
Above is a chart of the frequency distribution of speed observations for a week in March 2017. It is not in the form of the Normal Distribution, the classic bell-shaped curve for which common parametric statistics like Mean (average) and Standard Distribution are meaningful, but it is a skewed distribution. Without knowing the characteristics of the distribution, it is a misuse of parametric statistics to apply parametric statistics like Mean (average) and Standard Distribution.
So we must rely on non-parametric statistics.
A simple statistic is the median, the “middle” number of a set of ordered observations. 50% of the observations will be above the median and 50% below.
That concept can be applied to give the measure that was exceeded or not by any given percentage of observations. A percentile is a measure used in statistics indicating the value below which a given percentage of observations in a group of observations fall.
Above is a chart of the 90 percentile rank and 10 percentile rank of the services over some years.
A way to interpret the percentile is that the 10 percentile rank is the speed that is delivered 90% of the time, and it you think it is fair that the supplier should deliver the ‘rated’ speed most of the time, 90% of the time, the 10 percentile rank is a key measure.
At the start of the period, Telstra delivered a 90 percentile rank of about 70% of the service speed. That collapsed in the latter half of 2012 when they promoted T-box (IP TV) and closed the mail movie service. IPTV is a higher revenue earner. Service slowly degraded to the point that 10% of the time, transfers in a week ran below 0.5Mb/s (6% of the service speed).
Notwithstanding that Telstra said they could not improve the service, and why did they need to as ACCC had advised “that since speed was no longer stated on the bill, it was not an enforceable element of contract”, they did make some improvement about the time NBN fibre was being installed.
The interesting thing about the period from 05/2015 to 09/2016 was how close the 10 percentile rank is to the 90 percentile rank. The service now seems to have capacity to withstand peak usage by all users, albeit artificially capped at 2Mb/s.
You might ask the question, are they playing with us, turning service up and down at will, and imposing artificial constraints on speed at a level well below their original description of the service.
Above are the observations over the 40 odd weeks since iiNet cutover to the 12Mb/s service.
Note the clear stepwise drop in performance for the first two weeks of November. Service is restored to the October levels until almost three weeks in February runs 40% faster, and less variation from test to test. The October performance is again restored and that has continued until this day.
The two weeks of greatly improved performance in February questions whether the service at other times falls short of what an NBN AVC TC4 access service should deliver.
By early Jun 2017 the speeds had degraded seriously (eBay pages were very slow and often had to be refreshed to get full page display in less than a minute), so not only did the best speeds not exceed about half the ‘headline’ speed, but often measured speed was less than 10% of the ‘headline’ speed and online transactions became appallingly slow.
I reported this to iiNet, and after some days they quietly wrote the problem off as “NFAR” (which probably is code for No Further Action Required).
Not satisfied with this, I raised the matter with the Telecommunications Ombudsman (TIO).
iiNet’s follow up of the TIO matter acknowledged that the service had been congested and that that had been fixed, and they had me conduct a series of tests to support a fault report to nbnCo. My own analysis of those tests were that the nbnCo access service did not appear to be faulty or the cause of the performance problems measured in the previous weeks.
Above is a chart of the 10 and 90 percentile speeds since iiNet cutover.
On about the day of the TIO intervention, there was a step increase in 90 percentile speed to more than 10Mb/s, and observation over the following weeks showed the 10 percentile speed to be close to 10Mb/s, though it can be seen from the graph that it has declined, and if it continues to drop at that rate will drive another cycle of fault report to iiNet and possibly TIO complaint.
We also ditched the bundled iiNet VOIP service because diversion requests are not processed in real time but have latency of several hours. Use of an independent VOIP provider gives the freedom to change RSPs without changing phone number. iiNet’s supplied gateway not only had problems with access to the management screens, but it was locked to iiNet’s VOIP servers… a clever and deceptive lock-in tactic. For these reasons the gateway was replaced with one that has proven reliable and not locked to a provider.
For the moment the service is good, though the trend over the weeks since TIO intervention are a harbinger of performance issues again in the near term.