There are a plethora of articles and presentations on the ‘net showing how to use the Reverse Beacon Network (RBN) to make quantitative antenna comparisons over real propagation paths.
It is certainly an interesting subject to most hams with a deep interest in antenna systems.
So called A/B comparisons of antennas are as old as ham radio itself, and experience hams know that they are quite flawed.
Because ionospheric propagation paths vary from moment to moment, the challenge is to make a measurement that is directly comparable with one made at a slightly different place, or frequency or time. Accuracy is improved by making several measurements, and finding a central value, more observations tends to reduce uncertainty in that estimate of the population central value.
The challenge is finding that central tendency.
There are three common methods of estimating the central tendency of a set of figures:
- mean (or average);
- median (or middle value); and
- mode (or most common value).
The mean is a popular and well known measure of central tendency. It is a very good estimate of the central tendency of Normally distributed data, and in that case, we can compare means and calculate confidence levels for assertions about the difference between means. The mean is very susceptible to errors due to outliers, and skewed distributions.
The median is usually a better measure for skewed data.
The mode is if you like, the most frequent or popular value and has a great risk of being quite misleading on this type of data.
A recent article (Appleyard 2018) in Radcom provides a useful example for discussion.
Appleyard gives a summary table where he shows means of a set of RBN measurements of signals from two stations observed at 21 remote stations, and differences in those means.
There are some inconsistencies between the text and data recorded in the RBN database on the day.
It appears likely that callsign MX0NCA was used for the inland station, and the RBN shows 10 reports by DF7GB on that morning.
By eye, the full set of 10 observations do not appear to be Normally distributed, and in fact the IQR at 10.0dB is some 40% wider than would be expected of normally distributed data. A more sophisticated test for Normality is the Shapiro-Wilk test, and it gives a probability that falsely rejecting that the data is drawn from a Normally distributed population is 0.96%, in plain speak it is very unlikely that the 10 observations were drawn from a Normally distributed population. For this reason, the mean is not a very good estimator of its central tendency, and operations like finding the difference of the means (as shown in the table) is not valid.
For this data set, the mean is 15.5, median is 18 and mode is 20. What do you think the central tendency is of the graphed data? The median is probably a better estimate of the central tendency of this data (but note that there is no basis for taking the difference of the medians).
It appears that some data may have been excluded from the table summary as the given mean value of 14.8 is different to that of the full set of 10 observations.
An important attribute of Normally distributed data is that the mean of the sum (or difference) of two normally distributed variables is the sum (or difference) of the means of each.
The most important consequence of this is that since the antenna means are of non-Normal data, calculated difference in the third column is not a valid indicator of the difference in the antennas observed at that receiver.
Whilst it is not valid to find the difference of the means of non-normal data, the individual paired SNR observations may reveal a strong relationship between the two antennas.
Appleyard gives another summary table where he shows means of a set of RBN measurements of signals from four stations observed at 16 remote stations, and differences in those means.
The question again arises whether the observations are normally distributed, whether the mean is a good measure of central tendency. It is a rather complex two way table of measurements, one that probably cannot use methods that depend on Normal data.
The observations by each of the 16 remote sites could be seen as independent clusters of measurements of each of the transmitters.
Above, a plot of the means (even if they are not a good measure) doesn’t suggest a clear winner, and it can be seen that a transmitter that is clearly better at some remote sites, is not at others. Are the apparent differences due mainly to the antennas, or are they obfuscated by other variables.
The data defies a quantitative measure of the differences of the antennas with declared confidence limits.
So, you might ask, “what does this tell you about the relative merits of the various antennas tested?”
Count of observations
In terms of the short-term variation in S/N, we have found that the averaging of at least three successive reports mostly takes out these perturbations.
The analysis above of the set of 10 where 50% of the S/N observations were spread over 10dB (the IQR) contradicts that position.
Too few observations gives very wide uncertainty in any conclusions and it tends that for non-parametric analyses, even more observations are necessary.
Non-parametric studies comparing S/N observations in a two way analysis in another context become of useful accuracy with hundreds of paired observations, and that would seem to be impractical for RBN sourced observations.
Good experiments don’t usually happen by accident. The questions to be answered (the null hypotheses in statistical terms) need to be though through and the experiment design to capture enough data to hopefully provide valid results.
Capturing field data is an expensive process, and ability to do a first pass analysis while the experiment is set up can help avoid a wasted venture.
When the observation data cannot be shown to be Normally distributed, means are not a good measure of central tendency, and the whole raft of parametric statistical techniques premised on Normal distribution are unavailable.
It is likely that non-parametric techniques are needed for analysis, and the sheer volume of observations might not be practical from RBN.
References / links
- Appleyard, S. Jun 2018. Using the reverse beacon network to test antennas In Radcom.