WSPR for A/B tests – a discussion – part 1

The WSPR  User Manual sets out the purpose of WSPR:

The WSPR software is designed for probing potential radio propagation paths using low power beacon-like transmissions.

Though that talks about measuring radio paths, it is often used to compare transmitters or receivers over radio paths.

WSPR SNR measurements include the end to end radio path, which on some bands is highly variable, so using WSPR reported SNR values to compare two transmitters can be quite challenging.

A/B tests

We are all familiar with ad-hoc tests where a station might switch between two antennas and ask for comparative reports from receiving stations. At time when the radio path characteristics change greatly, changes in transmitter are often masked or confused by path variation.

Of course some practitioners will conduct several so-called A/B changes, perhaps as many as five and someone (receiver or transmitter) makes an informal judgement of the central tendency of the observations. The observations might be given in quite subjective terms, or in quantitative terms, possibly from an S meter of unknown calibration.

Normal distribution

Repeated measurements of the same thing, or same type of thing (eg 10 measurements of 1 new dry cell, or one measurement each of 10 new dry cells) tend to yield a set of slightly different observations.

For a lot of common physical things, the distribution of repeated measurements follows a bell shaped probability curve.

Most things that we repeatedly measure will return slightly different results from observation to observation due to various contributions in an imperfect world.

Above is a plot of the probability distribution of a normally distributed random variable with mean=1 and variance=1 (standard deviation=1).

There is a wealth of statistical techniques that can be applied to normally distributed data.

Log-normal distribution

Whilst the normal distribution is very common. some phenomena exhibit a distribution where the log of the variable is randomly distributed, a log-normal distribution.

An experiment

G3CWI recently conducted an experiment where two WSPR transmitters were combined to a single antenna, and observations collected from receivers that decoded both transmissions in a WSPR 2 minute measurement interval. There are more than 4000 paired observations of A, B and B-A.

In fact, the difference data B-A contains more information than the sets A and B in isolation, the pairing of the observations makes for increased statistical power and reduced confounding effects.

Above is a frequency histogram of the experiment log. You might notice a resemblance to the normal curve shown earlier. It is in fact an approximately log normal response to S/N, but normal response to S/N expressed in dB.

The parametric statistical methods that can be used for normally distributed data can be used with log-normal distributions (with appropriate log adjustments).

We will consider the WSPR SNR in dB to be approximately normally distributed (though the underlying SNR is log-normal), which leads to the question “how approximately?”

Continues at WSPR for A/B tests – a discussion – part 2.