On negative VSWR (read it first) discussed the case of negative VSWR results from some calculating tools and formulas, and more generally that simple formulas that depend on lossless line assumptions produce errors on practical lossy line scenarios.
Return Loss is defined as the ratio Pfwd/Prev, often given in dB.
Return Loss is usually calculated as 20*log(1/ρ), it yields negative calculated Return Loss for ρ>1. It would be a mistake to doctor the result to hide the negative return loss as it is a strong hint that the results may be invalid.
An important consideration here is the validity of the concept of Pfwd and Prev.
The forward and reverse waves are subject to superposition, but it is voltages and currents that superpose, not power.
The relation that P=Pfwd-Prev is valid only when Zref is purely real (see Power in a mismatched transmission line).
So, to speak in Return Loss is really only valid when Zref or Zo is purely real or approximately so.
Measurement instruments are almost always calibrated for a purely real Zref.
Almost all practical low loss transmission lines have approximately real Zo above 50MHz, and most are good down to 1MHz. Transmission lines using poor conductors, or composite conductors with thin cladding may depart significantly below 50MHz.
On negative VSWR – a worked example discussed various aspects, but did not touch on Return Loss.
Above is the plotted Return Loss (wrt Zo=50.71-j8.35Ω). The negative value plotted near to the load flags the underlying problem that Zo is not real, and therefore the calculated Return Loss may be invalid.