APRS duplicate removal – trial #2

The undetected long-delayed duplicate posits that are a feature of APRS VHF are a significant corruption of mapping.

Program code has been written to collect posit records direct from APRS-IS and where the record contains a HMS timestamp (as mine do), to apply a timestamp consistent with that timestamp, and then to not write records with duplicate timestamps to the database.

Screenshot - 04_02_16 , 15_57_38

The above map has two journeys, one a loop to the north of home to Mittagong, and another SW to Marulan and back. The Marulan trip extended to 40km from the prominent digi VK2RHR-1, and coverage in the area became poor, evidenced by the poorer APRS mapping in that region.

The map can be browsed interactively at http://owenduffy.net/map/20160204.htm. The red trace is APRS VHF derived using the processing detailed below.

OpenLog07

The cyan trace is from stand alone GPS logger above attached to the case of the TinyTrak 3+. The transmitter is 65W output to a quarter wave in the middle of the wagon roof.

On the Mittagong segment, 13/74  (18%) of the raw posit records from findu had delay of more than 30s, up to 138s. up to 2km positional error over most of this trip, >4km at highway speed. trip radius 3.2km. These duplicates were all due to the path via VK2RWG-1 and VK2KAW.

On the Marulan segment, 19/157 (12%) of the raw posit records from findu had delay of more than 30s, up to 191s. up to 6km positional error over most of this trip. These duplicates were all due to the path via VK2RWG-1 and VK2KAW.

The following is an example delayed (191s) duplicate record from APRS-IS.

20160203230921,VK2OMD-8>APT314,VK2RHR-1,WIDE1*,WIDE2-2,qAR,VK2OMD-3:/230916h3439.15S/15004.06E>251/000/A=002073
20160203231227,VK2OMD-8>APT314,VK2RHR-1,WIDE1,VK1RGI-1,VK2RWG-1,WIDE2*,qAS,VK2KAW:/230916h3439.15S/15004.06E>251/000/A=002073

The following is the PERL code used to fetch posits from APRS-IS and write a SQL database with corrected  / no duplicate timestamps.

#!/usr/bin/perl

use Ham::APRS::IS;
use Ham::APRS::FAP qw(parseaprs);
use Date::Format;
use DateTime::Format::Strptime;
use DBI;
use Data::Dumper;

my $is=new Ham::APRS::IS('sydney.aprs2.net:14580','N0CALL','appid' =>'aiposit v0.01','filter' =>'p/VK2OMD-8');

$username='';
$password='';
$dsn="dbi:SQLite:dbname=aprs"; #SQLite3
#$dsn="dbi:Pg:dbname=aprs"; #postgres

#connect to db
$dbh=DBI->connect($dsn,$username,$password) or die $DBI::errstr;

#connect to APRS-IS
$is->connect('retryuntil' => 3) || die "Failed to connect: $is->{error}";

#define strp format
my $strpgt=DateTime::Format::Strptime->new(pattern=>'%H%M%S',time_zone=>'UTC',);

while(1){  
  $tnc2 = $is->getline_noncomment(600);
  next if (!defined $tnc2);
  if($tnc2!~m%^[^:]+?:[/!=@'`]%m){next;}; #skip if not posit
  $src=$tnc2;
  $src =~s/^(.+?)>.*/$1/mg;
  $t1=time();
  if($tnc2=~m!:[/|@]([[:digit:]]{6})h!m){ #has HMS?
    $ts2=$1;
    my $tg=$strpgt->parse_datetime($ts2);
    $correction=($tg->epoch%(24*3600)-$t1%(24*3600));
    if($correction>-12*3600){$correction-=24*3600};
    if($correction<-12*3600){$correction+=24*3600};
    $t1+=$correction;
    }
  else{
    next; #skip if no HMS ts
    }
  #write text record
  $timestamp=time2str("%Y%m%d%H%M%S",$t1,'UTC');
  printf("%s,%s\n",$timestamp,$tnc2);

  #write db record
  my $sth=$dbh->prepare("SELECT EXISTS(SELECT 1 FROM posit WHERE timestamp='$timestamp' and src='$src')");
  $sth->execute();
  my $res=$sth->fetch();
  $sth->finish();
  if(@$res[0]==0){ #not duplicate
    $tnc2 =~ s/'/''/mg;
    my $sth=$dbh->prepare("INSERT INTO posit VALUES ($timestamp,'$src','$tnc2')");
    $sth->execute();
    $sth->finish();
    }
  }
  $dbh->disconnect(); 
  $is->disconnect()||die "Failed to disconnect: $is->{error}";

The red trace is about as good as APRS VHF gets, the reporting rate is above recommendation and network generated delayed duplicates have been detected and discarded. Though posits are ‘real time’ the map processing here to remove duplicates was batch processing and neither aprs.fi or findu.com utilise HMS timestamps in posits to detect and discard delayed duplicated posits.

The delayed duplicates are initiated by paths with latency, causes include mute stuck open on interference on substandard digis using carrier operated squelch (most digis), iGates / TNCs with flow control problems (eg some Kantronics), defective iGate software that slightly modifies some packets (eg aprsd)). APRS is designed to detect and discard duplicates within a 30s window… but clearly that window is not sufficient.

Whilst it is possible to identify likely causes of degraded APRS network performance, it is naive in the extreme to think that would lead to a fix. Most of the network is not monitored, it is not maintained, there is no coordination network of players in the APRS network. A possible solution to make a ill-conceived architecture and poor implementation work better is to build the core infrastructure more intelligently to discard corrupted and stale records. Further improvement could be achieved by filtering out traffic that has passed through identified corrupting digis and iGates to clean the traffic and possibly encourage rectification by owners of problems that are long-standing.

There is the information in posits that use HMS timestamps to apply a correct time within a 24h window. Clearly, the designer of APRS to have DHM or HMS or none didn’t regard it worthwhile to provide a more unambiguous timestamp, broadly the APRS core washed its hands of a time dimension on posit reports.

Conclusions

  • The architecture of the wider APRS network makes it vulnerable to corrupted posits including delayed duplicated posits and without a robust implementation of eradicating them in the core infrastructure and preventing them in the future.
  • It is possible to use optional APRS v1.0 compliant HMS timestamps to provide a unique key (per each source callsign) for detection of duplicates over a +/-12h window.
  • If you want good maps from APRS, use HMS timestamps and be prepared to post process raw posit records to discard delayed duplicates.
  • If you want better maps, albeit not real-time, use a standalone logger to archive the NMEA data, with or without a tracker.

In this part of the world, APRS VHF digi coverage is reducing, a harbinger perhaps of demise of APRS VHF, further reason to use a stand along NMEA logger to document journeys if that is desired. Another option is discussed at The future of APRS is here….

References

Wade, I. Aug 2000. APRS protocol reference v1.0.