ISP programming of the (tr)uSDX (trusdx) showed that filtering on the MOSI pin in that kit distorted the MOSI signal significantly and suggested a workaround (reducing SCK rate) for reliable programming.
Some correspondence prompts a little more information on the nature of the ATmega328P ISCP signals.
The line protocol used is actually SPI, quite a common protocol.
ISP uses SPI MODE 0 (CPOL=0, CPHA=0), shift out on the falling edge of SCK, and capture input on the rising edge.
Let’s look at a three channel capture of SCK, MOSI and MISO of a AVRDUDE / USBasp driving an Arduino Nano.
The capture shows SCK at around 750kHz rate, the default (-B1) rate for AVRDUDE in this setup.
In fact, this combination reliably programs at 3Mb/s, ie four time the clock rate above. Note that the maximum SCK rate also depends on the clock rate in the target MCU, it is bad practice to try to set the SCK rate at more than one third of the target clock, so not more than about 5Mb/s in this case.
Note that MOSI and MISO change on the -ve SCK edge and never on the +ve SCK edge. The +ve edge drives capture of the MOSI and MISO at their respective input pins.
The plot above is from the original article and shows MOSI distorted significantly by the heavy pin filtering.
The pin filtering in that application is for a reason, the MOSI pin is shared with another purpose… so the fix is to reduce the SCK rate so that the slow rise does not result in errors at the instant of capture of each bit.
Now you could look at the SPI signals using a logic analyser and SPI is a protocol that is decoded by lots of low cost logic analysers (eg the <$15 8ch logic analysers that work with sigrok). But realise that the logic analyser has its own threshold for capture of input and under degraded signals like that above, it might well decode bits differently to the target.