[<<][softarch][>>][..]

Mon Jun 22 13:02:47 EDT 2020

testability for low-level code

Trying to distill some lessons from last week's struggles, trying to
go fast when developing low level code.

The concrete context of this is a network of one "decent computer",
i.e. a linux box that can basically run any kind of high level
language to implement a test framework, and a bunch of low cost bare
metal memory-constrained high traffic networked devices that are
difficult to monitor.

Some rules:

1. Don't try to go fast by cutting corners on automated testing.

2. Use ONE software-configurable test haredware setup.  Do not build
   anything that requires manual intervention.  Invest the time
   up-front to make it completely software configurable such that
   configuration can be recorded in source control, and the test
   system can be specified and duplicated.  Time spent on this will be
   less than what would otherwise be lost in debugging anyway.

2. Spend the time to build assert tests.  Don't just rely on visual
   inspection of test logs or scope traces.  If necessary, add
   statistics gathering and special state queries to the bare metal
   devices, to allow assert testing on embedded device states.  Same
   argument as 1 wrt. debugging time: this is expensive and slow, but
   it pays off.

3. Try to split development in two phases: feature append only, and
   refactoring.  During append only work, also edit the append-only
   test suite.  This will then provide a test for everything that was
   developed, and can later be used as a regression test during
   refactoring.

   A possible exception: it is allowed to make the test configurable
   to _temporarily_ disable some tests to be able to zoom in onto an
   implementation issue, or to make it use less time, but then TURN IT
   ALL BACK ON AGAIN.

   The idea is to have ONE test that is kept running in all
   circumstances and doesn't cost any effort to run.  Simple
   pass/fail.

4. Writing the firmware is almost trivial if 1. you keep it simple and
   2. you have good tests that catch issues immediately.

5. Create a mechanism that can encapsulate assertions so they can be
   sent back to host.  I.e. each embedded device should have some kind
   of observer state machine of which the state needs to be queryable.

6. And last but not least: it is very easy to fool yourself.  At least
   for me.  Measure.  Be explicit about properties.


To boil it all down: refactoring is necessary. If your implementation
subtrate can not catch simple programming errors, you need a test
suite you can rely on to ensure that your application still works.

Another trick is pipelined/feedforward asserts, i.e. send command, let
controller change state and compute assert locally, send report
upstream, fail test suite on failed report.

[Reply][About]

[<<][softarch][>>][..]