Mon Jun 22 13:02:47 EDT 2020
testability for low-level code
Trying to distill some lessons from last week's struggles, trying to
go fast when developing low level code.
The concrete context of this is a network of one "decent computer",
i.e. a linux box that can basically run any kind of high level
language to implement a test framework, and a bunch of low cost bare
metal memory-constrained high traffic networked devices that are
difficult to monitor.
1. Don't try to go fast by cutting corners on automated testing.
2. Use ONE software-configurable test haredware setup. Do not build
anything that requires manual intervention. Invest the time
up-front to make it completely software configurable such that
configuration can be recorded in source control, and the test
system can be specified and duplicated. Time spent on this will be
less than what would otherwise be lost in debugging anyway.
2. Spend the time to build assert tests. Don't just rely on visual
inspection of test logs or scope traces. If necessary, add
statistics gathering and special state queries to the bare metal
devices, to allow assert testing on embedded device states. Same
argument as 1 wrt. debugging time: this is expensive and slow, but
it pays off.
3. Try to split development in two phases: feature append only, and
refactoring. During append only work, also edit the append-only
test suite. This will then provide a test for everything that was
developed, and can later be used as a regression test during
A possible exception: it is allowed to make the test configurable
to _temporarily_ disable some tests to be able to zoom in onto an
implementation issue, or to make it use less time, but then TURN IT
ALL BACK ON AGAIN.
The idea is to have ONE test that is kept running in all
circumstances and doesn't cost any effort to run. Simple
4. Writing the firmware is almost trivial if 1. you keep it simple and
2. you have good tests that catch issues immediately.
5. Create a mechanism that can encapsulate assertions so they can be
sent back to host. I.e. each embedded device should have some kind
of observer state machine of which the state needs to be queryable.
6. And last but not least: it is very easy to fool yourself. At least
for me. Measure. Be explicit about properties.
To boil it all down: refactoring is necessary. If your implementation
subtrate can not catch simple programming errors, you need a test
suite you can rely on to ensure that your application still works.
Another trick is pipelined/feedforward asserts, i.e. send command, let
controller change state and compute assert locally, send report
upstream, fail test suite on failed report.