Sat Oct 28 18:28:11 EDT 2006

pipes and deadlock

problem with multiprocessing: buffers need to be big enough to avoid
deadlock if 2 processes are trying to send to each other at the same

ways to solve this:

1) thread decouple reads
2) thread decouple writes
3) software flow control
4) write while waiting

is this an occasion to get rid of all threads?

3) merely avoids the problem. 1) and 2) use threads and can be quite
complicated. it seems number 4) is the easiest to get right: handle
writing at wait points, which means, select and read.

this does require some form of context saving for each postponed
write, so i do wonder if it's not easier to just use threads, since i
use libc ops everywhere: can't pre-empt them without writing to a
memory buffer first.

there is no need for a solution as long as the connectivity graph is
acyclic, which happens in quite a lot of applications that come to
mind, so maybe i just need a special 'write-atom-buffered' word which
buffers exactly one atom?

this will still give proper control flow (it will block on the 2nd
atom if the first one is never read)

so, i cannot give up atom based control flow, meaning:
- buffer size is counted in atoms and can to be specified, default = 1
- implementation: one thread per stream, or writer thread?

when i use the stdio streams, i need threads. the only way to work
around this is to print to a string buffer, and have some write struct
deal with the data later. so it's either

A) one thread per output stream
B) serialize to flat memory + use select based global write state machine.

i already have a mechanism to write to strings. this could be used to
go for option B. so it's going to be the 'serialize' word, together
with write-buffer.


writing this code, i'm thinking it might be a lot simpler to have only
one select word, as the central point of time keeping and I/O scheduling.


ok. select based stuff seems to work..
now the problem is, read might still block!!

suppose A and B nodes are sending each other. if they start sending at
the same time, they might both be tricked into a stdlib read, which
will prevent further sending because that read can block.

so the moral of the story is:
- select-input only garantees there is data
- this will block (for a long time) depends on wether the sender is blocked.

what i need is to explicitly use nonblocking I/O and perform a
pf_poll_output() before a retry call after EAGAIN is encountered.