updated: Wed Jan 11 12:31:43 CET 2006

Some notes on A/V sync. This is not specific to pf, in fact,
pf does not support audio natively yet.

This is really a hell of a problem. There are several solutions,
but i still haven't figured out a good one. If you're using pdp,
the current (darcs) version contains an object pdp_metro that
can do A/V sync using standard fractional sample rate ratios.

Without the use of special hardware, it is quite impossible to
get perfect A/V sync. Professional A/V studios use a 'studio
clock' from which audio and video sampling rates are derived.
This enables perfect sync without the need for drift compensation.
Such a studio clock is exactly what pdp_metro provides.


Using non-professional hardware, drift needs to be compensated.
The best way to do this is to use the audio as the definite timing 
source, and drop/dup video input/output frames, or use more elaborate 
interpolation schemes. Displaying on a computer monitor / projector
makes the issue rather trivial, but it is very important to keep in
mind when generating movie files or DV tapes.

So, if we don't have synced video hardware, how might we get to nearly
perfect sync? What are the approaches for connecting programs together?

An illustration: 3 execution threads
1 for audio i/o (like jack, or pd)
1 for video input
1 for video output

now, what is the 'tick' time for the video processing?
derived from the audio? the video in? or the video out?

the answer is: depends.

for a video game i.e., the video process tick could be
the refresh rate of the output monitor, the audio could
be completely decoupled, and only communicate through
time-stamped events.

for video (tape) processing, all 3 should be tightly synced
in software, with hardware drift issues solved by
frame drop/dup or interpolation.


in an os with blocking i/o, the most straighforward way
to do all of this, is to have one thread (or process)
per independent time source. the 'depends' above is about
how to connect these threads. or where exactly is the
drift solved?

* input frame dup/drop
* output frame dup/drop
* a/v decoupling through async events

some cases:

-- pd/pf in one process (one thread): 

* video input dup/drop
* video output dup/drop
* timing based on audio, all processing synchronous
problem here is that in single-threaded pd, audio latency
needs to be large, to cover the time spent in the more
coarse video tick.

-- pd/pf in 2 processes (two threads)

note: because running pf/pf in a separate thread in pd
introduces a lot of complexity, i prefer not to do that,
and use 2 different processes.

* input dup/drop
* video process tick based on output sync.
* audio process tick based on audio i/o sync.

the bridge between the 2 can be async io (netsend/netreceive)
using scalar events, together with shared memory for exchanging 
packets.

-- pd and pf in 2 processes
same as above, only that video output sync is easier to do
in pf (running it as a video-jack daemon).


so, conclusions:

* video i/o drift is a hardware problem: solve it by dup/drop video-in
  this is the 'video mixer' sync problem, it is quite impossible to
  avoid without a master clock. the thing that solves this is a
  TBC (time-base corrector).

* then there are basicly 2 approaches:

  A) the 'game framework' where audio and video are coupled only 
     through asynchronous events. the application time base is the
     video output rate. all audio generation is completely decoupled
     from video generation. it is driven only by asynchronous events.

  B) the 'A/V processing' framework, where the sampling rates are
     locked using a global master clock. if there is no professional
     hardware available (studio master clock connected to each digital
     device) this can be faked in software by using 1. prosumer audio
     hardware, 2. dup/drop for video input and output. (analogous
     to digital video mixer TBC and frame synchronization).

use the 'game' approach if you want fast snappy graphics, and use
the 'A/V proc' approach if you are using A/V files. the former is
rather trivial, and can be solved with tools available (pd, pf),
the 'A/V proc' is videojack. see the file tbc.txt