Mon May 11 09:54:21 CEST 2009

Counting votes.

This is a rehash of the previous post.

For bit vectors b of dimension n, the function 

  bits : 2^n -> [0..n]

introduces a partition[1] of 2^n.  The equivalence classes have an
order relation, determined by the bits function.

The central idea in using this partition to represent numbers in terms
of bit vectors is the distribution of bits(v) when v is drawn from a
uniform distribution.  It is a binomial distribution[2].

Such a distribution naturally encodes "agreement" (most bits zero or
one) as an exceptionally ordered state, and "disagreement" as the
natural disordered state.  

This non-linearity is quite a good match to the non-linear behaviour
of neural networks when modeled as an inner product followed by a
limiting nonlinearity, without ever using any explicit computations.
Simply concatenating a number of such vectors gives a resulting vector
that represents anything on the scale of no->disagrement->yes.  The
size of an individual bit vectors dermines its weight in the vote:

  v_out = [ v_1 | ... | v_n ]

Let's call a single bit vector v \in {0,1}^n a "vote".  Each vote has
a weight n which is its dimensionality.

Now, to make this managable, the only real "computation" that has to
happen is to reduce the weigth of votes so they can be combined with
other votes to form new ones.  The essential observation is that the
distribution of a vote with weight n is qualitatively similar to the
distribution of a vote with weigth n-1.  In other words, simply
discarding one element of the bit vector on average has no qualitative
effect.  Let's call this "weight loss".

The same reasoning goes for randomly flipping a bit.  On average this
has little effect, giving rise to good noise immunity.

This gives some form of computation (it can do anything feedforward
neural networks do) in a statistical way.  

The next step is to make weight loss programmable.  A vote should be
able to inhibit another vote.  The problem here is that it is a
physical operation.  Inhibition means to cut wires.  Is it possible to
do a non-phsyical inhibition?

Yes.  The and/or gate to introduce assymetry between yes and no.  One
vote could be combined with another vote (gate it).

So, what then is a nerve cell?  It's where multiple wires come
together and a random part of them is discarded.  Forgetting is the
essence of computation. ;)

So, given a (huge) set of binary nodes, there are only two operations:
    - reduction: taking a subset of signals to create a node
    - introduction of assymetry (inhibition through AND|OR) = gating

[1] http://en.wikipedia.org/wiki/Partition_of_a_set
[2] http://en.wikipedia.org/wiki/Binomial_distribution