# Restricted Boltzmann machine

A **restricted Boltzmann machine**, commonly abbreviated as **RBM**, is a neural network where neurons beyond the visible have probabilitistic outputs. The machine is restricted because connections are restricted to be from one layer to the next, that is, having no intra-layer connections.

As with contrastive Hebbian learning, there are two phases to the model, a *positive* phase, or *wake* phase, and a *negative* phase, or *sleep* phase.

## Model

We use a set of binary-valued neurons. Given a set of k-dimensional inputs represented as a column vector*m*neurons with (initially random, between -0.01 and 0.01) synaptic weights from the inputs, represented as a matrix formed by

*m*weight column vectors (i.e. a

*k*row x

*m*column matrix):

*i*and neuron

*j*.

During the positive phase, the output of the set of neurons is defined as follows:

*i*indicates the probability that neuron

*i*will output a 1.

*reconstruction*of the input

This completes one *wake-sleep* cycle.

To update the weights, a wake-sleep cycle is completed, and weights updated as follows:

where η is some learning rate. In practice, several wake-sleep cycles can be run before doing the weight update. This is known as *Gibbs sampling*.

A batch update can also be used, where some number of patterns less than the full input set (a *mini-batch*) are uniformly randomly presented, the wake and sleep results recorded, and then the updates done as follows:

*contrastive divergence*.

## References

- Hinton, Geoffrey (August 2, 2010). "A practical guide to training restricted Boltzmann machines". University of Toronto Department of Computer Science.

- Cho, KyungHyun (March 14, 2011) "Improved Learning Algorithms for Restricted Boltzmann Machines". Aalto University.