PCA as a generative model

We can use PCA as a generative model by sampling a $z_{\text{new}}\in {\mathbb{R}}^{\mathbb{k}}$ (for example we can take the average of two $z_1,z_2\in {\mathbb{R}}^k$) and using reconstruction in order to obtain the $x_{\text{new}}$ associated.

PCA alone is completely linear, since both the projection and reconstruction operations are linear, so in order to generalize this idea and generate more accurate or complex representations, we will replace the projection and reconstruction operations with two (highly non-linear) neural networks.