Neural networks (and LLMs are no exception) are all about deconstructing information, removing noise from signals, encoding signals and then deliberately adding noise (entropy) to create signals.

These processes of constructing and deconstructing information are at the heart of the emerging field of generative AI, which aims to create systems that can generate new and original content.

We reduce reality to the minimum expression necessary to recreate it. So we keep only the key information and discard all the entropy that just adds noise. We thus create an archetype or model (‘latent space’ in generative AI) from which we can create new instances (observations) by adding back entropy (randomness).

Contrary to popular belief, a specific model or neural network has almost nothing to do with the way a human brain works. But the process of modelling reality by subtracting / adding entropy does mimic the way we think.

The most practical implication of all this would be to use the entropy of the distribution of observations to develop an algorithm for designing the layers of a neural network.