Information as Representation

How raw information becomes a numerical representation, and why the shape of that representation matters.

In the previous chapter, we saw that a neuron does not operate on meaning. It operates on numbers. A network receiving $x_1 = 0.8, x_2 = -0.3$ has no idea whether those values came from a pixel, a sound wave, or a stock price. This raises a question we deliberately set aside: how does raw information become those numbers in the first place, and why does the shape of that numerical form matter so much?

This is the question of representation, and it turns out to be one of the most important ideas in the entire study of neural networks — arguably more important than any particular architecture built on top of it.

From raw signal to numbers

Every piece of information a network receives has to pass through some encoding step before it becomes usable. An image starts as light hitting a sensor, which becomes a grid of intensity values, one per pixel, per color channel. A sound starts as air pressure changing over time, which gets sampled at fixed intervals and stored as a sequence of amplitude values. A sentence starts as a string of characters, which gets split into tokens and then mapped to numerical identifiers.

None of these encoding steps are neutral. Each one makes decisions about what to keep and what to discard.

A photograph sampled at low resolution keeps the overall shape of a scene but discards fine texture. Audio sampled at a low rate keeps the general rhythm of a sound but discards high-frequency detail. A tokenizer that splits text into whole words keeps semantic units intact but struggles with unfamiliar words; one that splits into individual characters handles anything but loses the notion of a “word” as a single unit.

This means the representation a network receives is never simply “the information.” It is a particular numerical encoding of that information, shaped by choices made before the network ever sees it.

\text{raw signal} \to \text{encoding decisions} \to \text{numerical representation}

Everything the network subsequently learns is built on top of whatever survived that first step.

A representation is not the thing it represents

It is tempting to think of a vector like $x = [0.2, 0.7, -0.1, 0.4]$ as simply “being” the image or sentence it came from. It is not. It is one particular numerical arrangement that stands in for that information, chosen because it is useful for computation — not because it resembles the original thing in any visual or intuitive sense.

This distinction matters because the same underlying information can be represented in many different ways, and those different representations are not equally useful.

Consider a simple example. Suppose we want to represent which one of five colors an object is. One approach is to assign each color a single number: red is $1$ , blue is $2$ , green is $3$ , and so on. This works in the sense that every color maps to a distinct number, but it introduces a problem — it implies an ordering and a distance that doesn’t actually exist. It suggests that blue is “between” red and green in some meaningful sense, when nothing about color actually works that way.

A different approach is to represent each color as a vector where exactly one position is active:

\text{red} = [1, 0, 0, 0, 0], \quad \text{blue} = [0, 1, 0, 0, 0], \quad \text{green} = [0, 0, 1, 0, 0]

This is called a one-hot representation. It avoids the false ordering problem — every color is now equally distant from every other color — but it introduces a different limitation. Every color is treated as entirely unrelated to every other color, even when that isn’t true. Red and orange are, in most practical senses, more similar to each other than red and blue are, but a one-hot representation has no way to express that.

Neither representation is “correct.” Each one makes a different set of information available to whatever computation comes next, and hides a different set of information.

Distributed representations

Neural networks generally favor a third approach, one that neither of the two above examples uses: distributed, dense representations, where information is spread across many numerical dimensions at once, and no single dimension is required to mean something on its own.

Recall from the previous chapter that a neuron computes a weighted combination of its inputs. If a network is free to choose its own weights during training, it can construct representations where similarity between two inputs is reflected as proximity between two vectors — without anyone having to hand-design what each dimension of that vector means.

This is a significant shift in how we think about representation. In a one-hot encoding, we decided in advance what each position means. In a distributed representation learned by a network, no one decides that in advance. The meaning of a particular dimension, if it has one at all, emerges from training — and often, no individual dimension is interpretable by itself. It is the pattern across all of them together that carries information, in the same way that no single pixel tells you what an entire image contains.

This is why, when researchers inspect the internal representations of a trained network, they often cannot point to a single number and say “this is the concept of a cat.” The concept, if the network has learned something like one, is usually smeared across a combination of many values acting together.

Representation as geometry

Once information is expressed as a vector, it can be treated as a point in a space. This turns out to be an enormously useful shift in perspective, because it lets us borrow the language of geometry to describe relationships between pieces of information.

Two representations that are similar in some meaningful sense tend to end up close together in this space, while representations of unrelated information end up far apart. “Close” and “far” here are usually measured with an ordinary distance formula, such as

\|a - b\| = \sqrt{\sum_i (a_i - b_i)^2}

or with cosine similarity, which measures the angle between two vectors rather than the straight-line distance between them:

\text{sim}(a, b) = \frac{a \cdot b}{\|a\| \, \|b\|}

Neither formula knows anything about images, words, or sounds. Both are purely geometric. But if the representation was constructed well — either by careful design or, more commonly today, by training — geometric closeness in this space can end up corresponding to something meaningful in the original domain.

This is the property that makes distributed representations powerful. Once “similar” and “different” are expressed geometrically, an enormous range of mathematical tools built for vector spaces becomes available to the network for free.

A good representation is one where the geometry of the space reflects relationships that actually matter for the task at hand.

This single sentence explains a great deal about why certain representations work well and others don’t. A representation isn’t good or bad in the abstract — it’s good or bad relative to what you need geometric closeness to mean.

The network is not given a representation. It builds one.

Here is the idea that connects this chapter back to the previous one.

We have been discussing representation as though it were something decided in advance — a choice made by whoever designs the encoding scheme. In classical, hand-engineered systems, that is often true. But inside a trained neural network, most of the representations that matter are not designed at all. They are discovered.

Recall the closing idea from Chapter 1: a network is a parameterized computational system whose structure defines a space of possible transformations, and learning searches that space for a useful configuration. Representation is exactly what that search is searching for. The raw input encoding — pixels, tokens, audio samples — is only the starting point. Every layer after that is free to construct a new representation of the same underlying information, one that may make the next computation easier than the last representation did.

This is why depth mattered so much in the previous chapter. Each additional transformation isn’t just processing information — it’s an opportunity to re-represent it in a way that’s more useful for whatever comes next.

Where this leaves us

We now have two separate ideas that need to be brought together. Information must first be turned into a numerical representation before a network can do anything with it. And once inside the network, representations are not fixed — they are actively constructed and reconstructed as information passes through each layer.

What we haven’t yet examined carefully is exactly how that reconstruction happens — what a transformation actually does to a representation, step by step, and why applying the same kind of operation over and over can produce such different and increasingly useful representations at each stage.

That is the subject of the next chapter.