Neuron as a Mathematical Object

A neural network is often introduced as a collection of artificial neurons connected together. This description is useful, but it hides the more important question: what exactly is a neuron?

Before discussing neural networks, Transformers, language models, vision models, recurrent networks, or any other architecture, we need to understand the mathematical object from which these systems are constructed.

An artificial neuron is not a miniature brain cell. It is a mathematical function. Its purpose is to receive information represented numerically, transform that information according to a set of parameters, and produce another numerical representation. Once we understand this simple object, we can begin to understand why connecting many such objects together produces increasingly powerful computational systems.

The fundamental operation can be written as

$$z = w_1 x_1 + w_2 x_2 + \cdots + w_n x_n + b$$

The values $x_1, x_2, \dots, x_n$ are the neuron’s inputs. The values $w_1, w_2, \dots, w_n$ are its parameters, usually called weights, and $b$ is a bias. The neuron combines its inputs according to these parameters and produces an intermediate value $z$.

An activation function is then applied:

$$y = f(z)$$

The complete neuron can therefore be represented as

$$y = f\left(\sum_i w_i x_i + b\right)$$

This equation is small, but it contains several ideas that will remain important throughout our entire study. There is information entering the system, there is a transformation applied to that information, there are parameters controlling the transformation, and there is an output representation.

The neuron does not inherently know what any of these numbers mean.

If we give it $x_1 = 0.8, x_2 = -0.3$, the neuron does not know that one number represents a word, an image pixel, a sound measurement, a physical quantity, or something else. The meaning comes from how the numerical representation was constructed and how the network learned to use it.

This distinction is fundamental. A neural network does not operate directly on meaning. It operates on mathematical representations from which useful relationships can be learned.

Information must first become a mathematical object

Consider an ordinary piece of information such as an image. A human sees a picture and immediately recognizes shapes, objects, colors, boundaries, and relationships. A neural network cannot receive “a dog” as an abstract concept. It receives numerical data representing the image.

An image might be represented as a collection of pixel values. A sound recording can be represented as a sequence of numerical measurements. Text can be converted into tokens and then into numerical vectors. Sensor data can be represented as numerical time series.

The important step is always the same:

$$\text{information} \to \text{numerical representation}$$

Once information has been converted into numbers, mathematical transformations can be applied to it.

This is one reason the concept of a vector becomes so important.

A vector is simply an ordered collection of numerical values. For example,

$$x = \begin{bmatrix} 0.2 \\ 0.7 \\ -0.1 \\ 0.4 \end{bmatrix}$$

is a four-dimensional vector.

The word “four-dimensional” here does not mean four physical dimensions. It means that the vector contains four coordinates.

A neural network may work with vectors containing hundreds, thousands, or millions of values. The individual numbers are not necessarily understandable in isolation. What matters is the structure contained in the entire vector and the relationships between vectors.

This gives us the first important idea about neural representations:

A representation is a mathematical arrangement of information that allows subsequent computations to operate on that information.

The representation does not have to resemble the original information.

An image does not have to remain an image inside a neural network. A sentence does not have to remain a sentence. A physical measurement does not have to remain in its original units. A network can transform information into completely different numerical forms as it processes it.

Parameters define the transformation

Suppose a neuron receives a vector $x = [x_1, x_2, x_3]$. The weights determine how much influence each component has:

$$z = w_1 x_1 + w_2 x_2 + w_3 x_3 + b$$

If $w_1$ is large and positive, the first component has a strong positive influence on the neuron’s output. If $w_2$ is negative, the second component can suppress the result. If $w_3$ is close to zero, the third component has relatively little influence.

Two neurons receiving the exact same input can behave completely differently depending on their parameters. Given $x = [0.5, 0.2]$, a neuron with $w = [2, 1]$ responds strongly to the first component, while one with $w = [-1, 3]$ responds more strongly to the second. Neither is “detecting” anything in a meaningful sense — each is simply computing a different function of the same input.

Whether that function eventually corresponds to something recognizable to us depends on the larger network and how it was trained. Researchers sometimes find neurons or groups of neurons that correlate with recognizable concepts, but a neural network does not generally begin with a predefined collection of neurons labelled “object,” “verb,” “edge,” “dog,” or “reasoning.” These structures can emerge from optimization.

If we have thousands of neurons with different parameters, we obtain thousands of different transformations of the same input — this is the beginning of the computational richness of neural networks. But a single neuron can only perform a relatively simple transformation. To construct complicated functions, we combine many neurons.

From neurons to layers

Suppose a system has many neurons receiving the same input vector.

Instead of writing every equation individually, linear algebra gives us a compact representation:

$$z = Wx + b$$

Here, $W$ is a matrix containing the weights of many neurons.

The result is another vector:

$$z = [z_1, z_2, \dots, z_m]$$

An activation function can then transform each element:

$$y = f(z)$$

This collection of neurons forms a layer.

A layer therefore takes one representation and transforms it into another:

$$x \to y$$

This seemingly simple idea is responsible for much of what we mean when we say a neural network “processes” information.

The network repeatedly transforms representations.

For example,

$$x^{(0)} \to x^{(1)} \to x^{(2)} \to x^{(3)}$$

means that the original representation has been transformed several times.

Each transformation can reorganize information and create relationships that were not explicitly represented in the previous form.

Why depth matters

Suppose we have only one transformation:

$$y = f(Wx + b)$$

The network has limited computational structure.

Now suppose we compose several transformations:

$$x \to f(W_1 x + b_1) \to f(W_2 x^{(1)} + b_2) \to f(W_3 x^{(2)} + b_3)$$

The second transformation does not operate directly on the original input. It operates on the representation produced by the first transformation.

This means the transformations can build upon one another.

The first layer changes the representation. The next layer operates on the changed representation. The next layer operates on that new representation again.

Consequently, depth creates a hierarchy of transformations:

$$\text{input} \to \text{representation} \to \text{transformed representation} \to \text{further transformation} \to \text{output}$$

This is one of the fundamental reasons neural networks can represent complicated functions.

The architecture determines how information can move

Once we understand neurons and layers, we can understand what an architecture actually means.

An architecture is not merely the number of neurons in a model.

It defines how computational components are arranged and how information is allowed to move through them.

Consider a simple feed-forward network. Information moves forward through a sequence of transformations:

$$x \to L_1 \to L_2 \to L_3$$

A recurrent network changes this arrangement by allowing information from an earlier computational state to influence a later state.

A convolutional network imposes a particular structure on how information from nearby locations interacts.

A Transformer uses attention to create a different pattern of interaction between representations.

A spiking neural network introduces another dimension: neural state evolves through time and produces discrete events.

These architectures can all contain mathematical objects that resemble neurons, but they organize those objects differently.

This gives us a more useful definition of architecture:

An architecture defines the computational structure through which representations are transformed and information is allowed to interact.

This will become extremely important later when we compare static and dynamic neural systems.

The neuron is not the whole computational system

Understanding one neuron doesn’t mean understanding the network — the interesting behavior comes from how many such components interact. A single neuron transforms $x \to y$; thousands of neurons transform a whole representation $X \to Y$; and stacking layers composes those transformations into

$$X_0 \to X_1 \to X_2 \to \cdots \to X_n$$

An architecture can introduce further relationships between those layers on top of this. The computational possibilities grow not simply because there are more neurons, but because there are more ways for information to interact and be transformed.

What is computational structure?

A collection of numerical values isn’t, by itself, a useful computation — computation emerges from the operations connecting those values. We’ve already seen how stacking $x \to f(Wx+b)$ transformations builds more complex functions; swap in recurrence, attention, convolution, or temporal dynamics instead of simple stacking, and the structure changes again, even though the individual components can still look like ordinary neurons.

This is why, when studying neural networks, the more useful question isn’t “what is this architecture called,” but:

What information exists at this point, and what mathematical operation transforms it into the next representation?

Learning changes the computational structure

At initialization, the network’s parameters are usually not organized into a useful solution.

The network is given data and an objective. It produces an output. The output is compared with the desired result. A loss is calculated. Gradients are computed. The parameters are updated. Then the process repeats.

Mathematically, if the model has parameters $\theta$, training attempts to find parameters that minimize some objective:

$$\theta^* = \arg\min_\theta L(\theta)$$

Gradient descent provides a way of moving through parameter space:

$$\theta_{t+1} = \theta_t - \eta \nabla_\theta L$$

This means training can be viewed as a search process.

The model begins somewhere in a huge parameter space. The dataset defines an objective landscape. The gradient provides local information about which direction changes the objective. The optimizer repeatedly moves the parameters.

Eventually, the model reaches a region where its behavior is useful according to the training objective.

This is the beginning of our concept of search space.

The search space is not the dataset

This distinction is important.

Suppose a network contains one million parameters. Each parameter can take many possible values. The complete model therefore represents an enormous space of possible parameter configurations:

$$\theta \in \mathbb{R}^{1{,}000{,}000}$$

Every point in this space corresponds to a particular set of parameters and therefore to a particular function implemented by the network.

Training explores this space.

The dataset does not simply tell the model what answer to memorize. It provides constraints that make some regions of the parameter space more useful than others.

The optimizer uses those constraints to navigate the space.

This is why learning can be viewed as a search for computational structures that explain or predict the observed data.

The deeper question

We have now reached an important point.

A neural network begins with numerical parameters. Data enters the system as numerical representations. The architecture determines how those representations can interact. The neurons and layers transform those representations. The loss evaluates the resulting behavior. Gradients provide information about how the parameters should change. Optimization moves the model through parameter space.

And through this process, useful internal representations can emerge.

The remarkable thing is that the researcher does not need to explicitly specify every representation that the network should discover. The researcher defines the architecture, the data, and the objective. The optimization process searches for a configuration that performs well.

This gives us a much deeper way of looking at neural networks.

They are not simply collections of artificial neurons. They are parameterized computational systems whose structure defines a space of possible transformations, while learning searches that space for useful configurations.

That idea will be the foundation for everything that follows.

Before we can understand why different neural architectures exist, however, we need to go one level deeper. We need to understand exactly what happens to information when it enters a neuron, how vectors change as they pass through transformations, what activation actually means, how multiple representations interact, and how a sequence of transformations can create increasingly complex computational structures.

That is the subject of the next chapter.