Multilayer Perceptrons: Task-Specific Design and Self-Attention

Introduction

The Multilayer Perceptron, or MLP, is one of the oldest and most foundational architectures in deep learning. Larger and more specialized models have risen around it. Even so, it remains a strong, fast baseline whenever you can express the input as a fixed-length vector of numbers. That covers classic tabular data. It also stretches further than you might expect. Even image problems become MLP problems once a pretrained network turns each picture into a fixed-length feature vector. Most introductions to the MLP stop at the architecture itself: neurons, layers, activation functions, backpropagation. They leave a practitioner to figure out the rest alone.

This article picks up where those introductions usually stop. It does not re-explain what a neuron or a forward pass is. Instead, it asks a more practical question: how does the same MLP need to change depending on what you are trying to predict? An MLP’s hidden layers barely change across the four core tasks: binary classification, multi-class classification, multi-label classification, and regression. What changes almost entirely is the output layer, the loss function, and the way we judge the model. We will then look at a more exploratory idea. Can an MLP benefit from a self-attention block borrowed from Transformers? We will think through what that does and does not buy you in practice.

For a deeper primer on the basic building blocks of an MLP, including neurons, weights, biases, forward propagation, and backpropagation, AIML.com’s overview of Multilayer Perceptrons is a solid foundational reference. This article assumes that base and builds on top of it.

What an MLP Actually Is

An MLP is a stack of layers of artificial neurons. Each neuron computes a weighted sum of its inputs, adds a bias term, and applies a non-linear activation function. Information flows in a single direction. It moves from the input layer, through one or more hidden layers, to an output layer, with no loops or feedback. That is why MLPs are also called feedforward neural networks.

Two properties make this simple structure surprisingly powerful. The first is non-linearity. Activation functions such as ReLU let the network model complex relationships, not just straight lines and flat planes. A house’s price, for instance, does not rise in a neat straight line with income or location. It bends, plateaus, and interacts with other factors. Non-linearity is what lets the network capture those curves. The second property is flexibility. With enough hidden neurons, an MLP can in theory approximate almost any continuous function. That is precisely why you can repurpose the same basic architecture for very different prediction problems.

That last point is the thread running through the rest of this article. The body of the network, its hidden layers, is largely task-agnostic. What changes from task to task is the head of the network, meaning the output layer. The loss function, the language the network uses to measure its own mistakes, changes along with it.

How an MLP Learns

Before getting into task-specific design, it helps to walk through how any MLP learns from data.

Training begins with a forward pass. Data flows through the network’s layers, each one transforming it slightly, until the output layer produces a prediction. That prediction might be a probability, a set of probabilities, or a plain continuous number, depending on the task. A loss function then compares the prediction to the true answer. It reduces the mismatch to a single number the network tries to minimize. The choice of loss function is not arbitrary. It has to match the kind of output the network produces. That match is the central idea of the next section.

Next, the network works backward through its layers in a process called backpropagation. It calculates how much each weight contributed to the error. Then it nudges each weight slightly in the direction that reduces that error. An optimizer, commonly Adam, controls how large these nudges are and adapts the learning rate as training proceeds. The network does not process the entire dataset at once. It works through small batches instead, repeating this across many passes (epochs) over the data. Training continues until the weights converge toward values that produce accurate predictions.

Why Feature Scaling Matters

One detail matters for every task discussed below: feature scaling. Neural networks are sensitive to the scale of their inputs. Imagine a dataset where one column holds a proportion between 0 and 1 while another holds an income in the tens of thousands. The larger column dominates the gradients, and training becomes slow or unstable. Standardizing inputs to a mean of zero and a standard deviation of one is close to a universal first step. One caution is worth knowing early. Compute the scaling statistics from the training data only, after splitting the dataset. Then apply them to the validation and test sets. Computing them from the full dataset quietly leaks test information into training. The model’s evaluation then ends up more optimistic than it deserves to be.

The Real Differentiator: How the Output Layer Adapts to the Task

This is where the four problem types actually start to diverge. It helps to think of them less as four separate recipes and more as four answers to one question: what shape does the answer need to take?

Binary Classification: One Neuron, One Probability

In binary classification, the answer is a single yes-or-no judgment. Is a tumor malignant? Is an email spam? The network only needs one output neuron. The network passes that neuron’s raw value through a sigmoid function, which squashes it into a probability between 0 and 1: the likelihood of the positive class. A value of 0.92 reads as “very likely positive,” while 0.08 reads as “very likely negative.” The usual convention draws the decision line at 0.5. The natural loss function here is binary cross-entropy. It penalizes the model more heavily the further its predicted probability strays from the true label.

Multi-Class Classification: Categories That Compete

Multi-class classification asks a different question: which one of several mutually exclusive categories does this example belong to? An image is a cat or a ship or an airplane, but never two of those at once. Now the network needs one output neuron per category. Instead of a sigmoid, it uses softmax, or its numerically stable variant, log-softmax. Softmax converts raw scores into a probability distribution that sums to one across all classes. Because the categories are mutually exclusive, the activation forces the probabilities to compete with each other. A single sigmoid output never behaves this way. If the model grows more confident the image is a cat, its confidence in every other category must shrink to make room. The matching loss function is cross-entropy. Frameworks sometimes implement it as negative log-likelihood paired with log-softmax.

Multi-Label Classification: Independent Yes-or-No Decisions

Multi-label classification looks superficially similar to multi-class classification, since both involve more than two categories. But the underlying question is different: which of several labels apply here? Any number of them, including all or none, might be true at once. A single street photograph might contain a person, a bicycle, and a traffic light all together. A correct model should say yes to all three. Since the labels are not mutually exclusive, softmax would be the wrong tool. It would force the labels to compete for a shared probability budget. Raising confidence in “person” would artificially push down confidence in “bicycle,” even though both are plainly in the frame. Instead, each output neuron gets its own independent sigmoid. The loss applies binary cross-entropy to each label separately and then averages the results. This is, in effect, one shared network solving many small binary classification problems. It also introduces a complication the other tasks do not face as severely. Some labels appear far more often than others; people show up in photographs far more often than fire hydrants do. That makes multi-label problems particularly prone to class imbalance, which can distort both training and evaluation if it goes unaddressed.

Regression: Predicting a Number, Not a Category

Regression breaks the pattern entirely. The target is no longer a category at all but a continuous number, such as a house price or a temperature. There is nothing to squash into a bounded range here, so the output layer has no activation function. The raw value produced by the last linear layer is the prediction itself. The loss function shifts accordingly, from measuring probability mismatch to measuring numerical distance. Mean squared error and mean absolute error are the typical choices.

The Design Decisions at a Glance

Across all four tasks, the hidden layers in between could be nearly identical: a couple of linear layers with ReLU activations. The table below summarizes where they actually diverge.

The most common real-world mistake in MLP design lives in this table. Using softmax on a multi-label problem forces labels to compete when they shouldn’t. Leaving a sigmoid on a regression output forces predictions into a range nobody intended. Getting the right row of this table is, in practice, a bigger lever than almost any other architectural decision.

Judging Whether the Model Is Actually Good

Just as the output layer shifts by task, so does the right way to measure success, and defaulting to “accuracy” regardless of the problem can be quietly misleading or simply inapplicable.

Metrics for Binary and Multi-Class Problems

For binary and multi-class classification, accuracy is the proportion of correct predictions. It is a reasonable starting point when classes are fairly balanced. But it can hide poor performance on a minority class in an imbalanced class problem. Consider a medical screening model that labels every patient healthy. If only 5 percent of patients are actually ill, it scores 95 percent accuracy while missing every case that matters. A confusion matrix breaks predictions down by which classes are being mistaken for which. Precision, recall, and F1-score separate two different kinds of mistakes. Precision asks how many predicted positives were actually correct. Recall asks how many true positives the model found in the first place. Which of the two deserves more weight depends on the cost of each mistake. In that same screening example, a false negative is a missed illness. A false positive is a healthy patient sent for one extra test. The first is usually far more costly, so recall matters more than precision there.

Macro and Micro F1 for Multi-Label Problems

Multi-label problems need a slightly different lens, since a single sample can have several correct answers at once. Macro F1 averages the F1-score across all labels equally, regardless of how often each one occurs. That exposes weak performance on rare labels that a simpler average would bury. Micro F1, by contrast, aggregates predictions globally before computing F1. It ends up dominated by whichever labels are most frequent. A wide gap between the two is itself a useful diagnostic. It usually signals that the model handles common labels well while quietly failing on rare ones. That is a direct consequence of the class imbalance mentioned earlier. Think back to the street photographs from the previous section. A model can look impressive under micro F1 simply by being good at spotting people, while never once correctly tagging a fire hydrant. Macro F1 is what catches that failure.

Error Metrics for Regression

Regression calls for an entirely different set of metrics, since there is no notion of a correct class to begin with. Root mean squared error penalizes large mistakes disproportionately. That suits situations where big errors are especially costly. Being off by 200,000 on a house price is far worse than being off by 2,000 a hundred times. Mean absolute error gives a more directly interpretable measure of average error size, in the same units as the target itself. R-squared describes what proportion of the variance in the target the model can explain. That makes it useful for judging fit independent of the target’s units.

Can Self-Attention Make an MLP Better?

A more exploratory question, and a genuinely interesting one, is whether an MLP can be improved by borrowing an idea from Transformer architectures: self-attention. The twist is applying it to data that has no natural sequence at all, unlike text or time series, where attention was originally designed to shine.

What Self-Attention Does

Self-attention lets a model weigh the relationships between different elements of an input against one another, instead of treating each one in isolation. In language, this is what allows a model to connect a pronoun back to the noun it refers to several words earlier. Mechanically, the model computes, for every pair of elements, how much attention one should pay to the other, based on relationships it has learned during training.

Grafting Attention onto Feature Vectors

Feature vectors, such as a person’s age, income, and location, do not have an inherent order or sequence the way words in a sentence do. Attention also needs multiple elements to compare, and a feature vector, on its face, is just one object. The simplest way to graft attention onto an MLP is to treat the entire feature vector as a single token: a linear layer projects it into a higher-dimensional embedding, a Transformer encoder layer processes that embedding, and a small MLP head then turns the result into the prediction. It is worth being honest about what this actually does. With only one token in the sequence, there are no pairs of elements for the attention mechanism to relate; attention over a single token collapses into a learned re-weighting of that token, so the block behaves like an extra transformation layer with more parameters rather than a mechanism for modeling how features interact. The name suggests more than the mechanics deliver.

A design that genuinely engages the attention mechanism has to give it something to attend over, which means splitting the input into multiple tokens first: assigning each feature (or group of features) its own embedding, so the encoder can learn, for example, that age and income should be considered jointly. This is the direction serious tabular Transformer architectures take, and it comes at a cost: more parameters, more data needed to use that flexibility well, and a somewhat artificial premise, since there is no inherent reason to define tokens for tabular data the way there is for words in a sentence or patches in an image.

When the Extra Complexity Pays Off

This gap between the single-token shortcut and genuine multi-token attention is itself a useful lesson. In practice, on many tabular and small-to-medium datasets, a well-tuned plain MLP can match or beat a Transformer-augmented version, and the single-token case makes one reason plain: if the added block never actually models feature interactions, there is little reason to expect it to outperform the simpler network it wraps. Added architectural complexity does not automatically translate into better results, especially when the mechanism being added is not truly engaged, or when the underlying data does not have the kind of relational structure attention is built to exploit. The more useful framing is not that Transformers are strictly better, but that self-attention is one more tool worth trying when feature interactions are suspected to matter, when the design genuinely lets attention compare multiple elements, and when there is enough data to support a larger model.

Practical Lessons That Hold Across All Four Tasks

A few principles carry over no matter which of these problems you are solving. Matching the output activation to the loss function, and both to the task, is the single most consequential design decision in adapting an MLP to something new. Standardizing inputs, and for regression, often the target as well, keeps training stable and predictions interpretable once you convert them back to their original scale; computing those scaling statistics from the training split alone keeps the evaluation honest. Choosing metrics that fit the task, rather than defaulting to accuracy, prevents a reasonable model from looking like a failure, or an imbalanced one from looking better than it really is. Monitoring performance on a held-out validation set during training, and stopping early if it stalls, helps guard against overfitting regardless of the problem. Treat any architectural addition, including self-attention, as a hypothesis worth testing rather than an automatic upgrade, with the extra check that the added mechanism is genuinely being engaged rather than sitting in the network as expensive decoration.

Conclusion

The Multilayer Perceptron is best understood not as a single fixed recipe but as a flexible backbone whose hidden layers stay largely constant while its head, the output layer, activation function, and loss, gets redesigned for the task in front of it. Binary classification, multi-class classification, multi-label classification, and regression are four different translations of the same underlying architecture, each asking for a different way of reading the network’s final layer and a different way of judging whether it got things right.

Self-attention offers an intriguing way to extend that architecture further, with the promise of modeling interactions between features more explicitly, provided the design actually gives the mechanism multiple elements to compare. Whether the extension is worth its added complexity is ultimately a question that depends on the data, the problem, and how honestly the mechanism is put to work, but understanding the trade-off conceptually is what makes that decision a deliberate one rather than a default.

Video Explanations

• In this video, “Neural Networks Pt. 4: Multiple Inputs and Outputs”, presented by Josh Starmer of StatQuest, the simple one-input, one-output network from his earlier videos is extended step by step to handle multiple inputs and outputs. That structural extension is exactly what the four tasks we described put to different uses. (Watch time: ~14 mins)

• In this video, “But what is a neural network?”, created by Grant Sanderson of 3Blue1Brown, the structure of a feedforward network is built up visually using handwritten digit recognition as the running example. From 5:30 onward, the video walks through how neurons hold activations and how weights and biases connect the layers. (Watch time: ~19 mins)

Related Articles:

• What is Perceptron?
• What is a Multilayer Perceptron (MLP) or a Feedforward Neural Network (FNN)?
• What is an Activation function?