Support Vector Machine (SVM)

The idea: the widest street

Draw two groups of points on paper, say spam and normal email described by two numbers. Usually many straight lines separate them. Which one should a classifier use?

A line that squeezes past some points is risky: a new point that is only slightly different could fall on the wrong side. A support vector machine chooses the line that stays as far as possible from both classes. Imagine the widest possible street between the two groups, and draw the boundary down the middle of it.

The maths in one paragraph

A line (or, in more dimensions, a hyperplane) is the set of points with w · x + b = 0. Label the classes y = +1 and y = −1. The SVM requires every point to lie on its own side, outside the street:

yᵢ · (w · xᵢ + b) ≥ 1        for every training point

The street then has width 2 / ‖w‖, so the SVM minimises ‖w‖² subject to those constraints. This is a convex quadratic problem with a single best solution, which needs no lucky starting point (unlike training a neural network).

Support vectors

At the optimum, only a few points touch the edges of the street. These are the support vectors. Every other point could move (without crossing the street) or disappear, and the boundary would stay exactly the same. The final model depends only on the support vectors, which makes it compact.

When no line works: the kernel trick

Points arranged as a ring around a cluster can’t be split by any line. But add a third feature, z = x² + y² (the squared distance from the centre). Lifted into 3D, the middle points stay low and the ring rises high, and now a flat plane separates them. Viewed from above, that plane is a circle on the original floor.

Computing new features explicitly can be expensive. The SVM only ever needs dot products between points, so we can replace x · x′ with a kernel function K(x, x′) that equals a dot product in some richer space, without building that space. Popular kernels:

Soft margins

Real data overlaps and has outliers, so a perfect separating line may not exist. Soft-margin SVMs let some points break the rule, at a cost controlled by a parameter C. A large C means a hard margin with few mistakes allowed (risk of overfitting). A small C means a wider, more tolerant margin.

Code

from sklearn.svm import SVC
from sklearn.datasets import make_circles

# Linear SVM on separable data
X = [[1, 2], [2, 3], [3, 3], [6, 5], [7, 8], [8, 6]]
y = [-1, -1, -1, 1, 1, 1]
clf = SVC(kernel="linear", C=1e6).fit(X, y)
print(clf.support_vectors_)          # only the points on the margin
print(clf.predict([[4, 4], [7, 6]]))

# Ring data: a straight line fails, the RBF kernel succeeds
Xc, yc = make_circles(n_samples=200, factor=0.3, noise=0.05, random_state=0)
print(SVC(kernel="linear").fit(Xc, yc).score(Xc, yc))   # 0.69: a line cannot do it
print(SVC(kernel="rbf").fit(Xc, yc).score(Xc, yc))      # 1.0

SVM vs other classifiers

Common mistakes

• Forgetting to scale the features. SVMs measure distances, so a feature in rupees and one in kilograms must be put on similar scales first.
• Using a hard margin on noisy data. One outlier can then drag the boundary. Use a soft margin with a sensible C.
• Thinking the kernel actually builds the high-dimensional points. It doesn’t, and that is the whole trick.
• Expecting probabilities. A plain SVM outputs a class and a score, not a calibrated probability.