Skip to content

SVM classification🔗

Support-Vector Machines (SVM)🔗

Support-Vector Machines (SVM) are classification approaches that can handle classes that are not linearly separable.

Instead of finding a non-linear separating hypersurface directly in the original feature space, SVM transforms the feature space into a space of higher dimension, where the classes can become linearly separable. The feature vectors can then be classified using a linear classifier.

To avoid the curse of dimensionality, SVM does not explicitly transform the feature vectors into the higher-dimensional space. Instead, a kernel is used to perform the required transformation implicitly.

Common kernels include the inhomogeneous polynomial kernel and the Gaussian radial basis function kernel.

Figure 1: Transformation of a feature space using a kernel

For two classes, SVM constructs the separating hypersurface such that the margin between the two classes becomes as large as possible. The margin is defined as the closest distance between the separating hyperplane and any training sample.

The training samples that have the closest distance to the separating hypersurface are called support vectors. These support vectors determine the position of the separating hypersurface and therefore have a particularly strong influence on the classification.

Figure 2: Support vectors and separating hyperplane

Classification🔗

SVM naturally handles two-class classification problems. Multi-class classification can be achieved by combining several binary classifiers.

With the one-versus-one approach, a binary classifier is created for each pair of classes. The class that wins the most comparisons is selected.

With the one-versus-all approach, each class is compared with the remaining training data. The class with the maximum distance to the separating hypersurface is selected.

SVM can also be used for novelty detection. In this case, the classifier is trained using a single class and determines whether an unknown feature vector belongs to the trained class or to the rejection class.

Training🔗

During training, the separating hypersurfaces are optimized using the labeled training samples.

The quality of the trained classifier strongly depends on the selected kernel, its parameters, the regularization parameter, and the representativeness of the training data.

The parameters KernelParam and Nu are particularly important because they jointly influence the complexity of the separating hypersurface.

The training data should cover the expected variations within each class while avoiding irrelevant variations that may reduce the generalization capability of the classifier.

Further information on parameter optimization can be found in the Optimization chapter.

Typical applications🔗

SVM classifiers can be applied to a wide range of classification tasks. Typical applications include

  • general feature classification,

  • image segmentation, and

  • optical character recognition (OCR).

SVM can be particularly useful when the classes cannot be separated by a linear decision boundary in the original feature space.