Skip to content

Deep OCR๐Ÿ”—

More Info

Learn about Deep OCR Recognition and many other topics in interactive online courses at our MVTec Academy.

With Deep OCR, we want to detect or recognize text in an image, or both. Deep OCR detects and recognizes connected characters, referred to as โ€œwordsโ€ (in contrast to OCR methods used to read single characters).

Figure 1. Example for deep-learning-based optical character recognition: Words in an image are detected and recognized.

A Deep OCR model can contain two components, which are dedicated to two distinct tasks: detection, thus the localization of words, and recognition of words. By default, a model is created with both components, but the model can also be limited to either task.

HALCON already provides pretrained components, which are suited for a multitude of applications without additional training as the model is trained on a varied dataset and can therefore cope with many different fonts. Information on the available character set and model parameters can be retrieved using get_deep_ocr_param.
To further adjust the reading to a specific task, it is possible to retrain the recognition or detection component separately on a given application domain using deep learning operators.

Important

Only one component, detection or recognition, can be retrained at a time.

Figure 2. Fine-tuning the detection for an application by retraining the Deep OCR model with custom data

Figure 3. Fine-tuning the recognition for an application by retraining the Deep OCR model with custom data

In HALCON, the recognition component requires tight word crops around a word in an image to read the contained text. One way to obtain these tight crops is by applying the detection component, which is trained to find the exact position of multiple word boxes in an image.

For images containing a single word on a large background, HALCON offers the recognition alignment feature, which ensures that relevant text regions are properly aligned for the recognition component.

Figure 4. Fine-tuning the recognition to support alignment for an application by retraining the recognition component with custom data. The pretrained alignment generalizes well on a large variety of font and printing types and cannot be fine-tuned with custom data.

About data๐Ÿ”—

We distinguish between data used for training and evaluation, consisting of images with their information about the instances, and data for inference, which are bare images.

As a basic concept, the model handles data over dictionaries, meaning it receives the input data from a dictionary DLSample and returns the dictionaries DLResult and DLTrainResult. For more information about the data handling, see Deep learning > HALCON DL model.

Data for training and evaluation

The dataset consists of images and corresponding information. They have to be provided in a way the model can process them. For more information about the image requirements, see Images below.

The training data is used to train and evaluate a network for your specific application. With the aid of this data, the network can learn to detect or recognize text samples that resemble text that occurs during inference. The necessary information is provided by labeling each image with a word.

How the data has to be formatted for a DL model is explained in Deep learning > HALCON DL model. In short, a dictionary DLDataset serves as a database for the information needed by the training and evaluation procedures.

The data for DLDataset can be read using read_dl_dataset_ocr_recognition or read_dl_dataset_ocr_detection, depending on the model type.

  • Dataset based on images with word labels (for detection and recognition with alignment support)

    In this case, images with words that are labeled with rotated bounding boxes need to be provided. You can label your data using the MVTec Deep Learning Tool, available from the MVTec website.

    The dataset must be built as follows:

    • 'class_ids': Class IDs
    • 'class_names': Class names
      (needs to contain the class โ€œwordโ€; all other classes are ignored)
    • 'image_dir': Path to the image directory
    • 'samples': Tuple of dictionaries, one for each sample
      • 'image_file_name': Name of the image file
      • 'image_id': Image ID
      • 'bbox_col': Bounding box column coordinate
      • 'bbox_row': Bounding box row coordinate
      • 'bbox_phi': Bounding box angle
      • 'bbox_length1': First half edge length of the bounding box
      • 'bbox_length2': Second half edge length of the bounding box
      • 'label_custom_data': List of dictionaries containing custom label data for each bounding box
        • 'text': Word to be read
  • Dataset based on word crop images (only recognition without alignment support)

    In this case, only images that are cropped to a single word each are included in the dataset.

    The dataset must be built as follows:

    • 'image_dir': Path to the image directory
    • 'samples': Tuple of dictionaries, one for each sample
      • 'image_file_name': Name of the image file
      • 'image_id': Image ID
      • 'word': Word to be read in the image

The example program deep_ocr_prelabel_dataset.hdev can provide assistance by prelabeling your data.

Your training data should cover the full range of characters that might occur during inference. If a character is rarely or never included in the training dataset, the model may not learn to recognize it properly. To keep track of the character distribution within the dataset, the procedure gen_dl_dataset_ocr_recognition_statistics is provided, which generates statistics on how often every single character is contained in your dataset.

For the training of the recognition component supporting alignment, it is crucial that the dataset contains images with one word, labeled with rotated bounding boxes. The alignment feature ensures that relevant text regions are properly transformed and aligned for the recognition component.

You also want enough training data to split it into three subsets: for training, for validation, and for testing the network. These subsets are preferably independent and identically distributed (see also About data splitting).

Images

Regardless of the application, the model poses requirements on the images, such as the dimensions, the gray value range, and the type. The specific values depend on the network itself and can be queried with get_dl_model_param. For more information about specific values of different models, see read_dl_model.

To fulfill these requirements, you may have to preprocess your images. Standard preprocessing is implemented in the procedure preprocess_dl_dataset and, for a sample, in preprocess_dl_samples. In case of custom preprocessing, these procedures offer guidance on the implementation.

Requirements for images used for inference are described in apply_deep_ocr.

Model output

The network output depends on the task:

  • Training

    As output, the operator will return a dictionary DLTrainResult with the current value of the total loss as well as values for all other losses included in your model.

  • Inference and evaluation

    As output, the network will return a dictionary DLResult for every sample. This dictionary will include the recognized word as well as the candidates and their confidences for every character of the word.

Evaluation measures for Deep OCR results๐Ÿ”—

Deep OCR Detection

The following evaluation measures are supported in HALCON. To compute these metrics for testing or validation, ground truth annotation is needed.

  • Precision, Recall, and F-score

    The performance of Deep OCR Detection is evaluated using precision and recall on word boxes. The evaluation uses the Intersection over Union (IoU) to compare ground truth and predicted word boxes. The default IoU threshold for a match is 0.5, it can be increased or decreased if needed.

     

    Figure 5. Visual example of the IoU
    Input image with the ground truth bounding box (orange) and the predicted bounding box (light blue)
    IoU as the ratio between the area intersection and the union

    The precision is the proportion of true positives to all positives (true and false ones). Thus, it is a measure of how trustworthy the detection is:

    begin{eqnarray_} textrm{precision} = frac{textrm{TP}}{textrm{TP} + textrm{FP}} end{eqnarray_}

    The recall is the proportion of the number of correctly detected words to all labeled words:

    begin{eqnarray_} textrm{recall} = frac{textrm{TP}}{textrm{TP} + textrm{FN}} end{eqnarray_}

    The F-score, the harmonic mean of precision and recall, represents this with a single number:

    begin{eqnarray_} textrm{F-score} = 2 times frac{textrm{precision} times textrm{recall}}{textrm{precision} + textrm{recall}} end{eqnarray_}

  • Score of Angle Precision (SoAP)

    The SoAP value is a score for the precision of the inferred orientation angles. This score is determined by the angle differences between the inferred bounding boxes (I) and the corresponding ground truth annotations (GT):

    \[\begin{eqnarray*}\textrm{SoAP} = 1.0 - \frac{1}{\pi\, k} \sum_{k} \left| \varphi_{k}^{\textrm{I}} - \varphi_{k}^{\textrm{GT}} \right|\end{eqnarray*}\]

    where the index \(k\) runs over all inferred bounding boxes.

Deep OCR Recognition

The accuracy for a Deep OCR Recognition task is given as the percentage of correctly read words (CR) to the ground truth words (GT) of a dataset. The accuracy is then defined as:

begin{eqnarray_} textrm{accuracy} = frac{textrm{CR}}{textrm{GT}} times 100 end{eqnarray_}