Skip to content

Workflow for rule-based OCRđź”—

Basic workflowđź”—

The basic steps apply the following operators in the following sequence.

  1. Create an OCR classifier using create_ocr_class_mlp, create_ocr_class_svm, or create_ocr_class_knn.

  2. Extract the regions that represent the characters that have to be trained.

  3. Store the samples, i.e., the training regions and the corresponding class names to a training file. This can be done in different ways. Either

  • store all samples at once using write_ocr_trainf. Then the regions as well as the corresponding class names have to be available in a tuple. Or

  • successively add the individual regions (characters) and their corresponding class names to the training file using append_ocr_trainf.

  • Write characters into a training file with write_ocr_trainf_image. That is, regions, representing characters, including their gray values (region and pixel) and the corresponding class name are written into a file. An arbitrary number of regions within one image is supported. In contrast to write_ocr_trainf one image per character is passed. The domain of this image defines the pixels which belong to the character. The file format can be defined by the parameter 'ocr_trainf_version' of the operator set_system.

Additionally, several training files can be concatenated with concat_ocr_trainf.

  1. Read the training characters from the training file and convert them into images with read_ocr_trainf to check the correctness of the training file content.

  2. Train the OCR classifier with trainf_ocr_class_mlp, trainf_ocr_class_svm, or trainf_ocr_class_knn.

  3. Write the OCR classifier to a font file with write_ocr_class_mlp (default file extension: .omc), write_ocr_class_svm (default file extension: .osc), or write_ocr_class_knn (default file extension: .onc).

  4. Read the OCR classifier from the font file with read_ocr_class_mlp, read_ocr_class_svm, read_ocr_class_knn, or read_ocr_class_cnn.

  5. Extract the regions of the characters that have to be classified according to the trained font.

  6. Classify the regions of the characters to be classified. Here, you have different possibilities:

Additional optional steps

Besides the basic steps of an OCR classification, some additional steps and operators can be applied if suitable. In particular, you can

Besides these operators, some operators are provided that are available only for SVM:

Aligning and rectifying imagesđź”—

Because the reading of characters is not invariant to rotation, it may be necessary to correct the orientation of the image. This can be achieved either by directly determining the orientation of the text using the operator text_line_orientation, or by locating another object. Then, the part of the image containing the characters is cropped and aligned using the orientation of the text or the found object.

Similarly to alignment, it may be necessary to rectify the image, e.g., to remove perspective distortions. For slanted characters, the slant angle can be determined using the operator text_line_slant.

Preprocessingđź”—

Sometimes, the characters may be difficult to extract because of noise, texture, or overlaid structures. Here, operators like mean_image or gauss_filter can be used to eliminate noise. A fast but slightly less perfect alternative to gauss_filter is binomial_filter. The operator median_image is helpful for suppressing small spots or thin lines. The operator dots_image is optimized to emphasize a dot-print while suppressing other structures in the image. Gray value morphology can be used to eliminate noise structures and to adapt the stroke width of characters.

Segmenting the imageđź”—

Both for the training samples and for the online reading process, characters must be extracted from the image. This step is called segmentation. This means that the OCR operators like do_ocr_single_class_svm do not search for the characters within a given region of interest, but expect a segmented region, which then will be classified.

If the samples for training are taken from real application images, the same segmentation method will be applied for both training and reading. If the training images are more “artificial”, a simpler method might be used to segment the training images.

For the segmentation, various methods can be used. The Automatic Text Reader is very easy to use and provides robust results. It combines the two steps segmentation and classification into one call of the operator find_text. The Manual Text Finder can read engraved text, which cannot be read by the Automatic Text Reader, but it needs a greater effort in setting the parameters. Therefore, the Automatic Text Reader should be used, if possible. Both methods use a text model, which can be specified precisely.

Automatic Text Readerđź”—

The Automatic Text Reader segments and classifies text robustly, typically without the need for extensive parameter tuning. %HALCONEXAMPLES%/solution_guide/basics/simple_reading.hdev and %HALCONEXAMPLES%/hdevelop/Applications/OCR/bottle.hdev provide a good starting point to become familiar with the Automatic Text Reader.

For using the Automatic Text Reader, a model must be created with create_text_model_reader with the parameter Mode set to 'auto'. Here, an OCR classifier must already be passed. Segmentation parameters can then be specified with the operator set_text_model_param and can be queried with get_text_model_param. After this preparation, the text can be read with the operator find_text. This operator selects character candidates based on region and gray-value features and verifies them with the given OCR classifier. The character candidates are then further combined to lines which are subsequently tested if they qualify as a text line.

If the text must match a certain pattern or structure, the parameter 'text_line_structure' of the operator set_text_model_param can be set, which determines the structure, i.e., the number of characters for each character block of the text that shall be detected.

The Automatic Text Reader assumes approximately horizontal text. If the text is not horizontally aligned, the operators text_line_orientation and rotate_image can be used before the use of find_text.

The result of find_text is returned in TextResultID, which can be queried with get_text_result and get_text_object, respectively. get_text_result returns, e.g., the classification result. get_text_object returns the iconic result of the Automatic Text Reader, i.e., the character regions. To delete the result and the text model, use the operators clear_text_result and clear_text_model, respectively.

Manual Text Finderđź”—

If engraved text must be segmented or if no suitable OCR classifier can be provided, the Automatic Text Reader cannot be used. Instead, the Manual Text Finder can be used in these cases.

In order to segment images robustly, the parameters of the Manual Text Finder should be set carefully. For a practical introduction to the Manual Text Finder, please refer to the example %HALCONEXAMPLES%/hdevelop/Applications/OCR/find_text_dongle.hdev.

For using the Manual Text Finder, a model must be created with create_text_model_reader with the parameter Mode set to 'manual'. Note that no OCR classifier must be passed in this case. Segmentation parameters should then be specified with the operator set_text_model_param and can be queried with get_text_model_param. Note that the names of all parameters that can be used with the Manual Text Finder start with 'manual_'. After this preparation, the text can be segmented with the operator find_text. This operator selects character candidates based on region and gray-value features. The character candidates are then further combined to lines which are subsequently tested if they qualify as a text line.

If the text must match a certain pattern or structure, the parameter 'manual_text_line_structure' of the operator set_text_model_param can be set, which determines the structure, i.e., the number of characters for each character block of the text that shall be detected.

The Manual Text Finder assumes approximately horizontal text. If the text is not horizontally aligned, the operators text_line_orientation and rotate_image can be used before the use of find_text.

The result of find_text is returned in TextResultID which can be queried with get_text_result and get_text_object, respectively. get_text_result returns, e.g., the number of found text lines. get_text_object returns the iconic result of the Manual Text Finder, i.e., the character regions, which can then be classified with a suitable OCR classifier. To delete the result and the text model, use the operators clear_text_result and clear_text_model, respectively.

General character segmentationđź”—

For the common character segmentation you can either use the operator segment_characters to get a region containing all character candidates and then apply select_characters to select those parts of the region that are candidates for the individual characters, or you use blob analysis. There, the most simple method is the operator threshold, with one or more gray value ranges specifying the regions that belong to the foreground objects. Another very common method is dyn_threshold. Here, a second image is passed as a reference. With this approach a local instead of a global threshold is used for each position.

Training OCRđź”—

First, the characters from sample images must be extracted using a segmentation method. To each of the single characters a name must be assigned. This can be done either by typing it in, by a programmed input in the case of a well-structured image (having, e.g., multiple samples of each character in different lines), or by reading the character names from file. Then, the regions together with their names are written into training files. The most convenient operator to do this is append_ocr_trainf. Before applying the training, we recommend to check the correctness of the training files. This can, e.g., be achieved by using the operator read_ocr_trainf combined with visualization operators.

flowchart TD
    Image(["image"])
    Segmentation["segmentation"]
    Regions(["regions"])
    Combination["combination"]
    TrainingFile(["training file (*.trf)"])
    Inspection["training file inspection"]

    CharacterNames(["character names"])
    UserInteraction["user interaction"]
    Programmed["programmed"]
    TextFile["reading text file"]

    Image --> Segmentation
    Segmentation --> Regions
    Regions --> Combination
    Combination --> TrainingFile
    TrainingFile --> Inspection

    CharacterNames --> Combination
    UserInteraction --> CharacterNames
    Programmed --> CharacterNames
    TextFile --> CharacterNames

Note that you can also train your own system fonts. By altering and distorting the characters of a font, you can increase the number of different training samples for each class and thus also increase the detection rate. For this, the Training File Browser of HDevelop’s OCR Assistant can be used. Furthermore, also the example program %HALCONEXAMPLES%/hdevelop/Applications/OCR/generate_system_font.hdev shows how to derive training data and an OCR classifier from system fonts.

The actual training is depicted in the following flow chart.

flowchart TD

    TrainingFile(["training file (*.trf)"])
    OCRClassifierNew(["new OCR classifier"])
    Training["training"]
    OCRClassifierTrained(["trained OCR classifier"])
    Save["saving"]
    FontFile(["font file (*.omc or *.osc)"])
    Test["testing"]

    TrainingFile --> Training
    OCRClassifierNew --> Training
    Training --> OCRClassifierTrained
    OCRClassifierTrained --> Save
    Save --> FontFile
    FontFile --> Test

First, a new classifier is created. There are four different OCR classifiers available: a neural network (multi-layer perceptron or MLP) classifier, a classifier based on support vector machines (SVM), a classifier based on the k-nearest neighbor approach (k-NN), and the box classifier.

Note that if you want to use the Automatic Text Reader for the segmentation and classification of text, you must provide an MLP-based OCR classifier. Otherwise, an OCR classifier based on MLP, SVM, or k-NN can be used. The k-NN has advantages when only few samples are available, but is outperformed by MLP and SVM in typical OCR applications.

The two recommended classifiers differ as follows: The MLP classifier is fast at classification, but for a large training set slow at training (compared to the classifier based on SVM). If the training can be applied offline and thus is not time critical, MLP is a good choice. The classifier based on SVM leads to slightly better recognition rates than the MLP classifier and is faster at training (especially for large training sets). But, compared to the MLP classifier, the classification needs more time.

Dependent on the chosen classifier, you create the classifier using create_ocr_class_mlp or create_ocr_class_svm. Then, the training is applied using trainf_ocr_class_mlp or trainf_ocr_class_svm. After the training, you typically save the classifier to disk for later use by write_ocr_class_mlp or write_ocr_class_svm.

Reading symbolsđź”—

First, the characters must be extracted using an appropriate segmentation method. Here, you must use a method that returns the characters in a form similar to the ones used for training. After reading the classifier (font file) from file (read_ocr_class_mlp or read_ocr_class_svm), the classifier can be used for reading. The Automatic Text Reader performs the two steps segmentation and classification together in one single step.

flowchart TD

    Image(["image"])
    Segmentation["segmentation"]
    Regions(["regions"])
    FontFile(["font file (*.omc or *.osc)"])
    Classification["classification"]
    Symbols(["symbols"])
    Grouping["grouping"]
    Text(["text"])

    Image --> Segmentation
    Segmentation --> Regions
    Regions --> Classification
    FontFile --> Classification
    Classification --> Symbols
    Symbols --> Grouping
    Grouping --> Text
    Regions --> Grouping

For reading multiple operators are provided: In the easiest case, multiple characters are passed to the reading operators (do_ocr_multi_class_mlp or do_ocr_multi_class_svm). Here, for each region the corresponding name and the confidence are returned. Sometimes, it can be necessary not only to obtain the characters with the highest confidence but also others with lower confidences. A zero, e.g., might easily be mistaken for the character “O”. This information is returned by the operators do_ocr_single_class_mlp and do_ocr_single_class_svm.

As a final step it might be necessary to group digits to numbers or characters to words. This can be realized with the region processing operators used in blob analysis.

Additionally, HALCON provides operators for a syntactic and lexicon-based auto-correction. For example, you can use the operator do_ocr_word_mlp instead of do_ocr_multi_class_mlp to find sets of characters, that is, words, that match a regular expression or that are stored in a lexicon, which was created by create_lexicon or imported by import_lexicon.

Programming examplesđź”—

Some example applications are shown in the following programs:

  • Generating a training file
    %HALCONEXAMPLES%/solution_guide/basics/gen_training_file.hdev

  • Creating and training an OCR classifier
    %HALCONEXAMPLES%/solution_guide/basics/simple_training.hdev

  • Reading numbers
    %HALCONEXAMPLES%/solution_guide/basics/simple_reading.hdev

  • Reading “best before” dates on bottles
    %HALCONEXAMPLES%/hdevelop/Applications/OCR/bottle.hdev

  • Reading engraved text
    %HALCONEXAMPLES%/hdevelop/Applications/OCR/engraved.hdev

  • Reading forms
    %HALCONEXAMPLES%/hdevelop/Applications/OCR/ocrcolor.hdev

  • Segment and select characters
    %HALCONEXAMPLES%/hdevelop/OCR/Segmentation/select_characters.hdev

  • Use syntactic and lexicon-based auto-correction of OCR results
    %HALCONEXAMPLES%/hdevelop/OCR/Neural-Nets/label_word_process_mlp.hdev