Workflow for rule-based OCRđź”—
Basic workflowđź”—
The basic steps apply the following operators in the following sequence.
-
Create an OCR classifier using
create_ocr_class_mlp,create_ocr_class_svm, orcreate_ocr_class_knn. -
Extract the regions that represent the characters that have to be trained.
-
Store the samples, i.e., the training regions and the corresponding class names to a training file. This can be done in different ways. Either
-
store all samples at once using
write_ocr_trainf. Then the regions as well as the corresponding class names have to be available in a tuple. Or -
successively add the individual regions (characters) and their corresponding class names to the training file using
append_ocr_trainf. -
Write characters into a training file with
write_ocr_trainf_image. That is, regions, representing characters, including their gray values (region and pixel) and the corresponding class name are written into a file. An arbitrary number of regions within one image is supported. In contrast towrite_ocr_trainfone image per character is passed. The domain of this image defines the pixels which belong to the character. The file format can be defined by the parameter'ocr_trainf_version'of the operatorset_system.
Additionally, several training files can be concatenated with concat_ocr_trainf.
-
Read the training characters from the training file and convert them into images with
read_ocr_trainfto check the correctness of the training file content. -
Train the OCR classifier with
trainf_ocr_class_mlp,trainf_ocr_class_svm, ortrainf_ocr_class_knn. -
Write the OCR classifier to a font file with
write_ocr_class_mlp(default file extension: .omc),write_ocr_class_svm(default file extension: .osc), orwrite_ocr_class_knn(default file extension: .onc). -
Read the OCR classifier from the font file with
read_ocr_class_mlp,read_ocr_class_svm,read_ocr_class_knn, orread_ocr_class_cnn. -
Extract the regions of the characters that have to be classified according to the trained font.
-
Classify the regions of the characters to be classified. Here, you have different possibilities:
-
Classify multiple characters with an OCR classifier with
do_ocr_multi_class_mlp,do_ocr_multi_class_svm,do_ocr_multi_class_knn, ordo_ocr_multi_class_cnn. -
Classify a single character with an OCR classifier with
do_ocr_single_class_mlp,do_ocr_single_class_svm,do_ocr_single_class_knn, ordo_ocr_single_class_cnn.
Additional optional steps
Besides the basic steps of an OCR classification, some additional steps and operators can be applied if suitable. In particular, you can
-
compute the features of a character with
get_features_ocr_class_mlp,get_features_ocr_class_svm, orget_features_ocr_class_knn, -
return the parameters of an OCR classifier with
get_params_ocr_class_mlp,get_params_ocr_class_svm,get_params_ocr_class_knn, orget_params_ocr_class_cnn, -
compute the information content of the preprocessed feature vectors of an OCR classifier with
get_prep_info_ocr_class_mlporget_prep_info_ocr_class_svm(if Preprocessing was set to 'principal_components’ or 'canonical_variates’; note that this kind of operator is not available for k-NN classifiers, because they do not provide the respective preprocessing options), -
query which characters are stored in a training file with
read_ocr_trainf_names, -
read training specific characters from files and convert them to images with
read_ocr_trainf_select, or -
classify a related group of characters with an OCR classifier with
do_ocr_word_mlp,do_ocr_word_svm,do_ocr_word_cnn,do_ocr_word_knn, ordo_ocr_word_cnn. This is an alternative todo_ocr_multi_class_mlp,do_ocr_multi_class_svm,do_ocr_multi_class_cnn, ordo_ocr_multi_class_knnand is suitable when searching for specific words or regular expressions that are specified in a lexicon that has been created withcreate_lexiconor imported withimport_lexicon.
Besides these operators, some operators are provided that are available only for SVM:
-
To enhance the speed, the trained SVM-based OCR classifier can be approximated with a reduced number of support vectors by applying
reduce_ocr_class_svmafter the training. -
The number of support vectors that are stored within the SVM-based OCR classifier is returned with
get_support_vector_num_ocr_class_svm. -
The index of a support vector from a trained SVM-based OCR classifier is returned with
get_support_vector_ocr_class_svm.
Aligning and rectifying imagesđź”—
Because the reading of characters is not invariant to rotation, it may be necessary to correct the orientation of the image.
This can be achieved either by directly determining the orientation of the text using the operator text_line_orientation, or by locating another object.
Then, the part of the image containing the characters is cropped and aligned using the orientation of the text or the found object.
Similarly to alignment, it may be necessary to rectify the image, e.g., to remove perspective distortions.
For slanted characters, the slant angle can be determined using the operator text_line_slant.
Preprocessingđź”—
Sometimes, the characters may be difficult to extract because of noise, texture, or overlaid structures.
Here, operators like mean_image or gauss_filter can be used to eliminate noise.
A fast but slightly less perfect alternative to gauss_filter is binomial_filter.
The operator median_image is helpful for suppressing small spots or thin lines.
The operator dots_image is optimized to emphasize a dot-print while suppressing other structures in the image.
Gray value morphology can be used to eliminate noise structures and to adapt the stroke width of characters.
Segmenting the imageđź”—
Both for the training samples and for the online reading process, characters must be extracted from the image.
This step is called segmentation.
This means that the OCR operators like do_ocr_single_class_svm do not search for the characters within a given region of interest, but expect a segmented region, which then will be classified.
If the samples for training are taken from real application images, the same segmentation method will be applied for both training and reading. If the training images are more “artificial”, a simpler method might be used to segment the training images.
For the segmentation, various methods can be used.
The Automatic Text Reader is very easy to use and provides robust results.
It combines the two steps segmentation and classification into one call of the operator find_text.
The Manual Text Finder can read engraved text, which cannot be read by the Automatic Text Reader, but it needs a greater effort in setting the parameters.
Therefore, the Automatic Text Reader should be used, if possible.
Both methods use a text model, which can be specified precisely.
Automatic Text Readerđź”—
The Automatic Text Reader segments and classifies text robustly, typically without the need for extensive parameter tuning.
%HALCONEXAMPLES%/solution_guide/basics/simple_reading.hdev and %HALCONEXAMPLES%/hdevelop/Applications/OCR/bottle.hdev provide a good starting point to become familiar with the Automatic Text Reader.
For using the Automatic Text Reader, a model must be created with create_text_model_reader with the parameter Mode set to 'auto'.
Here, an OCR classifier must already be passed.
Segmentation parameters can then be specified with the operator set_text_model_param and can be queried with get_text_model_param.
After this preparation, the text can be read with the operator find_text.
This operator selects character candidates based on region and gray-value features and verifies them with the given OCR classifier.
The character candidates are then further combined to lines which are subsequently tested if they qualify as a text line.
If the text must match a certain pattern or structure, the parameter 'text_line_structure' of the operator set_text_model_param can be set, which determines the structure, i.e., the number of characters for each character block of the text that shall be detected.
The Automatic Text Reader assumes approximately horizontal text.
If the text is not horizontally aligned, the operators text_line_orientation and rotate_image can be used before the use of find_text.
The result of find_text is returned in TextResultID, which can be queried with get_text_result and get_text_object, respectively.
get_text_result returns, e.g., the classification result.
get_text_object returns the iconic result of the Automatic Text Reader, i.e., the character regions.
To delete the result and the text model, use the operators clear_text_result and clear_text_model, respectively.
Manual Text Finderđź”—
If engraved text must be segmented or if no suitable OCR classifier can be provided, the Automatic Text Reader cannot be used. Instead, the Manual Text Finder can be used in these cases.
In order to segment images robustly, the parameters of the Manual Text Finder should be set carefully.
For a practical introduction to the Manual Text Finder, please refer to the example %HALCONEXAMPLES%/hdevelop/Applications/OCR/find_text_dongle.hdev.
For using the Manual Text Finder, a model must be created with create_text_model_reader with the parameter Mode set to 'manual'.
Note that no OCR classifier must be passed in this case.
Segmentation parameters should then be specified with the operator set_text_model_param and can be queried with get_text_model_param.
Note that the names of all parameters that can be used with the Manual Text Finder start with 'manual_'.
After this preparation, the text can be segmented with the operator find_text.
This operator selects character candidates based on region and gray-value features.
The character candidates are then further combined to lines which are subsequently tested if they qualify as a text line.
If the text must match a certain pattern or structure, the parameter 'manual_text_line_structure' of the operator set_text_model_param can be set, which determines the structure, i.e., the number of characters for each character block of the text that shall be detected.
The Manual Text Finder assumes approximately horizontal text.
If the text is not horizontally aligned, the operators text_line_orientation and rotate_image can be used before the use of find_text.
The result of find_text is returned in TextResultID which can be queried with get_text_result and get_text_object, respectively.
get_text_result returns, e.g., the number of found text lines.
get_text_object returns the iconic result of the Manual Text Finder, i.e., the character regions, which can then be classified with a suitable OCR classifier.
To delete the result and the text model, use the operators clear_text_result and clear_text_model, respectively.
General character segmentationđź”—
For the common character segmentation you can either use the operator segment_characters to get a region containing all character candidates and then apply select_characters to select those parts of the region that are candidates for the individual characters, or you use blob analysis.
There, the most simple method is the operator threshold, with one or more gray value ranges specifying the regions that belong to the foreground objects.
Another very common method is dyn_threshold.
Here, a second image is passed as a reference.
With this approach a local instead of a global threshold is used for each position.
Training OCRđź”—
First, the characters from sample images must be extracted using a segmentation method.
To each of the single characters a name must be assigned.
This can be done either by typing it in, by a programmed input in the case of a well-structured image (having, e.g., multiple samples of each character in different lines), or by reading the character names from file.
Then, the regions together with their names are written into training files.
The most convenient operator to do this is append_ocr_trainf.
Before applying the training, we recommend to check the correctness of the training files.
This can, e.g., be achieved by using the operator read_ocr_trainf combined with visualization operators.
flowchart TD
Image(["image"])
Segmentation["segmentation"]
Regions(["regions"])
Combination["combination"]
TrainingFile(["training file (*.trf)"])
Inspection["training file inspection"]
CharacterNames(["character names"])
UserInteraction["user interaction"]
Programmed["programmed"]
TextFile["reading text file"]
Image --> Segmentation
Segmentation --> Regions
Regions --> Combination
Combination --> TrainingFile
TrainingFile --> Inspection
CharacterNames --> Combination
UserInteraction --> CharacterNames
Programmed --> CharacterNames
TextFile --> CharacterNames
Note that you can also train your own system fonts.
By altering and distorting the characters of a font, you can increase the number of different training samples for each class and thus also increase the detection rate.
For this, the Training File Browser of HDevelop’s OCR Assistant can be used.
Furthermore, also the example program %HALCONEXAMPLES%/hdevelop/Applications/OCR/generate_system_font.hdev shows how to derive training data and an OCR classifier from system fonts.
The actual training is depicted in the following flow chart.
flowchart TD
TrainingFile(["training file (*.trf)"])
OCRClassifierNew(["new OCR classifier"])
Training["training"]
OCRClassifierTrained(["trained OCR classifier"])
Save["saving"]
FontFile(["font file (*.omc or *.osc)"])
Test["testing"]
TrainingFile --> Training
OCRClassifierNew --> Training
Training --> OCRClassifierTrained
OCRClassifierTrained --> Save
Save --> FontFile
FontFile --> Test
First, a new classifier is created. There are four different OCR classifiers available: a neural network (multi-layer perceptron or MLP) classifier, a classifier based on support vector machines (SVM), a classifier based on the k-nearest neighbor approach (k-NN), and the box classifier.
Note that if you want to use the Automatic Text Reader for the segmentation and classification of text, you must provide an MLP-based OCR classifier. Otherwise, an OCR classifier based on MLP, SVM, or k-NN can be used. The k-NN has advantages when only few samples are available, but is outperformed by MLP and SVM in typical OCR applications.
The two recommended classifiers differ as follows: The MLP classifier is fast at classification, but for a large training set slow at training (compared to the classifier based on SVM). If the training can be applied offline and thus is not time critical, MLP is a good choice. The classifier based on SVM leads to slightly better recognition rates than the MLP classifier and is faster at training (especially for large training sets). But, compared to the MLP classifier, the classification needs more time.
Dependent on the chosen classifier, you create the classifier using create_ocr_class_mlp or create_ocr_class_svm.
Then, the training is applied using trainf_ocr_class_mlp or trainf_ocr_class_svm.
After the training, you typically save the classifier to disk for later use by write_ocr_class_mlp or write_ocr_class_svm.
Reading symbolsđź”—
First, the characters must be extracted using an appropriate segmentation method.
Here, you must use a method that returns the characters in a form similar to the ones used for training.
After reading the classifier (font file) from file (read_ocr_class_mlp or read_ocr_class_svm), the classifier can be used for reading.
The Automatic Text Reader performs the two steps segmentation and classification together in one single step.
flowchart TD
Image(["image"])
Segmentation["segmentation"]
Regions(["regions"])
FontFile(["font file (*.omc or *.osc)"])
Classification["classification"]
Symbols(["symbols"])
Grouping["grouping"]
Text(["text"])
Image --> Segmentation
Segmentation --> Regions
Regions --> Classification
FontFile --> Classification
Classification --> Symbols
Symbols --> Grouping
Grouping --> Text
Regions --> Grouping
For reading multiple operators are provided:
In the easiest case, multiple characters are passed to the reading operators (do_ocr_multi_class_mlp or do_ocr_multi_class_svm).
Here, for each region the corresponding name and the confidence are returned.
Sometimes, it can be necessary not only to obtain the characters with the highest confidence but also others with lower confidences.
A zero, e.g., might easily be mistaken for the character “O”.
This information is returned by the operators do_ocr_single_class_mlp and do_ocr_single_class_svm.
As a final step it might be necessary to group digits to numbers or characters to words. This can be realized with the region processing operators used in blob analysis.
Additionally, HALCON provides operators for a syntactic and lexicon-based auto-correction.
For example, you can use the operator do_ocr_word_mlp instead of do_ocr_multi_class_mlp to find sets of characters, that is, words, that match a regular expression or that are stored in a lexicon, which was created by create_lexicon or imported by import_lexicon.
Programming examplesđź”—
Some example applications are shown in the following programs:
-
Generating a training file
%HALCONEXAMPLES%/solution_guide/basics/gen_training_file.hdev -
Creating and training an OCR classifier
%HALCONEXAMPLES%/solution_guide/basics/simple_training.hdev -
Reading numbers
%HALCONEXAMPLES%/solution_guide/basics/simple_reading.hdev -
Reading “best before” dates on bottles
%HALCONEXAMPLES%/hdevelop/Applications/OCR/bottle.hdev -
Reading engraved text
%HALCONEXAMPLES%/hdevelop/Applications/OCR/engraved.hdev -
Reading forms
%HALCONEXAMPLES%/hdevelop/Applications/OCR/ocrcolor.hdev -
Segment and select characters
%HALCONEXAMPLES%/hdevelop/OCR/Segmentation/select_characters.hdev -
Use syntactic and lexicon-based auto-correction of OCR results
%HALCONEXAMPLES%/hdevelop/OCR/Neural-Nets/label_word_process_mlp.hdev