According to the Paper Diet report a misplaced document can cost a company $125 in lost productivity. It is a solid reason to digitize your paper-based processes if you have not done this before.
That’s when OCR image processing software saves the day. So if document digitization is a pressing issue for your company, we recommend that you build your own OCR system to get a unique solution tailored to the specific needs of your organization.
We use the term optical character recognition or OCR for short to define the technology that is used to recognize a printed or handwritten text inside an image and convert it into an electronic form for further data processing, as in searching or editing.
Businesses can derive the following benefits from using OCR systems:
In this article, we will discuss the most common and efficient approaches to OCR image processing and show how you can implement them on your own.
These days companies most commonly digitize the most important documents related to their operational activities, such as bills and invoices, official correspondences, receipts, financial statements, contracts, etc.
However, today we will delve into the business card recognition process. Wonder why? The thing is that this document is considered one of the most complicated ones for recognition since it contains lots of various data including name, address, phone number, email address.
Here we should mention that each business card has a unique design. It makes the OCR text recognition much more difficult.
The main problem is people have to enter information from the Visit Cards manually. And we wanted to create a business card reader that will allow us to use iOS devices’ cameras to extract Visit Cards’ owner name, email and the country name. And, possibly, other data on demand.
The solution can be split in several stages:
First we need to capture the video frame by frame from the camera. We create AVSession with the output that captures the video data:
- (AVCaptureSession*)startSessionWithInput:(AVCaptureDeviceInput*)input
{
AVCaptureSession* session = [[AVCaptureSession alloc] init];
AVCaptureVideoDataOutput* videoOutput = [AVCaptureVideoDataOutput new];
NSString* key = (__bridge NSString*)kCVPixelBufferPixelFormatTypeKey;
videoOutput.videoSettings = @{ key : @(kCVPixelFormatType_32BGRA) };
[videoOutput setSampleBufferDelegate:self queue:self.videoProcessingQueue];
return session;
}Capture frame needed to split video into frames sequence:
- (void)captureOutput:(AVCaptureOutput*)output
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection*)connection
{
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(imageBuffer, 0);
Sample sample;
size_t bpr = CVPixelBufferGetBytesPerRow(imageBuffer);
sample.width = CVPixelBufferGetWidth(imageBuffer);
sample.height = CVPixelBufferGetHeight(imageBuffer);
sample.bpp = bpr / sample.width;
sample.data = CVPixelBufferGetBaseAddress(imageBuffer);
sample.orientation = connection.videoOrientation;
// Process sample
CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
}Where the Sample is our custom structure to bridge between Objective-C and C++ code:
typedef struct
{
size_t width;
size_t height;
size_t bpp;
AVCaptureVideoOrientation orientation;
const void* data;
} Sample;Once we have raw image data we want to detect all the text on it. But here we have 2 issues:
To solve this problem we can detect the rectangular frame of the Visit Card, cut it out from the raw image and unwrap it horizontally. Let’s use OpenCV for this
First let’s prepare the image for edge detection:
// Make it grayscale
cv::cvtColor(img, gray, cv::COLOR_BGR2GRAY);
// Unsharp mask aka inverse gaussian blur
t0 = img.clone();
t1 = img.clone();
cv::GaussianBlur(img, t0, cv::Size(15, 15), 0);
cv::subtract(img, t0, t1);
// Pyramid filter
cv::Mat pyr;
cv::Mat timg;
cv::Mat res;
cv::pyrDown(img, pyr, cv::Size(img.cols/2, img.rows/2));
cv::pyrUp(pyr, res, img.size());Now let’s extract edges:
cv::Canny(res, gray, 0, thresh, 5);
cv::dilate(gray, gray, cv::Mat(), cv::Point(-1, -1));
std::vector< std::vector<cv::Point> > contours;
cv::findContours(gray, contours, RetrievalModes::RETR_LIST,
ContourApproximationModes::CHAIN_APPROX_SIMPLE);After that we can pick the best contour matching following criteria:
Now we have a contour, 4 vertices for quad bounding the card. For it we can build a matrix to transform the visit card area into aligned separated rectangular image
std::vector<cv::Point2f> src;
src.push_back(best.vertex(0));
src.push_back(best.vertex(1));
src.push_back(best.vertex(2));
src.push_back(best.vertex(3));
std::vector<cv::Point2f> dst;
dst.push_back(dest.vertex(0));
dst.push_back(dest.vertex(1));
dst.push_back(dest.vertex(2));
dst.push_back(dest.vertex(3));
Mat warpMatrix = cv::getPerspectiveTransform(src, dst);
cv::Mat rotated;
warpPerspective(img, rotated, warpMatrix, size,
INTER_LINEAR, BORDER_CONSTANT);Eventually we get something like that:
Now when we have an unwrapped visit card we can easily detect text on it. For this we can use Google Natural Language library. It’s the best available offline solution to recognize English text
FIRVision* vision = [FIRVision vision];
FIRVisionTextRecognizer* rec = [vision onDeviceTextRecognizer];
[rec processImage:img
completion:^(FIRVisionText* text, NSError* error) {
// Here we have a recognized text
}];Here is what we get for the particular image:
Stanford University
75 Alta Road
Stanford, CA 94305
T650.736.2290 (office admin)
F650.736.2460
amil@stanford.edu
http://med.stanford.edu/cerc.html
https:/aicare.stanford.edu/
ARNOLD MILSTEIN
Professor and Director
Clinical Excellence
Research CenterOnce we have the text from the card we can use iOS built-in libraries to get address, organization name and URLs (since email is kind or URL in Apple’s opinion). Here is how we can achieve this. First let’s search for names with NSLinguisticTagger:
NSArray* s = @[NSLinguisticTagSchemeNameType];
NSLinguisticTagger* tagger = [[NSLinguisticTagger alloc] initWithTagSchemes:s
options:0];
NSArray* tags = @[NSLinguisticTagPersonalName,
NSLinguisticTagPlaceName,
NSLinguisticTagOrganizationName];
tagger.string = s;
NSRange r = NSMakeRange(0, [s length]);
NSUInteger options = NSLinguisticTaggerOmitPunctuation |
NSLinguisticTaggerOmitWhitespace |
NSLinguisticTaggerJoinNames;
[tagger enumerateTagsInRange:r
unit:NSLinguisticTaggerUnitWord
scheme:NSLinguisticTagSchemeNameType
options:options
usingBlock:^(NSLinguisticTag tag,
NSRange tokenRange,
BOOL* stop)
{
// Here we have a result
}];After it let’s search for links and addresses with NSDataDetector:
NSUInteger types = NSTextCheckingTypePhoneNumber |
NSTextCheckingTypeAddress |
NSTextCheckingTypeLink;
NSRange r = NSMakeRange(0, [s length]);
NSDataDetector* detector = [NSDataDetector dataDetectorWithTypes:types
error:&error];
NSArray* matches = [detector matchesInString:s
options:0
range:r];We’re almost there. Point is the address is always written in freestyle. For example, the United States can be written as US, USA, United States, United States of America etc. Plus, some addresses may not include the country at all, however it’s clear that they belong to a particular country. To deal with this we can use a geocoding service. Apple’s Core Location allows to turn the address into a placemark which contains ISO country code.
CLGeocoder* geocoder = [[CLGeocoder alloc] init];
[geocoder geocodeAddressString:fullAddress
completionHandler:^(NSArray<CLPlacemark*>* placemarks,
NSError* error)
{
// placemarks.firstObject.ISOcountryCode is what we need
}];For this image the address is “75 Alta Road Stanford, CA”, but we don’t have an explicit mention of a country:
But this address is totally enough for geocoder to find a location. And this location has ISO country alpha-2 code:
75 Alta Road Stanford CA => USNow we actually parsed all the data. But the problem is every pipeline iteration gives us a bit different result. It can be caused by camera shake, illumination changes, interference in text recognition of designer’s text etc. But if we add a frequency filter for name, country and email we can pick the most stable result among recent ones. Which gives us a pretty nice accuracy in live recognition:
Current approach for OCR text recognition has some restrictions:
However, the solution appeared to be just good enough for solving the particular problem, which is OCR image processing of business cards. It works fast enough on any device not older than iPhone 6s and provides a good recognition accuracy in proper lighting conditions. Plus, it recognizes the email in 99.5% cases, which is already a good achievement
Solution can be improved and extended in case of necessity: