OCR Dynamics CRM email attachments

I am very excited about this blog. I was watching a Logic App demo in one of the Microsoft Ignite videos where an order record was created in an ERP system from an order image using Optical Character Recognition (OCR). I am trying to do the something similar using CRM email attachments.

Scenario

The scenario is when a new CRM email with the image attachments is received/created, the image attachment will be processed by OCR, and the response will be used to create a new Lead record in CRM.

Solution Overview

The most exciting part of this solution is the number of Azure services used in this solution. The Logic App is used as integration layer for the solution. The following diagram depicts the solution architecture and process workflow of the solution.


This is just a learning exercise.  In the above diagram, we could have merged steps 2 to 6 in the Azure Function (webhook receiver) and pass the end result of step 6 to the Logic App.

Process Flow Steps

• When a new attachment record is created/received in CRM, it passes the RemoteExecutionContext to Azure Function (webhook receiver).
• The webhook receiver reads the attachment body from the entity attributes and calls the Logic App with the attachment body.
• The Logic App calls the cognitive services Vision API and passes the body of the attachment.
• The Vision API runs OCR on the image and passes the response back to Logic App.
• The response received by the Logic App has some unwanted data, so that response is passed to a second Azure Function (OCR Parser).
• The OCR parser transforms the response, assigns it to a lead object (class) and sends it back as JSON string.
• The Logic App parses the JSON string.
• The Logic App uses the 'Dynamics 365-Create a new record' action to create a new Lead record in D365.

Components

Azure Function as webhook receiver

The first step is to create an Azure function which will receive the RemoteExecutionContext from CRM when a new email attachment is created.This Azure function receives the CRM context, extracts the attachment body and sends it to a Logic App in "binary" format with a content-type of “application/octet-stream”. This is important as Azure Vision API does not receive the image contents in  “application/json” content-types.

Webhook registration in CRM

Register a webhook and add a step on the Create event (Post) of the Attachment entity using the Plugin Registration Tool.

 

Azure Function for OCR parsing

The OCR data has a lot of extra information like regions, lines and boundaries. The parser reads the required data from the response and assigns it to a Lead class. This class is then serialized into a JSON string and sent back to the Logic App.
I got OCR processing classes from another blog. I can't find a name of it now.

Logic App

1. Create a new Logic App and add an HTTP trigger. The URL of the request is used by the Azure function (webhook receiver).
   
2. Add an Optical Character Recognition (OCR) to JSON action. For this step register here to get the URL and key for “Computer Vision API”. Enter the connection information and after the connection is established, set Image Source to Image Content, and Image Content to the Body of the HTTP trigger.
   
   
   If successful, this step will return the OCR response.
3. Add an HTTP action and enter the required information as shown. The action will return the serialized JSON string of lead object.
   
4. Add a JSON Parser action that will parse the JSON string received in the last step.
   
5. Add a new Dynamics 365-Create a new record action. Assign the values based on the schema of the JSON parser.
   
6. Save the Logic App and test the solution.

Outcome

I tested the solution by sending an email with the following image attachment.


This is the result in D365.

Issues I faced

1. Originally, I wanted to use a Logic App trigger  (Dynamic 365 -When a record is created) to get the attachment body. For some reason, this trigger does not work for the attachment (activity mime attachment) entity.
2. Even if you can get the attachment using the Get Lists action by passing the Email entity guid, the result does not contain the body of the attachment. I don’t know if this is by design. In short you cannot get attachment body by using the Logic App D365 connector. This is why I had to use CRM webhooks and Azure functions.
3. Sometimes the OCR API does not recognize all the text on the image. For example, in the screen shot above letter “l” is missing in the email address (satnam@live.com versus satnam@ive.com).

Conclusion

I believe this integration scenario has massive potential. It can be used to:

1. Create records based on hand written notes.
2. Automatically link attachments to appropriate records based on some value on an image or in PDF documents.
3. Some federal department applications, banking loan applications and RMS (Road and Maritime Services) still use paper based applications. This can save them tons of data entry.