Business Process Automation Accelerator

Responsible AI Check Required On First Deployment

Ensure that you have accepted terms and conditions for Responsible AI:
You must initiate the creation of a “Cognitive services multi-service account” from the Azure portal to review and acknowledge the terms and conditions by checking the checkmark.
Once accepted, you can create subsequent resources using any deployment tool (SDK, CLI, or ARM template, etc) under the same Azure subscription.

Without OpenAI

With OpenAI

With OpenAI and Redis Enterprise (check pricing) for Vector Search

Deploy to Azure Instructions

1. Get a Workflow Level Token: Create AND save personal access token
2. Fork the repository to a git account of which you are the Admin.
3. Click on the “Deploy to Azure” Button. If you do not have access to OpenAI, choose the first one.
4. Only the Resource Group, Project Name, Repo Token (from #2), and Forked Git Repo Url are needed. The remaining parameters are filled in for you.

Latest Backend : https://bpasource.blob.core.windows.net/source/backendv015.zip?sv=2020-04-08&st=2023-03-03T21%3A47%3A55Z&se=2023-11-21T21%3A47%3A00Z&sr=b&sp=r&sig=3rXVZCv%2BWvQ3TnMq33wsvQr4Gxvayt7%2FA3g%2BgQduKHw%3D

Deploy instructions below are only if you are doing development!! If you are not adding features to BPA, use the “Deploy to Azure” buttons above.

Overview

This accelerator provides a no code Studio for users to quickly build complex, multi-stage AI pipelines across multiple Azure AI and ML Services. Users can select, and stack, AI/ML Services from across Azure Cognitive Services (Speech, Language, Form Recognizer, ReadAPI), Azure Machine Learning, and even Hugging Face state-of-the-art models, into a single, fully integrated pipeline. Integration between services is automated by BPA, and once deployed, a web app is created. This customizable UI* provides and drag-n-drop interface for end users to build multi service pipelines. Finally, the user-created pipeline is triggered as soon as the first input file(s) are uploaded, storing the results in a CosmosDB.

And, optionally, the end-user can add a Cognitive Search Index, indexing each the output at each of stage of the pipeline, across all uploaded files. Index creation is handled entirely by the accelerator, where the end-user can access the index via their newly created Cognitive Search Index resource

High-level architecture

*Note: Azure Static Web Apps allows you to easily build React apps in minutes. Use this repo with the React quickstart to build and customize a new static site and automate the deployment of a functional, and customizable, POC for text and language processing.

Contents

Architecture

Once you’ve created a high-level Resource Group, you’ll fork this repository and importing helper libraries, taking advantage of Github Actions to deploy the set of Azure Cognitive Services and manage all of the new Azure module credentials, in the background, within your newly created pipeline. After pipeline deployment, a static webapp will be created with your newly customizable POC UI for building and triggering pipelines.

High-level technical architecture

The UI provides and drag-n-drop interface for end users to build multi-service pipelines:

1. Starting with specifying input file type(s), e.g. audio, texts, or emails
2. Pipeline stages are added by selecting which Cognitive Services to add to the pipeline
3. Once a new pipeline stage is added, if the data is transformed to a new format type (e.g. OCR’ing a PDF to text), the new pipeline stages will become available to be added based on the transformed format of the data at the latest stage of the pipeline
4. Once the pipeline is specified, the user will upload files through the UI, triggering the pipeline, and saving the result in a CosmosDB
5. Optionally, the end-user can add a Cognitive Search Index, indexing each the output at each of stage of the pipeline, across all uploaded files. Index creation is handled entirely by the accelerator, where the end-user can access the index via their newly created Cognitive Search Index resource

BPA UI

Sample Pipeline

A simple example pipeline for a call center mining use case is below is illustrated below. For a general call center mining use case, the Business Process Accelerator could be used to deploy a pipeline ingesting, transcribing, (+optionally translating), summarizing, and analyzing the sentiment for each call.
Resulting Pipeline Steps [AI Service Deployed]:

1. Transcribe audio calls [Speech Service]
2. Translate to another language (optional) [Speech Service]
3. Store the transcribed and translated calls [CosmosDB]
4. Apply suite of text analytics offerings (e.g. text summarization, sentiment analysis, call classification) [Language Service]

The resulting deployment pipeline leverages:
Azure Speech Service, Azure Language Service, Azure Cosmos DB, Azure Functions

Sample BPA pipeline for call center mining

Once the pipeline is completed – this process typically takes <1 min for smaller documents and simpler pipelines – the results are found in your newly created Azure Cosmos DB, where we can quickly inspect our results.

Currently Included Services

The current release of BPA allows you to build pipelines from multiple Cognitives Services, Azure Machine Learning Endpoints, and even Hugging Face models. New Services and Features are continuously being released. Please refer to each Service’s documentation for the latest reference.

Available Services

• Azure Cognitive Services
  • Form Recognizer
  • Language Service
  • Speech Service
  • Cognitive Search
• Azure Machine Learning Endpoints
• HuggingFace Tokenization Models

Form Recognizer Models

Type
Model
Description

Prebuilt
Read (preview)
Extract printed and handwritten text lines, words, locations, and detected languages.

General document (preview)
Extract text, tables, structure, key-value pairs, and named entities.

Layout
Extract text and layout information from documents.

W-2 (preview)
Extract employee, employer, wage information, etc. from US W-2 forms.

Invoice
Extract key information from English and Spanish invoices.

Receipt
Extract key information from English receipts.

ID document
Extract key information from US driver licenses and international passports.

Business card
Extract key information from English business cards.

Custom
Custom
Extract data from forms and documents specific to your business. Custom models are trained for your distinct data and use cases.

Custom
Composed
Compose a collection of custom models and assign them to a single model built from your form types.

Form Recognizer Models Documentation

Language Service Models

Type
Model
Description

Prebuilt
Named Entity Recognition (NER)
This pre-configured feature identifies entities in text across several pre-defined categories.

Personally Identifiable Information (PII) detection
This pre-configured feature identifies entities in text across several pre-defined categories of sensitive

Key phrase extraction
This pre-configured feature evaluates unstructured text, and for each input document, returns a list of key phrases and main points in the text.

Entity linking
This pre-configured feature disambiguates the identity of an entity found in text and provides links to the entity on Wikipedia.

Text Analytics for health
This pre-configured feature extracts information from unstructured medical texts, such as clinical notes and doctor’s notes.

Custom
Custom NER
Build an AI model to extract custom entity categories, using unstructured text that you provide.

Prebuilt
Analyze sentiment and opinions
This pre-configured feature provides sentiment labels (such as “negative”, “neutral” and “positive”) for sentences and documents. This feature can additionally provide granular information about the opinions related to words that appear in the text, such as the attributes of products or services.

Custom
Custom text classification (preview)
Build an AI model to classify unstructured text into custom classes that you define.

Prebuilt
Text Summarization (preview)
This pre-configured feature extracts key sentences that collectively convey the essence of a document.

Language Service Models Documentation

Speech Service

The Speech service provides speech-to-text and text-to-speech capabilities with an Azure Speech resource. You can transcribe speech to text with high accuracy, produce natural-sounding text-to-speech voices, translate spoken audio, and use speaker recognition during conversations.
Speech Service Documentation

Cognitive Search

Cloud search over private heterogeneous content, with options for AI enrichment if your content is unstructured or unsearchable in raw form.
Cognitive Search Documentation

Hugging Face Implementation

Many of the pretrained models from the Hugging Face library can be used, depending on the task selected! Find more information at https://huggingface.co/models?pipeline_tag=text-classification&sort=downloads

Hugging Face model repository

Prerequisities

1. Github account
2. Azure Command Line interface installed. The Azure Command-Line Interface (CLI) is a cross-platform command-line tool that can be installed locally on Windows computers. You can use the Azure CLI for Windows to connect to Azure and execute administrative commands on Azure resources. This accelerator uses Azure CLI to deploy a set of services provided by the templates within your cloned repo, in just a few lines of code.
   (https://docs.microsoft.com/en-us/cli/azure/install-azure-cli)
3. Ensure your subscription has Microsoft.DocumentDB enabled
   To confirm/enable:
   • Navigate to your subscription within portal.azure.com
   • Select Resource Providers at bottom of left navigation pane
   • Within the Filter by name menu, search for Microsoft.DocumentDB
   • Once Microsoft.DocumentDB is found, check if the status is marked as “Registered”. If marked as “NotRegistered”, Select “Register”
     Note: This process may take several seconds/minutes, be sure to refresh the entire browser periodically
4. Ensure that you have accepted terms and conditions for Responsible AI:
   You must initiate the creation of a “Cognitive services multi-service account” from the Azure portal to review and acknowledge the terms and conditions. You can do so here: Quickstart: Create a Cognitive Services resource using the Azure portal.
   Once accepted, you can create subsequent resources using any deployment tool (SDK, CLI, or ARM template, etc) under the same Azure subscription.

Installation Steps

1. Create a Resource Group in your Azure Portal

Create your Resource Group.
Note: When naming your Resource Group, please use lower case, alphanumeric characters only, as multiple Azure Services will be created later on, in the background, using your Resource Group name

Select your preferred Region

https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/manage-resource-groups-portal

It will take several seconds for your Resource Group to be created.

For more help, refer to https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/manage-resource-groups-portal

2. Fork the repo

Fork https://github.com/Azure/business-process-automation to your github account.
For basic instructions for forking repos, please refer to https://docs.microsoft.com/en-us/azure/devops/repos/git/forks?view=azure-devops&tabs=visual-studio
Note: a Microsoft organization GitHub account is not required

3. Create AND save personal access token

1. On your Git Hub repo page, click on your profile
2. Select Settings (under your profile icon in the top right)

3. Select Developer settings at bottom of left navigation pane

4. Select Personal access tokens
5. Select Generate new token
6. Under Select scopes, select the checkbox for “workflow”
   
7. Add your own note
8. Select Generate token (at bottom of page)
9. Copy your newly generated token
   Note: Be sure to save this token for completing pipeline setup, else this token will need to be regenerated

For more help refer to https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate

4. Update depoloyment template files within local repo

In step 4, you’ll clone your repo locally, and make small updates to two deployment template files, which will automate deployment of multiple Azure resources within your Resource group.

Navigate to and open for editing, templates/parameters.json.example in your local directory

1. Open a local command window
2. Clone the forked repo locally

git clone https://github.com/<your-account>/business-process-automation

3. Navigate to your templates/parameters.json.example within your local repo

cd business-process-automation/templates

4. Rename the file to “parameters.json”
   Note: This is an important step to ensure successful deployment. The file can be renamed via the command line, a local File Explorer browser, or after opening via Visual Studio Code (see the below note)

ren parameters.json.example parameters.json

5. Open parameters.json
   Note:*If you have Visual Studio Code installed, you can launch at the current directory by typing “code .”

C:\Users\<UserName>\business-process-automation\templates\code .

Update the three “value” fields below:

1. projectName: Must be a unique project name, only include lowercase, alphanumeric characters only, and keep length between ~10-16 characters. This name is used across multiple Azure resources, many with strict project naming conventions  
2. repository token: Copy the personal access token you recently created  
3. repository url: Paste the link of your forked repository - i.e., https://github.com/[your-account]/business-process-automation  

Save updates

6. Similarly, update your local deploy.yml file within the same directory
7. Open the file C:\Users<UserName>\business-process-automation\templates\deploy.yml
8. Update the field AZURE_FUNCTIONAPP_NAME with the name of your ResourceGroup
   Note: You will later use this file to create a GitHub workflow, kicking off configuration of your Resource Group pipeline components
   

Save updates

5. Run initial deployment configuration

1. In your local repository, navigate to the ‘templates’ directory
2. Run the below deployment code in your cmd window, after updating with your own project name and Resource Group

  az deployment group create --name [yourProjectName] --resource-group <YourResourceGroup> --template-file main.json --parameters parameters.json

Note: Be sure to maintain spaces while updating the project name and resource group
Note: This may take several minutes to run

3. When this has completed you should have the application infrastructure deployed to your resource group. You will see confirmation of numerous created Azure resources in your command window. Navigate to your Resource Group at your Azure Portal confirm to confirm your newly created resources.
   azure.portal.com

6. Collect the Published Profiles from your newly created Azure Function Apps

In this sequence of steps you will retrieve credentials from your two newly created Azure Function Apps, and add them as Secrets in your GitHub repo.

1. You will have two new function apps deployed within your Resource Group. Navigate to your Resource Group at your Azure Portal.
   One will start with the name “huggingface”.

Open the “huggingface” function app and in the “overview” tab there will be a button “Get publish profile” in the top center, which will then download a file. This will download as “[YourProjectName].PublishSettings.txt”
Note: It may take several seconds for the button to appear

2. Open the downloaded file, and copy the contents (to be pasted in upcoming steps)
3. Navigate back to your forked repo, go to Settings (local settings in the middle center) -> Secrets -> Actions

4. Select ‘New Repository Secret’

• Paste the below name into the “Name” field

AZURE_HF_FUNCTIONAPP_PUBLISH_PROFILE

• Paste the contents of your recently downloaded “[YourProjectName].PublishSettings.txt” file into the “Value” field

5. Repeat Steps 1-4 above the same process for the second newly created Azure Function App within your Resource Group, with the same name as your project name.
   Note: For step 4 above, this second secret will be named

AZURE_FUNCTIONAPP_PUBLISH_PROFILE

6. Create Github Action to your Function Apps, deploying your front and back end resources

1. Navigate to “actions” tab
2. Select “new workflow”
3. Select set up workflow yourself
   
4. This will take you to the editor for the main.yml file.
   Delete all of the contents within the main.yml file. Copy all of the contents from your deploy.yml file from your local directory.
   (C:\Users[UserName]\business-process-automation\templates\main.json) into the body.
   
   Finally, paste that selection into the editor window.
5. Run the workflow and select commit new file
   Note: Once you’ve run your workflow once, you’ll want to delete previous workflow runs to prevent buildup of old workflows.
   • Select “Start Commit”
   • Select “Commit New File”
6. View the progress of your actions under the “Actions” tab. This process will take several minutes to complete.
   You can also view real-time logs from your Azure Function Apps:
   • Navigate to your Azure Portal, and select your Function APP, named after your project name
   • Select “Log Stream” in the left navigation pane (towards the bottom; may have to scroll down)
   • Switch the stream from “File System Logs” to “App Insights Logs” via the drop down menu, directly above your log window

Navigate to your React App!

1. Navigate to your Resource Group within your Azure Portal
2. Select your static webapp
3. Within the default Overview pane, Click on your URL, which will take you to your newly launced WebApp!

Load Documents!

1. Select Configure a New Pipeline
   Important: When naming your pipeline, please choose a length greater then 5 characters to satisfy naming requirements of various Azure Services created through the pipeline
   
2. Select your input file(s) type
   Note: The .pdf input tile can accomodate several image input file format types (including, JPEG, PNG, BMP, TIFF) More information at https://docs.microsoft.com/en-us/azure/applied-ai-services/form-recognizer/faq
   Generally, please refer to the supported file types required by the various Azure Cognitive Services (documentation sections linked in Reference).
3. Next, continue building your pipepline by selelecting which analytical service you would like to apply:
   • Depending on your selection, new analytical services will appear. For example, if you start your pipeline with a .pdf document, your available Services within the pipeline builder will show Services compatible with .pdf inputs (e.g. Form Recognizer General Layout Pretrainted model). After adding a pipeline step, such as the aforementioned Layout Service, your data may be transformed to a different output (in this case text), and your pipeline builder will now show Services compatible with text inputs (e.g. Language Services such as text summarization).

Alternatively, you can first OCR the raw image to text, by selecting Form Recognizer’s OCR Service, after which your app will show new options that are available to you for processing converted text

4. To complete your pipeline, select the tile “Export Last Stage to DB”
5. Navigate to “Home”

6. Select “Ingest Documents”
7. Upload your first document! Once your document upload is completed, You’ll see a message indicating “Upload Successful”. You can upload more than one document at a time here.
Note: The first document loaded may take several minutes. However, all subsequent documents should be processed much faster

Using HuggingFace library

When using a custom Hugging Face model, you’ll be prompted select your desired model by specifiying the model name. To do so, you’ll need to navigate to https://huggingface.co/models, filter on “Token Classification” (task), “PyTorch” (libraries), and “transformers” (libraries). Select your model from the filtered model list, and copy the modelID. See Images below for further guidance – if you’re not using a HuggingFace model, skip to the next step.
Note: When pasting the model ID, be sure to paste without quotes – see below image

Specifying Hugging Face model name

Searching and filtering for BPA compatible Hugging Face models

Retreiving the Hugging Face model name

View Your Results

Your results will be stored in a Cosmos database within your Azure Resource Group.

• Navigate to your cosmosDB in your Azure Resource Portal (which will also have the same name as your project name)
  
• Navigtate to your Data Explorer
  • For help on using Azure Cosmosmos DB explorer for the first time, please see Work with data using Azure Cosmos DB Explorer
• The “items” folder (image below) will contain your pipeline output
  • You should see a container, named after your project name. Select that container
  • You will see a SQL query, named after your project name. Select that query
  • Within that query, Select Items. Here you should see multiple items
    • The first item will be your pipeline metadata
    • The second will be contain the output from your first document
    • An additional item will be created for each uploaded document
      

If a Cognitive Search index was added to the pipeline, you will need to navigate to your newly created Cognitive Search Resource within that same Resource Group.

Generated Cognitive Search Index

Further Customization

All code for the front end React-based UI is provided for further customization (see references for designing and customizing React apps).

Generally, Services with text outputs (e.g. Azure Language Service, and many Services from Form Recognizer and Speech Service) will return a JSON response, where the key output is passed to the next stage of the pipeline for continued processing, or finally stored in a CosmosDB (see image below). Each of these intermediate outputs, will also be added to a Cognitive Search Index, if the feature is added to the end of the pipeline.

Leveraging this accelerator as part of a broader pipeline is encouraged! The AI Rangers / AI Specialist CSA teams would love to hear about future use cases. See references for more contact information.

Contacts

Please reach out to the AI Rangers for more info or feedback aka.ms/AIRangers

Roadmap

Priority
Item

Impending
Adding instructions on basic UI customizations (e.g. adding header graphics, changing title, etc..)

TBD
…

References

Subject
Source (Link)

React source template
This project was bootstrapped with Create React App

Azure Form Recognizer
https://docs.microsoft.com/en-us/azure/applied-ai-services/form-recognizer/concept-model-overview

Azure Language Service
https://docs.microsoft.com/en-us/azure/cognitive-services/language-service/overview

Azure Speech Service
https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/overview

Azure Cognitive Search
https://azure.microsoft.com/en-us/services/search/

HuggingFace
https://huggingface.co/models?pipeline_tag=text-classification&sort=downloads

Additional Model Documentation
tbd