monorepo/LCU.API.Internal.DocumentIngestor/README.md

Introduction

---

The DocumentIngestor API was created to ingest large amounts of documents. These large requests generally come from an Index file which is comprised of many upload document requests. To accomplish this, DocumentIngestor API utilizes Landmarks micro service APIs such as FileOperations API, CoreApiWrapper, and DocumentTypeUploader API.

Getting Started

---

The required environment is .NET Core 6.0, Git, Docker Desktop, RabbitMQ, and a local FileOperationsApi instance.

• To run this API you will need a running RabbitMQ on Docker desktop
  • Run the following in a cmd
    • docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3-management
    • Access rabbitmq management UI http://localhost:15672/#
    • UserName: guest, Password: guest
• You will need a local FileOperations and DocumentTypeUploader running
  • FileOperations and DocumentTypeUploader are queue based APIs this application will need to be run locally. For the DocumentIngestor to hear back when this has been performed, it must give the API a ResponseItem. This uses the localhost rabbitmq set up. This should not use the QA RabbitMQ as the QA API would then pick up the request and can't communicate to your local rabbitmq.
  • FileOperations
    • DevOps Repo: https://landmarkcu.visualstudio.com/Platform/_git/LCU.API.Internal.FileOperations
    • When a FileOperations request such as MoveFile is requested it is placed on a queue.
  • DocumentTypeUploader
    • Devops Repo: https://landmarkcu.visualstudio.com/Platform/_git/LCU.API.Internal.DocumentTypeUploader
    • After an UploadRequest is completed, DocumentTypeUploader will return a resulting DocumentId to DocumentIngestor to be saved with its DocumentProcessingInfo.
• If not done already, you will need to publish your local instance of the DocumentIngestorSql database

Suggestions for IDE

• JetBrains Rider
• Visual Studio 2022

Pre Pull Request

Code is checked out from the Azure DevOps Repository. Before submitting your pull request, review the following:

• Did you commit with a commitizen style message?
• Did you add all needed unit testing, and ensure all existing tests are passing?
• Make sure you didn't accidentally commit secrets or passwords.

Configuration

---

Configuration is in appsettings.json, and updated with environment specific fields from vault and config mapping

Instances

Currently, multiple instances of DocumentIngestor are running and need to be updated in configuration:

• DocumentIngestor
• DocumentIngestor-Preprocessor (Going away)
• DocumentIngestor-Historical

Build and Test

---

Currently unit tests are the only level of testing that exist for this project.

Steps For Adding New Document Ingestors

---

Index file format has already been created:

1. Add DocumentCode(s) to LCU.Library.DocumentType
   • Make sure to put it under the correct ownership type enum.
   • Add what it is listed in the index file as the enum's "Name" annotation.
2. Add new DocumentCodeDetail(s) in DocumentManager

Index file format has NOT already been created:

1. Follow steps in "Index file format has already been created"

2. Create new Enum IngestorTypes

   • LCU.Library.Internal.DocumentIngestorModels --> Enums

3. Create new IndexFileDocument Model

   • This model will reflect what values are pulled from the Index file and is a part of the ProcessingRequest.
   • LCU.Library.Internal.DocumentIngestorModels --> Models --> IndexFileDocuments

4. Create a service class to parse the index file, create IngestorRunInfo, and create ProcessingRequest(s)

   • This class will house all the custom work to parse index files/get documents that need to be uploaded and to create a list of ProcessingRequest.
   • Every IndexParserServices class should contain a main method of HandleIndexRequest that returns a IngestorResult.
   • Exceptions should be handled and should log failure and messages to the IngestorResult.
   • Namespace: LCU.API.Internal.DocumentIngestor.Services.IndexParserServices
     • LCU.API.Internal.DocumentIngestor --> Services --> IndexParserServices
   • Example: FosTaxStatementIndexService

5. Add the new IngestorType to the ServicesFactoryConfiguration in the appsettings - IIndexParserService

   • This will add the newly created IndexParserService class to the IndexServices.cs --> method PerformIngestor

6. Determine if the documents need a PdfParserService to ingest

   • This would be if multiple documents are attached in one larger PDF file.
   • If so:
     1. Create PdfParserService for ingestion type
     2. Add the service to the ServicesFactoryConfiguration in appsettings - IPdfIndexParserService

7. Create new service class to handle document processing

   • This class will perform all the custom work.
     • Validation of the IndexFileDocument Model values
     • Calls classes such as PrimaryOwnerDeterminingService to get the Attributes
     • Builds out UploadRequest
     • Gets pdf content in bytes
     • Uploads Document using the DocumentTypeUploaderApiService
   • Namespace: LCU.API.Internal.DocumentIngestor.Services.DocumentProcessorServices
     • LCU.API.Internal.DocumentIngestor --> Services --> DocumentProcessorServices
   • Example: FosTaxStatementProcessingService

8. Add the new IngestorType to the ServicesFactoryConfiguration in the appsettings - IDocumentProcessingService

   • This will add the newly created DocumentProcessorService class to the DocumentProcessorService.cs --> method PerformDocumentProcess

9. Add unit tests for any services created

10. Add attributes component to DocumentManager for new ingestion type

After adding a new ingestion type:

1. Update NuGet packages for the following API's:
   • LCU.API.Internal.Dispatcher
   • LCU.API.Internal.DocumentTypeRetriever
   • LCU.API.Internal.DocumentTypeUploader
   • LCU.WebApps.Internal.DocumentManager
2. Add or update Dispatcher job to ingest new type

Services

---

IndexService

The purpose of this service class is to call IngestorType's index parsing classes.

1. This class is kicked off once an IngestorRequest has been consumed by the IngestorIndexConsumer.
2. The IngestorRequest has an IngestorTypes enum which is used in a switch to determine what Indexing class's HandleIndexRequest will be called.
3. Once the Ingestors Type's HandleIndexRequest was performed, an IngestorResult will be returned. This will contain a FatalFailure if a list of ProcessingRequest was not able to be created.
4. These ProcessingRequests will then be placed on the Processing queue (LCU.DocumentIngestor.Processing) with method SendProcessRequest.

DocumentProcessorService

The purpose of this service class is to call IngestorType's processing classes.

1. This class is kicked off once an IngestorRequest has been consumed by the IngestorProcessingConsumer.
2. The request will be checked if the document exists and isn't already archived.
   • If archived, processing complete. The ProcessingRequest and empty ProcessResult can be placed on LCU.DocumentIngestor.Completion queue.
3. The ProcessingRequest's IngestorTypes enum will be used in a switch to determine what Processor class's ProcessRequest method will be called.
4. ProcessingRequest method is called and returns a ProcessResult
   • If failed processing, process complete. The ProcessingRequest and ProcessResult can be placed on LCU.DocumentIngestor.Completion queue.
5. Processing successful document is set up to be archived by FileOperations
   • FileOperations will place the response on LCU.DocumentIngestor.Completion queue.
   • If the call to FileOperations throws an exception, the result is changed to false. The ProcessingRequest and ProcessResult are placed on LCU.DocumentIngestor.Completion queue.

CompletionService

The purpose of this service class is to Update the IngestorRunInfo count and to finalize the IngestorRunInfo if total count is equal to the Success+Failure count.

1. This class is kicked off once an LCU.Library.Internal.ApiResponse.Models.Response has been consumed by the queue IngestorProcessingConsumer (LCU.DocumentIngestor.Completion).

BaseOwnerDeterminingService

The purpose of this service class is to reach out to our micro service APIs to determine the owners of the document based off the information given in the index file.

• Different methods are available based on the information given for the ingestion type.
• Generally, we need at least two fields of data to verify the ownership of the document.
• Different services are available, based on the ownership type of the document type.

PdfPartitionerService

The purpose of this service is to determine what and how the larger PDF needs to be parsed into smaller documents for ingestion. The request is then sent to FileOperations to process.

• This service is only required if documents were sent from the vendor in a batch PDF.
• Currently, this process is kicked off by Dispatcher.
• Each ingestion type that requires this service, needs its own service class. (See "Steps For Adding New Document Ingestors")

MemberDocumentNotifierService

Resources: Original MemberDocumentNotifier Workflow Design

The purpose of this service is to generate email notifications to the owners of successfully ingested documents.

1. This class is kicked off in UploadCompletionService once a document has been successfully uploaded, and the resulting DocumentId returned.
2. The request is then sent to MemberNotificationInfoDeterminingService to gather all of the required EAgreeUserIds
3. MemberDocumentNotifierService then takes the request to:
   • Obtain the updated email from MemberPreferences
   • Update the status of the MemberNotificationInfo
   • Send a new email request to NotificationApi, grouped by EAgreeUserId and EmailAddress
4. MemberNotificationCompletionService then takes the response from the email request to:
   • Update the status of the MemberNotificationInfo, based on the result
   • Updates MemberPreferences if the email was either "Successful" or "Undeliverable" (Failed 3 times)

DocumentReprocessingService

The purpose of this service is to gather DocumentProcessingInfos to resubmit on the DocumentProcessing queue (LCU.DocumentIngestor.Processing) as a ProcessingRequest.

This is most commonly meant to override the owning entity from DocumentManager with an OwnerOverrideInfo. Queuing with an OwnerOverrideInfo will skip the OwnerDeterminingServices and ingest the document with the submitted entity values.

UploadCompletionReprocessingService

The purpose of this service is to gather DocumentInformation for DocumentTypeRetriever and resubmit a CompletionRequest to the UploadCompletion queue (LCU.DocumentIngestor.UploadCompletion). Mainly, to reprocess all DocumentProcessingInfos with a ProcessStatus of NotificationDataFailure.

This will only work if the DocumentProcessingInfo has a DocumentId, meaning the document was uploaded successfully. This will also kick off a new MemberNotificationInfo.

IngestionFolderCleanupService

Resources: Original IngestorFolderCleanUp Workflow Design

The purpose of this service is to delete zip files and directories that have already been ingested, therefore we no longer need the files.

Helpful Documents

---

DocumentManager Project File Repository | Platform Wiki

Project Files

• DocumentManager Overview
• DocumentManager High-Level Workflows
• DocumentManager Components
• DocumentType Attributes
• DocumentCodeDetails

Workflow Designs:

• Original DocumentIngestor Workflow Design
• Original MemberDocumentNotifier Workflow Design
• Original IngestorFolderCleanUp Workflow Design
• Original BatchIngestion Workflow Design