Landmark Credit Union Document API
This API is the layer that connects Landmarks Azure Blob Storage instance that stores document content and its Cosmos Mongo DB database that stores relative DocumentInformation and an AuditHistory for all documents in its Blob Storage. It's currently being developed as the central storage location for landmarks documentation. It has an audit log that captures and details changes to document information as an added level of security.
You should always review and update this document to encompass your changes.
Getting Started
The required environment is .NET Core 6.0, Git, and something to work with C#. Code is checked out from the Azure DevOps Repository.
Suggestion for IDE
- JetBrains Rider
- Visual Studio 2022
- Visual Studio Code
Pre Pull Request
- Did you commit with a commitizen style message?
- Did you add all needed unit testing, and ensure all existing tests are passing?
- Make sure you didn't accidentally commit and secrets or passwords.
Configuration
Environment changes are stored and applied as Azure Pipeline Variables, per environment.
MaliciousFileFoundEmailList
This is a set list of emails in the configuration. These emails will be notified through the UpdateDocumentScanStatus endpoint if Azure Defender scans a document and determines it is a malicious file.
FeatureManagement
- GetDocumentCheckOwnership
- This is a boolean flag, that enables a check on document ownership on the GetDocumentByBlobId and GetDocument endpoints. It will compare the given Person or Organization number to see if the returned DocumentInformation contains that number.
Libraries
This application contains the library 'LCU.Library.Internal.DocumentModels' that holds the current request and response models used to communicate with the DocumentAPI. This model is used by the DocumentTypeUploader and DocumentRetriever APIs in their communication with DocumentApi.
Build and Test
Current testing for this project are:
- LCU.API.Internal.DocumentTest
- LCU.API.Internal.DocumentLibTest
- LCU.API.Internal.DocumentModelsTest
- LCU.API.Internal.Document.PerformanceTest
PerformanceTest
This project utilizes BenchmarkDotNet.
Further documentation can be found here: BenchmarkDotNet
Database
There are two main databases that this API communicates with. DocumentService will return one one of these, or join the two in the response model.
Cosmos Mongo DB:
Database: "DocumentInformation"
This holds the DocumentInformation model. This is the metadata of the document, holding any ownership or document type information.
Azure Blob Storage:
Container: "documents"
This is the content byte array of the document itself. It is referenced in the Cosmos Mongo DB "DocumentsInformation" container as DocumentBlobId.
Data Migration
Using a document DB there is no schema in the database (it's in application code) therefore, unlike a relational database, an ALTER -> UPDATE transaction isn't possible. Data changes like an alter, update have to be done with application code which can read the old schema and translate it into the new one or the consuming application code needs to be able to handle data in both the old and the new schema. Past migration code can be found in the migrations solution folder.
Past Migrations
- OwnerNumberMigration
- Document owning numbers (PersonNumbers, OrganizationNumbers) were previously being held in the Attributes, and then parsed and searched through the filter. This was creating performance issues. To fix this, we had to move all of the numbers into new string list properties.
- AccountNumber was also moved into its own property to help with performance.
- We now use these for AccountOwner type document searches. PrimaryOwner type documents are still searched by Attributes.PersonNumber or Attributes.OrganizationNumber (singular). Both searches use the new AccountNumber property.
Helpful Documents
DocumentManager Project File Repository | Platform Wiki
Project Files
- DocumentManager Overview
- DocumentManager High-Level Workflows
- DocumentManager Components
- DocumentType Attributes
- DocumentCodeDetails