monorepo/LCU.API.Collections.RepoApi-old

Repo Api

This readme outlines the developer process, basic coding style guide and provides guidance and details around the technical approach to the Repo Manager API. You should always review and update this document to encompass your changes.

Getting Started

Before you can work with the code, build it, or deploy it you must setup your development environment. The required environment is .NET Core 2.2, Git, and something to work with C#. This environment code has been wrapped up into a Powershell script that should run on any Windows 10 machine and produce a viable development environment to carry you through this guide. The script is available in the root of this directory as setup-env.ps1.

You should start by reviewing and possibly running that script if needed.

Working With The Code

Code is checked out from the Azure DevOps Repository, which is currently [https://landmarkcu.visualstudio.com/Repo/_git/LCU.API.Collections.RepoApi] with the front end code located at [https://landmarkcu.visualstudio.com/Repo/_git/LCU.API.Collections.RepoApi].

It should be noted that this project has many related repositories:

• Collections API - An API to speak to Temenos
• Core API - An API to speak to the APEX/Core system about customers, accounts, and loans.
• RepoClient - The front end client that is the primary consumer of this API.

Work should be done in a branch from Develop.

Start by checking out the repository, and specifically the develop branch. You should be branching off of develop with your own working branch. When you are complete, you will create a pull request within Azure DevOps here: [https://landmarkcu.visualstudio.com/Repo/_git/LCU.API.Collections.RepoApi/pullrequests]

Before submitting your pull request, review the following:

• Did you commit with a commitizen style message? (see below)
• Did you add all needed unit testing, and ensure all existing tests are passing?
• Make sure you didn't accidentally commit any secrets or passwords.
• Have you created a change log if needed?
• Have you bumped the library version if you made changes?

In order to work and submit code, the project will need to be built and run; you will generally do this from Visual Studio.

Resources

There is additional documentation available within the Landmark shared folders; you should check the files tab under the RepoDB team in Microsoft Teams:

• Architecture Document: Landmark Credit Union - Repo Application - SAD
  • This is the software architecture description.
• Site Layout / Reference: Landmark_Site_Reference.XLSX
  • This shows urls/locations for source code and deployables and the Azure configuration

Suggestion for IDE

• Visual Studio 2017
• Visual Studio Code
• IDEA Rider

Releases

Releases are handled by anyone on the team with access to Azure DevOps. Generally when a pull request is accepted to develop, that will be automatically built and deployed to QA. The QA application is here: [https://qa-repoapi.landmarkcu.com]. The same is true for the front end / Client.

To get to UAT, develop must be merged into the master branch. This will automatically cause the main branch to be built and the resulting release to be pushed to UAT.

All releases to UAT are candidates for production. Go to [https://landmarkcu.visualstudio.com/Repo/_release] and get the production release stage that comes after UAT approved to go to production, or cancel it if the release will not go to production.

When creating releases, it's important not to forget to make a release and update the change notes as needed. There is information on creating a release below.

Style Concerns

• For this project, "Code to the codebase"; we use long variable, function, and class names so that the code reads as its own documenation.
• Single character variables are generally accepted for short Linq statements
• For rest services, try to be as RESTful as possible
  • POST/PUT should be idempotent:
  • POSTing to a url twice with the same data should create two uniquely identified objects.
  • PUTing to a url twice should result in the second call having now effect.
  • GET should not create anything
  • POST should be for creating things, but sometimes large forms just make sense to post, even for queries; be feasible.
• Models and other shareable should go in RepoApiLib, when you do this be sure to bump the version number and add a comment with what was added. This will trigger a release of the library within Azure DevOps

Libraries

• NewtonSoft.JSON - Used for conversion of objects to/from JSON
• Sqlite - Used for testing, see - https://jimmybogard.com/avoid-in-memory-databases-for-tests/
• Microsft.FeatureManagement for feature management

Security

Security is handled by a required JWT token that can be retrieved for Azure AD.

The relevant code is at the end of Startup.cs:

    //Set up Authentication
            services
                .AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
                // AzureAD config
                .AddAzureAdBearer(options => Configuration.GetSection("AzureActiveDirectory").Bind( options));
            services.AddIdentity<IdentityUser, IdentityRole>()
                .AddDefaultTokenProviders();

You can then annotate a controller end point like this:

[Authorize(AuthenticationSchemes = JwtBearerDefaults.AuthenticationScheme, Roles = "Admin")]
public async Task<IActionResult> Post([FromBody] AuctionHouse auction)

The other steps required are registering the app with Azure Ad (Azure Active Directory Blade, App Registrations)

There is a separate document in the share that walks through this setup. You have to register both the API and CLIENT, and then create and wire up any roles you want.

Security Groups

Security groups are basically AD Groups mapped to an application role. This application has two roles:

• RepoManagerAdmin
• Specialist

Multiple groups are mapped to these roles to handle both Azure and On-Premise group assignment for individual users. The relevant groups are:

• Azure: REPO_MANAGER_APP_ADMIN
• Azure: REPO_MANAGER_APP_SPECIALIST
• On-Prem: Repo Manager App Admin
• On-Prem: Repo Manager App Specialist

To manage roles or groups, you will need access to Azure; to simply give users access the on-prem groups can be used.

Hosted Services

This application uses queues and hosted services, these are the primary systems. They are all within the HostedServices folder.

• Schedule Service - On a CRON schedule, places a token on the schedule queue.
• Auto Ims - When a schedule token appears, processes Auto IMS imports (Auctions)
• Queue Producer - When a schedule token appears, fills queues with accounts to update, retry, and updates users cache.
• Queue Consumer - When items appear on the queue, they are processed.

Feature Management

This application uses Microsoft's FeatureManagement package to enable/disable features in the code.

This is initialized in startup like so: services.AddFeatureManagement();

The feature flags are configured at their default location in any of the appsettings:

"FeatureFlags" : {  
  "EnableControllers" : true,  
  "RunHostedServices": false  
}

After adding an item to the config it is suggested to add the Flag's string to the FeatureFlag.cs static class under the configuration folder of the main project.

Feature flags can be used in [FeatureGate(*nameOfFlag*)] attributes in controllers to enable and disable them dynamically by changing the config:

namespace LCU.API.Collections.RepoApi.Controllers  
{  
     [Route("api/[controller]")]  
     [ApiController]  
     [FeatureGate(FeatureFlags.EnableControllers)]  
      public class RefundStatusController : RepoControllerBase  
      {

After injecting IFeatureManagement they can also be used inline to prevent entire bloks of code from running:

 if (await featureManager.IsEnabledAsync(FeatureFlags.FeatureA)) { // Run the following code }

Learn more about Microsoft.FeatureManagement : (https://docs.microsoft.com/en-us/azure/azure-app-configuration/use-feature-flags-dotnet-core?tabs=core5x)

Learn more about the history and use cases of Feature Management: (https://martinfowler.com/articles/feature-toggles.html)

Business Details

Basics

Repo API is responsible primarily for three systems:

• CRUD operations on Repo entities such as Repo, Collateral, Borrower, Alert.
• Automated imports and updates of Repo Accounts sourced from Temenos.
• Automated creation and updates of AutoIms auctions; including price setting.

CRUD Operations

Most enties within Repo Manager are read only, or created by the automated process. So the changes that can come in via RESTful events are small. If you are looking for refactoring work, you could lower the RESTful footprint, as most controller methods are likely unused.

The methods used the most will be to update alerts (clear them), create and update auctions, update the receipts for an account, as well as closing accounts.

Automated Imports

One the hour existing accounts in watch status' as well as accounts that failed last hour are automatically re-queued, for new accounts; these now come over from Temenos as events via Azure->Vendor API->Relay API->Rabbit Queue.

Each account is updated to make sure it's fields are correct, and primarily to create Alerts for the business to call users to action on various situations. For example, a vehicle being O4R (out for repo) without a Repo Company assigned to it will create an alert asking a user to assign the repo company.

Automated Auctions

Users on the UX side can create auctions for REPO'd vehicles (vehicles in our possession) once they are beyond their redemption expired date (last date to pay to catch account up to good standing). At this point the vehicle will be retitled and an auction created. The application will submit the auction to this API, which will forward it to AutoIMS.

On schedule, this API will download a spreadsheet from AutoIMS containing a list of auctions and their status - such as received, on-the-block, sold. It will also include pricing and condition information which will be used to automatically update the price of the vehicle - which is sent back to AutoIMS. The auction info is saved locally for easy user access.

Primary Business Contacts

The known primary business contacts are:

• Dean Berendt
• Bill Johns
• Kyle Sternig
• Diana Vandebogert
• Benard Johnson

Temenos Notes and Known Business Rules

• L-WO does not load into Temenos, this is only in XP.
• Without CORS, it still doesn't matter if it is charged off.
• Written off accounts, should still be processed.

Temenos "Go-Live" and Repo Manager 2.0 Notes and Risks

This section outlines the go-live procedure for Repo Manager and outlines a series of risks/notes that developers should keep in mind if they are handling the roll out or the post go-live hypercare. The business has already been shown the application multiple times and Bill has demonstrated he is comfortable using the new Admin screen.

Checklist

• Database is correctly updated on release to production.
• Shutdown the App Server scheduled task runs, they are all now hosted services.
• Confirm accounts can be looked up in Temenos from the Repo Manager Admin screen.
• Monitor jobs via seq and processing receipts.
• Events are arriving to the Queue from Temenos via the Vendor API.

Database

The production database should be backed up before go-live. Since this is automatic, it shouldn't be a problem. The deploy to production will change/create some tables with the dacpac and after the deploy these tables should be checked:

1. ProcessingReceipt table should exist.
2. Repos table should now have these additional fields: ReferenceStatus, ReferenceId, ReferenceCaseId, AccountLastProcessed, and OrigAccountTypeCode

FTP / Auto-IMS

Repo Manager now does the FTP download as a background process known as a HostedService. There is only a production site, so this is difficult to test. Monitor the logs and make sure this goes through, if it doesn't it is likely a security error resulting from:

1. Certificate
2. Landmark Firewall
3. Auto IMS whitelist

You should be able to email [email protected] or get an updated contact from Dean Berendt. Work with him to make sure we're whitelisted. Work with IT on the firewall and cert issue. The privatekey.ppk is located in the project; so you should be fine but if something goes wrong that is a place to look.

Debugging The Auto-IMS FTP Process

1. Ensure the lcu/svc_Repo AD user has Read/Write access to the localDirectoryPath found in the appsettings
   • The user may change in the future, check the user the application is running as in the IIS pool, ensure that it has proper access
2. Ensure the WinSCP.exe is located at the ExecutablePath in the found in the appsettings
3. Ensure the WinSCP.exe version matches the WinSCP nuget package installed in the project
   • It is very important that you do not move an existing WinSCP.exe. If you need to update it, create a new folder in the parent directory for the new version and point the ExecutablePath to it. Other applications in the organization may rely on existing executables.

Repo Processing

The processing of Repo Accounts from Temenos has been designed specifically for debugging due to quality concerns with the Temenos product while we were working through the conversion. Specifically there are a few points of concern:

1. Knowing about events in Temenos is difficult because we don't receive specific events and we can't poll for relevant accounts.
2. The business can change workflows, tables, etc after the go-live. So the system could break on anything.

Once the new system is deployed to prod for the go-live; the processing runs will happen as a hosted service. The likely errors have been mitigated by creating a "Run Receipt". The code for processing accounts was refactored to make less web calls and process accounts in a saga like pattern. When an account is processed, a receipt is produced for that account. When the "batch" of accounts is done the run receipt is emailed; which consists of all of the account receipts.

The new admin screen was designed to help both business and developer alike. The business is empowered to do basic investigations and handle Temenos errors on their own. You can use the Admin screen in Repo Manager to:

1. lookup accounts in Temenos.
2. Force an account onto the queue.
3. View run receipts for that account.
4. Determine if there is a Repo Account for that Temenos Account.

On the main Repo Account screen in Repo Manager, you can also see the new Reference Account and Reference Case ID as well as a last processed date. These three fields should let you trace a Repo Account back to Temenos since those are the Temenos Account/Case numbers. If an account results in an Error or MissingRequirement status - it will be marked for Retry. When the processor runs it queries the processing receipt table for entries marked for retry. It does merge the accounts to those already pending so that an account isn't processed twice.

Finally, run receipt entries look like below. The pipes separate the known fields: REPOACCOUNT#|TEMENOSACCT#|TEMENOSCASE#|

1. The Collections team hasn't create a case for the account yet:
   • ########-###|||RepoManager-Retry -> MissingRequirement : The item ########-#### is missing Recovery Case : 893546 did not have case 0..
2. The account has matching status codes/etc, so for now it is considered unconverted and skipped:
   • ########-###|||RepoManager -> Unconverted : Success
3. Something went wrong, check SEQ:
   • ########-###|||RepoManager -> Error : The item ########-### produced an error. Please investigate.
4. Success!
   • ########-###|519534|2|RepoManager -> Created : Success
   • ########-###|519534|2|RepoManager -> Updated : Success

Temenos Events

Temenos events are posted to the Vendor API in Azure, which posts them via the HCM to the Relay API on premise. This puts the items on the Repo Queue for processing. Bill, Kyle S, or Jim can help you confirm this by creating a Case. You should be able to check the Azure logs (no Seq, it is in Azure, so bring up Kudu, go to the sites folder and open the log). Relay API should receive the event from Vendor API, and that does log to SEQ. Finally; you should see a run receipt as this will cause an item to queue and process.

Other Notes

• When someone forces an account onto the Queue, the receipt will identify them:
  • ########-###|519534|2|[email protected] -> Created : Success
• Failing accounts can retry forever, so the business should do something about them.

Repo Receipt Common Fixes

Missing Letter Descriptions

########-###|||RepoManager-Retry -> MissingRequirement : Letter table requires LetterDescription Column entry."

This error occurs when there was an update to the Letter Descriptions in Temenos that have not been updated in RepoDB. To fix this:

1. Find the 'LetterUpdate.Sql' Query in the LCU.API.Collections.RepoApiSql project
2. Update the Letter Descriptions for Each Action code using the information available in temenos
3. Update the query
4. Copy and Past the query into a RepoDB script and run the script

Missing Repo Company Row

########-###|||RepoManager-Retry -> MissingRequirement : The item ########-### is missing RepoCompanyAssignment - RepoCompanyID : Update or Create a RepoCompany Row with RepoAgentID:

This error occurs when RepoCompany References IDs in Temenos are not properly being mapped to Repo Companies in RepoManager. To fix this:

1. Call this CollectionsAPI route from Postman or the swagger page: https://collectionsapi.lcu-internal.com/api/Configuration
2. Find the "repoAgents" section. Keep this open as you will need to reference this data.
3. Go to the Repo Company admin page in Repo manager: https://repomanage.lcu-internal.com/#/RepoCompanies/
4. For each RepoCompany set its referenceID to the ID in the matching entry in the CollectionsAPI results

 "repoAgents": [
 {
  "id": 20000XX, //This is the referenceID
  "description": "Name of Agent",
  "code": "AgentCode",
  "type": "CUSTOM_REPO_AGENT"
},

If there are any new repoAgents in the collections query, be sure to reach out to the RepoAPI business contacts and confirm the addition before adding it to RepoManager yourself.