monorepo/LCU.API.Internal.Document/LCU.API.Internal.Document.PerformanceTest

Running Benchmarks

dotnet run -c Release

past results

2022.09.09 - local mongodb - "old" regex matching

note: default mongodb indexing is not the same as CosmosDB Mongo default indexing

BenchmarkDotNet=v0.13.2, OS=Windows 10 (10.0.19042.1826/20H2/October2020Update)
11th Gen Intel Core i5-1145G7 2.60GHz, 1 CPU, 8 logical and 4 physical cores
.NET SDK=6.0.400
  [Host]     : .NET 6.0.8 (6.0.822.36306), X64 RyuJIT AVX2
  Job-HPFGMU : .NET 6.0.8 (6.0.822.36306), X64 RyuJIT AVX2

InvocationCount=1  UnrollFactor=1  

Method	DocumentCount	Mean	Error	StdDev
GetDocumentInformationByFilterRun	1	3.756 ms	0.6389 ms	1.874 ms
GetDocumentInformationByFilterRun	10	3.868 ms	0.6174 ms	1.820 ms
GetDocumentInformationByFilterRun	100	8.027 ms	1.0094 ms	2.976 ms
GetDocumentInformationByFilterRun	1000	41.938 ms	7.9785 ms	23.525 ms
GetDocumentInformationByFilterRun	10000	337.205 ms	51.8841 ms	152.167 ms
GetDocumentInformationByFilterRun	100000	3,429.605 ms	632.3863 ms	1,864.605 ms
GetDocumentInformationByFilterRun	250000	7,896.811 ms	1,431.8274 ms	4,221.775 ms
GetDocumentInformationByFilterRun	500000	16,629.300 ms	2,749.5569 ms	8,107.131 ms
GetDocumentInformationByFilterRun	750000	23,883.445 ms	4,083.3474 ms	11,911.316 ms
GetDocumentInformationByFilterRun	1000000	32,725.371 ms	5,751.8744 ms	16,778.487 ms


2022.09.12 - local mongodb - "new" regex matching

BenchmarkDotNet=v0.13.2, OS=Windows 10 (10.0.19042.1826/20H2/October2020Update)
11th Gen Intel Core i5-1145G7 2.60GHz, 1 CPU, 8 logical and 4 physical cores
.NET SDK=6.0.400
  [Host]     : .NET 6.0.8 (6.0.822.36306), X64 RyuJIT AVX2
  Job-ZGOEWK : .NET 6.0.8 (6.0.822.36306), X64 RyuJIT AVX2

InvocationCount=1  UnrollFactor=1  

Method	DocumentCount	Mean	Error	StdDev	Median
GetDocumentInformationByFilterRun	1	3.924 ms	0.6009 ms	1.724 ms	3.724 ms
GetDocumentInformationByFilterRun	10	3.675 ms	0.6997 ms	2.063 ms	3.096 ms
GetDocumentInformationByFilterRun	100	5.679 ms	0.8358 ms	2.464 ms	5.332 ms
GetDocumentInformationByFilterRun	1000	35.936 ms	5.6629 ms	16.697 ms	37.860 ms
GetDocumentInformationByFilterRun	10000	305.437 ms	53.4368 ms	157.560 ms	291.339 ms
GetDocumentInformationByFilterRun	100000	2,946.267 ms	483.7081 ms	1,418.632 ms	2,793.991 ms
GetDocumentInformationByFilterRun	250000	8,456.390 ms	1,315.9931 ms	3,796.941 ms	8,575.654 ms
GetDocumentInformationByFilterRun	500000	15,775.558 ms	2,830.8954 ms	8,167.781 ms	15,527.663 ms
GetDocumentInformationByFilterRun	750000	24,948.355 ms	3,861.0450 ms	11,323.777 ms	25,630.684 ms
GetDocumentInformationByFilterRun	1000000	30,470.661 ms	5,342.0572 ms	15,583.032 ms	29,073.994 ms


2022.09.28 - indexing strategies

BenchmarkDotNet=v0.13.2, OS=Windows 10 (10.0.19042.1826/20H2/October2020Update)
11th Gen Intel Core i5-1145G7 2.60GHz, 1 CPU, 8 logical and 4 physical cores
.NET SDK=6.0.400
  [Host]   : .NET 6.0.8 (6.0.822.36306), X64 RyuJIT AVX2
  ShortRun : .NET 6.0.8 (6.0.822.36306), X64 RyuJIT AVX2

Job=ShortRun  IterationCount=3  LaunchCount=1  
WarmupCount=3  

Method	Strategy	Mean	Error	StdDev
DocumentSearch_ByName	Default	171.51 ms	68.40 ms	3.749 ms
DocumentSearch_ByName	FieldSpecific	91.36 ms	29.19 ms	1.600 ms
DocumentSearch_ByName	Wildcard	94.74 ms	78.58 ms	4.307 ms

required tools

• dotnet 6+ -- choco install dotnet -- probably installed with visual studio
• R -- choco install r.project then add "C:\Program Files\R\R-4.2.1\bin" to your path, assuming you used the default install directory and the version is 4.2.1 --- better to manually add to the path because setx will only give you 1024 characters. Your path is likely already longer than that and will get truncated. -- this is for outputting visual graphs, not just text tables into standard out

output

• written to the console
• .\BenchmarkDotNet.Artifacts relative to where ever you ran the code from

graphs

The plots should be produced automatically and you'll find them in the results folder.

If not you can manually generate them:

• cd .\BenchmarkDotNet.Artifacts\results
• rscript BuildPlots.R

benchmarkdotnet docs

https://benchmarkdotnet.org/

gotchas

Debugging

This will be more difficult than you might be accustomed to. By default benchmark dotnet takes control of a lot of parts of the runtime and even rebuilds some parts of the assembly to ensure isolation and consistent optimzation. See their docs for enabling debugging.

It's possible to run or attach to a release optimized assembly. Some of the things you might expect to be there won't be availabe to the debugger; you're missing debug symbols and the IL has been optimized.

There's always good old console output and file logging.

You might also want to try setting up the problematic scenario in a unit test and then debug that instead of debugging the benchmark itself.

Singletons

Benchmarkdotnet ends up invoking msbuild in order to build separate assemblies (for controlling optimization and isolation). This means that, e.g. Environment.Type = EnvironmentType.Azure; in program.cs will not affect the benchmarks when they are run; Environment.Type will be recreated every benchmark because benchmarks are running in a different assembly than the BenchmarkRunner.Run<T>(); statement which caused the benchmark to run.

One option to deal with this is setting the option inside the benchmark class e.g. in a static constructor (ensuring the value is set before anything else). There are almost surely other options, perhaps a way to pass arguments (other than [Params()]) to the benchmark class.

Azure Rate Limiting

Despite having an Azure environment with code to spin it up and tear it down trying to run a benchmark against a real Azure CosmosDB will likely run you into rate limiting issues. Specifically inserting enough documents to produce a notable database load in a short amount of time. As of 2022.09.13 the rate limit is 1000 records per 5 minutes... You can see where this might produce a problem trying to produce a database on the order of hundreds of thousands of records.

See https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/request-limits-and-throttling#network-throttling for current information.