8.0 KiB
Running Benchmarks
dotnet run -c Release
past results
2022.09.09 - local mongodb - "old" regex matching
note: default mongodb indexing is not the same as CosmosDB Mongo default indexing
BenchmarkDotNet=v0.13.2, OS=Windows 10 (10.0.19042.1826/20H2/October2020Update)
11th Gen Intel Core i5-1145G7 2.60GHz, 1 CPU, 8 logical and 4 physical cores
.NET SDK=6.0.400
[Host] : .NET 6.0.8 (6.0.822.36306), X64 RyuJIT AVX2
Job-HPFGMU : .NET 6.0.8 (6.0.822.36306), X64 RyuJIT AVX2
InvocationCount=1 UnrollFactor=1
| Method | DocumentCount | Mean | Error | StdDev |
|---|---|---|---|---|
| GetDocumentInformationByFilterRun | 1 | 3.756 ms | 0.6389 ms | 1.874 ms |
| GetDocumentInformationByFilterRun | 10 | 3.868 ms | 0.6174 ms | 1.820 ms |
| GetDocumentInformationByFilterRun | 100 | 8.027 ms | 1.0094 ms | 2.976 ms |
| GetDocumentInformationByFilterRun | 1000 | 41.938 ms | 7.9785 ms | 23.525 ms |
| GetDocumentInformationByFilterRun | 10000 | 337.205 ms | 51.8841 ms | 152.167 ms |
| GetDocumentInformationByFilterRun | 100000 | 3,429.605 ms | 632.3863 ms | 1,864.605 ms |
| GetDocumentInformationByFilterRun | 250000 | 7,896.811 ms | 1,431.8274 ms | 4,221.775 ms |
| GetDocumentInformationByFilterRun | 500000 | 16,629.300 ms | 2,749.5569 ms | 8,107.131 ms |
| GetDocumentInformationByFilterRun | 750000 | 23,883.445 ms | 4,083.3474 ms | 11,911.316 ms |
| GetDocumentInformationByFilterRun | 1000000 | 32,725.371 ms | 5,751.8744 ms | 16,778.487 ms |
|  |
2022.09.12 - local mongodb - "new" regex matching
BenchmarkDotNet=v0.13.2, OS=Windows 10 (10.0.19042.1826/20H2/October2020Update)
11th Gen Intel Core i5-1145G7 2.60GHz, 1 CPU, 8 logical and 4 physical cores
.NET SDK=6.0.400
[Host] : .NET 6.0.8 (6.0.822.36306), X64 RyuJIT AVX2
Job-ZGOEWK : .NET 6.0.8 (6.0.822.36306), X64 RyuJIT AVX2
InvocationCount=1 UnrollFactor=1
| Method | DocumentCount | Mean | Error | StdDev | Median |
|---|---|---|---|---|---|
| GetDocumentInformationByFilterRun | 1 | 3.924 ms | 0.6009 ms | 1.724 ms | 3.724 ms |
| GetDocumentInformationByFilterRun | 10 | 3.675 ms | 0.6997 ms | 2.063 ms | 3.096 ms |
| GetDocumentInformationByFilterRun | 100 | 5.679 ms | 0.8358 ms | 2.464 ms | 5.332 ms |
| GetDocumentInformationByFilterRun | 1000 | 35.936 ms | 5.6629 ms | 16.697 ms | 37.860 ms |
| GetDocumentInformationByFilterRun | 10000 | 305.437 ms | 53.4368 ms | 157.560 ms | 291.339 ms |
| GetDocumentInformationByFilterRun | 100000 | 2,946.267 ms | 483.7081 ms | 1,418.632 ms | 2,793.991 ms |
| GetDocumentInformationByFilterRun | 250000 | 8,456.390 ms | 1,315.9931 ms | 3,796.941 ms | 8,575.654 ms |
| GetDocumentInformationByFilterRun | 500000 | 15,775.558 ms | 2,830.8954 ms | 8,167.781 ms | 15,527.663 ms |
| GetDocumentInformationByFilterRun | 750000 | 24,948.355 ms | 3,861.0450 ms | 11,323.777 ms | 25,630.684 ms |
| GetDocumentInformationByFilterRun | 1000000 | 30,470.661 ms | 5,342.0572 ms | 15,583.032 ms | 29,073.994 ms |
|  |
2022.09.28 - indexing strategies
BenchmarkDotNet=v0.13.2, OS=Windows 10 (10.0.19042.1826/20H2/October2020Update)
11th Gen Intel Core i5-1145G7 2.60GHz, 1 CPU, 8 logical and 4 physical cores
.NET SDK=6.0.400
[Host] : .NET 6.0.8 (6.0.822.36306), X64 RyuJIT AVX2
ShortRun : .NET 6.0.8 (6.0.822.36306), X64 RyuJIT AVX2
Job=ShortRun IterationCount=3 LaunchCount=1
WarmupCount=3
| Method | Strategy | Mean | Error | StdDev |
|---|---|---|---|---|
| DocumentSearch_ByName | Default | 171.51 ms | 68.40 ms | 3.749 ms |
| DocumentSearch_ByName | FieldSpecific | 91.36 ms | 29.19 ms | 1.600 ms |
| DocumentSearch_ByName | Wildcard | 94.74 ms | 78.58 ms | 4.307 ms |
 |
required tools
- dotnet 6+
--
choco install dotnet-- probably installed with visual studio - R
--
choco install r.projectthen add "C:\Program Files\R\R-4.2.1\bin" to your path, assuming you used the default install directory and the version is 4.2.1 --- better to manually add to the path because setx will only give you 1024 characters. Your path is likely already longer than that and will get truncated. -- this is for outputting visual graphs, not just text tables into standard out
output
- written to the console
.\BenchmarkDotNet.Artifactsrelative to where ever you ran the code from
graphs
The plots should be produced automatically and you'll find them in the results folder.
If not you can manually generate them:
cd .\BenchmarkDotNet.Artifacts\resultsrscript BuildPlots.R
benchmarkdotnet docs
gotchas
Debugging
This will be more difficult than you might be accustomed to. By default benchmark dotnet takes control of a lot of parts of the runtime and even rebuilds some parts of the assembly to ensure isolation and consistent optimzation. See their docs for enabling debugging.
It's possible to run or attach to a release optimized assembly. Some of the things you might expect to be there won't be availabe to the debugger; you're missing debug symbols and the IL has been optimized.
There's always good old console output and file logging.
You might also want to try setting up the problematic scenario in a unit test and then debug that instead of debugging the benchmark itself.
Singletons
Benchmarkdotnet ends up invoking msbuild in order to build separate assemblies (for controlling optimization and isolation). This means that, e.g. Environment.Type = EnvironmentType.Azure; in program.cs will not affect the benchmarks when they are run; Environment.Type will be recreated every benchmark because benchmarks are running in a different assembly than the BenchmarkRunner.Run<T>(); statement which caused the benchmark to run.
One option to deal with this is setting the option inside the benchmark class e.g. in a static constructor (ensuring the value is set before anything else). There are almost surely other options, perhaps a way to pass arguments (other than [Params()]) to the benchmark class.
Azure Rate Limiting
Despite having an Azure environment with code to spin it up and tear it down trying to run a benchmark against a real Azure CosmosDB will likely run you into rate limiting issues. Specifically inserting enough documents to produce a notable database load in a short amount of time. As of 2022.09.13 the rate limit is 1000 records per 5 minutes... You can see where this might produce a problem trying to produce a database on the order of hundreds of thousands of records.
See https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/request-limits-and-throttling#network-throttling for current information.