Performance failure mode #10061

mjkkirschner · 2019-10-14T21:36:09Z

Purpose

still a work in progress

Adds an additional check to the comparison command of the DynamoPerformanceTest CLI -

Compare command will now also return a Pass/Fail state
This state is logged to the console as part of the comparison results
this state is determined by a function - the default function just checks the increase in mean time is not > 10% - the ability to pass different functions for different tests in the future is possible.
I have temporarily removed all but one performance graph just to speed up my tests while developing this and related CI changes.
If any test is marked as FAIL - then the CLI will exit with a non zero error code and log a message to the console.

TODO:

verify that non zero exit code will mark the jenkins build a failure
fix log file - ~~result is not serialized there for some reason even though it shows in console log.~~ rather it is just on the wrong line.

see related changes:
https://git.autodesk.com/Dynamo/DynamoPerformance-CI/pull/8

Declarations

Check these if you believe they are true

The code base is in a better state after this PR
Is documented according to the standards
The level of testing this PR includes is appropriate
User facing strings, if any, are extracted into *.resx files
All tests pass using the self-service CI.
Snapshot of UI changes, if any.
Changes to the API follow Semantic Versioning, and are documented in the API Changes document.

Reviewers

@QilongTang
@reddyashish

add log text for result add comparisonData property to each Comparer and use it

mjkkirschner · 2019-10-15T18:29:37Z

 Comparison of Model tests: 
12:10:31  
12:10:31  | Method |   Graph | Version |        Mean |    Error |   StdDev | Result |
12:10:31  |--------|---------|---------|-------------|----------|----------|--------|
12:10:31  |   Open | Aniform |    Base |   126.83 ms |  2.07 ms |  0.54 ms |
12:10:31  |        |         |     New |   134.50 ms |  2.59 ms |  0.67 ms |
12:10:31  |        |         |         |      +6.05% |   25.28% |   25.28% |PASS
12:10:31  |    Run | Aniform |    Base | 3,779.43 ms | 82.77 ms | 49.26 ms |
12:10:31  |        |         |     New | 3,678.10 ms | 66.58 ms | 39.62 ms |
12:10:31  |        |         |         |      -2.68% |  -19.56% |  -19.56% |PASS
12:10:31  
12:10:31  Comparison of View tests: 
12:10:31  
12:10:31  exception while comparing results System.IO.FileNotFoundException: Could not find file 'C:\Jenkins\workspace\Dynamo\DynamoPerformance-CI\dyn1884\baselines\DynamoPerformanceTests.DynamoViewPerformanceTestBase-report.csv'.
12:10:31     at Microsoft.VisualBasic.FileIO.TextFieldParser.ValidatePath(String path)
12:10:31     at Microsoft.VisualBasic.FileIO.TextFieldParser.InitializeFromPath(String path, Encoding defaultEncoding, Boolean detectEncoding)
12:10:31     at DynamoPerformanceTests.ResultsComparer.ImportResultsCSV(String csvPath) in c:\DynamoPerformance-CI\Dynamo\tools\Performance\DynamoPerformanceTests\ResultsComparer.cs:line 329
12:10:31     at DynamoPerformanceTests.ResultsComparer.CreateComparisons(String basePath, String newPath) in c:\DynamoPerformance-CI\Dynamo\tools\Performance\DynamoPerformanceTests\ResultsComparer.cs:line 270
12:10:31     at DynamoPerformanceTests.Program.Compare(String baseResultsPath, String newResultsPath, String savePath) in c:\DynamoPerformance-CI\Dynamo\tools\Performance\DynamoPerformanceTests\PerformanceTestConsoleApp.cs:line 143 
12:10:31   Could not find file 'C:\Jenkins\workspace\Dynamo\DynamoPerformance-CI\dyn1884\baselines\DynamoPerformanceTests.DynamoViewPerformanceTestBase-report.csv'.
12:10:31

QilongTang

LGTM with a few comments

tools/Performance/DynamoPerformanceTests/PerformanceTestConsoleApp.cs

tools/Performance/DynamoPerformanceTests/ResultsComparer.cs

reddyashish · 2019-10-16T18:38:27Z

Looks good to me.

mjkkirschner · 2019-10-16T19:08:05Z

@QilongTang addressed your comments, I'm having trouble getting to build so that I can check the failure - I'm going to close the PR and reopen after I am done testing. - whoops, need to close PR on the CI side - sorry.

PTAL.

mjkkirschner · 2019-10-17T00:25:57Z

when theres a failure we'll see something like this:
@aparajit-pratap @smangarole @QilongTang

20:24:03  |--------|---------|---------|-------------|-----------|----------|--------|
20:24:03  |   Open | Aniform |    Base |   126.83 ms |   2.07 ms |  0.54 ms |
20:24:03  |        |         |     New |   123.60 ms |   1.09 ms |  0.39 ms |
20:24:03  |        |         |         |      -2.55% |    -47.3% |  -27.63% |PASS
20:24:03  |    Run | Aniform |    Base |   779.43 ms |  82.77 ms | 49.26 ms |
20:24:03  |        |         |     New | 3,797.70 ms | 152.90 ms | 90.99 ms |
20:24:03  |        |         |         |    +387.24% |    84.73% |   84.73% |FAIL
20:24:03  
20:24:03  Comparison of View tests: 
20:24:03  
20:24:03  exception while comparing results System.IO.FileNotFoundException: Could not find file 'C:\Jenkins\workspace\Dynamo\DynamoPerformance-CI\dyn1884\baselines\DynamoPerformanceTests.DynamoViewPerformanceTestBase-report.csv'.
snip
  Comparison failed, some model performance benchmarks failed. Please see log above for details.
20:24:03  localBuildContainer
20:24:03  failed during test stage
20:24:03  + throw "failed during test stage"

This reverts commit 6134760.

mjkkirschner added 2 commits October 14, 2019 15:17

temp remove graphs to speed up testing

6134760

add comparison failure exit code

c92602f

add log text for result add comparisonData property to each Comparer and use it

mjkkirschner added the WIP label Oct 14, 2019

mjkkirschner added 3 commits October 14, 2019 17:47

fix saved file

ccaabe2

remove extra pipe | from log

2625559

remove new line

66052f7

mjkkirschner requested a review from QilongTang October 15, 2019 18:30

mjkkirschner added PTAL Please Take A Look 👀 and removed WIP labels Oct 15, 2019

mjkkirschner requested a review from reddyashish October 15, 2019 18:30

mjkkirschner changed the title ~~[WIP] Performance failure mode~~ Performance failure mode Oct 15, 2019

QilongTang reviewed Oct 15, 2019

View reviewed changes

address comments

008feb2

mjkkirschner closed this Oct 16, 2019

mjkkirschner reopened this Oct 16, 2019

QilongTang added LGTM Looks good to me and removed PTAL Please Take A Look 👀 labels Oct 16, 2019

Revert "temp remove graphs to speed up testing"

d1abfc5

This reverts commit 6134760.

mjkkirschner merged commit a97eeee into DynamoDS:master Oct 17, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Performance failure mode #10061

Performance failure mode #10061

Uh oh!

mjkkirschner commented Oct 14, 2019 •

edited

Loading

Uh oh!

mjkkirschner commented Oct 15, 2019

Uh oh!

QilongTang left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

reddyashish commented Oct 16, 2019

Uh oh!

mjkkirschner commented Oct 16, 2019 •

edited

Loading

Uh oh!

mjkkirschner commented Oct 17, 2019

Uh oh!

Uh oh!

Performance failure mode #10061

Performance failure mode #10061

Uh oh!

Conversation

mjkkirschner commented Oct 14, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

still a work in progress

Declarations

Reviewers

Uh oh!

mjkkirschner commented Oct 15, 2019

Uh oh!

QilongTang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

reddyashish commented Oct 16, 2019

Uh oh!

mjkkirschner commented Oct 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mjkkirschner commented Oct 17, 2019

Uh oh!

Uh oh!

mjkkirschner commented Oct 14, 2019 •

edited

Loading

mjkkirschner commented Oct 16, 2019 •

edited

Loading