Actionscript Performance Tests

Performance tests for tamarin are located in the test/performance directory.  Similar to the acceptance tests set the environment variables AVM, ASC, BUILTINABC.  The runtests.py test harness works by parsing the abc stdout for lines like: 'metric time 566'.  The testconfig.txt file defines groups of tests to run (e.g. sunspider, scimark, language).  By default the time for each test is output.  Use the --memory flag to capture the maximum private bytes memory (high water mark) for a test.  By default each test is run a single time.  To run each test multiple times use the --iterations flag.  When the number of iterations is greater than 2 all times are displayed in [], faster time is displayed in the output column, and the 95% confidence interval is calculated.  The 95% confidence interval is the calculated percentage of the mean value to fall within 95% of the measured results.  For example if the mean is 100 and the 95% confidence interval is 2%, 95% of the expected values should fall between 98 and 102.  To compare two vms against each other export AVM2=<path to 2nd vm> or use --avm2=<path to 2nd vm>.   The percentage difference is calculated between the two vms. See the --help usage for more details about performance tests.

# to run a group of tests (e.g. sunspider)
$ ./runtests.py sunspider/*.as sunspider/as3/*.as
test                                                   avm  metric
sunspider/access-binary-trees.as                       584    time 
sunspider/access-fannkuch.as                           364    time 
...
# to set vm args (e.g. interp)
$ ./runtests.py --vmargs="-Dinterp" sunspider/*.as
test                                                   avm  metric
sunspider/access-binary-trees.as                       690    time 
sunspider/access-fannkuch.as                           467    time 
...
# to measure memory 
$ ./runtests.py --memory sunspider/*.as
test                                                   avm  metric
sunspider/access-binary-trees.as                      2.4M  memory
sunspider/access-fannkuch.as                          1.5M  memory
...
# change the iterations
$ ./runtests.py --iterations=10 sunspider/*.as
test                                                   avm     95% conf  metric
sunspider/access-binary-trees.as                       565        1.0%    time  [565, 586, 573, 577, 567, 567, 579, 576, 565, 585] 
sunspider/access-fannkuch.as                           361        0.3%    time  [365, 366, 362, 363, 366, 361, 363, 363, 364, 364] 
...
# compare 2 builds
$ ./runtests.py --avm2=build2/avmshell sunspider/*.as
test                                                   avm    avm2     %sp  metric
sunspider/access-binary-trees.as                       565     564     0.2    time 
sunspider/access-fannkuch.as                           366     362     1.1    time 
...
# compare 2 builds with > 2 iterations
$ ./runtests.py --iterations=10 --avm2=build2/avmshell sunspider/*.as

test                                                                avm                   avm2
                                                       min :   max     avg       min :   max     avg   %diff  metric
                                                   -----------------------   -----------------------   -----
sunspider/access-binary-trees.as                   [ 567.0 :   585]  575.9   [ 564.0 :   580]  572.0     0.5    time 
sunspider/access-fannkuch.as                       [ 363.0 :   386]  370.0   [ 362.0 :   399]  371.8     0.3    time 
...