Running Tamarin acceptance tests

The Tamarin acceptance tests can be used to verify your installation and or local changes that you have made to the source. To run acceptance tests on the android shell see the "Testing the Android Shell" heading below. Running the Tamarin test suite requires the following steps:

1. Set the AVM environment variable to the path of the AVMPlus executable (avmshell). Note: the name of the avmshell executable varies among the build systems; use the name of the AVM shell produced from the build process you used, above.
2. Set the BUILTINABC environment variable to the path of the generated/builtin.abc.
3. You may build the .abc files build by downloading the latest asc.jar. The latest asc.jar can be downloaded from from ftp://ftp.mozilla.org/pub/js/tamarin...latest/asc.jar. Also the source to asc.jar may be downloaded and rebuilt from http://opensource.adobe.com/wiki/dis...exsdk/Flex+SDK.
4. Set the ASC environment variable, as described above.
5. Set the SHELLABC environment variable to the path of the generated/shell_toplevel.abc.

$ cd tamarin-redux/test/acceptance
$ export ASC=/Users/build/hg/tamarin-redux/utils/asc.jar
$ export BUILTINABC=/Users/build/hg/tamarin-redux/generated/builtin.abc
$ export SHELLABC=/Users/build/hg/tamarin-redux/generated/shell_toplevel.abc
$ export AVM=/Users/build/hg/tamarin-redux/objdir-release/shell/avmshell
$ python runtests.py 
Tamarin tests started: 2010-09-28 10:37:06.410676
current configuration: x64-mac-tvm-release
avm version: 5260:6d1899261bac
Executing 2532 tests against vm: /Users/build/hg/builds/5260-6d1899261bac/mac/avmshell_64
2532 running abcasm/abs_helper.as
  skipping... reason:
2531 running abcasm/adhoc.abs
  skipping... reason: inconsistencies in different debug output, need to implement regex matching of diffs
2527 running abcasm/branchToCommon.abs 
2530 running abcasm/arithmetic.abs 
2529 running abcasm/bkpt.abs 
2528 running abcasm/bkptline.abs 
2526 running abcasm/bug_476556.abs 
2524 running abcasm/bug_491056.abs
...
test run PASSED!
Tests complete at 2010-09-28 10:39:26.751797
Start Date: 2010-09-28 10:38:07.221457
End Date  : 2010-09-28 10:39:26.751797
Test Time : 0:01:19.530340
passes               : 59291
failures             : 0
expected failures    : 75
tests skipped        : 76

The -f flag tells runtests.py to force recompilation of all the scripts. For help on runtests.py options, see below or use python runtests.py -h.

Notes on using acceptance runtests.py

Threading

In order to speed up the execution time of the acceptance testsuite, the runtests script automatically detects the number of cores on the machine and sets an equivalent number of threads.  This can be overrided with the threads flag:

python ./runtests.py --threads=3

Threading on cygwin / windows

Due to an issue with cygwin python, threading does not work, and threads will always be set to 1.  In order to use threading on windows you must install windows python and invoke runtests using that python executable in cygwin.  The locations of AVM, ASC, BUILTINABC and SHELLABC must also be defined using windows paths (replace backslashes [\] with forward slashes [/]):

export AVM=c:/location/of/avmshell.exe
export ASC=c:/location/of/asc.jar
...
c:/Python26/python.exe ./runtests.py --threads=3

Other flags of interest

(For a list of all flags, run runtests.py with -h/--help)

--nohtml : will suppress the creation of an html output file
--rebuildtests : will only rebuild test .abc's, will not attempt to run them.  Can also be using in conjunction with --threads
-f --forcerebuild force rebuild all test files
-q --quiet : will print a period (.) for each test run regardless of results, then print the test run summary when finished
--ascargs= : additional arguments to pass to asc.jar
--vmargs= : additional arguments to pass to avmshell
--random : run tests in random order
--timeout : max time for testrun
--verify : run a verify pass instead of running abcs
--verifyonly : run a -Dverifyonly pass: only checks test exitcode (only works with debugger vms)

Testing the Android Shell

The instructions above apply to running tests on an android device as well, but with a few differences. You should set environment variables as described above except for AVM, which should be:

$ export AVM=$TAMARIN_BUILD_TOP/platform/android/android_shell.py

where $TAMARIN_BUILD_TOP is your main Tamarin repo folder. Note that AVM needs to point to android_shell.py which is a proxy for running the tests on the phone. The /android-public/android-vars.sh script has commented-out examples of the environment settings for running tests on the android avm shell.  To recap the other environment settings:

- Set the BUILTINABC environment variable to the path of the generated/builtin.abc.

- You may build the .abc files build by downloading the latest asc.jar. The latest asc.jar can be downloaded from from ftp://ftp.mozilla.org/pub/js/tamarin...latest/asc.jar. Also the source to asc.jar may be downloaded and rebuilt from http://opensource.adobe.com/wiki/display/flexsdk/Flex+SDK. 

- Set the ASC environment variable, as described above.

- Set the SHELLABC environment variable to the path of the generated/shell_toplevel.abc.

adb

'adb' stands for Android Debug Bridge. adb is used to access your android device and can run both direct and shell-type commands. If you've correctly edited and run the android-vars.sh script mentioned on the Tamarin Build Documentation page you should already be pathed to the adb executable in the public sdk/ndk, but if not it exists at /android-public/android-sdk-mac_86/platform-tools.

adb supports ssh-like commands to an android device. adb automatically connects to a phone if connected by usb. adb shell "ls /" is a good test to see if connected. The command:  

$ adb devices

lists the device id of each phone. If you have more than 1 phone connected to usb, use "adb -s " for each phone. If this fails, do you have a dev OS installed? Under Settings->Applications->Development, is USB Debugging enabled? 

Testsuite Setup

Example commands: 

Copy avmshell to /data/local/tamarin. The shell can be named variously in the build but on the phone the executable must be 'avmshell' 


$ adb shell 'mkdir /data/local/tamarin'
$ adb push avmshell /data/local/tamarin/avmshell
$ adb shell 'chmod 777 /data/local/tamarin/avmshell'

Copy android_runner.sh, if it doesn't already exist on the phone in /data/local/tamarin


$ adb push tamarin-redux/platform/android/android_runner.sh /data/local/tamarin/android_runner.sh
$ adb shell 'chmod 777 /data/loca/android_runner.sh'

Test it out with a simple .abc or no args for usage (should return EXITCODE=0)

$ tamarin-redux/platform/android/android_shell.py hello.abc
hello
EXITCODE=0

Test it out by retrieving the version information of the shell on the android device

$ $AVM -Dversion
shell 1.4 debug build 6299:455bca954565 
features AVMSYSTEM_32BIT;AVMSYSTEM_UNALIGNED_INT_ACCESS;AVMSYSTEM_LITTLE_ENDIAN;AVMSYSTEM_ARM;AVMSYSTEM_UNIX;
AVMFEATURE_JIT;AVMFEATURE_ABC_INTERP;AVMFEATURE_SELFTEST;AVMFEATURE_EVAL;AVMFEATURE_PROTECT_JITMEM;
AVMFEATURE_SHARED_GCHEAP;AVMFEATURE_CACHE_GQCN;AVMFEATURE_SAFEPOINTS;AVMFEATURE_SWF12;AVMFEATURE_SWF13;AVMTWEAK_EXACT_TRACING;

Running Acceptance Tests

--androidthreads sets the number of threads to the number of phones found via usb. Running with 2 or more phones uses threads to run tests in parallel and reduces running times dramatically. You must, in any case, at least use --androidthreads or --threads=1 (or will fail due to multiple tests running). 

$ cd tamarin-redux/test/acceptance
$ ./runtests.py --androidthreads

Note on using an emulator: the emulator does seem to support hardware floating point but gives a BusError when running avmshell -Darm_vfp. It's unreliable, losing its adb connection frequently. An emulator is ok for testing a few abc files but for any consistency a real device is better.