About
-----
The project aimed to do an empirical study of Ekstazi tool by running the tool across multiple projects and revisions. The respective project results are then compared against executions without the Ekstazi tool with respect to test reduction time and reduction count. 
The report details the approach we have followed from the initial discussion with Professors and TA, through the evolution of code and finally analyzing the results.

Project approach
-----------------
We started our project by working to create automated scripts which can execute tests ( without Ekstazi changes ) across a number of revisions for some given list of projects and gather test results. We had planned to target GIT + Maven projects first and then extend to SVN + Maven and other combinations.
To execute the same set of projects with Ekstazi pom changes, we initially made manual pom changes for few projects and compared the results. To automated this process, we planned to have a set of reference poms for each project and use it for subsequent revisions.

Procedure followed for populating reference poms.
------------------------------------------------
We start with the pom changes for the base revision of the project. 
As long as the pom for the next successive revision match the base revision pom, we can directly use the pom changes made on the base revision to execute. We found this observation to hold true for many projects and thus implemented the code which compares the poms across revision and in case they are found to be same, it saves the modified pom of the base version to be used for the respective version. This procedure is followed only once per project and revision. For successive test runs, we thus avoid prompting the user with vimdiff as long as have saved revision. 

Learning from progress1 report
------------------------------
Change in the order of execution.
During the execution without Ekstazi , the order of execution ( older to newer or otherwise ) does not matter much, for execution with Ekstazi changes, we need the order of execution to be from older to newer to properly gauge the performance of Ekstazi. 

Preserving the Ekstazi dependencies and sanitizing the clone.
---------------------------------------------------------------
We had to include the check to clean the clone after every execution to prevent any undesirable effects of stale. Also since the Ekstazi depends on the exclusion list under .Ekstazi, we had to move around the folder before cleaning the directory and bring it back at the appropriate location before each execution. There was a challenge in code to extend this functionality even in cases where we had multiple .Ekstazi folders under various sub modules.

Populating fresh snapshots
---------------------------
Some times it was observed that the saved snapshots were older and had to be reinstalled. This was fixed by adding mvn install -DskipTests prior to test execution.
The above learnings were applied in code for SVN repositories as well.

Learnings from progress2 and progress3 report
---------------------------------------------
Execution of large projects with Ekstazi
----------------------------------------
As per the suggestion to run Ekstazi tool on larger projects, we integrated Ekstazi for guava-libraries and Netty projects, both took more than 8 minutes to execute the test cases. The logs for both can be found at logs/woEks and logs/wEks folder for without and with Ekstazi executions respectively.

Change in execution time calculation:
-------------------------------------
The script was calculating thus far the total time elapsed by the summation of time taken for each individual test per revision. This was wrong since for runs executed with Ektazi , the time elapsed always will be zero in case it selects no test case to run. Instead we modified the code to calculate time taken by reading the maven test summary result directly. The script has been corrected and tables updated for all the previous run. 

Change in POM modification code.
----------------------------------
We were made aware of another team working in parallel to modify the pom for Ekstazi. The team had used java pom parser code to directly modify / append pom as per the requirements. While our method of populating a reference pom and using the saved reference poms for subsequent execution still works, we tried to integrate their parser logic into our project for the pom modification module of the code alone. 

We started with few projects where the team had reported success using their tool. We compared the results of executions with our code and with their tool and found it to be same for the initial set of projects. We now support both the methods of executing tests for Ekstazi with an optional flag passed at runtime to either make use of reference POMs or modifiy poms directly using their tool.

Project Directory Structure:
----------------------------------------
/scripts
	All our scripts are available under the “scripts” directory

/logs
	All execution logs uploaded under “logs” directory and is organized as below.

/repoDir
	This is the default location where the repositories are cloned by the script. Please note this directory is under svn ignore list.

/wEkstazi & /woEkstazi
	The two directories contain the execution logs for with Ekstazi changes and without respectively. Their nomenclature is in <project>_<with/withoutEkstazi>.log format.

/issues
	This contains the results of few projects where failures were observed during execution. They also contain the re run logs of projects available under wEkstazi/woEkstazi directories

/pomReferenceDir
	We store every project’s reference pom here. They are structured as
Pom modified for Ekstazi is saved under
	<project_directory>/SHA/BASE_<base_revision_number/SHA>/<project_module(if_present)>/<pom_file>
Base version POM , saved to compare against successive revisions saved under,
	<project_directory>/<project_module(if_present)>/<pom_file>
	
import.csv
	This file contains the cummulative tabulated execution result. The same file is the source for Tableau charts uploaded online.
 
Script / Code execution details
-----------------------------
includeFunctions.sh
	All the functions used in the code are available in this file. They are included in the wrapper scripts demoWithEkstazi.sh and demo.sh 
Of the 23 helper functions, listing few groups of important functions which have been unit tested well. The tests for few sanity functions are availabe under /logs and listed under test.suite

Repository related functions.
-----------------------------
pullClone() 
Input args : 3 : <repoType> <cloneUrl> <ProjectDirectoryName>
    About : creates a new clone for the given <cloneUrl> under <ProjectDirectoryName>

getLastNCommits()
Input args : 1 : <repoType>
    About : Fetches the last <REVCOUNT> number of revision numbers in chronological order.
[ Note the default git / svn log displays in reverse chronological order ] . This order is important for Ekstazi so that we can see the test reduction properly.

cloneCheckout()
Input args : 1 : <repoType> <SHA/revisionNumber>
    About : This checkouts the given <SHA/revisionNumber> as per the <repoType>

checkAndCloneRepo()
    About : Wrapper function which checks for presence of already existing clone and invokes pullClone otherwise.

Generic Helper Functions.
------------------------
getInputs()
    About: Helper function which gathers all the necesary inputs and initializes the global variables.

coloredEcho()
    About : This function is used to print the test execution output in different color on the console for easy viewing. 
The below functions operate over the below project scope global variable as set by the getInputs function.


Functions related to Ekstazi
---------------------------------
installEkstazi()
Input args : 1 <ekstaziVersion>
    About : Installs ekstazi for the given <ekstaziVersion> from mir.cs.illinois.edu/gliga/projects/ekstazi/release/ under the $HOME/.m2/repository directory

runWithEkstazi()
    About : Main wrapper function which executes the test for ${REVCOUNT} number of revisions with Ekstazi pom changes. 
This intern invokes the below helper functions.

cloneCleanup()
    About : This function is used to cleanup the local files generated during the testing process.

backupEkstazi()
    About : Backups all the .ekstazi folders under the project directory to /tmp/<project>. This function is called prior to cloneCleanup()

restoreEkstazi()
    About : Restores all the .ekstazi folder for the given project from /tmp/<project> to all the respective modules. This function is called prior to test execution during Ekstazi runs.

copyPOMAndExecute()
    About : This function replaces the current version’s pom with the saved pom for the project under pomReferenceDir. This function is invoked only if the main code is executed with reference pom flag set i.e no “auto” flag set.

updatePOMandExecuteRun()
    About : This function modifies the pom as per Ekstazi requirements and executed only when main code is executed with “auto” flag set. This makes use of the other team’s pom parser code to modify the pom.

removeEkstaziDir()
    About : Used to remove the /tmp/<project>/<.ekstazifolders> prior to the start of test execution

resyncPom()
    About : This is a wrapper function to invoke cloneCleanup() and cloneCheckout() functions. It also takes care to remove the temporary pom changes that occur during the ekstazi executions with the help of repullPom() function.

repullPom()
    About : Function to update only the modified poms in the clone. 

saveReferencePOMS()
    About : Function which compares the checked out revisions poms against respective project’s base version reference poms saved under pomReferenceDir and prompts the user to save the changes for Ekstazi related modifications. It does not prompt if the revision’s modified pom is already available in repo and hence the user prompt occurs only once.

Sanity Testing
-------------
We have few test scripts to check the sanity of the project repository and the availability of Ekstazi for testing. The test scripts are available under /scripts/tests/ directory. 

Test Scripts
------------
Test scripts can be included in "test.suite" file. They invoke scripts inside logs/tests/

Test logs
----------
Test logs are available under /scripts/tests/logs
Test log file name nomenclature
		<test_name>.log
Test log data nomenclature
		<DATA>|<TYPE_OF_LOG>|<MSG>
		TYPE_OF_LOG may be any of the below
				INFO : For information related message logs
				ERROR : Log error messages for debug
				PASS : Logs pass after condition check
				SUCCESS : To log test success
				FAILURE : To log test failure
Example log:
------------
Tue Dec  9 00:29:30 CST 2014|[INFO]|BEGIN CLONE CLEANUP TEST
Tue Dec  9 00:29:41 CST 2014|[INFO]|CLONE CLEANUP CODE CHECK BEINGS
Tue Dec  9 00:29:41 CST 2014|[PASS]|GIT CLONE CLEANUP PASSED 
Tue Dec  9 00:29:41 CST 2014|[INFO]|SVN CLONE TEST
Tue Dec  9 00:29:45 CST 2014|[PASS]|SVN CLONE PASSED 
Tue Dec  9 00:29:45 CST 2014|[SUCCESS]|CLONE TEST PASSED

Sanity test for ekstazi reduction:
----------------------------------
/scripts/tests/test_ekstaziReductionCheck.sh
This test is takes a sample project and executes testing with Ektazi changes twice on the same revision. The expected behaviour is that second time the test should run ideally zero cases. 
It prints the test log as below
Tue Dec  9 03:23:08 CST 2014|[INFO]|BEGIN TEST
Tue Dec  9 03:26:16 CST 2014|[INFO]|TESTING COMPLETE; VERIFICATION BEGINS
Tue Dec  9 03:26:16 CST 2014|[SUCCESS]|TEST SUCCESS : REDUCTION 99 % 
Tue Dec  9 03:26:16 CST 2014|[INFO]|EKSTAZI TEST ENDS


Observation and Analysis
------------------------
Projects executed with and without Ekstazi tool.

1. chukwa
2. closure-compiler
3. commons-coll4
4. commons-config
5. commons-lang3
6. commons-math
7. commons-pool
8. cucumber-jvm
9. gora
10. guava-libraries
11. zxing
12. tika
13. jgit
14. joda-time
15. log4j
16. netty
17. phoenix


From the data collected for the above 17 projects from both Git and SVN across 20 revisions for without and with Ekstazi, we came across some interesting observations with respect to number of tests run and the total time of test case execution. 

The execution results can be viewed online at https://public.tableausoftware.com/views/SEv1/EmpiricalEvaluationOfEkstazi?:embed=y&:display_count=no.  

Observation and analysis for the anamolies are given below. The detailed observation results along with the screenshots can also be found at "report.pdf" 

Observations Without Ekstazi
--------------------------------
1. The number of tests run fluctuated across different revisions of the same project. i.e. for revision 2 the number of tests run decreased when compared to revision 1.
2. The number of tests run constantly increased from one revision to the next.
3. For certain revisions, more time was taken to execute tests compared to the previous revision, although the number of test cases executed remained the same.
4. For some projects, no tests were executed. This was observed only for some revisions.

Observations With Ekstazi
-----------------------------
1. The number of test cases run dropped to 0, compared to the tests run for the revision before. Ex. For Revision 1, the total number of tests were X, however in Revision 2, the tests run were completely reduced to 0.
2. Number of tests run between successive revisions remained 0 i.e. the technique of regression test selection was effective.
3. Time taken to reduce tests with Ekstazi was more compared to test execution without Ekstazi changes.
4.Test reduction was unusually low for some specific intermediate revisions.

To determine whether the above observations were indeed expected or dubious, we carried out thorough investigation on projects which showed the above anomalies.
The result of the investigation is as below:


Analysis for various projects
----------------------------
1. Chukwa
project url: http://svn.apache.org/repos/asf/chukwa/trunk/

Without Ekstazi
Observation - 1:
 	For initial revision 1611855, the number of tests run (148) were more compared to the number of tests run(139 tests) in the next revision 1611856.
Reason:
    To identify the reason for the above anomaly, we used the below svn command
	svn diff -r 1611855:1611856, which shows the differences between the two revisions mentioned.
    From the above result, we observed that in revision 1611856, total of 9 tests under file TestAdaptorController.java were removed. Methods removed are testPrintNvp(), testPrintNvpIndented(), testPrintNvpIndentedWithDash(), testPrintNvpIndentedWithStringLiteral(), testPrintStartTag(), testPrintStartTagWithAttributes(), testPrintElement(), testPrintElementWithAttributes(), testPrintEndTag()

Observation - 2:
 	For revision r1612600 the total tests run were 139, however for next revison r1612604 the total tests run were 140.
Reason:
    Using svn diff between the above 2 revisions we found that a new test case was added to the file TestRestAdaptor.java. This explains the increase of number of test executions by 1.

With Ekstazi
Observation:
	Ekstazi successfully reduced the number of tests run(142) from previous revision r1611855
	to 0 tests run in the next revision r1611856.
Reason:
	To understand the change between the 2 revisions, we used the command
	 svn log -r 1611855:1611856
	The above command gives the log message between the mentioned revisions.
	Using the logs we observed that in revision 1611856, the log message was “Updated Chukwa Agent REST API document and generation method”
	Since only the documentation was changed and no extra tests were added to the existing code, the tests were reduced to 0 as expected.

2. Closure-Compiler
project url: https://github.com/google/closure-compiler

Without Ekstazi
Observation - 1:
 	There was increase in a test case execution from the revision 0fc234b  to revision 21a2f2a
Reason:
	we used git diff 0fc234b 21a2f2a to find the modifications between the two revisions.
	From the above output, we observed that In file IntegrationTest.java a new test method named testSuppressBadGoogRequire() was added in the new revision. This explains the increase in a test case between the two revisions
Observation - 2:
  	The number of test cases executed between revisions 1bc2e0e and f02c5d2 were same
Reason:
	Using git diff 1bc2e0e f02c5d2, we identified that there was no test cases added in the new revision. This is the result of same number of test case exections between the two revisions.

With Ekstazi
Observation:
 	 The number of tests run from first revision 0fc234b to the next revision 21a2f2a, dropeed to 0.
Reason:
	Using git log -p -2, we observed that there was no new functionality added in the new revision
	21a2f2a, and hence Ekstazi tool behaved as expected by reducing total number of tests run for this revision.

3. Commons-Lang3
project url: http://svn.apache.org/repos/asf/commons/proper/lang/trunk

Without Ekstazi
Observation:
	Between revisions 1632416 and r1632874, the number of tests executed was increased by 17.
Reason:
	svn diff -r r1632416:r1632874
	When we analyzed the diff from the above two revisions, we observed that exactly 17 test cases (annotated with @Test)were added in the new revision 1632874 in the files ArrayUtilsTest.java, CharUtilsTest.java, BooleanUtilsTest.java and NumberUtilsTest.java

With Ekstazi
Observation:
	Ekstazi reduced the number of tests run to 0 in both revisions 1630269 and 1630277.
Reason:
	svn diff -r r1630269:r1630277, this command revealed that there was no additional functionality added between the two revisions. The changes that was introduced were only contributor tag addition in pom.xml file and a comment addition in MethodUtils.java file. Thus, Ekstazi tool worked as expected by not running any test cases in the new revision.

4. Commons-Config
project-url: http://svn.apache.org/repos/asf/commons/proper/configuration/trunk

With Ekstazi:
Observation:
	Ekstazi tool reduced the number of tests cases run in revision 1629051, which is 2646 to 0 in revision 1629231.
Reason:
	using the comand svn diff -r r1629051:r1629231, we identified that in the new revision 1629231, only a non functional change was introduced in site.xml file. Hence, as expected Ekstazi tool drastically reduced the number of tests run for this revision. Hence this case proves that Ekstazi tool is very effective RTS tool.

5. Commons-Pool
project-url: http://svn.apache.org/repos/asf/commons/proper/pool/trunk

Without Ekstazi:
Observation:
	For revision r1622424, the number of tests run were only 10, whereas for the previous revision r1622092 the number of tests run was 256. This was strange as svn diff -r r1622092:r1622424 showed that 2 test cases were added. Hence, the number of tests run for this revision should have been 258 instead of 10.

Reason:
	When we executed mvn test for revision r1622424, initially there was a build failure after executing only 10 revisions and we got the below error:
	[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project commons-pool2: There was a timeout or other error in the fork ->
	Hence, we tried running only this revision in a different machine and observed that this time it ran successfully and the total number of test cases was 258 as expected.

6. Joda-time, zxing
project-url: 
	https://github.com/JodaOrg/joda-time.git
	https://github.com/zxing/zxing.git
Observation:
	For both of the above projects, the time taken to reduce tests ( with Ekstazi ) was more compared to test execution without Ekstazi changes.
Reason:
	To confirm whether this is a system issue, we ran the test for both joda-time and zxing with ekstazi again for 20 revisions on a different machine. This time, we also ensured that there are no other process running in parallel. On completion across 20 revisions, we observed that this time the time taken to reduce tests were significantly lower and matched the expectation of the tool.

7. common-coll4, common-math, common-config, common-lang3
project-url:
	http://svn.apache.org/repos/asf/commons/proper/collections/trunk,
	http://svn.apache.org/repos/asf/commons/proper/math/trunk/
	http://svn.apache.org/repos/asf/commons/proper/configuration/trunk
	http://svn.apache.org/repos/asf/commons/proper/lang/trunk
Observation:
	With Ekstazi changes, the number of tests reduced was unusually low for some specific intermediate revisions.
Reason:
	The previous data had been collected for Ekstazi version 3.4.2. They were re excuted with Ekstzai version 4.2.0 ( as with all other project runs ). This time the reductions were as expected ( no unsualy full test count selections for revisions were observed )

8. Gora
project-url: https://github.com/apache/gora.git
Observation:
	Without Ekstazi, there were random number of tests executed, sometimes number of tests run dropped to zero. 
	Example : r1560407 version had 0 tests executed [ log at revision # -r3317 for the log logs/woEkstazi/gora_withoutEkstazi.log ] because of below error

	[ERROR] COMPILATION ERROR :
	[INFO] -------------------------------------------------------------

Reason:
	We tried re executing this project [ revision # r3384 for the log logs/woEkstazi/gora_withoutEkstazi.log  ] but since its now moved to GIT from SVN, we ran a fresh set of 20 revisions for both with and without Ekstazi. This time we see that there was no anomaly in the test result. The same can be viewed from the chart shared

9. Phoenix
project-url: https://github.com/apache/phoenix
Observation:
 	For one revision a715a79, execution without ekstazi ran no tests.
Reason:
	After re executing this specific revision now,  we found that build failed with below reason-
	“Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile.”
	log can be found at  logs/issues/phoenix_issue.log
	This justified the reason for not running any test cases for that particular revision.
	However, after professor’s suggestion, we re ran the tests for 20 revisions around that revision i.e. starting from revision de7c4df. This time all the tests executed successfully without any build failure.

10. commons-lang3, commons-config
project-url:
	http://svn.apache.org/repos/asf/commons/proper/configuration/trunk
	http://svn.apache.org/repos/asf/commons/proper/lang/trunk
Observation:
	The total time taken to execute tests with ekstazi for all the revisions was more compared to without ekstazi execution even though there was reduction in number of tests run.
Reason:
	To confirm whether this is a system issue, we ran the test for both commons-lang3 and commons-config with ekstazi again for 20 revisions on a different machine. This time, we also ensured that there are no other process running in parallel. On completion across 20 revisions, we observed that this time the time taken to reduce tests were significantly lower compared to that of without Ekstazi and matched the expectation of the tool.

Issues with Ekstazi tool integration:
------------------------------------
storm
guice
httpclient
wildfly

The above 4 projects execution failed when we tried to integrate it with Ekstazi tool. We tried to execute the above projects along with respective POM changes in differnt machines, however no tests were executed because of Build Failure. Logs for the same can be found at logs/issues folder in our SVN repository.


Conclusion:
----------
The projects studied the effectiveness of RTS technique using Ekstazi. 
Though the study targetted a small set of projects, the empirical evaluation of the Ekstazi across projects has bolstered the regression test selection technique's aim of reducing the test count while keeping the time to reduction low. From the online chart (https://public.tableausoftware.com/views/SEv1/EmpiricalEvaluationOfEkstazi?:embed=y&:display_count=no), it can be observed that Ekstazi successfully reduced more than 99% of test case execution (from 629964 to 61) for 17 different projects across 20 revisions. Also, there is an average reduction of 37% of time taken (6014 seconds to 3737 seconds) to execute the test cases using Ekstazi tool when compared to without ekstazi test execution results.