Details
-
Type:
Improvement
-
Status: Open
-
Priority:
Minor
-
Resolution: Unresolved
-
Affects Version/s: EJBCA 7.4.0
-
Fix Version/s: None
-
Component/s: None
-
Labels:
-
Provenance:Internal Delivery
-
Epic Link:
Description
The problem:
Typical configuration per JUnit test target execution in ant's file (./ejbca/test.xmli) is:
<target name="test:run" description="run both stand-alone JUnit test cases and system test" depends="deleteDirectories">
<antcall target="test:runsa" inheritall="true" inheritrefs="true"><param name="aggregate.results" value="true"/></antcall>
<antcall target="test:runsys" inheritall="true" inheritrefs="true"><param name="aggregate.results" value="true"/></antcall>
<antcall target="createreport"/>
</target>
It contains:
- [test:runsa] - tests execution for target: executes tests, outputs test results in XML format and generates a HTML report on top of XML result files;
- [test:runsys] - tests execution for target: executes tests, outputs test results in XML format and generates a HTML report on top of XML result files;
- [createreport] - generates a HTML report on top of all XML result files
As outcome of this, you usually have the report file ./ejbca/reports/test/html/index.html The generation of this report (XML->XSLT->HTML) is memory heavy operation, thus require additional memory for the report generation itself, eg (./ejbca/jenkins-files/EE_COS7_OpenJDK8_WF10_NOHSM_MSSQL2017/run.sh):
... # Options for ant itself. The report building can be memory heavy, otherwise it shouldn't need much memory export ANT_OPTS="-XX:+UseG1GC -XX:+UseCompressedOops -XX:OnOutOfMemoryError='kill -9 %p' -Xms64m -Xmx768m" ...
However, in jenkins, as a build report (named as 'Latest Test Result' / 'Test Result' ) we use results generated by jenkins via pipeline command, eg (./ejbca/jenkins-files/EE_COS7_OpenJDK8_WF10_NOHSM_MSSQL2017/Jenkinsfile) :
... stage('save test results') { steps { junit "ejbca/**/reports/**/TEST-*.xml" } } ...
Means, we don't actually use the './ejbca/reports/test/html/index.html', but regenerate the report again.
Possible solution:
1. Separate the process of tests execution and report creation via intermediate command, eg:
Before:
<target name="test:run" description="run both stand-alone JUnit test cases and system test" depends="deleteDirectories">
<antcall target="test:runsa" inheritall="true" inheritrefs="true"><param name="aggregate.results" value="true"/></antcall>
<antcall target="test:runsys" inheritall="true" inheritrefs="true"><param name="aggregate.results" value="true"/></antcall>
<antcall target="createreport"/>
</target>
<target name="test:runsys" description="run system test, use -Dtest.includep11=true to include P11 tests" depends="deleteDirectories">
<ant antfile="build.xml" dir="modules/systemtests" target="run" inheritall="true" inheritrefs="true"/>
<antcall target="optionaltestconfigdump-system" inheritall="true" inheritrefs="true"/>
...
<antcall target="createreport" inheritall="true" inheritrefs="true"/>
...
</target>
<target name="test:runsa" description="run stand-alone JUnit test cases, use -Dtest.includep11=true to include P11 tests" depends="deleteDirectories">
<antcall target="optionaltestacme-unit" inheritall="true" inheritrefs="true"/>
<ant antfile="build.xml" dir="modules/cesecore-common" target="test" inheritall="true" inheritrefs="true"/>
...
<antcall target="createreport" inheritall="true" inheritrefs="true"/>
</target>
After:
<target name="test:run:jenkins" description="run both stand-alone JUnit test cases and system test" depends="deleteDirectories">
<antcall target="test:runsa:jenkins" inheritall="true" inheritrefs="true"><param name="aggregate.results" value="true"/></antcall>
<antcall target="test:runsys:jenkins" inheritall="true" inheritrefs="true"><param name="aggregate.results" value="true"/></antcall>
</target>
<target name="test:run" description="run both stand-alone JUnit test cases and system test" depends="deleteDirectories, test:run:jenkins">
<antcall target="createreport"/>
</target>
<target name="test:runsys:jenkins" description="run system test, use -Dtest.includep11=true to include P11 tests" depends="deleteDirectories">
<ant antfile="build.xml" dir="modules/systemtests" target="run" inheritall="true" inheritrefs="true"/>
<antcall target="optionaltestconfigdump-system" inheritall="true" inheritrefs="true"/>
</target>
<target name="test:runsa:jenkins" description="run stand-alone JUnit test cases, use -Dtest.includep11=true to include P11 tests" depends="deleteDirectories">
<antcall target="optionaltestacme-unit" inheritall="true" inheritrefs="true"/>
<ant antfile="build.xml" dir="modules/cesecore-common" target="test" inheritall="true" inheritrefs="true"/>
</target>
<target name="test:runsys" description="run system test, use -Dtest.includep11=true to include P11 tests" depends="deleteDirectories, test:runsys:jenkins">
<antcall target="createreport" inheritall="true" inheritrefs="true"/>
</target>
<target name="test:runsa" description="run stand-alone JUnit test cases, use -Dtest.includep11=true to include P11 tests" depends="deleteDirectories, test:runsa:jenkins">
<antcall target="createreport" inheritall="true" inheritrefs="true"/>
</target>
NB! some targets in this example are not covered, eg 'ant antfile="build.xml" dir="modules/cesecore-common" target="test"', might be as well:
...
<ant antfile="build.xml" dir="modules/cesecore-common" target="test:jenkins" inheritall="true" inheritrefs="true"/>
...
The benefits here are:
- we break the cyclic dependency tree test-createreport:
test:run test:runsa optionaltestacme-unit createreport modules/cesecore-common" target="test" createreport createreport test:runsys optionaltestconfigdump-system createreport createreport - we dont' call report(s) generation for jenkins
2. Update the jenkins job to call proper target (eg. ./ejbca//jenkins-files/EE_COS7_OpenJDK8_WF10_NOHSM_MSSQL2017/run.sh)
Before:
... echo '=================== Starting system tests ==================================' ant test:runsys -Dtests.jvmargs="$TEST_OPTS" echo '=================== System tests are done ==================================' ...
After:
... echo '=================== Starting system tests ==================================' ant test:runsys:jenkins -Dtests.jvmargs="$TEST_OPTS" echo '=================== System tests are done ==================================' ...
3. Optimize memory limits (experimental) (eg. ./ejbca//jenkins-files/EE_COS7_OpenJDK8_WF10_NOHSM_MSSQL2017/run.sh):
Before:
... # Options for ant itself. The report building can be memory heavy, otherwise it shouldn't need much memory export ANT_OPTS="-XX:+UseG1GC -XX:+UseCompressedOops -XX:OnOutOfMemoryError='kill -9 %p' -Xms64m -Xmx768m" ...
After:
# Options for ant itself. export ANT_OPTS="-XX:+UseG1GC -XX:+UseCompressedOops -XX:OnOutOfMemoryError='kill -9 %p' -Xms64m -Xmx128m"
