Uploaded image for project: 'EJBCA'
  1. EJBCA
  2. ECA-9164

Jenkins: reduce memory consumption for JUnit reports generation

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: EJBCA 7.4.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      The problem:
      Typical configuration per JUnit test target execution in ant's file (./ejbca/test.xmli) is:

          <target name="test:run" description="run both stand-alone JUnit test cases and system test" depends="deleteDirectories">
          	<antcall target="test:runsa" inheritall="true" inheritrefs="true"><param name="aggregate.results" value="true"/></antcall>
          	<antcall target="test:runsys" inheritall="true" inheritrefs="true"><param name="aggregate.results" value="true"/></antcall>
          	<antcall target="createreport"/>
          </target>
      

      It contains:

      • [test:runsa] - tests execution for target: executes tests, outputs test results in XML format and generates a HTML report on top of XML result files;
      • [test:runsys] - tests execution for target: executes tests, outputs test results in XML format and generates a HTML report on top of XML result files;
      • [createreport] - generates a HTML report on top of all XML result files

      As outcome of this, you usually have the report file ./ejbca/reports/test/html/index.html The generation of this report (XML->XSLT->HTML) is memory heavy operation, thus require additional memory for the report generation itself, eg (./ejbca/jenkins-files/EE_COS7_OpenJDK8_WF10_NOHSM_MSSQL2017/run.sh):

      ...
      # Options for ant itself. The report building can be memory heavy, otherwise it shouldn't need much memory
      export ANT_OPTS="-XX:+UseG1GC -XX:+UseCompressedOops -XX:OnOutOfMemoryError='kill -9 %p' -Xms64m -Xmx768m"
      ...
      

      However, in jenkins, as a build report (named as 'Latest Test Result' / 'Test Result' ) we use results generated by jenkins via pipeline command, eg (./ejbca/jenkins-files/EE_COS7_OpenJDK8_WF10_NOHSM_MSSQL2017/Jenkinsfile) :

      ...
      		stage('save test results') {
      			steps {
      				junit "ejbca/**/reports/**/TEST-*.xml"
      			}
      		}
      ...
      

      Means, we don't actually use the './ejbca/reports/test/html/index.html', but regenerate the report again.

      Possible solution:
      1. Separate the process of tests execution and report creation via intermediate command, eg:
      Before:

          <target name="test:run" description="run both stand-alone JUnit test cases and system test" depends="deleteDirectories">
          	<antcall target="test:runsa" inheritall="true" inheritrefs="true"><param name="aggregate.results" value="true"/></antcall>
          	<antcall target="test:runsys" inheritall="true" inheritrefs="true"><param name="aggregate.results" value="true"/></antcall>
          	<antcall target="createreport"/>
          </target>
      
          <target name="test:runsys" description="run system test, use -Dtest.includep11=true to include P11 tests" depends="deleteDirectories">
          	<ant antfile="build.xml" dir="modules/systemtests" target="run" inheritall="true" inheritrefs="true"/>
              <antcall target="optionaltestconfigdump-system" inheritall="true" inheritrefs="true"/>
              ...
          	<antcall target="createreport" inheritall="true" inheritrefs="true"/>
              ...
          </target>
      
          <target name="test:runsa" description="run stand-alone JUnit test cases, use -Dtest.includep11=true to include P11 tests" depends="deleteDirectories">
              <antcall target="optionaltestacme-unit" inheritall="true" inheritrefs="true"/>
             	<ant antfile="build.xml" dir="modules/cesecore-common" target="test" inheritall="true" inheritrefs="true"/>
              ...
          	<antcall target="createreport" inheritall="true" inheritrefs="true"/>
          </target>
      

      After:

          <target name="test:run:jenkins" description="run both stand-alone JUnit test cases and system test" depends="deleteDirectories">
          	<antcall target="test:runsa:jenkins" inheritall="true" inheritrefs="true"><param name="aggregate.results" value="true"/></antcall>
          	<antcall target="test:runsys:jenkins" inheritall="true" inheritrefs="true"><param name="aggregate.results" value="true"/></antcall>
          </target>
      
          <target name="test:run" description="run both stand-alone JUnit test cases and system test" depends="deleteDirectories, test:run:jenkins">
          	<antcall target="createreport"/>
          </target>
      
          <target name="test:runsys:jenkins" description="run system test, use -Dtest.includep11=true to include P11 tests" depends="deleteDirectories">
          	<ant antfile="build.xml" dir="modules/systemtests" target="run" inheritall="true" inheritrefs="true"/>
              <antcall target="optionaltestconfigdump-system" inheritall="true" inheritrefs="true"/>
          </target>
      
          <target name="test:runsa:jenkins" description="run stand-alone JUnit test cases, use -Dtest.includep11=true to include P11 tests" depends="deleteDirectories">
              <antcall target="optionaltestacme-unit" inheritall="true" inheritrefs="true"/>
             	<ant antfile="build.xml" dir="modules/cesecore-common" target="test" inheritall="true" inheritrefs="true"/>
          </target>
      
          <target name="test:runsys" description="run system test, use -Dtest.includep11=true to include P11 tests" depends="deleteDirectories, test:runsys:jenkins">
          	<antcall target="createreport" inheritall="true" inheritrefs="true"/>
          </target>
      
          <target name="test:runsa" description="run stand-alone JUnit test cases, use -Dtest.includep11=true to include P11 tests" depends="deleteDirectories, test:runsa:jenkins">
          	<antcall target="createreport" inheritall="true" inheritrefs="true"/>
          </target>
      
      

      NB! some targets in this example are not covered, eg 'ant antfile="build.xml" dir="modules/cesecore-common" target="test"', might be as well:

             	...
             	<ant antfile="build.xml" dir="modules/cesecore-common" target="test:jenkins" inheritall="true" inheritrefs="true"/>
             	...
      

      The benefits here are:

      • we break the cyclic dependency tree test-createreport:
        test:run
          test:runsa
            optionaltestacme-unit
              createreport
            modules/cesecore-common" target="test"
              createreport
            createreport
          test:runsys
            optionaltestconfigdump-system
              createreport
          createreport
        
      • we dont' call report(s) generation for jenkins

      2. Update the jenkins job to call proper target (eg. ./ejbca//jenkins-files/EE_COS7_OpenJDK8_WF10_NOHSM_MSSQL2017/run.sh)
      Before:

      ...
      echo '=================== Starting system tests =================================='
      ant test:runsys -Dtests.jvmargs="$TEST_OPTS"
      echo '=================== System tests are done =================================='
      ...
      

      After:

      ...
      echo '=================== Starting system tests =================================='
      ant test:runsys:jenkins -Dtests.jvmargs="$TEST_OPTS"
      echo '=================== System tests are done =================================='
      ...
      

      3. Optimize memory limits (experimental) (eg. ./ejbca//jenkins-files/EE_COS7_OpenJDK8_WF10_NOHSM_MSSQL2017/run.sh):
      Before:

      ...
      # Options for ant itself. The report building can be memory heavy, otherwise it shouldn't need much memory
      export ANT_OPTS="-XX:+UseG1GC -XX:+UseCompressedOops -XX:OnOutOfMemoryError='kill -9 %p' -Xms64m -Xmx768m"
      ...
      

      After:

      # Options for ant itself.
      export ANT_OPTS="-XX:+UseG1GC -XX:+UseCompressedOops -XX:OnOutOfMemoryError='kill -9 %p' -Xms64m -Xmx128m"
      

        Attachments

          Activity

            People

            Assignee:
            Unassigned
            Reporter:
            andrey_helmes Andrey Sergeev
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:

                Time Tracking

                Estimated:
                Original Estimate - 1 day
                1d
                Remaining:
                Remaining Estimate - 1 day
                1d
                Logged:
                Time Spent - Not Specified
                Not Specified