Uploaded image for project: 'EJBCA'
  1. EJBCA
  2. ECA-7754

Reproducable Builds

    Details

    • Type: Epic
    • Status: Closed
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: EJBCA 7.0.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Epic Name:
      Reproducable Builds
    • Issue discovered during:
      Integration

      Description

      Being able to reproduce an exact build of an application from its sources allows third party to sample and verify that we are not cheating.

      For deliveries where the same EJBCA library appear multiple times, having the exact same files allows better compression of the overall delivery or sym-linking instead of copying.

      SignServer has come a long way on the subject: DSS-1042

      Background

      Java itself is deterministic and will produce the same bytes in each .class-file each time a .java-file is compiled as long as the JDK is sufficiently similar.

      JARs are zip-files. Files needs to always be added in the same order and additionally each entry has a timestamp that needs to stay the same.
      (Same applies for WARs, the EAR and the RAR.)

      Outline

      When unzipping an EJBCA release, timestamps are preserved and will stay the same when included in JARs.
      Built classes and generated files are however not.

      Changing timestamp of built classes

      Ant's touch task https://ant.apache.org/manual/Tasks/touch.html can be used to modify timestamps.

      Since we use inner classes, the following would not be sufficient (which has the benefit of simply re-using the timestamp of the source for the compiled class):

      <touch>
          <fileset dir="${module.dir}/src" includes="**/*.java"/>
          <mapper type="regexp" from="(.*)\.java" to="${module.dir}/build/\1.class"/>
      </touch>
      

      Instead we need a solution where we provide the timestamp and also modify the timestamp of created directories (which are also entries in the zipfile):

          <macrodef name="modify-timestamp" description="Modify timestamp to make build reproducable">
              <attribute name="millis"/>
              <attribute name="directory"/>
              <sequential>
                  <condition property="modify-timestamp.disable-pattern" value="" else="modify-timestamp.disable-pattern/">
                      <isset property="@{millis}"/>
                  </condition>
                  <touch millis="${@{millis}}">
                      <fileset dir="@{directory}" includes="${modify-timestamp.disable-pattern}**"/>
                  </touch>
              </sequential>
          </macrodef>
      

      Deterministic timestamp of META-INF/MANIFEST.MF

      Ant's manifest task https://
      https://ant.apache.org/manual/Tasks/manifest.html can be used to generate a MANIFEST.MF before the assembling the JAR, so the timestamp can be modified using touch.

      This also implies using the zip task https://ant.apache.org/manual/Tasks/zip.html instead of jar https://ant.apache.org/manual/Tasks/jar.html to assemble the JAR.

      Deterministic order of added files

      We can use sort https://ant.apache.org/manual/Types/resources.html#sort to ensure that files are always added in the same order if we sort by name:

      <zip destfile="${module.dir}/dist/module.jar" duplicate="fail">
          <sort>
              <resources>
                  <fileset dir="${module.dir}/resources" includes="META-INF/**/*"/>
                  <fileset dir="${module.dir}/build"/>
                  ...
              </resources>
              <name/>
          </sort>
      </zip>
      

      Selecting a time that should be used when touching files

      One option is to use the timestamp of src/internal.properties which is modified at the time of release:

      	<script language="javascript"><![CDATA[
      		// Override 'app.timestamp' unless it is already defined with the lastModified of internal.properties
      		project.setNewProperty('app.timestamp', new java.util.Date(new java.io.File(project.getProperty('internal.properties.file')).lastModified()).getTime());
      		]]>
      	</script>
      

      But this will require JavaScript to be available during build. A more robust and efficient solution would probably be to set app.timestamp in src/internal.properties during ant ziprelease, since this require JavaScript anyway and there is less room for local deviations from the production build.

      Conclusion

      With more static (build time) configuration and defined release build JDK, this will probably get us pretty close to being reproducable.

      Once the pattern to use has been decided on, implementing this is not a huge task.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              johan Johan Eklund
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 2 days
                  2d
                  Remaining:
                  Remaining Estimate - 2 days
                  2d
                  Logged:
                  Time Spent - Not Specified
                  Not Specified