Uploaded image for project: 'EJBCA'
  1. EJBCA
  2. ECA-6745

Manual tests with asynchronous replication

    Details

    • Type: Task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Study ECA-6754 before proceeding to get an idea of why this ticket matters.

      Conceptual test setup

      VM host

      3 isolated networks

      CAs

      Two EJBCA CA nodes (VMs) running Linux.
      Each node should have 3 NICs in order to allow simulation of network cuts without cutting off EJBCA or SSH access to the machine.

      Setup MySQL/MariaDB master-master replication over the third NIC roughly based on https://www.digitalocean.com/community/tutorials/how-to-set-up-mysql-master-master-replication . Use ROW-based replication and last write wins (e.g. overwrite in case of conflict).
      Install Java, JDBC driver, appserver and EJBCA on both nodes using the replicated database.

      VAs

      Two EJBCA VA nodes (VMs) running Linux.
      Each node should have 2 NICs in order to allow simulation of network cuts without cutting off EJBCA or SSH access to the machine.

      Setup standalone MySQL/MariaDB databases.
      Install Java, JDBC driver, appserver and EJBCA on both nodes.

      VA publishing

      Setup VA publishing from the CAs to the VAs using the secondary NIC of CAs and VAs.

      Test tools

      Use https://wiki.linuxfoundation.org/networking/netem to simulate a WAN connection delay between the nodes on the secondary NIC.

      Use iptables to simulate network disconnections/split on the second NIC.

      Use EJBCA WS CLI to generate certificate requests and revocations. For convenience, setup two copies targeting the different EJBCA nodes from the VM host.
      It is probably also wise to have a DB client and DB-accounts on all machines for easy access to raw database data (way faster than checking GUI or making OCSP requests).

      Tests

      For all tests it makes sense to do the permutations:

      • Throw away: true | false
      • CertificateData | NoConflictCertificateData

      →Create 4 different CAs in order to quickly iterate over these combinations from the test client.

      Sanity

      Issue and revoke things at both nodes. Ensure that data is replication as expected after a short while.

      Split brain

      Killl database connection between CA nodes with iptables.
      Issue and revoke things at both nodes. Ensure that there are conflicting updates.
      Allow traffic again.

      When using CertificateData was used there should have been loss of info when there were conflicting updates.
      When using NoConflictCertificateData was used all info should be present in database and OCSPs should have the most correct version of the data.
      Check VAs as well.

      Unreachable VA

      What happens when the VAs are temporarily unavailable as well...e.g canode1 can talk only to vanode1 and canode2 can only talk to vanode2?
      ....

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              robinsed Robin Sedvallson (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - 3 days
                  3d
                  Remaining:
                  Remaining Estimate - 3 days
                  3d
                  Logged:
                  Time Spent - Not Specified
                  Not Specified