Uploaded image for project: 'EJBCA'
  1. EJBCA
  2. ECA-9711

AWS KMS request throttling when reading public keys results in unusable keys

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: EJBCA 7.4.3.2
    • Component/s: None
    • Labels:
      None
    • Issue discovered during:
      Customer
    • Sprint:
      EJBCA Team Alice - 2020 w50

      Description

      The specifiic method of reading public keys have very low request throttling in AWS KMS. This means that if the crypto token have more than 5 keys, reading those will (randomly) fail with a message in the log:

      2020-12-18 23:28:35,480+0000 DEBUG [org.apache.http.impl.conn.PoolingHttpClientConnectionManager] (default task-12) Connection released: [id: 726][route: {s}->https://kms.us-west-1.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 2; total allocated: 1 of 20]
      
      2020-12-18 23:28:35,480+0000 DEBUG [org.ejbca.keys.token.AWSKMSCryptoToken] (default task-12) getPublicKey JSON response: {"__type":"ThrottlingException","message":"You have exceeded the rate at which you may call KMS. Reduce the frequency of your calls."}
      
      2020-12-18 23:28:35,481+0000 DEBUG [org.ejbca.keys.token.AWSKMSCryptoToken] (default task-12) No public key found (HTTP 400 returned) with alias: defaultKey001. Error message: You have exceeded the rate at which you may call KMS. Reduce the frequency of your calls.
      
      2020-12-18 23:28:35,481+0000 DEBUG [org.cesecore.keys.token.CryptoTokenManagementSessionBean] (default task-12) Ignored key alias 'defaultKey001' in crypto token 'AWSKMSManagement' since it is missing a public and/or private key. Perhaps it is a symmetric key?
      

      We need to implement a "backoff and retry" algorithm for these calls.
      For example as documented here: https://docs.aws.amazon.com/general/latest/gr/api-retries.html

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              tomas Tomas Gustavsson
              Reporter:
              tomas Tomas Gustavsson
              Verified by:
              Amin Khorsandi
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 2 hours
                  2h
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 2 hours
                  2h