Copy-pasta from private mailbox:
I discussed this internally, and the way it has been solved for other customers is to let the syslog server trigger a script when there is a message in the log indicating a publisher error. The script will put the VA into maintenance mode by creating a maintenance file on the VA's filesystem (see https://doc.primekey.com/ejbca/ejbca-operations/ejbca-operations-guide/ca-operations-guide/ejbca-maintenance/monitoring-and-healthcheck). When the VA is in maintenance mode, EJBCA Healthcheck will fail, and the VA load balancer will then make sure that traffic is not being sent to that VA.
This solution is not going to work on the appliance since it is not possible create files on the filesystem. While we discussed other ways of taking the VA offline, e.g. by letting the script on the syslog server contact the load balancer directly).
I feel that these solutions are somewhat convoluted and not in line with the PKI-in-a-box offering that PrimeKey is promoting. I would therefore like to propose an enhancement of EJBCA which we could put in a future release, and I'm wondering if that's something XXXXX would be interested in. Basically, the load balancer for the VAs would query the healthcheck servlet for a CA to get the publisher status:
This would return a JSON payload, containing the hostnames of the VAs which are out of sync.
A VA would be defined as out of sync if there is at least one item in the VA publisher queue older than X seconds, and the contents are not the same in all VA publisher queues (which would indicate that all VAs are equally out of sync, but that would probably have to be corrected by the network guy manually, and we don't want to take the OCSP service offline completely).
X would be configurable in the GUI and would be 60 seconds for your installation.
Make it happen.
Customer XXXXX is using Citrix load balancers and they will check if JSON is an appropriate format.