In general, better debug logging. Figure out what you want to know, add it to the code, and have that in the logs so that you can work it out. Capturing more details of the environment at the time also help - what request, when, etc.
In specific, I would look for a common pattern in clients hitting this - and if you found one optimise - but then go and capture the TCP layer traffic.
Looking at the SSL messages exchanged should give you some idea what is going wrong in the protocol, or at least what the common properties of the request are. Once you have that it should be closer to being debugged.
As a guide, I would guess this comes from one of three things:
- Something that isn't SSL talked to the SSL port. (port scans are common, but HTTP to the HTTPS port also happens.)
- The client doesn't share an acceptable set of ciphers with the server.
- The client offers a certificate, and the server has a hissy-fit. (Uncommon, but possible.)