I found the issue,for the retry queue,the email is sent one by one,if the one is still sending,it will not open another thread and send other one. 

This will be problem if one of big email sending to downstream server,and many small emails keep stay in the queue.

Best suggestion:

For the retry queue,if one of the email is not sent finished within 1 minutes,open another new thread to deliver other emails,this can avoid email queue issue.