Practically you can't do anything about it. But there are some people in the forums and they are providing some advice to increase that. But those are core-hack and not recommended.
"Add to cart process" is taken care by JSON with AJAX, and it takes time. The speed may vary depends on your server's respond time, ping and visitor's internet speed.
Alternatively you can disable the "show modal popup box when product is added to cart" option through Virtuemart Configuration. So the page will be redirected to shopping cart directly once the product is added to cart. This will minimize this problem.
Regards.
Support & Development