You really can't speed up the process. It all comes down to some very bad decisions they made when they designed WordPress and their stubborn refusal to fix them afterwards.
WordPress uses an unstable, insecure, outdated and slow format (PHP serialised data) to unnecessarily store absolute URLs, absolute paths and hardcode table names. This means that we need to go record by record, run expensive regular expressions to see if any of the static things WordPress is unnecessary storing in there is present and replace it. It is extremely slow and the only way to deal with an atrocious database design like that,
Furthermore, WordPress' database schema is not normalised, making any kind of access to it very slow.
My only advice is to use Joomla. Just steer clear from templates built on crappy template "frameworks" that try to simultaneously satisfy the requirements of 50 radically different designs and work the same across a laundry list of disparate CMS and PHP scripts (typically: Joomla, WordPress and phpBB which is like putting apples, burgers and masonry in the same category). If you take the time to design your own template, like we did, Joomla is much more flexible and much faster than WordPress. It also restores in no time. The server you have is an overkill for what you're using it for – except that you're using WordPress so you need all the extra power to waste it on WordPress' bad code and bad database design...
Nicholas K. Dionysopoulos
Lead Developer and Director