I can guarantee that this is the reason. The slow death spiral you describe happens when Apache starts swapping like hell.
This is completely outside of scope of our support but I can give you some good pointers.
First read the Apache Performance Tuning
page in Apache's documentation. It gives you a concise list of all the things you need to look into and how they affect Apache's performance.
The next must-read article is 3 small tweaks to make Apache fly
which goes into minute detail about how much memory will be used by Apache and how to tune down
MaxClients to prevent the slow death due to extreme swapping. In fact, if you read the last (Real-world example) paragraph in the MaxClients section I'm sure you'll recognize it to be the very problem you described having intermittently!
Armed with this knowledge and trawling through your logs you will be able to find out what is the best approach to follow. Decrease some Apache threading limits or add more RAM? Is the PHP memory limit and / or execution time unrealistically high, allowing for pages going rogue? Also ask yourself if you need all the services running on your server (Ubuntu enables quite too much by default if you ask me). Then start considering how you can make your site lighter. Is caching plausible or do you have a mostly personalized site like ours? Does using Redis or memcached make more sense than file caching for your use case? Can you get rid of a slow-loading, framework-based template and use a nimbler, speedier and less memory intensive hand-built one? Are all modules and plugins on your site necessary? Can you move your backups to a different time of the day, away from high-traffic periods and / or your site being crawled by search engines?
I hope that helps!