I intend to keep this part of the article pretty vague. I want to speak in concepts to help you understand what to buy. If I said you should get X brand motherboards and X brand hard drives, you won't really learn anything and we'll start a bunch of Ford vs Chevy arguments that benefit no one.
Case
A case is a fairly simple purchase. It's plastic and metal. It holds the computer. That's it. There is no reason to buy a fancy case. I always just buy a $20 computer case and call it a day. You'll want what's called an ATX case which mean it meets the size standards of motherboard manufacturers today.
Power Supply
The power supply is where you will plug the power chord into. It delivers power to all components inside the computer. I've purchased many power supplies that were included within cheap $20 cases. These power supplies are usually cheap but noisy. When building a recording computer, noise is always something to consider depending on your room and situation. On most computers, I've actually had pretty good luck with the cheap, default power supplies. If you do decide to go with the cheap power supply that's included with a case, I also recommend that you buy a nice power supply and keep it for backup. (Or maybe use the nice power supply and keep the other one for backup).
All modern computer power supplies come with a fan that shoots hot air out the back of the computer. I highly recommend that you purchase a power supply that also has a fan on the bottom as well as these tend run much cooler.
Some brands specialize in quite power supplies. If your worried about fan noise in your PC, these are worth a look.