Cooling

Air-fans are the most common cooling mechanism found in a PC and also the reason why they make so much noise. You need at least one in your case to expel the heat that builds up inside the case (the exhaust), one on the CPU, and now video cards and power supplies come with their own as well. Some note about fans: the bigger they are, the less energy they take to push the same amount of air as the smaller ones. They also don’t need to rotate as fast, which reduces their noise level a little bit. My case has four 140 mm fans (front, rear and two at the top). I put an Arctic Cooling Freezer Pro on top of my CPU (I’ve owned that CPU fan since 2004), which makes less noise then the default fan that came with the CPU.

Also, with air-fans comes dust. It’s a good idea to clean the inside of the PC every year. If you use the vacuum cleaner, make sure you have unplugged the electricity cable and that you discharge all the electrostatic by touching the metal frame first (and avoid carpet!), otherwise you might destroy electronic components. Personally, I think the best way to clean the inside of a PC is with compressed air.
Finally there is the expensive solution that most people won’t ever use: watercooling. You still need a few fans to expel heat and if you want to cover all of your hardware pieces that need cooling, you will have to break a few warranty labels. You also need good technical skills to set it up. I never tried it, but I read that you also need to change the “water” every 2 years. Only useful if you overclock a lot and buy high-end hardware.
Case setup
While having an air-fan and heat sink is important, it’s not the most important part of cooling a PC. What you really want is good airflow; without it, the heat will just build up in the PC. You want cold air to enter from the front and get expelled from the rear of the case. This means you need space between the rear of the case and anything that stands behind it. It also means that the front should be free to get fresh new air into the case. Beside that, trying to free as much space inside the case will help the air to move in the right direction. Example of air-flow:

Overclocking
Overclocking is a double-edged sword. You can get a bit more power out of your hardware at the cost of shorter lifespan, more heat, slightly higher electricity bills, and crashes. Manufacturers have made it easy to overclock your CPU and video cards, but I never saw that much boost in doing either, unless you can unlock stuff to a higher level. Also, there is as much luck as technical knowledge that can affect the success rate of the operations. Following discussion on specialized websites is a good idea.
Websites
Here are a few helpful websites:
Newegg: They might not always have the best price, but people love to put feedback about products on it. I mostly use it to compare qualities and see what people say about a specific product.
Official websites/forums/support centers: Great way to see if there is a problem with something before buying it. It’s also a great way to learn about compatibility issues.
Tom’s Hardware: I mostly use it for the comparative charts and sometimes the forums. Great to find in what category you stand, depending on your needs and money.
HardwareCanuck: Detailed reviews and easy to read for neophytes.
Anandtech & Bit-tech: Great for tech news and reviews.
Guru3D: The forums are great when you have problems with a video card or just need to know what is better. They also have handy links to video card drivers.
JohnnyGuru: The best place for power supply reviews.
Overclock.net: Title says it all, they also do reviews now and have helpful forums.