Don't forget that the tempratures are highly affected from the room temprature. In a warm room without ventilation it is rather difficult to keep your system cool, unless you spend some extra money on this.
Your case' airflow is responsible for that the high tempratures you get, do you have any fans on the case? Do you also have any fanless/silent VGA card?
I have a great CPU cooler for my AMD 4200+ X2 and it runs at 20~25C. My system is close to 35~40C. All this when the room temprature is close to 22C.
When you remove the side panel the temprature in your case is decreased, that's normal there is more air.
I have also the Q-Fan disabled, I don't need the fans to run at 100%, there is a lot of noise this way.
You can install a better cooler for your AMD from Arctic Cooling or Thermaltake while two fans on the case would be great, one in the front to blow air inside and one on the back to take the air out of the case.