John, you should be fine with leaving your speakers at their current level. However, I am a little worried about what you just said. I'm sure you've calibrated your system with an SPL meter at least and have accurately adjusted the speaker distances and have messed with the equalizers and tone controls, right?
I would consult your receiver's manual for detailed instructions on adjusting more complex settings...