my set up, how should it be set up

[crxvtec]

I looked but did not understand.
I have a receiver from 2001. Kenwood vr-615. Surround sound speakers, 5.1. Receiver has optical input, no HDMI. DTS, Dolby Pro Logic II, Dolby Digital.
PS3 as Blu Ray player
Samsung 4661 lcd tv.

One thing I do not get is what to boxes to check for sound in the sound settings for optimal sound? I had the video and sound out of the PS3 to TV via HDMI and sound only to the receiver via Optical. I have only DTS checked.( And the defaults)It has been good but I know it can sound better. I tried messing with settings and while watching Cars, there would be some dialouge that was 100% gone. What did I check to make it this way?

What would be the best set up. Optical straight to the receiver? HDMI to TV and then from TV to receiver optical? What boxes do I checkmark? Please, help me