Older Denon AVR 5800, Sony BDP S-1100, using digital coaxial but no DTS or Dolby

[hometheatergeek]

Page 21 of your manual (audio settings):

[Digital Audio Output]
[Auto]: Normally select this. Outputs
audio signals according to the status of
the connected devices.
[PCM]: Outputs PCM signals from the
DIGITAL OUT (COAXIAL)/HDMI
OUT jack.

Change this setting to PCM

x [BD Audio MIX Setting]
[On]: Outputs the audio obtained by
mixing the interactive and secondary
audio to the primary audio.
[Off]: Outputs the primary audio only.
Select this to output HD audio signals to
an AV amplifier (receiver).

Change this setting to off

x [Downmix]
[Surround]: Outputs audio signals with
surround effects. Select this when
connecting an audio device that supports
Dolby Surround (Pro Logic) or DTS
Neo:6.
[Stereo]: Outputs audio signals without
surround effects. Select this when
connecting an audio device that does not
support Dolby S

Change this setting to surround.

This is what I would try.