Advice on upgrading Mini-ITX for gaming performance

[oddbox83]

Hi,

I'm looking for a bit of opinion on upgrading my Mini-ITX self-build for gaming performance.

It's about 3-4 years old but is a bit slow for casual Steam gaming. I recall giving it decent stuff back then so doesn't appear totally outdated today with processor and RAM still seeming comparable to current standards. I've tried overclocking the RAM which did help but left an unstable system that won't always boot so I've had to clock back to defaults.

I think what's important in the specs:
AMD A8-6500 APU 3.5Ghz
8GB (2x4GB) DDR3 RAM - Was set at 1866, overclocking seems to have made the actual clock speed random in UEFI which has puzzled me. I hope I haven't damaged it.
Asrock FM2A88x ITX board with spare PCIE slot

So, without wanting to go to the expense of a new system, which is going to give me a cost effective yet obvious performance boost in gaming, as the frame rate can get ludicrously low at the moment. Updating the RAM, allocating more to the APU and a higher native clock speed, or getting a dedicated GPU that is compatible with the board. I can't just add more RAM, there are no spare sockets so would have to be a total swap.

I'm thinking the former, but given PC hardware moves so quickly and I'm 3-4 years out of date since this was built I'm a bit lost as to what I should be looking for that is compatible with this system.

Also, I'm not totally familiar with how APUs work when the onboard graphics aren't used, is the full processor and RAM capacity given over to actual system then rather than shared?

Thanks in advance.