CEC issue, x3800h input keeps switching back to last working one, on S95D

Between my S95D and x3800h, where all devices are connect to, I am having an issue where I am unable to change inputs on x3800h sometimes. Let's say I had input working on PS5 and played a game. Next day, when I turn on the TV and switch to XSX input or any other input, within a second or two (as per the x3800h display), it'll switch back to PS5. This happens the other way around i.e. XSX input will become sticky. I then have to just try random things or disable CEC to try to switch the inputs on x3800h.

CEC is disabled on PS5 and XSX. When I disable CEC on Samsung TV, or for example testing network on Samsung TV, I can do the input switch.

This could be an x3800h issue or a S95D issue, I am just not sure how to fix this. ————Oddly, when I disable the setting for "switch to TV input when an appropriate signal comes" on x3800h, this issue seems to go away, despite none of these inputs I am trying to switch between are eARC. Edit: I just had the issue with this setting off so unrelated. ———-

Obviously I can keep that option disabled but that means I just have to switch to eARC manually each time I switch to TV gui and/or apps.

Any thoughts on what could be causing this issue and if there's a nice workaround?