I've read through every post and thread on this here and on other sites over the last couple days, solving this annoying problem. I'm on the latest everything, top of the line PC, great WiFi, aTV4k, etc. What I was running in to since moving up to 4k rips and remuxes, were buffering for the really high bitrate files and a 20-30 start delay for the smaller ones, even though both bitrates were well within all the limits of the network.
Long story short, it's the Windows SMB implementation. Yes I tried every possible setting and mode, everywhere, with no effect. All I had to do was setup a basic Plex server and connect to it's share, rather than the SMB shared folder. Instant fix, instant movie start, zero buffering of anything, smooth scrubbing. All over Google Wi-Fi 3 hops away.
If like me you simply store everything on a PC with huge internal storage, do yourself a favor and just switch from a shared folder to Plex. It makes everything better.