All Kodi Krypton versions I was using have excactly the same problem.
Video starts fine and sometimes after pasuing it for a couple of mintues or seeking video the frame rate drops and it doesn’t go back to normal until I press stop and play video once more.
This also happens without doing anything.
I’m using an Amlogic S905X device with 2GB RAM.
Videos are played by local network (SMB).
I’m out od ideas. Tried to change everything in the settings. Nothing helps.
Last thing I didn’t test is to play video from USB.
But older versions of Kodi don’t have that problem.
So it must be a Kodi Krypton bug.