thekappe: I'd suggest using devtool for modifications like that.
- export modifications to patch file
- move patch file outside of the build dir
- clean kernel sources
- devtool modify linux-yocto
- apply the mofifications from your patch files in build/workspace/source/linux-yocto
- build & test
- commit all changes you want to keep in build/workspace/source/linux-yocto
- devtool update-recipe linux-yocto --append path/to/your/layer