I found the easiest way to color the line, and it was so obvious that I'm almost ashamed of how dumb I was.
Just take the layer with flats, copy and clip it to the layer with line. Use tonal correction (saturation/luminosity) and adjust it to your liking until the colors of the line look good. Since I'm a messy disaster, I need to fix some something here and there, but in the end it looks nice.