Unfortunately, Diffusion always mucks with the color palette even when prompted not to.
First, keep in mind that Diffusion is NOT intended to replaceproper rendering applications.
You can turn off Foreground Photo in the Style’s Match Photo settings so that the image from Diffusion doesn’t override the applied textures.
With Foreground on.
With Foreground off.
Don’t forget to update the style.