The detail texture is not suited to be used as a normal, SReesor. Different job description altogether... though the result does not look too bad... I will certainly try to use details als normals in my next project, for the rougher patches - might actually add a little something to the rough...
Note that normal maps usually have to be set very weakly in Unity - on my fairway, out of a max of 1, they are at strength 0.05.
You have reached the outer limit of how far you can go without CF, I'm afraid - you will need CF to progress further. You have, imo, demonstrated that you are ready for it in your Fun with Unity thread.