The manual states that enabling fast mode has the "advantage of recovering very quickly whenever the S/PDIF receiver looses lock due to glitches in the signal".
What is the disadvantage, or, to phrase it another way, is there ever any reason not to enable fast mode?