ADG704 not work for photo diode switching

I try to use  ADG704 to switch 4 photo diodes. The anodes of the diodes are connected together and to the ground. The cathodes are connected to S1 to S4. D is the output and connected to a transimpedance amplifier (AD8616). The switch is working fine if I apply a 3V signal on each input (S1-S4). However if the signal source is changed to a photo diode, I get light signals at the output of the transimpedance amplifier with and without the diode channel being selected. It seems there is a cross talk or leakage current between the switches. According to the datasheets the leakage current should be less than 0.1nA and I am sure the photo current is much larger than that. Can some one explain to me why I  get signals even the photo diode is not connected?