I would like to use the adn4652 as an galvanic isolation interface between an incoming LVDS and an outgoing LVDS.
Both, the incoming and the outgoing LVDS use shielded CAT5e RJ45 connectors.
Clearly, both shields, the one on the incoming RJ45 and the one on the outgoing RJ45 connectors must be interconnected and electrically attached to the case.
I assume I should use the layers of the board as follows:
Layer1: Isolated signals
Layer2: Isolated GNDs: One side GND1 for incoming, other side GND2 for outgoing connectors as shown in CN-0256. Big gap in between.
Layer3: Extensive shield layer
Layer4: Isolated signals
1) Are there any arguments speaking against one extensive full board covering shield layer?
2) Should the layer ordering be different? Like
Layer2: Isolated signals
Layer3: Isolated GNDs
Layer4: Extensive shield layer
Please let me know what you think.
Your cable shield (or shields) should be earthed. There are different RJ-45 connector footprints (e.g. surface-mount vs. through-hold, different shield connections) but in general it should be possible to connect the shield of the connector (connects to cable shield) to a distinct PCB copper plane that is coupled to chassis/earth.
I recommend keeping this at the edge of the board, avoiding overlap with other PCB planes (e.g. isolated power, ground). An example is shown below (RJ-45 + ADN4651 + ADuM5000 footprints) - the area on the left is for the shield - the area on the right is isolated (top area) or non-isolated PCB ground (this might also be coupled to earth, or even to the shield area). Typically the shield area would be connected to the device chassis. The isolated planes may only have the isolators connecting them to the rest of the circuit (sometimes a high-voltage filter capacitor might be connected). The rest of the PCB ground can be coupled to chassis, maybe with some filtering.
Some other boards have the connector side ground coupled to chassis, with the MCU side kept floating (main PCB ground is the "isolated" ground).
The shield area could be an internal layer if you need to maintain larger PCB clearance on the outside layers. You really only need one layer for this section - the rest of the PCB can follow a standard stack-up (e.g. Layer 1: Signal+Ground, Layer 2: ground, Layer 3: Power, Layer 4: Signal + ground)