In the circuit mentioned in CN-0399, ADL5904 and AD7091R is used. ADL5904 is a true rms responding power detector. ADL 5904 gives output in the form of DC voltage.
What is the resolution of this detector ?
Resolution of the circuit dependent on ADC used ?
I'm not aware of any inherent limitation on ADL5904 VRMS analog output resolution. Trying to get down to 0.001dB resolution could be difficult, as things like noise and drift and even very slight temperature differentials and Seebeck thermoelectric effects might start to come into play.
There will be some added noise contributed by the ADL5904, as with any active circuit, so you might want to use a sensitive multimeter or low-noise amplifier setup on the output, to measure how much VRMS output AC noise is present. You may find it difficult to measure accurately, because the RF power source will contribute it's own residual AM noise.
Regarding CN-0399 resolution of the RF Detector + ADC:
Detector output slope is approx. 35mV per dB (+ or -, depending on frequency).
At the AD7091R, each LSB is 610 uV (ref. CN0399).
Therefore ADC resolution in dB at the detector input will be 0.61mV / (35mV/dB) = 0.017 dB