I simulated my circuit in LTSpice, as the spectrum analyzer I placed 50 ohm resistors. It works fine.
A schematic would help us analyze what's wrong. What ADC are you using?
For a simpler solution, use ADC drivers. To optimize the 65MSPS sampling rate of LTM9008-14, you can use either LTC6404-1 or AD8139. However, if you will only operate at 5 MHz (shown in your simulations), please use ADA4940 or LT1994 instead. It is much simpler to implement than using to op-amps to convert single-ended to differential signals.
For the attenuation you observed in the output, do you have other components after the amplifier outputs? It seems to have created a voltage divider network with the analyzer. In 1 Mohm termination, it's not a big factor.
Let me be more clear.
1) I have a single ended signal and I want to convert it to digital using LTM9008-14. Since LTM9008-14 accepts differential input I use a driver circuit whose schematic is below.As operational amplifiers I used LTC6254. 50 ohm resistors represent Spectrum analyzer's input.
When I simulate this circuit I get the result below, which is quite good enough for me.
I simulate the same circuit, changing 50 ohm resistors with 1mega ohm resistors to simulate the input of the oscilloscope, nothing changes as seen below.
2) I produced the circuit's pcb and then tested it by connecting the probe with one of the differential lines. When the oscilloscope's input impedance is set to 1 Megaohm everything seems fine as you can see below.
3) However, when I set the input impedance as 50ohm voltage level collapses.
Such a result makes me think that the operational amplifier cannot supply the necessary current. This situation affects 2 things.
a. Input impedance of the spectrum analyzer is 50 ohm. Since the voltage level collapses dramatically I see very weak signals. Harmonics cannot be seen, probably because they stay below noise floor. As a result I cannot analyze the quality of my circuit pproperly.
b. I am not sure whether or not LTC6254 can supply the necessary current to drive LTM9008-14.
Do you habe any idea about the situation?
You can give advice for another operational amplifier with a really high GBP, or another single ended to differential converter.
Thank you very much for your interest.
Give me some time to work on the solution.