High speed low power analog switch

Hello, I've been trying to create an Inexpensive circuit that can generate a frequency adjustable saw wave with a PWM signal as input such that when the PWM is high the saw wave rises and when it is low it falls while maintaining the correct amplitude and frequency.

TLDR since I am very wordy is I need to somehow switch around 50 mA with a switching time of 1-2ns with VCC voltages of 4-6V. The switches i am using don't seem to be capable of that and so I'm asking for suggestions, the approach doesn't necessarily have to be switch IC's the requirements are >50MOhm Off resistance and <20Ohm on resistance, If it is possible to realize this with transistors/Fets or whatever else I don't mind, the only restriction is the cost/switch should be low as I need a lot of these things in the future, preferably less than 70c/switch

The approach I've chosen is a very simple one, I am switching a capacitor between a current source and a current sink at the desired PWM while adjusting the strengths of the source/sink to produce the proper waveform. Testing at lower frequencies has shown that this method has promise and is fairly reliable. However the problem occurs at higher frequencies of above 10MHz where the switches I am using appear to not be able to switch fast enough to produce the desired outputs.

The switches used are the SN74LVC1G3157 from Texas Instruments which claim a turn on/ turn off time of around 5ns, however in testing the waveforms produced suggest that either there is a problem with the supplied input signal not having a high enough amplitude or just simply the switches not being fast enough. I unfortunately do not posses a frequency generator capable of generating a square wave above 5Mhz, the tests above 5 MHZ have been performed with a AD9833 Frequency generator IC with a maximum output frequency of 12.5MHz going into a ICS501 PLL Clock multiplier which multiplies the frequency by 8x. The issue with this setup is that the ICS501 at 100MHz outputs a sine wave with a peak of 2V and a minimum of 1V which is just barely above the "Low" Voltage threshold of 30% VCC of the SN74LVC1G3157 switches, thus I conclude that the switches are at best very poorly driven at these frequencies. I do not have the hardware available to me at this moment to fix this issue however using a high speed comparator it should be fairly trivial to fix, the thing i am most concerned about is weather the switches are truly fast enough to switch at 100MHz or weather I am barking up the wrong tree here.

Here is a simplified schematic that has all the moving parts necessary to make this work along with oscilloscope measurements of the real circuit