Reference noise can be a main contributor to overall system noise. Analog Dialogue article Why Does Voltage Reference Noise Matter? (analog.com) provides an in-depth look at how the reference contributes noise and some approaches to filtering and minimizing its impact.
