LT8708 not regulating Vout

Hi, I'm looking into the issue you're seeing with the simulation, particularly at which amplifiers or how many you can disable before you start running into issues. The power flow verification table (below) indicates the part might need more than just FBOUT to function correctly. To answer your questions:

1. I noticed there is no output cap, but that isn't the root cause of the issue, although it should be added of course. I'll follow up with more details on the error amplifier configuration, as I mentioned earlier.
2. I've reached out to the LTspice team to inquire about the model's restrictions. The part itself is certainly not limited to applications with voltage sources on input and output.

Could you tell me more about your application? Will you be only using this part as a buck-boost in one direction, or is the setup in your simulation just an initial test, and your final application will have bidirectional capability, different conduction modes, etc.?

I tweaked the simulation a bit and was able to get it to start regulating the output to 20V by connecting Vinchip to your 24Vin source, instead of a separate 12V. I also added 2x220uF Cout.

Here's your sim with Vinchip tied to a 12V source:

Here's with it tied to Vin = 24V:

Changing SS cap to 1nF improves the initial overshoot:

However, I think the real issue is that the converter is not stable. Seeing that kind of overshoot on startup is an indicator of instability, particularly too little phase margin. After connecting your main Vin source to Vinchip, try tweaking the RC values on the VC pin. Increasing the cap from 10nF to 22nF and the resistor from 10k to 15-20k is a good start.

Startup with Css=1nF, Ccomp=22nF, Ccomp_par=680pF, Rcomp=5k, 10k 17.4k (black, blue, red)

This will likely need further experimentation as you move on with testing different load transient conditions and what switching mode the part is in (buck, boost, buck/boost).

Are you designing the circuit so that you can adjust the output voltage in real time? This might also cause complications with the compensation, as the FB networks aren't meant to be manipulated on the fly.

 
Here is my modified copy of your sim.

Do you see the output settle if you just place a 0.1uF cap from VC to GND? This will slow down the loop a lot but will provide sufficient phase margin for startup. 

With VC network set as (22nF + 17.4k) || 680pF, I see output rise to 20V, slightly dip to about 19.92V, then settle out. Switching looks as expected.

If you are still seeing issues, try making sure LTspice is up to date. Otherwise, default settings are what I'm using.

Glad we could get it working. If you want to really dial in the control loop response, you can use the .fra tool in LTspice. Here's some info on the tool. You can also find sample circuits in Open Examples → Educational → FRA in LTspice.