AnsweredAssumed Answered

mathematic result precision 21469 EzKit lite

Question asked by Uhendan on Jun 21, 2010
Latest reply on Jun 24, 2010 by Uhendan

hi, everyone!

I`ve got a problem with mathematics in visual DSP 5.0:


I simulate a sharc 21469 EzKit lite board and write a code for filter coefficient calculation...

just for example (not a part of my code, but the core of the problem, i guess):

//code ....

float x =2;
float y;
  y=sqrt(x);                   //*** or division, or something else..



result would be 1.41421350000000000000


...but, for precise mathematical operations,
the true result should rather be:



how can i fix the problem with the precision of math operations? Another C Compiler, i tested, does not do this..


why is there a rounding after 8 digits?