AnsweredAssumed Answered

Floating Point Precision within Expressions Pane

Question asked by usernameGoesHere on Mar 19, 2015
Latest reply on Jan 5, 2016 by Kader.M



I am currently having issues with the expressions pane rounding floating point numbers. For example if I enter:


The expressions pane shows: 7464.6

I know that the processor is not losing any digits because the hex value in memory is set correctly to: 0x45e944ca


I did find this article which is similar to the bug that I am seeing:


The current method that I have to get the values from VDSP is the processor.EvaluateToString function which has worked well until we saw the issue with VDSP rounding the float values.


I tried to change the way that I get the values in order to use the ConvertToDouble Method that is documented within the VDSP API help.

my code is currently:

( javascript )

var memory_type_list = processor.MemoryTypeList

var memory_type = memory_type_list( "Data(DM) Memory" )

var value_list = memory_type.GetMemory( VariableAddressAsInteger, 1, 1 )

float_values = value_list.ConvertToDouble("Data(DM) Memory", 1)


The problem is when I try to access the members of the float_values my external editor does not know that it is an array with variable contents. I cannot access the converted values.


I have tried the same code within the VDSP console and had the same results.


Is there anything that I can do to fix this issue?