AnsweredAssumed Answered

Floating Point Precision within Expressions Pane

Question asked by usernameGoesHere on Mar 19, 2015
Latest reply on Jan 5, 2016 by Kader.M

Hi,

 

I am currently having issues with the expressions pane rounding floating point numbers. For example if I enter:

7464.5986

The expressions pane shows: 7464.6

I know that the processor is not losing any digits because the hex value in memory is set correctly to: 0x45e944ca

 

I did find this article which is similar to the bug that I am seeing: https://ez.analog.com/message/25811#25811

 

The current method that I have to get the values from VDSP is the processor.EvaluateToString function which has worked well until we saw the issue with VDSP rounding the float values.

 

I tried to change the way that I get the values in order to use the ConvertToDouble Method that is documented within the VDSP API help.

my code is currently:

( javascript )

var memory_type_list = processor.MemoryTypeList

var memory_type = memory_type_list( "Data(DM) Memory" )

var value_list = memory_type.GetMemory( VariableAddressAsInteger, 1, 1 )

float_values = value_list.ConvertToDouble("Data(DM) Memory", 1)

 

The problem is when I try to access the members of the float_values my external editor does not know that it is an array with variable contents. I cannot access the converted values.

 

I have tried the same code within the VDSP console and had the same results.

 

Is there anything that I can do to fix this issue?

 

Thanks,

David

Outcomes