Quantcast

At Precision Point

Last modified by
on
Jul 24, 2020, 6:28:07 PM
Created by
on
Nov 16, 2013, 9:16:38 PM
` = "At Precision Point"`
`"Input value"`
`"Tolerance"`
`"Expected Value"`
Tags
UUID
613e3c59-4f04-11e3-9569-bc764e049c3d

This equation determines if a value is within a specified tolerance (+/-) of a specified precision point.  This can be used in equations where you are trying to assure the output value is set to a specific value when precision errors cause the resultant to be off slightly.

Ex:  An equation's inputs should result in a known value of 1.0 but precision in Big Decimal calculations result in the value being displayed as 0.999999999999...  You can use this equation to test for closeness to the right result at a specified precision.  So, you could choose to set the display value to exactly 1.0 when the difference from 1.0 is less than, let's say, 1E-15.


This equation, At Precision Point, references 1 page
  • Comments
  • Attachments
  • Stats
No comments
This site uses cookies to give you the best, most relevant experience. By continuing to browse the site you are agreeing to our use of cookies.