C-more Frequently Asked Questions
Question
Why is it that only 7 digits can be entered in the "Numeric Entry Object "when using Floating point numbers as the data format?
Answer
32-bit Floating point (single precision floating point) only has a decimal precision of 7 significant digits.The significand of the 32 bit Floating point format is only 24 bits.
Which seems to like it should be 8-digits, but you can not get up to 99999999, you can only get to 16777215.
That is why it is about 7 decimals. It's overall max precision is 3.403 x 1038, but that is represented exponentially.
64-bit (or Double Precision) Floating point has a decimal precision of 16 significant digits
FAQ Subcategory: None
This item was last updated on 2011-09-16
Document ID: 1190
The information provided in the AutomationDirect knowledge base is provided "as is" without warranty of any kind. AutomationDirect disclaims all warranties, either express or implied, including the warranties of merchantability and fitness for a particular purpose. In no event shall AutomationDirect or its suppliers be liable for any damages whatsoever including direct, indirect, incidental, consequential, loss of business profits or special damages, even if AutomationDirect or its suppliers have been advised of the possibility of such damages. Some states do not allow the exclusion of limitation of liability for consequential or incidental damages so the foregoing limitation may not apply.