Arithmetic Precision

Pronunciation: /ærɪθˈmɛ tɪk prɪˈsɪ ʒən/ ?

Arithmetic precision refers to the number of digits that are meaningful when expressing a value. In pure mathematics, the digits in a number can go on forever. In practical mathematics, computers can only handle a finite number of digits and measuring instruments can only measure a finite number of digits. The number of digits that are meaningful is the arithmetic precision. One says that there are 'n' significant digits.

Measurement and Arithmetic Precision

Metric ruler being used to measure banana seeds.
Figure 1: A metric ruler being used to measure banana seeds.

Measuring devices can only measure a finite number of digits. Figure 1 shows banana seeds being measured using a metric ruler. The smallest unit of measure on the ruler is millimeters. Each of the banana seeds measures approximately 16 millimeters. Since there are 2 digits in 16, the measurement is accurate to 2 significant digits. See also Measurement Error.

Computation and Arithmetic Precision

Computation is necessarily limited to a finite number of digits. To compute the square root of 2, for example, to an infinite number of digits would take an infinite amount of memory and an infinite amount of time. So, when computing values, one chooses an arithmetic precision. For computing devices that store numbers in binary format, such as most computers and many calculators, arithmetic precision is measured in binary digits. For computing devices that store numbers in decimal format, such as many calculators, arithmetic precision is measured in decimal digits.

More Information

  • Warren-Smith, R.F.. Arithmetic Precision. starlink.rl.ac.uk. Council for the Central Laboratory of the Research Councils. 2009-11-23.
  • INFO: Visual Basic and Arithmetic Precision. support.microsoft.com. Microsoft. 2009-11-23.
  • McAdams, David. Measurement Error. allmathwords.org. Life is a Story Problem LLC. 2011-03-18.
  • McAdams, David. Computational Error. allmathwords.org. Life is a Story Problem LLC. 2011-03-18.

Cite this article as:


Arithmetic Precision. 2011-03-19. All Math Words Encyclopedia. Life is a Story Problem LLC. http://www.allmathwords.org/en/a/arithmeticprecision.html.

Translations

Image Credits

Revision History


2011-03-19: Added links to Measurement Error and Computational Error (McAdams, David.)
2009-11-23: Authored (McAdams, David.)

All Math Words Encyclopedia is a service of Life is a Story Problem LLC.
Copyright © 2005-2011 Life is a Story Problem LLC. All rights reserved.
Creative Commons License This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 License