Floating‑Point Precision — Why Calculators Disagree — GetCalcMaster

Why calculators sometimes disagree

You may see small differences between tools like:

• 0.1 + 0.2 showing 0.30000000000000004 in one place and 0.3 in another
• two calculators matching to 10 digits but not the last 2 digits

This usually isn’t “a bug” — it’s how floating‑point arithmetic works.

The core idea: computers store numbers in binary

Many decimal fractions cannot be represented exactly in base‑2. Just like 1/3 can’t be represented exactly in finite decimal digits (0.333…), 0.1 can’t be represented exactly in finite binary digits.

So computers store the closest representable value, and tiny rounding errors appear in downstream calculations.

Try it

• 0.1 + 0.2 − 0.3
• (0.1+0.2-0.3)==0 (depends on exact/approx modes)

Catastrophic cancellation: the “difference of two close numbers” trap

Some expressions lose precision when they subtract nearly equal quantities. A classic example is:

sqrt(x + 1) - sqrt(x)

For large x, both square roots are close, and their difference is small — the subtraction wipes out meaningful digits.

A stabilized form

Multiply by a conjugate to rewrite it:

sqrt(x + 1) - sqrt(x)
= ( (x + 1) - x ) / (sqrt(x + 1) + sqrt(x))
= 1 / (sqrt(x + 1) + sqrt(x))

Try both forms

• Unstable: sqrt(1e12+1) − sqrt(1e12)
• Stable: 1/(sqrt(1e12+1)+sqrt(1e12))

What “precision” means in practice

Most web calculators use double‑precision floating point (IEEE‑754). It gives ~15–16 decimal digits of precision, but:

• not every operation keeps all digits (especially subtraction of close values)
• transcendental functions (sin, log, exp) involve approximation algorithms
• different libraries may round slightly differently

How to work safely

• Use a tolerance when comparing floats (e.g., “equal within 1e‑9”), not strict equality.
• Rearrange formulas to avoid subtracting nearly equal numbers.
• Cross‑verify using a second method (graph vs numeric, identity check, alternate formula).
• Keep guard digits in Notebook, round at the end.

GetCalcMaster tips

• The calculators have an Exact mode for cases where symbolic/exact simplification is available.
• The Explain panel helps you see how an expression was parsed (which often reveals the real mistake).
• For reproducible work, store the expression and checks in Notebook.

FAQ