The tag has no usage guidance.

learn more… | top users | synonyms (1)

4
votes
4answers
107 views

Avoiding Division by Zero Using Float Comparison

Source code analyzers (like SonarQube) complain about float (or double) equality comparisons because equality is a tricky thing with floats; the values compared can be the results of computations ...
-3
votes
1answer
57 views

Floating point error questions

Books teach us that single precision has 24 significands digits this is aproximatly 7 decimal significant digits. Now what would be the sense of printing a float with more than 7 digits? Im wondering ...
0
votes
0answers
56 views

How do the conversions between signed, unsigned and float types work? [migrated]

The compiler I use is g++ (Ubuntu 4.8.4-2ubuntu1~14.04) 4.8.4. I compile my programs with the following command: g++ -std=c++11 -pedantic -Wall program.cpp The program no. 1.: #include ...
0
votes
0answers
35 views

avoid nan and inf in floating point multiplication and division

I have the following hypothesis: It is better to reorder mult and div operation in such a way that we can do our best effort to avoid nan, inf and operating with subnormals. The reorder should follow ...
7
votes
2answers
128 views

Should `Vector<float>.Equals` be reflexive or should it follow IEEE 754 semantics?

When comparing floating point values for equality, there are two different approaches: NaN not being equal to itself, which matches the IEEE 754 specification. NaN being equal to itself, which ...
-2
votes
1answer
85 views

How to convert byte-array (4 bytes) back into float? [closed]

A piece of code I'm trying to debug generated a sequence of 4 bytes from a known float number. I'm trying to verify the float number given the four bytes. The (known) float 8.03 generated the bytes ...
3
votes
0answers
81 views

Rationale of IEEE754 direct rounding behavior near infinity [closed]

A preface: it's well known that IEEE754 defines five rounding modes (in 2008 edition terms, with my abbreviations): rounding to nearest, ties to even (RNE) - the default mode for binary arithmetic; ...
3
votes
3answers
107 views

Is order of arguments in an arithmetic expression important to achieve as most exact result as possible (speed is not necessary)?

Actually, in this question I do not ask about particular language or architecture, however I understand there might be some differences. In physics / engineering it is usually better to handle ...
1
vote
2answers
666 views

How to calculate min/max values of floating point numbers?

I'm trying to calculate the min/max, or the lowest to highest value range of a 48 bit floating point type MIL-STD-1750A (PDF) (WIKI). Ex: How a double range is 1.7E +/- 308 I've looked around for ...
4
votes
2answers
563 views

Why does Python's math.ceil return a float?

While I understand that the difference between integral and long values is blurred in Python, the difference between floats and integral values is not. Therefore, I'm having a difficult time ...
0
votes
1answer
123 views

floating-point number stored in float variable has other value [duplicate]

I'm reading a book about C programming By "King.K.N" and I'm reading the following statement: If we store 0.1 in a float variable, we may later find that the variable has a value such as ...
4
votes
0answers
164 views

How can unums emulate IEEE's negative zero?

I am currently reading "The End of Error - Unum Computing" by John Gustafson (Youtube). What I am still not sure about is how the cases handled in IEEE by negatively signed zero are handled with ...
2
votes
2answers
157 views

How can a computer round the last digit in a floating point representation?

I'm confused by how a computer rounds off the last digit in the floating point representation. For example, I'm told that x=1.24327789 is stored in a computer with a 6-digit capacity then it;s ...
1
vote
2answers
78 views

How to implement rounding in an all-purpose stack language using different types?

Disclaimer: If you are not terribly interested in numerics and mathematical processes, this is most likely nothing for you. I am currently a bit stuck in a development process of a private project I ...
33
votes
4answers
911 views

Why is negative zero important?

I'm confused about why we care about different representations for positive and negative zero. I vaguely recall reading claims that having a negative zero representation is extremely important in ...
3
votes
1answer
200 views

Is the behaviour of floating-point arithmetic defined by the c++ standard?

I started writing this as a SO question, but I think it's better suited here. While I'm happy for a simple answer I'm also fond of the "Teach a Man to Fish" philosophy so I'm happy for people to point ...
1
vote
1answer
480 views

adding the digits of a number

I am trying to write a program that asks the user for a decimal number and then calculates the sum of its digits. for example if the number is 123.25 then the sum will be 1+2+3+2+5=13. I decided to ...
-1
votes
1answer
71 views

Is this statement correct for floating point numbers?

The decimal point can "float" to accommodate larger numbers while staying in 32-bits which is why float is considered "in-accurate" Is this an accurate statement? I want to know if I understand ...
27
votes
10answers
6k views

Why are floating point numbers used often in Science/Engineering?

While investigating the accuracy of floating point numbers, I've seen in a few places a statement similar to "float and double are (designed for/used often in) engineering and scientific ...
0
votes
1answer
916 views

How is a floating point number represented in Java?

Which algorithm does java uses to convert floating point number (Ex:0.15625) into binary(0.00101)? Whether java uses normalized form or denormalized format? If I type float ab=0.15625 in Java source ...
10
votes
1answer
243 views

Implied precision of floating-point functions

While reviewing another programmer's implementation of a function to calculate the normal distribution CDF, I made a suggestion to either replace the entire implementation with Python's built-in ...
2
votes
2answers
158 views

How are some IEEE 754 floats printed succinctly?

Using this calculator, I can see how the decimal ".8" and the expression ".7 + .1" have different representations: .8 = 1.1001100110011001100110011001100110011001100110011010 *2-1 .7 + .1 = ...
2
votes
0answers
45 views

Time passage arithmetic explanation

I ported this from http://www.effectgames.com/effect/article.psp.html/joe/Old_School_Color_Cycling_with_HTML5 some time ago. However i'm now wanting to modify it for the purpose of changing it from ...
40
votes
3answers
5k views

Are there numbers that are not representable in base 10 but can be represented in base 2?

C# has the decimal type which is used for numbers that needs exact representation in base 10. For instance, 0.1 cannot be represented in base 2 (e.g. float and double) and will always be an ...
0
votes
4answers
209 views

Do commercial statistical calculators calculate with a high degree of arbitrary precision or just floating point precision? [closed]

Do commercial statistical calculators like Minitab or SAS (and open source like R) perform statistical analysis with very high precision? Or do they just use floating point (or double) precision? I ...
4
votes
3answers
7k views

Why is BigDecimal the best data type for currency? [duplicate]

I was reading this question and the accepted answer says that BigDecimal is the best type for representing currency values. I've also seen several other questions support the use of big decimal. Why ...
4
votes
1answer
561 views

Custom class for monetary amount (money value) instead of BigDecimal

On StackOverflow I found multiple advices to stick with BigDecimal in Java instead of reinventing the wheel. Despite that, I decided to create my custom class, and I would ask if that was a good idea ...
4
votes
3answers
1k views

When do rounding problems become a real problem? Is the least significant digit being one off really a big deal? [duplicate]

If I do .1 + .1 + .1 in Python, I get 0.30000000000000004. (I am not asking about Python in particular, and do not want Python specific answers.) The only problem I can see with this is ...
1
vote
1answer
267 views

Will we go back(?) to fixed-point arithmetic in the near future? [closed]

As far as I recall, this has been the ongoing trend of the past, I'm just a student, so I might be wrong: Long ago: Integer numbers and calculations. Very precise, however cannot present a big ...
24
votes
11answers
4k views

Why do we still use floats? [duplicate]

I understand why floats served a purpose in the past. And I think I can see why they're useful in some simulation examples today. But I think those example are more exceptional than common. So I ...
3
votes
2answers
323 views

Why does IEEE 754 Single Float have 8 bits for exponent but only allow 38 base ten Max Value

Why is it that the IEEE 754 standard provide 8 bits for the exponent but puts a range maximum of 38 on the largest acceptable exponent? Seems kind of ridiculous to have the data storage there but not ...
6
votes
3answers
516 views

Unit (regression) testing scientific algorithms given floating point behavior

I have been working on a project and running into a very difficult problem. The problem can be stated simply as how to unit-test numerical algorithms. However if you just took this simple statement ...
2
votes
2answers
6k views

Can anyone explain representation of float in memory?

This is not duplicate question as I read the previous question. Can anyone help me in understanding how float values are stored in the memory . My doubt is here float values contain '.' (for example ...
0
votes
1answer
125 views

How can I handle clock hands using doubles that suffers at most O(log n) corrupt bits at nth frame of display?

I presently have a clock app that calculates from scratch at every iteration. This means O(1) corrupt bits in my doubles and heavy object creation and deletion as well. I am wary of running ...
2
votes
4answers
330 views

Generic term for “objects” vs “fundamental types”?

What are the exact terms to call data types with a logic structure (like C structures, C++ or Java objects) versus fundamental data types (like numeric types, characters, booleans...) independently of ...
2
votes
3answers
410 views

Why is float the default in the majority of languages?

In programming languages, for represent numbers you have (mainly) two types: Int(s) and floats. The us of Int is very easy. However floats cause a lot of unexpected surprises: ...
8
votes
2answers
5k views

How Lua handles both integer and float numbers?

As far as I remember myself programming I was taught not to compare floating point numbers for equality. Now, while reading Programming in Lua about Lua number type, I found following: The number ...
2
votes
1answer
3k views

Addition of double's is NOT Equal to Sum of the double as a whole

I am aware of the floating point errors as I had gained some knowledge with my question asked here in SE Floating Point Errors. What I am finding it difficult to understand is, the output of the ...
7
votes
2answers
2k views

Solutions for floating point rounding errors

In building an application that deals with a lot of mathematical calculations, I have encountered the problem that certain numbers cause rounding errors. While I understand that floating point is not ...
1
vote
2answers
625 views

Implementing base-10 floating point division

I'm implementing floating-point arithmetic, for a micro-controller which does not support floating point numbers, in either hardware or software. (Software being "written" in a sort of electrical ...
3
votes
7answers
870 views

Is (1/(1/x)) always a perfect round trip?

Is the following guaranteed to return true for all numerical and non-zero values of x? bool IsRoundTrip(double x) { double y = 1 / (1 / x); return x == y; } What conditions would cause a ...
5
votes
2answers
2k views

Performance and other issues with using floating point types in C++

Being interested in C++ performance programming there is one aspect I really have no clue about- and that is the implications of using floating point calculations vs doubles vs normal integer ...
3
votes
1answer
885 views

Dummy float values to force floating-point operations

It is common to use idioms such as: x/60.0 to force a floating-point division when x is an integer in languages which do not have distinct operators for integer and decimal division. Is this an ...
-2
votes
1answer
2k views

in Objective-C, value of type double/float can only be NAN, INFINITY, & normal number?

I knew double or float value can be not only normal value(-1.3, 0, 1.0, 2.3) but also NAN and INFINITY in Objective-C. Is there other special values except for NAN and INFINITY for double/float value ...
6
votes
2answers
2k views

How to remove the boundary effects arising due to zero padding in scipy/numpy fft?

I have made a python code to smoothen a given signal using the Weierstrass transform, which is basically the convolution of a normalised gaussian with a signal. The code is as follows: #Importing ...
41
votes
12answers
3k views

Is there an alternative to bits?

Is there an alternative to bits as the smallest unit of data? Something that won't be only 0 or 1, but actually hold many possible states in between? Wouldn't it be more natural to store floats like ...
0
votes
1answer
609 views

Conversion of a number from Single precision floating point representation to a Half precision floating point [closed]

I have a code where I have to work on Half precision floating point representation numbers. To achieve that I have created my own C++ class fp16 with all operators(arithmetic logical, relational) ...
37
votes
7answers
21k views

What causes floating point rounding errors?

I am aware that floating point arithmetic has precision problems. I usually overcome them by switching to a fixed decimal representation of the number, or simply by neglecting the error. However, I ...
27
votes
12answers
4k views

Why do you need float/double?

I was watching http://www.joelonsoftware.com/items/2011/06/27.html and laughed at Jon Skeet joke about 0.3 not being 0.3. I personally never had problems with floats/decimals/doubles but then I ...