#community-help

Precision Issues with Int64 Numeric Field in Go Client

TLDR SamHendley expressed concerns about precision and rounding problems with an int64 numeric field using a timestamp. The problem was traced back to a conversion issue in the go client, prompting SamHendley to suggest updating the openapi spec to use double for reduced errors.

Powered by Struct AI
5
11mo
Solved
Join the chat
Nov 03, 2022 (11 months ago)
SamHendley
Photo of md5-a9a351e11d64f05b41fec183816a0cda
SamHendley
01:28 AM
What is the precision and rounding on min/max stats for a int64 numeric field? I am using it with timestamp (unix seconds) and the values appear to be getting rounded. The lowest field in my collection is 1662633479 but the reported min in facet response is 1662633500 which is a little bit higher. Is there any chance this is being converted to float during the stats phase?
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:37 AM
We do use a double for facet stats computation so it's possible that has an impact on the precision. Do you have a small reproduceable example that we can use for testing?
SamHendley
Photo of md5-a9a351e11d64f05b41fec183816a0cda
SamHendley
01:11 PM
I couldn’t reproduce in curl. Turns out the conversion is happening in the go client when it parses the result. In the openapi spec the min/max/avg fields are declared to be float which is then declared a float32 in the generated code. It looks like that could be at least a float64 for golang if the spec format was ‘double’ instead of ‘float’.
Kishore Nallan
Photo of md5-4e872368b2b2668460205b409e95c2ea
Kishore Nallan
01:11 PM
Ah yes, that must be it.
SamHendley
Photo of md5-a9a351e11d64f05b41fec183816a0cda
SamHendley
01:16 PM
I’d recommend updating the openapi spec to use double since the underlying data is often 64 bits. Fortunatley this is the only place the float term is used in the openapi spec so the impact to generated clients/consumers would be minor but it would probably need a ‘major’ version bump for semver sake