I'm working on a project that uses a BMP280 sensor. The raw temperature / pressure values have to be compensated using calibration values stored in the device. This is now about the temperature compensation calculation, more precisely about the integer algebra version of it published in the datasheet.
We are on page 22/23. The calculation of var1 is:
var1 = (adc / 16384 - t1 / 1024) * t2
which according to the datasheet in 32 bit systems (assuming 32 bit signed integer variables and adc as a 20 bit unsigned value, t1 and t2 are as 16 bit signed values):
var1 = [(adc>>3 - t1<<1) * t2]>>11
The trick is that adc is a 20 bit unsigned value. I think it's wrong because multiplying a 20 bit value right shifted 3 (yields a 17 bit value) with a 16 signed is 33 bit data with the sign bit. The maximum value for adc ic 0xFFFFF shifted -> 1FFFF * 7FFF = FFFD8001. The max value of a signed 32 int is 7FFFFFFF. It will overflow at high temperatures. (What a lovely way to fail)
I think the correct calculation would be:
var1 = [(adc>>4 - t1) * t2]>>10
PS: the datasheed does say to use these calculations at your own risks but it doesn't make this any better.
Any comments on this?
Edit: yes, there is the -t1 part that could in theory knock down a bit from the adc value. But t1 can be very little, even zero, counting on this to knock down another bit worth of value is wishful thinking.