Monday, October 17, 2011

Integer overflow in Microsoft C++ SizeOf operator

I was reviewing some code in C++ and came up to a line in the code that was performing calculations using "SizeOf" ,something in the lines of:
value = sizeof(A)+sizeof(B)+sizeof(D)
At this point a thought came to me and I needed a break anyway, so I've decided to do a little test, I decided to write a small program that will edge case test the SizeOf operator.

The SizeOf operator in C++ is a unary operator that is being calculated during compile time, meaning the compiler replaces the result value with constant values in the final assembly code.
So a SizeOf(int) would be 4 in the final assembly.
The SizeOf function return a result of type size_t which is defined by Microsoft like this:(From crtdefs.h)

#ifdef _WIN64
       typedef unsigned __int64    size_t;
#else  /* _WIN64 */
       typedef _W64 unsigned int   size_t;
#endif  /* _WIN64 */
#endif  /* _SIZE_T_DEFINED */

/*__int64 nHuge;      // Declares 64-bit integer*/

So far so good, everything looks to be in order, right?
Not quite…

Now, back to the test:
I started a new C++ console project in VS2010 and I defined a 4Gig struct in it like so:

typedef struct {
char bytes1[1073741824]; //1 GIGA
char bytes2[1073741824]; //1 GIGA
char bytes3[1073741824]; //1 GIGA
char bytes4[1073741824]; //1 GIGA
Then later in the main function I calculated it's size into an unsigned 64 bit variable,
unsigned long long d = sizeof(DATA);
I added a printf to the variable to see what turns out, Then I compiled the code in Visual Studio 2010 under a 64bit configuration (i.e. with the _WIN64 define set) successfuly with no errors or warnings.

Any guesses on what is the output?

Is it "4294967296"? Or is it "0"? Or "1"?

Drum roll please!

The result is "1"! Que the "WTF?!?"

When looking into the output assembly it looks more odd:
mov edx, 1
Que the second "WTF?!?"

Wait... Didn't I just compile this in x64? with all x64 flags set?, yes I did... so why in the hell is the edx register being used?

In short - Yes.
This is a bug in the Microsoft C++ compiler. I have submitted it to Microsoft and they confirmed it as an issue,
However... it is not a security issue. just a bug :) 

1 comment:

  1. You know what else is a bug? having a 4GB executable, that's a bug.