Recently work has been extremely stressful, and I haven’t updated the blog for a long time. I finally found a suitable question, so I did a deep dive and recorded it here. Along the way I also checked my assembly skills; at least for now I can still trace the low-level details contained in simple code…
Question and Analysis
The question came from a brainstorm by my former advisor and colleague (Mr. Heng): suppose there are two classes A and B. A contains a method A::show(), while B contains a function-pointer member B::func_ptr. And by some means, we forcibly make the func_ptr member of an instantiated object b of class B point to the address of the function A::show(). Then when we call b.func_ptr(), who will the this pointer point to?
Personally I think this is an excellent interview question. Strictly speaking, it doesn’t really have a correct answer; you could even say the question itself is wrong. But how you analyze it can strongly reveal how well someone understands C++. We know that C++’s so-called object-oriented mechanism is essentially a technique for binding data together with the methods that operate on that data. Among them, class methods (ignoring static methods) must operate on some data—that is, on an object. Therefore, the object’s memory address must be passed as an implicit argument to the class method. In C++ semantics, this is designed as passing an implicit this pointer; through this pointer, we can directly access the current object’s memory address.
Since object b only stores an A member-function address A::show(), but not the information about how to call this function, even if the call really happens, how would the compiler know whose address should be passed as the this pointer to invoke this function? So in fact this is a question that completely violates object-oriented semantics; it is itself incorrect, and naturally there is no correct answer.
Back to the question: for an instantiated object, what happens if it tries to call a method that does not belong to it? Let’s first write the first demo:
Demo Code I
#include <cstdio>
class A {
public:
void show() {
printf("Address of object: %p\n", this);
}
};
typedef void (*show_func)();
class B {
public:
show_func show;
};
int main()
{
A *a = new A();
a->show();
B *b = new B();
b->show = a->show;
b->show();
printf("Address of a: %p\n", a);
printf("Address of b: %p\n", b);
return 0;
}
This code can pass static program analysis completely, but the compiler will not accept it, with the following error:
error: reference to non-static member function must be called; did you mean to call it with no arguments? b->show = a->show;
From this we can see that the C++ compiler strictly restricts references to non-static member-function symbols—you must reference these symbols in the form of a function call, otherwise the compiler refuses to accept it. This successfully avoids most code that attempts to reference member function addresses with incorrect semantics.
However, programmers’ ingenuity is always limitless. With Google, we can find some special tricks to achieve our goal. As compilers evolve, many early tricks are no longer effective, which shows that compiler developers have long been restricting programmers from abusing the language semantics. But the trick demonstrated by the following code is, for now, still workable (clang-700.1.76):
Demo Code II
#include <cstdio>
class A {
public:
void show() {
printf("Address of object: %p\n", this);
}
};
typedef void (*show_func)();
class B {
public:
show_func show;
};
int main()
{
union {
void *pv;
void (A::*pfn)();
} u;
u.pfn = &A::show;
A *a = new A();
B *b = new B();
b->show = (show_func)u.pv;
b->show();
a->show();
printf("Address of a: %p\n", a);
printf("Address of b: %p\n", b);
return 0;
}
In short, what this code does is define a special type inside a union to complete the function-pointer assignment, and then use another union member to read out the stored value. From this we can see that an object’s methods actually belong to the namespace of the class they belong to, so you must define a variable whose type is in the same namespace in order to access the class method in that namespace and assign it. From this perspective, without considering inheritance and in the simple case, static methods and ordinary class methods are essentially the same; it’s just that for class methods, an implicit this pointer is passed at call time. But in terms of the generated assembly code itself, there is no difference.
In my work environment (MacBook Pro, Intel i5, LLVM version 7.0), compiling and running this code produces the following output:
Address of object: 0x7ffd83400260
Address of object: 0x7ffd834013d0
Address of a: 0x7ffd83400260
Address of b: 0x7ffd834013d0
It seems that after the pointer assignment, a->show() and b->func_ptr() are the same? The this pointer passed to them correctly equals pointers a and b, respectively? So it is as if class B can obtain a show() method just like class A in this way? However, things are not that simple. If we simply swap the order of the code as follows:
A *a = new A();
B *b = new B();
b->show = (show_func)u.pv;
a->show();
b->show();
Then we get the following output:
Address of object: 0x7ffa80400260
Address of object: 0x7fff73c84118
Address of a: 0x7ffa80400260
Address of b: 0x7ffa804013d0
That is, in the call b->show();, the obtained this pointer is no longer equal to pointer b! So the question is: how did this strange phenomenon appear?
So I brought out the debugger, set a breakpoint at the key location, and from the assembly perspective, see what the CPU actually did.
Low-Level Analysis
Using lldb (LLVM’s debugger), disassemble the following lines:
B *b = new B();
b->show = (show_func)u.pv;
b->show();
a->show();
The result is as follows:
callq 0x100001eac ; symbol stub for: operator new(unsigned long)
movq %rax, %rdi ; Note here: %rax holds the function return value, i.e., the address allocated by new, which is the address pointer b points to
movq %rax, %rdx ; The same value is copied to %rdx
movq %rdi, -0x50(%rbp) ; Save the original value of %rdi here first
movq %rax, %rdi ; Then put %rax, i.e., the newly allocated memory address, into %rdi as an input parameter (for object construction)
movq %rdx, -0x58(%rbp) ; Then save the value of %rdx, i.e., save the value of %rax
callq 0x100001e9a ; symbol stub for: B::B() call the constructor of object B
movq -0x58(%rbp), %rax ; After construction, restore the value of %rax (since the constructor has no return value, %rax is useless at this point)
movq %rax, -0x38(%rbp) ; Save the value of %rax, i.e., the value of pointer b, to a stack location -0x38(%rbp)
movq -0x18(%rbp), %rcx ; Fetch the value of u.pv here
movq -0x38(%rbp), %rdx ; Then locate b’s position on the stack
movq %rcx, 0x8(%rdx) ; Perform assignment: b->show = (show_func)u.pv;
; Note that %rdi’s value is not modified in this context;
; and if we print the register values, we can also see that %rdx has the same value as %rdi
movq -0x38(%rbp), %rcx ; Store the pointer to object b into %rcx
callq *0x8(%rcx) ; Equivalent to (*b->show)(b): dereference the pointer and call directly, without passing any parameters.
movq -0x20(%rbp), %rdi ; But here, a correct member function call needs to use %rdi to pass an implicit argument, i.e., the this pointer
callq 0x100001e88 ; symbol stub for: A::show() then directly jump to the corresponding member function address
Each line above includes its corresponding comment. By analyzing these assembly statements, we can basically reconstruct the entire process—what happened when the CPU executed these statements.
Let’s first analyze how a normal member function call works. Look directly at the last two assembly lines, corresponding to a->show();. We see that since show() does not need ordinary parameters, the compiler-generated assembly first assigns some value from stack memory to the %rdi register, and then directly issues the function call. From analyzing the overall context, we can determine that the stack memory address -0x20(%rbp) is actually the memory address of pointer variable a. From this we can guess: when issuing a member function call, the this pointer is passed via the %rdi register. In fact, System V (i.e., Unix) on x86_64 has the following calling convention [1]:
The first six integer or pointer arguments are passed in registers RDI, RSI, RDX, RCX, R8, and R9.
And the this pointer happens to be the first and also the only argument, so it will naturally be passed in %rdi. What if there are more than six arguments? Then they have to be pushed onto the stack.
Next, let’s look at this assembly from the beginning. When constructing an object, memory must be allocated for it. Therefore, the starting address of a free region is returned by the memory allocation function and stored in %rax (following the calling convention). Note that this uses the new operator, so it allocates a heap address. Then the starting address of this new memory is saved into %rdi and %rdx respectively, as well as the stack memory location -0x50(%rbp). Next, %rdx is backed up and saved at -0x58(%rbp), which is likewise equivalent to saving the new memory address at that location. Clearly, the new memory address is the location pointer b points to. After memory allocation completes, the constructor of object B is called. After construction, the value of %rax is restored; since the constructor has no return value, the value in %rax is useless at this point and it doesn’t matter if it gets overwritten. Then the value of %rax, i.e., the memory address of object b, is saved to a stack location -0x38(%rbp). This location is actually where the pointer variable b lives on the stack, so at this point the statement B *b = new B(); is fully executed.
After object b is constructed, the value of u.pv is read out and assigned to some offset position after the start address of object b, which naturally corresponds to the location of a member variable of class B. Now if we review the context of this part, we can notice that %rdi has not been modified here. And if we print the register values, we can also see that %rdx and %rdi have the same value—that is, both equal the value of pointer b. This indicates that during the call to B’s constructor, the value in register %rdi was not modified and remained as-is, i.e., the value of pointer b. As mentioned earlier, this pointer clearly points to a heap address. Therefore, in the subsequent function call, even though the this pointer value is not correctly set, because the agreed-upon register %rdi happens to contain the correct value, the phenomenon shown earlier appears: while object b forcibly calls a member function of class A, it “magically” gets a correct this pointer to itself. But in reality, this is just a coincidence. When, as in Demo II, we slightly swap the execution order of the constructors, the value in %rdi gets changed, and the coincidence no longer occurs.
So how can we make object b work normally when it “borrows” class A’s member function? Let’s look at the third demo:
Demo Code III
#include <cstdio>
class A {
public:
const char *type_name = "A";
void show(int v1, int v2, int v3, int v4, int v5, int v6) {
printf("This is class %s\n", this->type_name);
}
};
typedef void (*show_func)(void *, int v1, int v2, int v3, int v4, int v5, int v6);
class B {
const char *type_name = "B";
public:
show_func show;
};
int main() {
union {
void *pv;
void (A::*pfn)(int, int, int, int, int, int);
} u;
u.pfn = &A::show;
A *a = new A();
B *b = new B();
b->show = (show_func)u.pv;
a->show(0, 0, 0, 0, 0, 0);
b->show(b, 0, 0, 0, 0, 0, 0);
return 0;
}
We only need to cleverly declare the function pointer and explicitly pass the object’s memory address as the this pointer, and then object b can successfully “borrow” class A’s member function. Of course, the prerequisite for doing this is that class A and class B have similar memory layouts. Strictly speaking, for the member variables accessed by the “borrowed” member function, there must be the same memory layout.
Finally, the output of the code above is:
This is class A
This is class B
As you can see, even with six parameters passed in, the whole program still runs successfully! This also indicates that under normal circumstances, in the assembly generated by the compiler, the this pointer is implicitly passed as the first parameter to the member function.
Python Member Function Calls
To further broaden the discussion, let’s examine what similar behavior would look like in Python. Consider the following code:
Demo Code IV
class A:
def show(self):
print self
class B:
def __init__(self):
self.show = None
if __name__ == '__main__':
a = A()
b = B()
b.show = a.show
a.show()
b.show()
The output is:
<main.A instance at 0x10d6e1cb0>
<main.A instance at 0x10d6e1cb0>
That is, when calling class A’s member function on object b, the function’s self still points to the instance object a of class A! The deeper reason behind this different behavior involves the fundamental implementation differences between interpreted and compiled languages. For the latter, in order to achieve the highest possible efficiency, functions are compiled into machine code, containing as little runtime information and type information as possible. For the former, highly flexible dynamic languages like Python effectively implement functions as “self-contained” objects—each function object can fully locate its owning object and so on through its internal member variables. Therefore, unlike C++, when executing b.show(), the interpreter does not pass any additional information to this member function; instead it directly obtains information about the owning object a from the function object a.show, and passes that as the self parameter to complete the function call. The explicitly written self reference when defining a member function is merely a syntactic rule; in fact, no external parameter passing is performed based on it.
Therefore, if we write the following code, the effect is the same:
Demo Code V
class A:
def show(self):
print self
if __name__ == '__main__':
a = A()
f = a.show
a.show()
f()
This example clearly shows that member functions in Python are indeed self-contained. So why did Python design them as completely “self-contained” objects? On the one hand, it benefits from the flexibility of dynamic-language implementations, allowing the interpreter to record as much runtime information as possible in an object. On the other hand, this design can also be seen as syntactic sugar, allowing us to write concise and flexible code in special scenarios such as callbacks. For example, in the tkinter GUI library, we can write code like the following to create a button:
import Tkinter as tk
tk.Button(master=self.root, text='Do it',
command=self.do_func).pack(side=tk.LEFT)
If function objects were not self-contained, then we would pass only a member function as the callback when the button is pressed—but when the callback actually happens, how would we know which object this function should operate on? In that case, more awkward syntax would be required to make it work. Therefore, Python’s design has deeper considerations behind it.
References
[1] https://en.wikipedia.org/wiki/X86_calling_conventions
[2] Bryant, Randal E., et al. Computer Systems: A Programmer’s Perspective.
Bonus
On the day before Mr. Heng left, he was still doing phone interviews; a few days later he joined a major company. If he still has interviewing duties there and happens to meet a previous candidate, that scene would be amusing… the interviewee would probably be mentally collapsing: how can I run into this guy everywhere! 😂😂😂