I try to use convexHull and convexityDefects to define a hand. But when the program hit the convexityDefects, there is always an error
It looks like the issue is that convexityDefects() expects an array for the contourDefects parameter:
void convexityDefects(InputArray contour, InputArray convexhull, OutputArray convexityDefects);
Both contours and hull are vector types, which is correct, but defects also needs to be a vector. That way when you loop through the contours array, defects[i] will contain an array to pass to convexityDefects():
...
vector> defects( contours.size() );
...
Later on in your code, you will have to loop through the defects[i] vector to print the depth as well, if that is what you were trying to do.