a part of an assignment of mine is based on an array (its size is given by the user) which contains random numbers from 1 to 10^10. Then we have to find the k-th smaller number
rand()*rand() is a lot different than a single rand(), it decreases the randomness and changes its distribution. See this question for a deeper explanation.
Also, an integer usually is 4 bytes. It can contain a value as big as 2^31 (2 billions and something) or 2^32 (4 billions and more) if it's unsigned. You can see the max number it can contain checking the INT_MAX macro defined in limits.h. 10^10 is 10 billions, it won't fit in an integer, you'll have to use a bigger type (long long usually is 64 bytes thus more than you need).
rand, also, returns numbers up to RAND_MAX, and since it returns an int, it won't bigger than INT_MAX. You should use some other way to generate a number as big as 10^10.
If you don't care about randomness and random number distributions you could sum n random numbers (obtained by rand) so that n=10^10 / RAND_MAX.