Best practices for measuring the run-time complexity of a piece of code

Question

I have a gnarly piece of code whose time-efficiency I would like to measure. Since estimating this complexity from the code itself is hard, I want to place it in a loop and time

Answer 1

Suppose you run the following in a loop. At iteration i = 0, 1, 2, ...., for some fixed n_0 > 0 and very large n, you sample the function at 2 i + n_0 equi-distanced (up to rounding) points in the range 1, ..., n. You then do either one of the following or a combination of both:

1. Train a spline using the even points and test it on the odd points (and also vice versa). Decide the iteration is enough if the l2 error is below some threshold.

2. Train a spline using all the points, and test it on the values at, say 2n. Again, decide the iteration is enough if the l2 error is below some threshold.

Point 1. emphasizes the interpolation error, and point 2. emphasizes the extrapolation error. Realistically speaking, I think you will at best be able to identify functions described by a spline.

---

Depending on the fitting method you use, you might need to fit some meta-parameters for the spline method. In this case, you might need to use more than ~2 i samples per iteration, as you might need to use some of them for parameter-tuning cross validation.