optimization

问题 I am using pyspark to estimate parameters for a logistic regression model. I use spark to calculate the likelihood and gradients and then use scipy's minimize function for optimization (L-BFGS-B). I use yarn-client mode to run my application. My application could start to run without any problem. However, after a while it reports the following error: Traceback (most recent call last): File "/home/panc/research/MixedLogistic/software/mixedlogistic/mixedlogistic_spark/simulation/20160716-1626

问题 I am using pyspark to estimate parameters for a logistic regression model. I use spark to calculate the likelihood and gradients and then use scipy's minimize function for optimization (L-BFGS-B). I use yarn-client mode to run my application. My application could start to run without any problem. However, after a while it reports the following error: Traceback (most recent call last): File "/home/panc/research/MixedLogistic/software/mixedlogistic/mixedlogistic_spark/simulation/20160716-1626

问题 gcc features likely/unlikely hints that help the compiler to generate machine code with better branch prediction. Is there any data on how proper usage or failure to use those hints affects performance of real code on some real systems? 回答1: The question differs, but Peter Cordes's answer on this question gives a clear hint ;) . Modern CPU's ignore static hints and use dynamic branch prediction. 回答2: I don't know of any thorough analysis of such particular hints. In any case, it would be

问题 I have identified one pandas command timeseries.loc[z, x] = y to be responsible for most of the time spent in an iteration. And now I am looking for better approaches to accelerate it. The loop covers not even 50k elements (and production goal is ~250k or more), but already needs a sad 20 seconds. Here is my code (ignore the top half, it is just the timing helper) def populateTimeseriesTable(df, observable, timeseries): """ Go through all rows of df and put the observable into the timeseries

问题 I have identified one pandas command timeseries.loc[z, x] = y to be responsible for most of the time spent in an iteration. And now I am looking for better approaches to accelerate it. The loop covers not even 50k elements (and production goal is ~250k or more), but already needs a sad 20 seconds. Here is my code (ignore the top half, it is just the timing helper) def populateTimeseriesTable(df, observable, timeseries): """ Go through all rows of df and put the observable into the timeseries

问题 I need to do multiplication on matrices. I'm looking for a library that can do it fast. I'm using the Visual C++ 2008 compiler and I have a core i7 860 so if the library is optimized for my configuration it's perfect. 回答1: FWIW, Eigen 3 uses threads (OpenMP) for matrix products (in reply to above statement about Eigen not using threads). 回答2: BLAS is a de facto Fortran standard for all basic linear algebra operations (essentially multiplications of matrices and vectors). There are numerous

问题 Besides the fact that the standard defines it to be contiguous, why is std::vector contiguous? If it runs out of space, it needs to reallocate a new block and copy the old block to the new one before continuing. What if it wasn't contiguous? When the storage fills up, it would just allocate a new block and keep the old block. When accessing through an iterator, it would do simple >, < checks to see which block the index is in and return it. This way it doesnt need to copy the array every time

问题 I have a numpy array and I can simply append an item to it using append, like this: numpy.append(myarray, 1) In this case I just appended the integer 1 . But is this the quickest way to append to an array? I have a very long array that runs into the tens of thousands. Or is it better to index the array and assign it directly? Like this: myarray[123] = 1 回答1: Appending to numpy arrays is very inefficient. This is because the interpreter needs to find and assign memory for the entire array at

问题 Using the latest gcc compiler, do I still have to think about these types of manual loop optimizations, or will the compiler take care of them for me well enough? 回答1: If your profiler tells you there is a problem with a loop, and only then, a thing to watch out for is a memory reference in the loop which you know is invariant across the loop but the compiler does not. Here's a contrived example, bubbling an element out to the end of an array: for ( ; i < a->length - 1; i++) swap_elements(a,

问题 I understand that this could be argued as a non-issue, but I write software for HPC environments, so this 3.5x speed increase actually makes a difference. In [1]: %timeit 10 / float(98765) 1000000 loops, best of 3: 313 ns per loop In [2]: %timeit 10 / (98765 * 1.0) 10000000 loops, best of 3: 80.6 ns per loop I used dis to have a look at the code, and I assume float() will be slower as it requires a function call (unfortunately I couldn't dis.dis(float) to see what it's actually doing). I