optimization

Optimization of json.load() to reduce in-memory usage and time in Python

余生颓废 提交于 2020-07-22 05:57:37
问题 I have 10K folders each with 200 records in 200 JSON format files. Trying to compile all records into one data frame then finally into a CSV (other format suggestions welcome) Here is my working solution which takes around 8.3hrs just for the dataframe building process. (Not converting into CSV) %%time finalDf = pd.DataFrame() rootdir ='/path/foldername' all_files = Path(rootdir).rglob('*.json') for filename in all_files: with open(filename, 'r+') as f: data = json.load(f) df = pd.json

Split a vector into chunks such that sum of each chunk is approximately constant

若如初见. 提交于 2020-07-21 08:29:10
问题 I have a large data frame with more than 100 000 records where the values are sorted For example, consider the following dummy data set df <- data.frame(values = c(1,1,2,2,3,4,5,6,6,7)) I want to create 3 groups of above values (in sequence only) such that the sum of each group is more or less the same So for the above group, if I decide to divide the sorted df in 3 groups as follows, their sums will be 1. 1 + 1 + 2 +2 + 3 + 4 = 13 2. 5 + 6 = 11 3. 6 + 7 = 13 How can create this optimization

Split a vector into chunks such that sum of each chunk is approximately constant

爷,独闯天下 提交于 2020-07-21 08:29:08
问题 I have a large data frame with more than 100 000 records where the values are sorted For example, consider the following dummy data set df <- data.frame(values = c(1,1,2,2,3,4,5,6,6,7)) I want to create 3 groups of above values (in sequence only) such that the sum of each group is more or less the same So for the above group, if I decide to divide the sorted df in 3 groups as follows, their sums will be 1. 1 + 1 + 2 +2 + 3 + 4 = 13 2. 5 + 6 = 11 3. 6 + 7 = 13 How can create this optimization

Can I use a machine learning model as the objective function in an optimization problem?

半城伤御伤魂 提交于 2020-07-18 16:00:18
问题 The bounty expires in 7 hours . Answers to this question are eligible for a +50 reputation bounty. Neel is looking for an answer from a reputable source . I have a data set for which I use Sklearn Decision Tree regression machine learning package to build a model for prediction purposes. Subsequently, I am trying to utilize scipy.optimize package to solve for the minimized solution based on a given constraint. However, I am not sure if I can take the decision tree model as the objective

Can I use a machine learning model as the objective function in an optimization problem?

…衆ロ難τιáo~ 提交于 2020-07-18 15:59:26
问题 The bounty expires in 7 hours . Answers to this question are eligible for a +50 reputation bounty. Neel is looking for an answer from a reputable source . I have a data set for which I use Sklearn Decision Tree regression machine learning package to build a model for prediction purposes. Subsequently, I am trying to utilize scipy.optimize package to solve for the minimized solution based on a given constraint. However, I am not sure if I can take the decision tree model as the objective

Why is a ternary operator with two constants faster than one with a variable?

南楼画角 提交于 2020-07-17 06:35:51
问题 In Java, I have two different statements which accomplish the same result through using ternary operators, which are as follows: num < 0 ? 0 : num; num * (num < 0 ? 0 : 1); It appears that the second statement is unnecessarily complex and would take longer than the first, however when I recorded the time that each took, using the following code, the results were as follows: final long startTime = System.currentTimeMillis(); Random rand = new Random(); float[] results = new float[100000000];

Why is a ternary operator with two constants faster than one with a variable?

邮差的信 提交于 2020-07-17 06:35:19
问题 In Java, I have two different statements which accomplish the same result through using ternary operators, which are as follows: num < 0 ? 0 : num; num * (num < 0 ? 0 : 1); It appears that the second statement is unnecessarily complex and would take longer than the first, however when I recorded the time that each took, using the following code, the results were as follows: final long startTime = System.currentTimeMillis(); Random rand = new Random(); float[] results = new float[100000000];

How to pass the parameters into solnl() function in R?

僤鯓⒐⒋嵵緔 提交于 2020-07-10 10:36:05
问题 I have the non-linear optimization problem I have tried to use the solnl() function from the NlcOptim package. Unfortunately, I don't know how to pass the param into the solnl() function correctly. And I have the Error in objfun(X) : argument "param" is missing, with no default Calls: solnl -> objfun My code is below: library(MASS) library(NlcOptim) objfun=function(x, param, n){ return( x[n] * (param[1] - param[2] / sum(x[1:(n-1)]) )) } #constraint function confun=function(x, param, n){ f

Rounding Non-LinearExpr with google or-tools SAT solver

与世无争的帅哥 提交于 2020-07-10 10:27:07
问题 Using CP-SAT of google or-tools I'm trying to write this constraint: q >= (50x + 100y + 150z + 200k + 250p + 300v) / (x + y + z + k + p + v) Where q is a simple integer. The thing is I need to round the right side of the equation (let's call it expression ) as follows: if(expression < 75) { expression = 50; } else if(expression < 125) { expression = 100; } else if(expression < 175) { expression = 150; } else if(expression < 225) { expression = 200; } else if(expression < 275) { expression =

How to use 'cbc' in pyomo's SolverFactory in Windows 10

不羁岁月 提交于 2020-07-10 10:26:46
问题 I have been trying to use 'cbc' solver in pyomo as following: """ import pyomo.environ as pe solver = pe.SolverFactory('cbc') result = solver.solve(m) """ Currently, I have been relying on the embedded solver with pyomo installation, which is 'glpk', and I have found that 'cbc' can do better job in many circumstances. I am using: python 3.XX Anaconda environment Windows 10 Pyomo (installed via Conda) It would be great to upgrade the ability of pyomo by using advanced solver! 回答1: You will