batch-processing | 易学教程

Best approach using Spring batch to process big file

阅读更多关于 Best approach using Spring batch to process big file

问题 I am using Spring batch to download a big file in order to process it. the scenario is pretty simple: 1. Download the file via http 2. process it(validations,transformations) 3. send it into queue no need to save the input file data. we might have multiple job instances(of the same scenario) running in the same time I am looking for best practice to handle this situation. Should I create Tasklet to download the file locally and than start processing it via regular steps? in that case I need

extracting specific lines of data from multiple text files, to convert to a single csv file

阅读更多关于 extracting specific lines of data from multiple text files, to convert to a single csv file

问题 First, apologies for my poor coding ability, however I have spent a few hours reading the forums and giving it a crack, so I would really appreciate any help with the following problem: I have 3 text files, from which I would like to take the filename, 3rd line of data, 5th line, and 7th line and pop them into a single CSV, such as follows: filename1, linedata3, linedata5, linedata7 filename2, linedata3, linedata5, linedata7 filename3, linedata3, linedata5, linedata7 Simples, eh? not so, for

BigQuery - same query works when submitted from UI and reports SQL syntax error from batch

阅读更多关于 BigQuery - same query works when submitted from UI and reports SQL syntax error from batch

问题 I have a SQL query involving two joins on different fields. When I run this query interactively in the UI, I get back a result set, no problem. When I submit the exact same query in batch, I get back a SQL syntax error: Ambiguous field name 'video' in JOIN. Please use the table qualifier before field name. But the joins are already fully qualified: SELECT t1.video AS video t1.session AS session ... FROM (select video, session, ...) t1 LEFT JOIN EACH (select video, ...) t2 ON t1.video = t2

batch file + convert LF to CR+LF

阅读更多关于 batch file + convert LF to CR+LF

问题 We have a shell script file named LineFeed.sh which does a function of converting a Linefeed( LF ) to Carriage Return + LineFeed. We want the same to be done by a batch file in windows . Is it possible? Linux shell file E_WRONGARGS=65 cat OutputList|while read -r Line do if [ -z "$Line" ] then echo "Usage: `basename $0` filename-to-convert" exit $E_WRONGARGS fi NEWFILENAME=$Line.unx CR='\015' # Carriage return. # 015 is octal ASCII code for CR. # Lines in a DOS text file end in CR-LF. # Lines

Batch OCR Program for PDFs [closed]

阅读更多关于 Batch OCR Program for PDFs [closed]

This has been asked before, but I don't really know if the answers help me. Here is my problem: I got a bunch of (10,000 or so) pdf files. Some were text files that were saved using adobe's print feature (so their text is perfect and I don't want to risk screwing them up). And some were scanned images (so they don't have any text and I will have to settle for OCR). The files are in the same directory and I can't tell which is which. Ultimately I want to turn them into .txt files and then do string processing on them. So I want the most accurate OCR possible. It seems like people have

How to execute a for loop in batches?

阅读更多关于 How to execute a for loop in batches?

问题 for x in records: data = {} for y in sObjectName.describe()['fields'] data[y['name']] = x[y['name']] ls.append(adapter.insert_posts(collection, data)) I want to execute the code ls.append(adapter.insert_post(collection, x)) in the batch size of 500, where x should contain 500 data dicts. I could create a list a of 500 data dicts using a double for loop and a list and then insert it. I could do that in the following way, , is there a better way to do it? : for x in records: for i in xrange(0

Check if a process is running or not?

阅读更多关于 Check if a process is running or not?

问题 I want to check if a process is running or not ? if the process is not running, then i execute it (in this example I took the calculator with process name = calc.exe) I started the batch script, but I have a syntax problem I believe ! @echo off Set MyProcess=calc.exe echo %MyProcess% pause for /f "tokens=1" %%i In ('tasklist /NH /FI "imagename eq %MyProcess%"') do set ff=%%i echo %ff% If /i %ff%==%MyProcess% (Echo %ff% est en cours d^'execution) Else (Start %MyProcess%) pause 回答1: Here's

SAPUI5 - Batch Operations - how to do it right?

阅读更多关于 SAPUI5 - Batch Operations - how to do it right?

I got multiple EntititySets which I want to update in my SAP Backend, therefore I fetch my data (payload) as JSON and put it in a request (its successfully in the node "data"): Code: var oTreeJSON = oTreeTable.getModel().getProperty("/root"); var oModel = sap.ui.getCore().getModel(); var batchChanges = []; for (var i = 0; i < oTreeAll.length; i++) { batchChanges.push(oModel.createBatchOperation("/sap/opu/odata/sap/MY_SERVICE/?$batch", "POST", oTreeAll[i])); } oModel.submitBatch(); My Request looks like this: Where should it arrive in SAP (which method)? What am I doing wrong, there is no error

Training broke with ResourceExausted error

阅读更多关于 Training broke with ResourceExausted error

I am new to tensorflow and Machine Learning. Recently I am working on a model. My model is like below, Character level Embedding Vector -> Embedding lookup -> LSTM1 Word level Embedding Vector->Embedding lookup -> LSTM2 [LSTM1+LSTM2] -> single layer MLP-> softmax layer [LSTM1+LSTM2] -> Single layer MLP-> WGAN discriminator Code of he rnn model while I'm working on this model I got the following error. I thought My batch is too big. Thus I tried to reduce the batch size from 20 to 10 but it doesn't work. ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape

Automated httr authentication with twitteR , provide response to interactive prompt in “batch” mode

阅读更多关于 Automated httr authentication with twitteR , provide response to interactive prompt in “batch” mode

I am using the R package twitteR to post items to Twitter. I put everything inside of a function and it works fine. However, I would like to run the function without being prompted for a response, and I haven't figured out how to do that. Any suggestions? Here are the bare bones of my function: doit <- function(<snip>) { <snip> # connect to Twitter setup_twitter_oauth(api_key, api_secret, access_token, access_token_secret) <snip> } When I run the function from the command line, I am prompted for an interactive response. [1] "Using direct authentication" Use a local file to cache OAuth access