text-files

I have a recurring task of splitting a set of large (about 1-2 GiB each) gzipped Apache logfiles into several parts (say chunks of 500K lines). The final files should be gzipped again to limit the disk usage. On Linux I would normally do: zcat biglogfile.gz | split -l500000 The resulting files files will be named xaa, xab, xac, etc So I do: gzip x* The effect of this method is that as an intermediate result these huge files are temporarily stored on disk. Is there a way to avoid this intermediate disk usage? Can I (in a way similar to what xargs does) have split pipe the output through a

I have to read and modify some JSON files. The file encoding must be UTF8 without BOM or the JSON file will be not accepted. I tried the following Code: const Utf8Bom = #$EF#$BB#$BF; Utf16BomLE = #$FF#$FE; // little endian // Utf16BomBE = #$FE#$FF; // big endian // Utf16Bom = Utf16BomBE; CP_UTF16 = 1200; CP_UTF8 = 65001; function WideStringToString (const wStr: string; codePage: Word): string; var len: Integer; begin len := WideCharToMultiByte (codePage, 0, wStr, -1, '', 0, 0, 0); if len > 0 then begin SetLength (Result, len-1); WideCharToMultiByte (codePage, 0, wStr, -1, Result, Length

I'm trying to convert the data from a large number of PDF files to data frames in R. I've been converting the PDF files to .txt files using read.fwf(), but the issue is that the widths of all .txt files are not the same. Is there a way to determine the widths of the columns, or is there a way to use a function other than read.fwf()? I have a large number of files to convert and they all have different formats to begin with, so finding the specific column widths for each file is getting very tedious. Is there a more efficient way to convert data from PDF files to data frames in R? Here's one

How to write a text header in text file? for example in the example below, how to write the header code salay month just once? Code Salary Month 12 1000 12 14 1020 11 11 1212 9 The code: fid = fopen('analysis1.txt','wt'); for i=1:10 array = []; % empty the array .... array = [code salary month]; format short g; fprintf(fid,'%g\t %g\t %g\n',array); % write to file end fclose(fid); Is there any reason for not using simple solution like following? ... fid = fopen('analysis1.txt','wt'); fprintf(fid, '%s\t %s\t %s\n', 'Code','Salary','Month'); for i=1:10 array = []; % empty the array ... Just to

I am trying to read a .txt file into a multi-line text box with the following code. I have gotten the file dialog button to work perfectly, but I am not sure how to get the actual text from the fiile into the textbox. Here is my code. Can you help? private void button_LoadSource_Click(object sender, EventArgs e) { Stream myStream = null; OpenFileDialog openFileDialog1 = new OpenFileDialog(); openFileDialog1.InitialDirectory = "c:\\"; openFileDialog1.Filter = "txt files (*.txt)|*.txt|All files (*.*)|*.*"; openFileDialog1.FilterIndex = 2; openFileDialog1.RestoreDirectory = true; if

I have a file full of ascii data. How would I append a string to the first line of the file? I cannot find that sort of functionality using fopen (it seems to only append at the end and nothing else.) gnovice Option 1: I would suggest calling some system commands from within MATLAB . One possibility on Windows is to write your new line of text to its own file and then use the DOS for command to concatenate the two files . Here's what the call would look like in MATLAB: !for %f in ("file1.txt", "file2.txt") do type "%f" >> "new.txt" I used the ! (bang) operator to invoke the command from within

The text file contains two columns- index number(5 spaces) and characters(30 spaces). It is arranged in lexicographic order. I want to perform binary search to search for the keyword. Here's an interesting way to do it with Python's built-in bisect module. import bisect import os class Query(object): def __init__(self, query, index=5): self.query = query self.index = index def __lt__(self, comparable): return self.query < comparable[self.index:] class FileSearcher(object): def __init__(self, file_pointer, record_size=35): self.file_pointer = file_pointer self.file_pointer.seek(0, os.SEEK_END)

I have to write a textfile in VB6. I need to do it in appending and utf-8 encoded. I tried two solutions, one with "TextStream" and another one with "ADODB.Stream". The first one: Set fsoFile = fso.OpenTextFile(FileIn(fi), ForAppending, True) fsoFile.WriteLine "<tag>kkkjòòkkkkjlòlk</tag>" fsoFile.Close Works good in appending but how can I write it utf-8 encoded? The second one: Dim ST As ADODB.Stream Set ST = New ADODB.Stream ST.Mode = adModeReadWrite ST.Type = adTypeText ST.Charset = "UTF-8" ST.Open ST.LoadFromFile FileIn(fi) ST.Position = ST.Size ST.WriteText "<tag>kkkjòòkkkkjlòlk</tag>" ST

sample user id User Name U456 Mathew U457 Leon U458 Cris U459 Yancy U460 Jane and so on up to 500k. I need to read this text file and insert to MySQL in two columns say User ID and User Name. How do I do this in PHP? LOAD DATA INFILE Example: NOTE: if you run this from Windows you need to escape the forward slashes in the file path. EXAMPLE: C:\\path\to\file.txt Looks like: C:\\\\path\\to\\file.txt Here is the query: LOAD DATA INFILE '/path/to/sample.txt' INTO TABLE `database_name`.`table_name` FIELDS TERMINATED BY ',' LINES TERMINATED BY '\r\n' IGNORE 1 LINES ( user_id, user_name ) Delete the

I have many HTML files from which I need to extract text. If it's all on one line, I can do that quite easily but if the tag wraps around or is on multiple lines I can't figure how to do this. Here's what I mean: <section id="MySection"> Some text here another line here <br> last line of text. </section> I'm not concerned about the <br> text, unless it will help wrap the text around. The area that I want always begins with "MySection" and then is ended with </section> . What I'd like to end up with is something like this: Some text here another line here last line of text. I'd prefer something