large-files | 易学教程

Read and parse a Json File in C#

阅读更多关于 Read and parse a Json File in C#

问题 I have spent the best part of two days \"faffing\" about with code samples and etc., trying to read a very large JSON file into an array in c# so I can later split it up into a 2d array for processing. The problem I was having was I could not find any examples of people doing what I was trying to do. This meant I was just editing code a little an hoping for the best. I have managed to get something working that will: Read the file Miss out headers and only read values into array. Place a

Is there a memory efficient and fast way to load big json files in python?

阅读更多关于 Is there a memory efficient and fast way to load big json files in python?

问题 I have some json files with 500MB. If I use the \"trivial\" json.load to load its content all at once, it will consume a lot of memory. Is there a way to read partially the file? If it was a text, line delimited file, I would be able to iterate over the lines. I am looking for analogy to it. Any suggestions? Thanks 回答1: Short answer: no. Properly dividing a json file would take intimate knowledge of the json object graph to get right. However, if you have this knowledge, then you could

Parsing Huge XML Files in PHP

阅读更多关于 Parsing Huge XML Files in PHP

问题 I\'m trying to parse the DMOZ content/structures XML files into MySQL, but all existing scripts to do this are very old and don\'t work well. How can I go about opening a large (+1GB) XML file in PHP for parsing? 回答1: There are only two php APIs that are really suited for processing large files. The first is the old expat api, and the second is the newer XMLreader functions. These apis read continuous streams rather than loading the entire tree into memory (which is what simplexml and DOM

Reading large text files with streams in C#

阅读更多关于 Reading large text files with streams in C#

问题 I\'ve got the lovely task of working out how to handle large files being loaded into our application\'s script editor (it\'s like VBA for our internal product for quick macros). Most files are about 300-400 KB which is fine loading. But when they go beyond 100 MB the process has a hard time (as you\'d expect). What happens is that the file is read and shoved into a RichTextBox which is then navigated - don\'t worry too much about this part. The developer who wrote the initial code is simply

Processing large JSON files in PHP

阅读更多关于 Processing large JSON files in PHP

问题 I am trying to process somewhat large (possibly up to 200M) JSON files. The structure of the file is basically an array of objects. So something along the lines of: [ {\"property\":\"value\", \"property2\":\"value2\"}, {\"prop\":\"val\"}, ... {\"foo\":\"bar\"} ] Each object has arbitrary properties and does not necessary share them with other objects in the array (as in, having the same). I want to apply a processing on each object in the array and as the file is potentially huge, I cannot

Git with large files

阅读更多关于 Git with large files

问题 Situation I have two servers, Production and Development. On Production server, there are two applications and multiple (6) databases (MySQL) which I need to distribute to developers for testing. All source codes are stored in GitLab on Development server and developers are working only with this server and don\'t have access to production server. When we release an application, master logs into production and pulls new version from Git. The databases are large (over 500M each and counting)

Text editor to open big (giant, huge, large) text files [closed]

阅读更多关于 Text editor to open big (giant, huge, large) text files [closed]

问题 As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance. Closed 7 years ago . This post is a Community Wiki . Edit existing answers to improve this post. It is not currently accepting new answers. I mean 100+ MB

Managing large binary files with Git

阅读更多关于 Managing large binary files with Git

问题 I am looking for opinions of how to handle large binary files on which my source code (web application) is dependent. We are currently discussing several alternatives: Copy the binary files by hand. Pro: Not sure. Contra: I am strongly against this, as it increases the likelihood of errors when setting up a new site/migrating the old one. Builds up another hurdle to take. Manage them all with Git . Pro: Removes the possibility to \'forget\' to copy a important file Contra: Bloats the

Number of lines in a file in Java

阅读更多关于 Number of lines in a file in Java

问题 I use huge data files, sometimes I only need to know the number of lines in these files, usually I open them up and read them line by line until I reach the end of the file I was wondering if there is a smarter way to do that 回答1: This is the fastest version I have found so far, about 6 times faster than readLines. On a 150MB log file this takes 0.35 seconds, versus 2.40 seconds when using readLines(). Just for fun, linux' wc -l command takes 0.15 seconds. public static int countLinesOld

Parsing Huge XML Files in PHP

阅读更多关于 Parsing Huge XML Files in PHP

I'm trying to parse the DMOZ content/structures XML files into MySQL, but all existing scripts to do this are very old and don't work well. How can I go about opening a large (+1GB) XML file in PHP for parsing? There are only two php APIs that are really suited for processing large files. The first is the old expat api, and the second is the newer XMLreader functions. These apis read continuous streams rather than loading the entire tree into memory (which is what simplexml and DOM does). For an example, you might want to look at this partial parser of the DMOZ-catalog: <?php class