Fastest way to add new node to end of an xml?

Question

I have a large xml file (approx. 10 MB) in following simple structure:

   .......
   .......

        

Answer 1

The quickest method is likely to be reading in the file using an XmlReader, and simply replicating each read node to a new stream using XmlWriter When you get to the point at which you encounter the closing </Errors> tag, then you just need to output your additional <Error> element before coninuing the 'read and duplicate' cycle. This way is inevitably going to be harder than than reading the entire document into the DOM (XmlDocument class), but for large XML files, much quicker. Admittedly, using StreamReader/StreamWriter would be somewhat faster still, but pretty horrible to work with in code.

Answer 2

Here's how to do it in C, .NET should be similar.

The game is to simple jump to the end of the file, skip back over the tag, append the new error line, and write a new tag.

#include <stdio.h>
#include <string.h>
#include <errno.h>

int main(int argc, char** argv) {
        FILE *f;

        // Open the file
        f = fopen("log.xml", "r+");

        // Small buffer to determine length of \n (1 on Unix, 2 on PC)
        // You could always simply hard code this if you don't plan on 
        // porting to Unix.
        char nlbuf[10];
        sprintf(nlbuf, "\n");

        // How long is our end tag?
        long offset = strlen("</Errors>");

        // Add in an \n char.
        offset += strlen(nlbuf);

        // Seek to the END OF FILE, and then GO BACK the end tag and newline
        // so we use a NEGATIVE offset.
        fseek(f, offset * -1, SEEK_END);

        // Print out your new error line
        fprintf(f, "<Error>New error line</Error>\n");

        // Print out new ending tag.
        fprintf(f, "</Errors>\n");

        // Close and you're done
        fclose(f);
}

Answer 3

Try this out:

        var doc = new XmlDocument();
        doc.LoadXml("<Errors><error>This is my first error</error></Errors>");

        XmlNode root = doc.DocumentElement;

        //Create a new node.
        XmlElement elem = doc.CreateElement("error");
        elem.InnerText = "This is my error";

        //Add the node to the document.
        if (root != null) root.AppendChild(elem);

        doc.Save(Console.Out);
        Console.ReadLine();

Answer 4

You need to use the XML inclusion technique.

Your error.xml (doesn't change, just a stub. Used by XML parsers to read):

<?xml version="1.0"?>
<!DOCTYPE logfile [
<!ENTITY logrows    
 SYSTEM "errorrows.txt">
]>
<Errors>
&logrows;
</Errors>

Your errorrows.txt file (changes, the xml parser doesn't understand it):

<Error>....</Error>
<Error>....</Error>
<Error>....</Error>

Then, to add an entry to errorrows.txt:

using (StreamWriter sw = File.AppendText("logerrors.txt"))
{
    XmlTextWriter xtw = new XmlTextWriter(sw);

    xtw.WriteStartElement("Error");
    // ... write error messge here
    xtw.Close();
}

Or you can even use .NET 3.5 XElement, and append the text to the StreamWriter:

using (StreamWriter sw = File.AppendText("logerrors.txt"))
{
    XElement element = new XElement("Error");
    // ... write error messge here
    sw.WriteLine(element.ToString());
}

See also Microsoft's article Efficient Techniques for Modifying Large XML Files

Answer 5

I would use XmlDocument or XDocument to Load your file and then manipulate it accordingly.

I would then look at the possibility of caching this XmlDocument in memory so that you can access the file quickly.

What do you need the speed for? Do you have a performance bottleneck already or are you expecting one?

Answer 6

I attempted to use code other answers had suggested but ran into an issue where sometimes calling .length on my strings was not the same as the number of bytes for the string so I was inconsistently losing characters. I modified it to get the byte count instead.

var endTag = "</Errors>";
var nodeText = GetNodeText();
using (FileStream file = File.Open("my.log", FileMode.Open, FileAccess.ReadWrite))
{
    file.BaseStream.Seek(-(Encoding.UTF8.GetByteCount(endTag)), SeekOrigin.End);
    fileStream.Write(Encoding.UTF8.GetBytes(nodeText), 0, Encoding.UTF8.GetByteCount(nodeText));
    fileStream.Write(Encoding.UTF8.GetBytes(endTag), 0, Encoding.UTF8.GetByteCount(endTag));
}