How to calculate HttpWebRequest spent outbound and inbound internet traffic

Question

I have the below function to fetch a page. My question is i want to calculate how much internet connection is spent

Both inbound (download) and outbound traffic (sent)

How can i do that ? Thank you

My function

 public static string func_fetch_Page(string srUrl, int irTimeOut = 60,
    string srRequestUserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0",
    string srProxy = null)
    {
        string srBody = "";
        string srResult = "";
        try
        {
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(srUrl);

            request.Timeout = irTimeOut * 1000;
            request.UserAgent = srRequestUserAgent;
            request.KeepAlive = true;
            request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";

            WebHeaderCollection myWebHeaderCollection = request.Headers;
            myWebHeaderCollection.Add("Accept-Language", "en-gb,en;q=0.5");
            myWebHeaderCollection.Add("Accept-Encoding", "gzip, deflate");

            request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;

            using (WebResponse response = request.GetResponse())
            {
                using (Stream strumien = response.GetResponseStream())
                {
                    using (StreamReader sr = new StreamReader(strumien))
                    {
                        srBody = sr.ReadToEnd();
                        srResult = "success";
                    }
                }
            }
        }
        catch ()
        {

        }

        return srBody;
    }

C# .net 4.5 WPF application

@Simon Mourier how do i calculate spent traffic

public static long long_GlobalDownload_KByte = 0;
public static long long_GlobalSent_KByte = 0;

Time  _timer_fetch_download_upload = new Timer(getDownload_Upload_Values, null, 0, 100 * 1000);

public static void getDownload_Upload_Values(Object state)
{
    using (Process p = Process.GetCurrentProcess())
    {
        foreach (var cnx in TcpConnection.GetAll().Where(c => c.ProcessId == p.Id))
        {
            Interlocked.Add(ref long_GlobalDownload_KByte, Convert.ToInt64(cnx.DataBytesIn));
            Interlocked.Add(ref long_GlobalSent_KByte, Convert.ToInt64(cnx.DataBytesOut));
        }
    }
}

Answer 1

You can create a proxy using FiddlerCore (just a single dll to reference) and set a WebProxy to your HttpWebRequest to count the sent and received bytes

public class WebConnectionStats
{
    static int _Read = 0;
    static int _Written = 0;

    public static void Init(bool registerAsSystemProxy = false)
    {
        Fiddler.FiddlerApplication.OnReadRequestBuffer += (s, e) => Interlocked.Add(ref _Written, e.iCountOfBytes);
        Fiddler.FiddlerApplication.OnReadResponseBuffer += (s, e) => Interlocked.Add(ref _Read, e.iCountOfBytes);
        Fiddler.FiddlerApplication.Startup(8088, registerAsSystemProxy, true);
    }

    public static int Read
    {
        get { return _Read; }
    }

    public static int Written
    {
        get { return _Written; }
    }
}

---

WebConnectionStats.Init(); //call this only once

var client = HttpWebRequest.Create("http://stackoverflow.com") as HttpWebRequest;
client.Proxy = new WebProxy("127.0.0.1", 8088);
var resp = client.GetResponse();
var html = new StreamReader(resp.GetResponseStream()).ReadToEnd();

Console.WriteLine("Read: {0}   Write: {1}", WebConnectionStats.Read, 
                                            WebConnectionStats.Written);

PS1: This counts will not include the Tcp Headers' length

PS2: You can get more info about Fiddler core here

Answer 2

One Windows API can give you this information: GetPerTcpConnectionEStats for IPV4 connections and the associated IPV6 GetPerTcp6ConnectionEStats function. Note you need to use SetPerTcpConnectionEStats before being able to get any stat, and that usually needs admin rights...

To get the list of all connections, you can use the GetExtendedTcpTable function. It can also give you the connection's process id which is very useful.

These are native APIs not that easy to use, but I have created a TcpConnection class that wraps all this. It's available here in a small WPF app called IPStats: https://github.com/smourier/IPStats

So, the difficulty here is to link your .NET HttpWebRequest to a TcpConnection from the list of connections. You can't get any stat before the connection exists, but once you have the connection created, you can get the corresponding one with a code like this:

    static IEnumerable<TcpConnection> GetProcessConnection(IPEndPoint ep)
    {
        var p = Process.GetCurrentProcess();
        return TcpConnection.GetAll().Where(c => ep.Equals(c.RemoteEndPoint) && c.ProcessId == p.Id);
    }

    HttpWebRequest req = ...

    // this is how you can get the enpoint, or you can also built it by yourself manually
    IPEndPoint remoteEndPoint;
    req.ServicePoint.BindIPEndPointDelegate += (sp, rp, rc) =>
        {
            remoteEndPoint = rp;
            return null;
        };
    // TODO: here, you need to connect, so the connection exists
    var cnx = GetProcessConnection(remoteEndPoint).FirstOrDefault();

    // access denied here means you don't have sufficient rights
    cnx.DataStatsEnabled = true;

    // TODO: here, you need to do another request, so the values are incremented
    // now, you should get non-zero values here
    // note TcpConnection also has int/out bandwidth usage, and in/out packet usage.
    Console.WriteLine("DataBytesIn:" + cnx.DataBytesIn);
    Console.WriteLine("DataBytesOut:" + cnx.DataBytesOut);

    // if you need all connections in the current process, just do this
    ulong totalBytesIn = 0;
    ulong totalBytesOut = 0;
    Process p = Process.GetCurrentProcess();
    foreach (var cnx in TcpConnection.GetAll().Where(c => c.ProcessId == p.Id))
    {
        totalBytesIn += cnx.DataBytesIn;
        totalBytesOut += cnx.DataBytesOut;
    }

There are 3 drawbacks:

• you need to be admin to enable data stats;
• you can't get the stats for the very first request. This may not be a problem depending on your context;
• the matching between the HttpWebRequest's connection and the the TCP connection can be trickier to determine because you can have more than one connection to a remote endpoint, even in the same process. The only way to distinguish is to determine the local endpoint (especially the port) and extend the GetProcessConnection with this local endpoint. Unfortunately, there is no easy way to do this. Here is a related answer about this: How do I get the local port number of an HttpWebRequest?

Update: if you want to continuously monitor all connections for a given process, I've written a ProcessTcpConnections utility class that remembers all connections and sums their usage. It would be used like this (in a console app sample):

class Program
{
    static void Main(string[] args)
    {
        ProcessTcpConnections p = new ProcessTcpConnections(Process.GetCurrentProcess().Id);
        Timer timer = new Timer(UpdateStats, p, 0, 100);

        do
        {
            // let's activate the network so we measure something...
            using (WebClient client = new WebClient())
            {
                client.DownloadString("http://www.example.com");
            }
            Console.ReadKey(true); // press any key to download again
        }
        while (true);
    }

    private static void UpdateStats(object state)
    {
        ProcessTcpConnections p = (ProcessTcpConnections)state;
        p.Update();
        Console.WriteLine("DataBytesIn:" + p.DataBytesIn + " DataBytesOut:" + p.DataBytesOut);
    }
}

public class ProcessTcpConnections : TcpConnectionGroup
{
    public ProcessTcpConnections(int processId)
        : base(c => c.ProcessId == processId)
    {
        ProcessId = processId;
    }

    public int ProcessId { get; private set; }
}

public class TcpConnectionGroup
{
    private List<TcpConnectionStats> _states = new List<TcpConnectionStats>();
    private Func<TcpConnection, bool> _groupFunc;

    public TcpConnectionGroup(Func<TcpConnection, bool> groupFunc)
    {
        if (groupFunc == null)
            throw new ArgumentNullException("groupFunc");

        _groupFunc = groupFunc;
    }

    public void Update()
    {
        foreach (var conn in TcpConnection.GetAll().Where(_groupFunc))
        {
            if (!conn.DataStatsEnabled)
            {
                conn.DataStatsEnabled = true;
            }

            TcpConnectionStats existing = _states.Find(s => s.Equals(conn));
            if (existing == null)
            {
                existing = new TcpConnectionStats();
                _states.Add(existing);
            }
            existing.DataBytesIn = conn.DataBytesIn;
            existing.DataBytesOut = conn.DataBytesOut;
            existing.LocalEndPoint = conn.LocalEndPoint;
            existing.RemoteEndPoint = conn.RemoteEndPoint;
            existing.State = conn.State;
            existing.LastUpdateTime = DateTime.Now;
        }
    }

    public ulong DataBytesIn
    {
        get
        {
            ulong count = 0; foreach (var state in _states) count += state.DataBytesIn; return count;
        }
    }

    public ulong DataBytesOut
    {
        get
        {
            ulong count = 0; foreach (var state in _states) count += state.DataBytesOut; return count;
        }
    }

    private class TcpConnectionStats
    {
        public ulong DataBytesIn { get; set; }
        public ulong DataBytesOut { get; set; }
        public IPEndPoint LocalEndPoint { get; set; }
        public IPEndPoint RemoteEndPoint { get; set; }
        public TcpState State { get; set; }
        public DateTime LastUpdateTime { get;  set; }

        public bool Equals(TcpConnection connection)
        {
            return LocalEndPoint.Equals(connection.LocalEndPoint) && RemoteEndPoint.Equals(connection.RemoteEndPoint);
        }
    }
}

Answer 3

It's hard to know exactly how much network traffic is occurring because of some unpredictable Ethernet traffic and that sort of thing, but essentially the gist of any HTTP request is:

method request-uri version
* (header : value)
CRLF
body

Given that, you can calculate out how long the request string will be:

HttpWebRequest req = ...;
StringBuilder requestText = new StringBuilder();

requestText.AppendFormat("{0} {1} HTTP/{2}.{3}", req.Method, req.RequestUri, req.ProtocolVersion.Major, req.ProtocolVersion.Minor);

requestText.AppendLine();

foreach (var header in req.Headers)
{
    requestText.AppendFormat("{0}: {1}", v, webReq.Headers[v]);
    requestText.AppendLine();
}

requestText.AppendLine();

// somehow add on the contents of the request stream, or just add that length later. I won't put that in this code because of stream positioning and all that

return System.Text.Encoding.UTF8.GetByteCount(requestText.ToString());

Then it's pretty similar for the response side. The format for an HTTP response is:

version status-code status-description
* (header : value)
CRLF
body

So,

HttpWebResponse resp = ...;
StringBuilder responseText = new StringBuilder();

responseText .AppendFormat("HTTP/{0}.{1} {2} {3}", resp.ProtocolVersion.Major, resp.ProtocolVersion.Minor, (int)resp.StatusCode, resp.StatusDescription);
responseText .AppendLine();

foreach (var header in resp.Headers)
{
    responseText .AppendFormat("{0}: {1}", v, resp.Headers[v]);
    responseText .AppendLine();
}

responseText.AppendLine();

Here, there's a bit of a decision to be made. You need to get the length of the response body. There might be more options, but what I'm thinking now are that you could:

1. Write a Stream wrapper that grabs the number of bytes copied. That way you don't have to worry about chunked transfers, and it could be kind of fun to write.
2. Use the content-length header, although then you won't get it if there isn't that header, as is the case for transfer-encoding: chunked.
3. Read the stream using whatever method you want, then add the length of that in.
4. Copy the stream to a MemoryStream, then pass that off as the new response stream whilst grabbing the length on the way.

That all said, you have the extra frustration of worrying about content compression. I see you use it in yours, in fact. Given simplicity, I'm going to assume GZip and go with the fourth option. You might want to expand on what I write here to make it a little more comprehensive.

// webReq.AutomaticDecompression = DecompressionMethods.None; is required for this, since we're handling that decompression ourselves.

using (var respStream = resp.GetResponseStream())
using (var memStream = new MemoryStream())
{
    respStream.CopyTo(memStream);

    using (var gzip = new System.IO.Compression.GZipStream(respStream, System.IO.Compression.CompressionMode.Decompress))
    using (var reader = new StreamReader(gzip))
    {
        var content = reader.ReadToEnd();

        // you may or may not actually care about this, depending on whether this is just light testing or if you'll actually have these metrics in production
    }

    return System.Text.Encoding.UTF8.GetByteCount(responseText.ToString()) + memStream.Length;
}

Now, a few caveats that I'm seeing as I look now. Others might notice more, and I'll add them in if people comment.

• As I mentioned way back at the start of this, there can be more network traffic from one request than this will tell you.
• Depending on the client and server, you could find that actual headers could be listed with a different number of spaces. I don't believe that's beyond the HTTP spec, and even if it is people will do it. So there's a chance, for random example, that you'll see one server set content-type: text/html and another set content-type : text/html. There's at least (well, exactly, since we're using UTF-8) one byte difference. Again, little, but that's a possible discrepancy.

This is just an estimate. It's a good one, but it's just an estimate.

If you're looking for extra accuracy at the cost of some simplicity, you could also write yourself a proxy. The idea of an HTTP proxy is that it simply gets the verbatim request, so it's "super easy" to then get the length. The biggest issue, of course, is that you then have to write a fully functioning proxy that, in all likelihood, parses and re-requests your incoming requests, and parses and relays all their respective responses. Certainly doable, but it's non-trivial. Particularly if you have to worry about SSL.

At the most basic level, of course, you could write, in essence, an alternative to a program like WireShark. I wouldn't know how to do that, and I wouldn't suggest you bother unless you really need it. That still wouldn't give you a perfect idea, either. Just closer.

And now, that all said, and boy, that was quite a bit said, if you're doing this for the purposes of profiling, there's a good chance the tools built into Visual Studio will let you do it. Admittedly I'm far from knowledgeable on them--I've never even opened them--but I believe there are network-traffic profilers available. Of course, I doubt they work on every platform like this would. But that would certainly be an easier approach.

Also, if anyone happens to notice any typos in here, I did copy-paste a couple times and I think I got them all, but feel free to let me know or just fix them.

Answer 4

it bit ugly but you can use network tracing and then parse the log. see http://msdn.microsoft.com/en-us/library/ty48b824(v=vs.110).aspx for enable network tracing

i add basic code ( i know that i have bug in parsing the log but the idea should be clear )

NetworkListner.cs

using System;
using System.Diagnostics;
using System.Globalization;
using System.Text.RegularExpressions;

namespace NetworkTracing
{
    /// <summary>
    /// Description of NetworkListner.
    /// </summary>
    public class NetworkListner : TraceListener
    {
        public static int BytesSent { get; private set;} 
        public static int BytesReceived { get; private set;}
        private bool _inSend = false;
        private bool _inReceived = false;
        private string _lastMessage = "";
        private Regex _lengthRegex = new Regex(@"(\[\d*\]) ([\dA-F]*)");
        public NetworkListner()
        {
            BytesSent = 0;
            BytesReceived = 0;          
        }

        private int ExtractNumOfBytes(string message){
            string lengthUntilThisLineStr = null;
            try {           
                var match = _lengthRegex.Match(message);                
                lengthUntilThisLineStr = match.Groups[2].Value;
            } catch (ArgumentException ex) {
                // Syntax error in the regular expression
            }
            if (String.IsNullOrEmpty(lengthUntilThisLineStr)) {
                return 0;
            }
            var lengthUntilThisLine = int.Parse(lengthUntilThisLineStr,NumberStyles.HexNumber);
            return lengthUntilThisLine;
        }

        public override void Write(string message) {
            if (message.Equals("System.Net.Sockets Verbose: 0 : ")) {
                return;
            }
            if (message.Contains("Exiting Socket#")) {
                int bytes = ExtractNumOfBytes(_lastMessage);
                if (_inSend) {
                    _inSend = false;
                    BytesSent += bytes;
                }else if (_inReceived) {
                    _inReceived = false;
                    BytesReceived += bytes;
                }   
            }           
            else if (message.Contains("Data from Socket")){
                if (message.Contains("Send")) {
                    _inSend = true;
                }
                else if (message.Contains("Receive")) {
                    _inReceived = true;
                }
            }
            _lastMessage = message;
        }

        public override void WriteLine(string message) {
            Write(message + Environment.NewLine);
        }
    }
}

Program.cs

using System;
using System.Configuration;
using System.Diagnostics;
using System.IO;
using System.Net;

namespace NetworkTracing
{
    class Program
    {
        public static void Main(string[] args)
        {           
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.google.com");
            request.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0";
            using (WebResponse response = request.GetResponse())
            {
                using (Stream strumien = response.GetResponseStream())
                {
                    using (StreamReader sr = new StreamReader(strumien))
                    {
                        var res = sr.ReadToEnd();
                        Console.WriteLine("Send -> {0}",NetworkListner.BytesSent);
                        Console.WriteLine("Recieve -> {0}",NetworkListner.BytesReceived);
                        Console.ReadLine();
                    }
                }
            }
        }
    }

App.config

<configuration>
<system.diagnostics>
    <sources>
      <source name="System.Net.Sockets" tracemode="includehex" maxdatasize="1024">
        <listeners>
          <add name="network"/>
        </listeners>
      </source>
    </sources>
     <switches>
      <add name="System.Net.Sockets" value="Verbose"/>      
    </switches>
    <sharedListeners>
      <add name="network"
        type="NetworkTracing.NetworkListner, NetworkTracing"
      />
    </sharedListeners>
    <trace autoflush="true"/>
  </system.diagnostics>
</configuration>

you also need to compile your project with trace flag ( this flag is default in debug ) by :

1. With a project selected in Solution Explorer, on the Project menu
2. click Properties.
3. Click the Compile tab.
4. Click the Advanced Compile Options button to open the Advanced Compiler Settings dialog box.
5. Select the Define TRACE constant check box, and then click OK.

Answer 5

If your client app is communicating with your known IIS server(s), then I would try getting this data from the IIS logs. Check out the cs-bytes (client to server bytes aka request bytes) and sc-bytes (server to client bytes aka response). http://technet.microsoft.com/en-us/library/cc754702(v=ws.10).aspx

Answer 6

I think you can use .NET CLR Networking performance counter as it can give you sent and received bytes per AppDomain

http://msdn.microsoft.com/en-us/library/70xadeyt%28v=vs.110%29.aspx

I wrote a Helper class to verify the accuracy and the results are similar to Windows ResourceMonitor, so I believe the accuracy should be acceptable.

How to Use:

here is the Helper class:

using System.Diagnostics;
using System.Linq;

public class NetworkMonitor
{
    private PerformanceCounter _bytesSent;
    private PerformanceCounter _bytesReceived;
    private readonly int _processId;
    private bool _initialized;

    public NetworkMonitor(int processID)
    {
        _processId = processID;
        Initialize();
    }

    public NetworkMonitor()
        : this(Process.GetCurrentProcess().Id)
    {

    }
    private void Initialize()
    {
        if (_initialized)
            return;

        var category = new PerformanceCounterCategory(".NET CLR Networking 4.0.0.0");
        var instanceNames = category.GetInstanceNames().Where(i => i.Contains(string.Format("p{0}", _processId)));
        if (!instanceNames.Any()) return;

        _bytesSent = new PerformanceCounter
        {
            CategoryName = ".NET CLR Networking 4.0.0.0",
            CounterName = "Bytes Sent",
            InstanceName = instanceNames.First(),
            ReadOnly = true
        };

        _bytesReceived = new PerformanceCounter
        {
            CategoryName = ".NET CLR Networking 4.0.0.0",
            CounterName = "Bytes Received",
            InstanceName = instanceNames.First(),
            ReadOnly = true
        };

        _initialized = true;
    }

    public float GetSentBytes()
    {
        Initialize(); //in Net4.0 performance counter will get activated after first request
        return _initialized ? _bytesSent.RawValue : 0;
    }
    enter code here
    public float GetReceivedBytes()
    {
        Initialize(); //in Net4.0 performance counter will get activated after first request
        return _initialized ? _bytesReceived.RawValue : 0;
    }
} 

you should add this part to your app.config

  <system.net>
    <settings>
      <performanceCounters enabled="true" />
    </settings>
  </system.net>

and here is the sample I used to verify accuracy based on your own method:

   private static void Main(string[] args)
        {
            var netMonitor = new NetworkMonitor();

            var received = netMonitor.GetReceivedBytes();
            var sent = netMonitor.GetSentBytes();

            Console.WriteLine("received:{0}, sent:{1}", received, sent);
            func_fetch_Page("http://www.google.com");

            received = netMonitor.GetReceivedBytes();
            sent = netMonitor.GetSentBytes();

            Console.WriteLine("received:{0}, sent:{1}", received, sent);
            Console.ReadKey();
        }