How to consume real-time ETW events from the Microsoft-Windows-NDIS-PacketCapture provider?

Question

The larger question is how to consume real-time ETW network stack events in general but I'm particularly interested in the Microsoft-Windows-NDIS-PacketCapture provider. All other network stack providers partially work but the NDIS-PacketCapture (NDIS-PC) doesn't work at all so this is probably the simplest question I can ask here.

I'm using the following code as the base and modified very little for it to work in real time: http://msdn.microsoft.com/en-us/library/windows/desktop/ee441325(v=vs.85).aspx

Changes I've made are:

1. Call StartTrace to start the NDIS-PC session before doing anything else. In the property struct EVENT_TRACE_PROPERTIES, setting LogFileMode = EVENT_TRACE_REAL_TIME_MODE, LogFileNameOffset = 0, and Wnode.Guid = something random GUID I made up.

2. Calling status = EnableTraceEx2(hSession, &Current_Guid, EVENT_CONTROL_CODE_ENABLE_PROVIDER, TRACE_LEVEL_VERBOSE, 0, 0, 0, NULL); where hSession is the session started using StartTrace and and Current_Guid is

   {0x2ED6006E,0x4729,0x4609,{0xB4,0x23,0x3E,0xE7,0xBC,0xD6,0x78,0xEF}};

3. Then calling OpenTrace with LoggerName = some wide string, LogFileName = NULL, and LogFileMode = EVENT_TRACE_REAL_TIME_MODE;

4. Finally call ProcessTrace on the trace handle that was just opened above.

5. Again, leaving everything else the same as provided in the MSDN example

Using the identical code with a single change of the provider to anything else such as Microsoft-Windows-Winsock-AFD or Microsoft-Windows-TCPIP gets me calls into the record callback I'd defined (However, I'm still not able to retrieve the properties but I won't delve further yet to keep this prob as simple as I can). When I use NDIS-PC, I get 0 callbacks. I've tried flushing manually using ControlTrace without any success. I've also tried defining "EventCallback" instead of the "EventRecordCallback" with no success.

I've looked at ALL data structures involved in this process and compared between each provider and they all looked correct and the same. I've looked at all return values from functions and returned data structures and they also look the same between the providers I've tried.

I've looked at the session properties by calling "logman "My Trace Session 04" -ets" and it looks identical for NDIS-PC and TCPIP:

C:\windows\system32>logman "My Trace Session 04" -ets

Name: My Trace Session 04 Status:
Running Root Path: %systemdrive%\PerfLogs\Admin Segment:
Off Schedules: On

Name: My Trace Session 04\My Trace Session 04 Type:
Trace Append: Off Circular: Off Overwrite:
Off Buffer Size: 64 Buffers Lost: 0 Buffers Written: 0 Buffer Flush Timer: 1 Clock Type: Performance File Mode: Real-time

Provider: Name: Microsoft-Windows-NDIS-PacketCapture Provider Guid: {2ED6006E-4729-4609-B423-3EE7BCD678EF} Level:
5 (win:Verbose) KeywordsAll: 0x0 KeywordsAny:
0xffffffffffffffff (Ethernet802.3,WirelessWAN,Tunnel,Nativ e802.11,PacketStart,PacketEnd,ut:SendPath,ut:ReceivePath,ut:L3ConnectPath,ut:L2C onnectPath,ut:ClosePath,ut:Authentication,ut:Configuration,ut:Global,ut:Dropped, ut:PiiPresent,ut:Packet,ut:Address,ut:StdTemplateHint,ut:StateTransition,win:Res ponseTime,Microsoft-Windows-NDIS-PacketCapture/Diagnostic,0x2,0x4,0x8,0x10,0x20, 0x40,0x80,0x100,0x400,0x800,0x1000,0x2000,0x4000,0x20000,0x40000,0x80000,0x10000 0,0x200000,0x400000,0x800000,0x1000000,0x2000000,0x4000000,0x8000000,0x10000000, 0x20000000,0x400000000000,0x800000000000,0x2000000000000,0x4000000000000,0x80000 00000000,0x10000000000000,0x20000000000000,0x40000000000000,0x80000000000000,0x1 00000000000000,0x200000000000000,0x400000000000000,0x800000000000000,0x100000000 0000000,0x2000000000000000,0x4000000000000000) Properties: 0 Filter Type: 0

The command completed successfully.

I've also tried starting the sessions manually using logman and just opening it in code to process but that didn't work for me either. I've also tried to just write to an ETL file and that doesn't work either. There are many more things I've tried but nothing is working.

I've devoured everything on the Internets that had to do with real-time ETW consumption (MSDN, Google search, Stackoverflow, etc) and I haven't seen a single complete example of real-time ETW event consumptions. All examples show event consumption from an ETL file or exporting recorded events to an ETL file then just says to make a few parameter changes to get real-time consumption working. I believe the code changes I summarized above reflect those changes.

I'm on Win7 Ultimate using VS2010 SP1 creating 32bit console app. I've also tried creating 64bit app with no improvement.

Follwing two posts are relevant but didn't make any difference for me when I tried/enforced. In real-time mode, the code copies the session name to the end of the property struct and log file offset needs to be 0 anyway. I don't think I have any alignment issues as all other providers work just fine:

Windows ETW: Kernel consumer receives no EventCallback or BufferCallback events Windows ETW: StartTrace failing with error 87 (ERROR_INVALID_PARAMETER)

I feel like I'm missing something small and trivial and this should just work. I'd appreciate any help.

Answer 1

If you look into what the "netsh trace" commands do internally you'll see they attach an NDIS lightweight filter driver to the various network interfaces. Only with this filter attached and activated will you get events from this provider. The details of this facility are undocumented and subject to change. All the logic of the netsh trace commands is implemented in nettrace.dll which you can reverse engineer with the assistance of Microsoft's public symbols. Specifically, the CInboxCapture class has code which determines whether the driver has been started, binds it to appropriate network interfaces and starts it. If you start the capturing filter driver the way nettrace.dll does, you'll get your packet capture events.

Good luck.

Answer 2

For what it's worth, I did find one example of a real-time ETW consumer (the example is for IIS web server): http://blogs.iis.net/eokim/archive/2009/05/15/consume-iis-etw-tracing.aspx

Answer 3

Here's a commented c++ example application that demonstrates simultaneous real-time ETW sessions for packet capture and kernel events.

https://github.com/packetzero/etwrealtime

Answer 4

Instead of running netsh trace start etc. you could try this:

net start ndiscap

When running netsh trace etc. it will do this for you, and I think that's the missing part here, that the lightweight filter driver which is injected into ndis to capture packets (i.e. the etw provider) is not running and emitting events.

When you are finished you can stop it using:

net stop ndiscap