I want to learn about how nl80211 and cfg80211 works in detail. Function flow, how nl80211 interact with network tools like wpa_
To be able to control wireless drivers from userspace, some IPC communication processes between kernel and userspace are used.
ioctl with vendor dependent APIs was used.The Wireless Extension (WE) is a generic API allowing a driver to expose to the user space configuration and statistics specific to common Wireless LANs.
In 2006, John Linville creates mac80211 and Johannes Berg creates cfg80211 and nl80211. Together it is intended to replace wireless extensions.
+-------------+
| |
| Userspace |
| |
+-------------+
^
- - - | - - - -
| nl80211
v
+-------------+
| |
| cfg80211 |
| |
+-------------+
+-------------+
| |
| mac80211 |
| driver |
| |
+-------------+
An important point is that nl80211/cfg80211/mac80211 no longer use ioctl, they use netlink.
So, tools like iw, hostapd or the wpa_supplicant use some netlink libraries (like libnl or libnl-tiny) and the netlink interface public header which is of course nl80211.h.
There is not so much documentations, but I advise you to read the libnl documentation and then the iw source code (because iw use libnl).
I've created a basic code flow diagram for the wireless stack in linux,
all the way from wpa_supplicant > cfg80211 > mac80211 > ath9k_htc.
The code has been traced for linux kernel 5.4.31.
Here is the link.
A slightly more detailed picture of how nl80211 and cfg80211 work with other parts of the system (user space, kernel, and hardware).
nl80211 is the interface between user space software (iw, wpa_supplicant, etc.) and the kernel (cfg80211 and mac80211 kernel modules, and specific drivers). cfg80211_ops is a set of operations that Full-MAC drivers and mac80211 module register to cfg80211 module.ieee80211_ops is a set of operations that Soft-MAC drivers register to mac80211 module.See my reply to How to learn the structure of Linux wireless drivers (mac80211)?
In wpa_supplicant, you can follow the code in src/drivers/driver_nl80211.c. This is a wpa_supplicant driver (not a kernel driver but an abstraction used in wpa_supplicant code) which uses libnl to communicate with the kernel cfg80211 module. When wpa_supplicant issues a scan for example then wpa_driver_nl80211_scan gets called. It builds the netlink message with a command called NL80211_CMD_TRIGGER_SCAN and with all the parameters required for the scan.