Answer
Windows 10 underlines Active Icons on the Task Bar. This is default behavior, and you would need a custom third party theme to remove the underline (basic research about this feature).
Accordingly, Windows 10 and Windows 11 are the same in this regard assuming native setup.
Follow up: Build 22000.120 is out tonight (Aug 5) and all is still the same between Windows 10 and Windows 11 with respect to underlining active task bar icons.
Attribution
Source : Link , Question Author : Prab-mat , Answer Author : John