Exploiting Windows’ vulnerabilities with Hyper-V: A Hacker’s swiss army knife
Disclaimer⌗
I constructed this post entirely for educational purposes only, and this blog post is limited in depth with respect to Hyper-V internals.
Introduction⌗
Hyper-V is Microsoft’s virtualization technology for Windows operating systems, providing a virtualization layer that allows multiple virtual machines to run on a single physical machine. While it provides a secure and isolated environment for running virtual machines, the Windows implementation of Hyper-V also presents a significant attack vector that is not entirely well-known to many security professionals and is compliant with security technologies deployed by Windows, such as KPP (PatchGuard) and VBS (Virtualization Based Security) if loaded before they are fully initialized.
Analyzing the Hyper-V technology⌗
Hyper-V developed by Microsoft, is a widely used technology that runs before the operating system has fully loaded and paravirtualizes various components of the kernel; it’s job is to also establish a barrier between itself and other components using either Intel VMX or AMD SVM and give the guest capabilities of doing specific tasks by going through the virtualization layer. Guests aware of being present under Hyper-V can utilize many of the features known as “Enlightenment” by the host.
Communication between the hypervisor-aware system and Hyper-V is a crucial part of the virtualization process, and the process is straightforward with Hyper-V utilizing hypercalls to communicate between the guest and host. These hypercalls are implemented through the use of the “vmcall/vmmcall” instruction in x86-based systems, which triggers a vmexit that the hypervisor can intercept and respond to accordingly.
The inner workings of Hyper-V can be analyzed by analyzing the modules for Hyper-V present inside the system root directory, where it is stored as hvix64.exe and hvax64.exe for Intel and AMD processors, respectively.
The hypercall information utilized for communication is stored in the CONST section in the binary. When we try to analyze it, we are met with something like this.
It follows a distinct pattern, which we can try to analyze and understand when we cross-verify it with the hypercall interface documentation provided by Microsoft here and ntoskrnl.exe, which is the Windows kernel.
The final structure looks something like this:
struct HVCallEntry_t
{
void* Callback;
uint16_t CallCode;
uint16_t RepCount;
uint16_t InputSize;
uint16_t InputSize2;
uint16_t OutputSize;
uint16_t OutputSize2;
uint16_t HypercallGroupNumber;
};
Although the Hyper-V module’s hypercalls are not entirely public in Microsoft’s documentation, we can still dive deeper into their functionality. Reversing all 239 hypercalls would be a daunting task, and definitely beyond the scope of this blog post. However, by examining the exported functions of ntoskrnl.exe such as HvlInvokeFastExtendedHypercall and HvcallInitiateHypercall, we can gain a better understanding of what each hypercall does by following the control flow of each of the handlers it references where it can provide some insight, if not the whole description of their purpose by the symbols. To be precise, it should be noted that a few other drivers are responsible for hypercalling to Hyper-V, such as winhvr.sys and securekernel.exe.
Now that we know how Hyper-V deals with hypercalls, let’s dive deeper into how the Windows kernel handles Hyper-V when it’s aware of the system being virtualized under it.
Hyper-V control flow in the Windows kernel⌗
The Windows kernel’s startup control flow graph looks something like the following:
The Hyper-V related functions in the Windows kernel are typically prefixed with Hvl, and the initialization of Hyper-V is split into three phases, each performed at different stages of the kernel initialization process. The initialization phase that most interests us is the first one, HvlPhase0Initialize. This phase sets up the hypercall interface and the hypercall callbacks both critical components in allowing the operating system to communicate with the hypervisor.
To hijack Hyper-V we need to set up a few things, which are initialized at the first stage of the enlightenment phase.
NTSTATUS HvlpTryConfigureInterface( _LOADER_PARAMETER_BLOCK* LoaderBlock )
{
HviGetHypervisorFeatures( &HypervisorFeatures );
if ( !HviIsHypervisorMicrosoftCompatible( ) || !HypervisorFeatures.PartitionPrivileges.AccessHypercallMsrs )
{
HvlpHypercallCodeVa = 0;
return ERROR_HV_NOT_PRESENT;
}
if ( LoaderBlock )
{
Extension = LoaderBlock->Extension;
MappedPhysPage = Extension->HypercallCodeVa;
SecureKernelRunning = Extension->IumEnabled != 0;
}
else
{
MappedPhysPage = 0;
SecureKernelRunning = 0;
}
if ( MappedPhysPage )
goto $SetupHypercallPtrs;
__writemsr( HV_X64_MSR_GUEST_OS_ID, uint16_t(NtBuildNumber) | ((*(uint8_t*)&CmNtCSDVersion | 0x1040A0000) << 16) );
HypercallPhysicalPage = __readmsr( HV_X64_MSR_HYPERCALL ) | 1;
if ( HyperVisorFeatures.PartitionPrivileges.AccessMemoryPool || SecureKernelRunning )
{
AllocatedPhysPage.QuadPart = HypercallPhysicalPage & -0x1000;
MappedPhysPage = HalpMapEarlyPages( AllocatedPhysPage, 1, PAGE_EXECUTE_READ, SecureKernelRunning );
if ( MappedPhysPage )
goto $SetupHypercallPtrsAndHypercallMSR;
}
else
{
if ( !LoaderBlock )
{
PhysicalAddress = MmGetPhysicalAddress( HvlpHypercallCodeVa );
MappedPhysPage = HvlpHypercallCodeVa;
AllocatedPhysPage = PhysicalAddress;
goto $SetupHypercallPtrsAndHypercallMSRAndSetHypercallPhysicalPage;
}
MappedPhysPage = HalpAllocateEarlyPages( LoaderBlock, MEMORY_CACHING_TYPE::MmCached, &AllocatedPhysPage, PAGE_EXECUTE_READ );
if ( MappedPhysPage )
{
$SetupHypercallPtrsAndHypercallMSRAndSetHypercallPhysicalPage:
HypercallPhysicalPage = AllocatedPhysPage.QuadPart ^ (LOWORD( AllocatedPhysPage.LowPart ) ^ (unsigned __int16)HypercallPhysicalPage) & 0xFFF;
$SetupHypercallPtrsAndHypercallMSR:
__writemsr( HV_X64_MSR_HYPERCALL, HypercallPhysicalPage );
$SetupHypercallPtrs:
HvcallCodeVa = MappedPhysPage;
_InterlockedExchange64( &HvlpHypercallCodeVa, MappedPhysPage );
return STATUS_SUCCESS;
}
}
return STATUS_INSUFFICIENT_RESOURCES;
}
The function above initializes the hypercall stub by checking for multiple things to verify if it should go ahead and set up the hypercalling interface. When a write MSR is executed to the HV_X64_MSR_HYPERCALL MSR with the hypercall stub’s physical address provided as the parameter. The host intercepts this request and then eventually calls the function below, which is responsible for translating the guest’s physical address to the host’s physical address, mapping it, and then copying over the vmcall and return stub, and finally writing the rest of the page with nops.
NTSTATUS InitializeHypercallPageForGuest(QWORD FContext)
{
Status = 0;
Idx = 0;
HypercallPhysicalPage = 0;
VmcallStubAddress = VmcallRetStub;
Size = (unsigned int)VmcallRetStub0 - (unsigned int)VmcallRetStub1;
if ( *(int*)(FContext + 0x10B04) < 0x501 )
Size = (unsigned int)&VmcallRetStub1Align - (unsigned int)VmcallRetStub1;
if ( *(int*)(FContext + 0x10B04) < 0x501 )
VmcallStubAddress = VmcallRetStub1;
while ( 1 )
{
v6 = *(QWORD*)(FContext + 0x120);
v7 = (v6 & 0x20000) != 0 ? 3 : ((v6 & 0x10000) != 0) + 1;
if ( Idx >= v7 )
break;
Status = TranslateGuestToHostPA(*(QWORD *)(FContext + 304), (DWORD *)(FContext + 8816), &HypercallPhysicalPage);
if ( Status )
{
sub_FFFFF800002B0564(FContext);
return Status;
}
HypercallPagePfn = HypercallPhysicalPage >> 12;
*(QWORD*)(FContext + 0x46C0 * Idx + 0x3770) = HypercallPhysicalPage >> 12;
MappedPage = MapHostPA(NtCurrentTeb()->NtTib.ExceptionList, HypercallPagePfn, 6);
// Copy the vmcall and return stub to the mapped page.
memcpy((void *)MappedPage, VmcallStubAddress, (unsigned int)Size);
// Basic sanity check.
if ( Size != 4096 )
memset((void *)(Size + MappedPage), 0x90, 4096 - Size); // Setting the rest of the page with nops.
UnmapHostVA(NtCurrentTeb()->NtTib.ExceptionList, MappedPage, 0);
++Idx;
}
return Status;
}
The second function which would be required to initialize the Hyper-V interface would be HvlpSetupBootProcessorEarlyHypercallPages
NTSTATUS HvlpSetupBootProcessorEarlyHypercallPages( _LOADER_PARAMETER_BLOCK* LoaderBlock )
{
// Allocate 6 RW pages.
VirtualAddress = HalpAllocateEarlyPages( LoaderBlock, 6, &PhysicalPage, PAGE_READWRITE );
if ( !VirtualAddress )
return STATUS_INSUFFICIENT_RESOURCES;
KeGetCurrentPrcb()->HypercallCachedPages = VirtualAddress;
for ( int i = 0; i < 2; i++ )
*(uint64_t*)(VirtualAddress + 16 + i * 0x1000) = uint64_t(PhysicalPage.QuadPart) + i * 0x1000;
return STATUS_SUCCESS;
}
This function allocates a writable page and initializes the HypercallCachedPages field in the current processor block. Then it initializes the structure of the cached page, this is important as certain hypercalls try to lock the page for thread safety by calling HvlpAcquireHypercallPage beforehand, and hence access the HypercallCachedPages pointer, which would point to a null pointer if Hyper-V is not initialized and connected to the guest and will end up bug-checking the system due to a page fault. Hence we have no option but to initialize the field in the processor block structure.
Solving this issue is relatively easy as we can copy what the original function is supposed to do.
/*
* IPI callback for setting the HypercallCachedPages pointer in the KPRCB.
*/
ULONG_PTR SetHypercallCachedPagesIPICallback( _In_ ULONG_PTR CachedPagePtr )
{
uint64_t* HypercallCachedPages = (uint64_t*)(uint64_t( KeGetPcr( )->CurrentPrcb ) + GetHypercallCachedPagesOffset( ));
*HypercallCachedPages = CachedPagePtr;
return 0;
}
// ... Omitted code ...
// Check if Hyper-V is NOT running.
if (!HyperVRunning)
{
// When Hyper-V is off, HypercallCachedPages is null, and this is
// accessed in multiple places and will cause a page fault if not initialized.
HypercallCachedPages = MmAllocateContiguousMemory( 0x6000, PHYSICAL_ADDRESS{ .QuadPart = -1 } );
if (!HypercallCachedPages)
return false;
// Zero out the newly allocated memory.
memset( HypercallCachedPages, 0, 0x6000 );
int64_t HypercallCachedPagesPhys = MmGetPhysicalAddress( HypercallCachedPages ).QuadPart;
for (int i = 0; i < 2; i++)
*(uint64_t*)(uint64_t( HypercallCachedPages ) + 16 + i * 0x1000) = HypercallCachedPagesPhys + i * 0x1000;
// Do an IPI on all cores to set HypercallCachedPages for every core’s processor block.
KeIpiGenericCall( SetHypercallCachedPagesIPICallback, ULONG_PTR( HypercallCachedPages ) );
}
The next function which we are interested in is HvlpDetermineEnlightenments. The function is moderately large to fit in this blog post, so it’s best to just summarize what the function does and the important part we are focusing on.
HvlpDetermineEnlightenments queries the capabilities provided by Hyper-V and sets the flags respectively to HvlpRootFlags and HvlpFlags and also sets up certain things, such as the enlightenment flags. However, the most essential aspect of this function is the below:
Note: pHvlGetEnlightenmentInfo is undocumented and is just a name I used.
pHvlGetEnlightenmentInfo = HvlGetEnlightenmentInfo;
HvlpEnlightenments = ~HvlpRescindedEnlightenments & EnlightenmentFlags;
HvlEnlightenments = ~HvlpRescindedEnlightenments & EnlightenmentFlags;
The code above sets various entries of a structure provided as the first parameter to several values such as the enlightenments, function addresses, and more. At first glance, the structure may appear to be difficult to interpret, but fortunately, the structure passed to the function is called HAL_INTEL_ENLIGHTENMENT_INFORMATION. By setting the pointer type to this structure, we can easily understand the significance of each entry.
However, this function is not invoked anywhere else, so we must look back and look at the references for pHvlGetEnlightenmentInfo, and we can find that HalpHvInitDiscard calls it.
The global variable HalpEnlightenment follows the structure of HAL_INTEL_ENLIGHTENMENT_INFORMATION. However, due to missing information in the PDB, IDA could not understand the entire structure. It is necessary to keep this limitation in mind while analyzing the binary. Additionally, most of the fields in this global structure are not initialized as Hyper-V is not connected and initialized for the guest. Therefore, we need to initialize the relevant entries. It can be a tedious task to implement every function in this list. Fortunately, Microsoft made most of the function members in the structure optional. As a result, we can initialize only the entries that we want to intercept instead of all the entries.
Back to exploitation!⌗
Since we now know the basic initialization route of Hyper-V, we can look into how we can abuse the information we learned to get control over the operating system.
First, we need to figure out how to intercept these hypercalls. And as discussed in the previous section, HvcallInitiateHypercall, and HvcallFastExtended are responsible for handling hypercalls, which both call HvcallCodeVa, which points to a vmcall/vmmcall as mentioned in the previous section of the post. Intercepting this would be easy as one can overwrite the hypercall stub pointer without dealing with any hindrance, such as VBS or KPP (Patchguard). Secondly, we need to set the Enlightenments for the OS to know that Hyper-V provides x capabilities for the guest.
Now the issue with this is that when Hyper-V is not running, and the OS is not enlightened, the OS throws the heavy lifting work to Hyper-V rather than doing it all themselves as it would then defeat the purpose of “enlightenments”. Well, this is precisely the issue we will be facing when we apply this to a real machine that is not enlightened, and we must emulate what the OS expects the hypervisor to do. For example, look at address space switches in SwapContext, which is the function responsible for switching address spaces.
The code above shows the part responsible for switching process contexts, and there is a check if the address space switch enlightenment is available; if it is, it will expect Hyper-V to handle the CR3 switch and TLB flush.
However, the nice thing is that most of the hypercall enlightenments already have an implementation of the feature in the kernel but use Hyper-V as a means to do something faster or better. Thus we can reverse engineer the feature and reimplement it in our code or jump back to the code responsible for it.
Another drawback is that tons of features are initialized via HalpHvInitDiscard, which will only run if Hyper-V was initialized at boot. Hence, we would be required to set them up if we need to intercept said feature; now, we can achieve this with a pattern scan for the structure by its many references. One feature that suffers from this issue is the virtualized sleep state, HalpHvEnterSleepState, which can only trigger a hypercall when the function pointer is initialized with HvlEnterSleepState.
Though moving on, this still allows us to intercept quite a lot of things, such as spinlocks, where we can hook even more stuff by checking for specific calls using the return address in the stack. For example, we can take the SwapContext function where the HvlNotifyLongSpinWait function is invoked as a hint for the virtualized scheduler to delay the next instruction to save performance. Now the function HvlNotifyLongSpinWait is a hypercall to Hyper-V that can be intercepted using the abovementioned method.
Closing notes⌗
It should be noted that the HvcallCodeVa IS protected by KPP (Patch Guard), hence it should be noted that this project must be either loaded before patch guard has fully initialized.
Source⌗
Now this wouldn’t be a complete blog post without providing some easy to paste code wouldn’t it?
HyperDeceit is the ultimate all in one library which includes an easy to use interface to intercept Hypercalls. HyperDeceit also includes a basic emulator which allows it to operate on systems where Hyper-V is disabled.
Yumekage is a demo proof of concept which utilizes HyperDeceit to intercept context swaps to create hidden memory regions for a usermode process.