This week we had a clients 2016 RDS server require a reboot. The reboot was done because it’d been up for 60 days and was being cantankerous, you know the typical kind of behaviour that we’ve seen when Windows needs a reboot. So our tech rebooted the server. This server was a new one that was built just a few months back now, and at the time was fully patched etc. It runs on Hyper-V and it’s been performing fine. After a few minutes however the server was not back up and running. It was in fact showing the spinning circle with the black screen that Windows normally does briefly during the boot process. This concerned us, we left it for a further 5 minute before deciding that there might be issues.
We power cycled the machine again and it gave exactly the same results. Given that this client had Hyper-V replica – we decided to do a test failover of the VM to the other host. Guess what… it did exactly the same thing too. Ok – so the client has a Datto backup of this server as well. We tried to boot it using the Datto boot features on the Datto appliance. Guess what… it did the exact same thing there. We tried backups over the past 5 days and all had the same thing. At this point we began to panic and become concerned. We also tried to boot it without having the network card connected as we’d seen issues like that in the past. No luck there either.
I broke out the Windows Recovery environment and dug into the system – we were looking to see if maybe there was a pending.xml file there that was causing patching issues, but there was not. We tried a few other things but nothing seemed to work. I even decided to mount the VHDX file that was the C: drive of the VM and run a chkdsk on it – it too proved to be fine.
About this time, we broke this issue out of the two people working on it to our wider team to see if anyone else in the team had other ideas. One of our team recalled a similar situation a few weeks back on a Windows 10 machine that did the same thing. It was a physical machine. His solution at that time was to just leave it alone and 20 minutes later it had come back. We figured we had nothing to loose by leaving the machine booting while we dug into other potential solutions, so we left it alone. Sure enough – 20 minutes after boot the machine came up without issue. Around the same time frame (ie 20 minutes) the machine we were trying to boot on the Datto device ALSO came up. Wow.
At that point we got the client back online as the downtime was more than was planned by the simple reboot. Later that night we rebooted the machine again and it came back within 90 seconds without issue.
Later that day I was at the Sydney SMB IT Professionals meeting and after the meeting without me mentioning it to others, I had two individual people come and ask if I’ve seen an issue like this!!! They have seen it on both Windows 10 and Windows Server 2016 over the last few months all without resolution aside from leaving it alone. It’s been seen on brand new machines right out of the box, it’s been seen on machines that have been running for 18 months as well. I’m looking for any kind of link here, but as yet don’t have one.
We are still digging into this issue and don’t have a solution, but I wanted to post this in case others came across it as well. Give it time… 20 to 30 minutes seems to be the timeframe. I’ll update this post with more info as I have it. If you’ve seen this then please share whatever info you have.
Following
I’m having a similar situation with a HyperV Server 2016.. The Console has a blue screen with a white circle. I can RDP into the Server as well as the virtual servers it houses without incident. It’s just the console itself. I select Ctrl+Alt+Del to enter my crfedentials, and there is the booting screen. I have remoted in and rebooted the HyperV Server and it still does it. I’m thinking of cold booting it. I’m hesitant because everything else is working fine..
Wayne I too saw this behaviour on a newly built 2016 server std box running on hyper v. Let it go and it eventually started 20 minutes later. ???????
That’s really interesting Rog. I’ve not seen that before – could it have been installing new components at the time or something like that?
I am not sure why this post/blog was created? Why (and I mean WHY) would you say to just wait for 20-30 minutes and all is good? I am pretty sure most (IT Professionals anyways) would do just this, but yet figure out the issue as to why this is happening. BTW, this issue is NOT just a problem with Windows 10 or Server 2016…
Do not just let this ride out and ignore it.. doesn’t make sense at all to see this posting/recommendation. Figure out the issue and resolve it. There can be several reasons.. but IMHO.. DO NOT do what this post says and just let it ride itself out. Especially if you are supporting another business. Figure it out and FIX IT!
I still can’t believe I seen this… sorry..
Thanks for your comment, although with the email address you left, I’m not certain you’ll ever see this response. The blog post exists because the issue has occurred ONE TIME, RANDOMLY to a few machines. Once you wait the 20-30 minutes, the machine continues to reboot without the long delay. Attempts to investigate it while the issue exists have not revealed anything much, and by the time you lodge a PSS call with Microsoft and then wait 8 hours for a callback, guess what… the issue has “resolved” itself. As I said in the original post, we’re still looking for candidates for the issue to investigate as it’s not a problem we can reproduce.
I would like to chime in here. I am having this issue right now and the server has been doing the spinning circle for 3 hours now. Seems there is absolutely no fix to this problem.I am in a VMware Environment and I have a backup solution just like Datto and same thing, Server does not boot.. This similar thing happened on one of my VM’s a few months back. The 2016 server finally came up after a few days!. During the time I was waiting, I was trying to fix this by trying to spin it up in a separate environment and using DISM to see if I could remove recent updates. Nothing worked and it just came back on after a few days. No rhyme or reason. I find it hard to believe I don’t see more posts about this issue.
I have one server (out of 10) that this happens on every boot (at the moment).
Microsoft’s solution as we have premier support, restore a working backup!
That had me try a few other things, but nothing resolved it.
Great find.
I have a very similar experience. Windows Server 2016, RDS running on an extremely powerful host.
ANY reboots and the server sits on Hyper-V splash screen for over an hour but does boot eventually.
Clearly, this is an issue. We don’t run off any special configurations. Single Server, 48 Cores, Two Servers currently on the host. 14 TB of space, Quad-Nics on host.
VM is set as Gen2, was built on a different host with previous Windows Server version, less memory and drive space. 64 gb ram, Dynamic RAM, 12 processors, Single SCSI drive….. nothing complicated and used on other VM’s on another host without the same issues.
It happened on that host as well.
Host is 2019, newly installed, fully patched. Hardware, fully patched as of three months ago.
Did you ever find a solution, Jeff? I have a 2019 RDS server running on HyperV. host. It takes it 40 minutes to reboot. Adequate resources. Move to another host same result. All other VMs on system boot up within a couple minutes. Would think there is a cache or something inside the OS to fix.
Based on this article, and its comments section here, I have left a physical, Windows Server 2016 server powered on now for *10* days straight now, and the boot-up spinning circle is *still* there! On the one hand, this was never a super fast server, but on the other hand, I don’t think it was ever 10-days-of-spinning-circle slow! Should I give up? (that is the way I am leaning). If so, than this post of mine can serve as an FYI to others that find this article in the future, that one possible outcome could be that your server with this problem will never boot again (where “never” equals at *least* 10+ days), as in my case 🙁
See this on two virtual Windows Server 2019… first I thought to reboot the machine again but I waited … and waited … 20-30 minutes later the machine was booted up.
Sawing the same loading screen on a virtual Windows 10 but there was a check and repair of drive C in progess.
Maybe this happens also to the server but you don’t see it, cause the message is missing?
Just a thinking about this very long boot….
Cheers
I have the spinning circles of death also on a windows server 2019 domain controller where I added a license key to go from standard trial to standard edition.
Waiting several days.
Tried rebooting and still in cycle.
What troubleshooting options? Where is there no startup repair?
Also seeing this with a customer’s 2016 VM. The VM will come up after about 6-8 hours. Other VMs built from the exact same template do not seem to have the issue.
vSphere 6.7, all updates applied to ESXi and Windows. Guest is a SQL server. Very frustrating.
Never found a solution. Actually, rebuilding the server on a different host.
I tried replacing the drives, stepped through each setting, moved to another host and back.
The VM was simply moved off another host, worked great for months, then… this.
I have the same issue. I have been able to confirm that if I move my video card to the primary pci-e slot (in place of the RAID card), I do not have the issue. As soon as I reinstall the RAID card in that slot and move the video card to a 1x slot (as that’s all it requires), I freeze moments after the wheel behind to spin. I have to RDP after that to manage the server.
Same issue here. Waiting 2 hours with no luck.
I’ve been seeing this hit 2012, 2016 and 2019 servers running on Nutanix AHV, Nutanix support couldn’t find anything and just advised to turn on boot logging, ofc, it rarely strikes twice so i’ve yet to have an incident on the same server I’d just enabled boot logging on.
Well just happened again on the same server, but ntbtlog.txt file doesn’t even contain entries for the failed boot.
Dont know if Nutanix AHV has this dynamic memory allocation option, will need to ask.
Hello, we are plagued with this problem as well on all three of our ESXI VM’s. Does anyone have a real fix? I don’t seem to see a firm solution in this thread.
We have a tweak to get them to boot — we use Macrium Reflect for backups, so we can boot up the VM from a macrium “rescue CD” and perform a “repair boot problems” and then the VM boots normally, however this changes the Disk ID’s so this solution is far from optimal.
It is VERY non-reproducible – it will hang on reboot, then I do the “repair boot problems” workaround, and it boots up fine, then I immediately can shut down and reboot with no problem.
This is driving me crazy, can anyone help?
Thanks,
We had the exact issue with a Windows 2016 VM server with a large amount of dynamic memory assigned. (64GB). The VM would sit at the Hyper V screen for 20-30 mins before loading successfully. We shut down the VM and assigned static memory (64GB) and the VM now boots in 45secs to the OS. I notice that a dynamic memory configuration has not been mentioned in this thread, so not sure it will help but resolved the issue for us.
Very interesting Nathan! I am sure that will help some people in the thread, so thanks for sharing that! In my case, this problem happen on a non-virtualized copy of Windows Server 2016 (i.e. the OS was installed directly on the hardware). Therefore, your intriguing solution wouldn’t have applied min my case, but again I am guessing it will help other here, both now and in the future. Therefore, thanks again for your post!
Thanks for sharing that. I’ve not been able to repeat this with enough certainty to confirm it myself, so next time I see it, I’ll give it a shot!
Hi All,
If anyone is still interested, I did find the answer, at least in my case. We are running Malwarebytes Endpoint Protection for Servers, and there is an option in the “Protection” section of the Server Policy settings, “Enable Self-Protection Module earlier in the boot process”… when this was checked ON, VMWare was VERY unhappy and caused this “spinning dots”/boot hanging… as soon as I unchecked that option, booting immediately resumed to normal!
Hope this helps someone!
Our issue was MWB also, thank you for this. You literally saved a massive amount of work.
I confirm Nathan experience. Assign memory as static solve problem for us
Exact 20 minutes to start booting. Thank you a lot!!!!!!!