I had a situation today when working with our team, where our ConnectWise server had something go wrong overnight following an upgrade. Long story short, it filled up the disk space on the C: drive and caused things to come to a screaming halt before we woke up this morning! We cleared out some disk space by removing other files first up (note always leave SOMETHING you can get rid of later) and then set about trying to figure out exactly WHAT it was that chewed up the disk space.
Figuring out what changed is often hard. Lucky for us we run StorageCraft ShadowProtect backups every hour, and we could see that the backup done just before the upgrade and just after the upgrade, as well as backups done hourly since then.
We mounted the backup that occurred right after the upgrade and the most recent one (12+ hours later) and then used TreeSize Free to look at the size of various folders in both. It took only a few minutes to locate the folder that grew massively and from there the actual file that was causing the issues. The file in question was a text based log file and even though on disk it was using NTFS compression, it had grown from its normal size of sub 1GB to over 40GB overnight.
Now – Notepad won’t open a file that big – it just won’t so no need to try. My normal goto log file viewer is a utility called trace32 from the old Microsoft Systems Management Server toolkit. Here’s a blog post from a while back on it – http://sbsfaq.com/how-can-i-view-log-files-live/ . I love it because it opens files live and you can see updates live, but I’ve never thrown a 40GB file at it. I tried and didn’t have the patience because after 5 minutes of NOT RESPONDING I killed it. Had I left it an hour it might have handled it, but I’ll never know.
So I did a quick google search for large text file log viewers and found this utility funnily enough called “Large Text File Viewer”. You can download it from here https://large-text-file-viewer.en.softonic.com/
This cool tool is very simple, requires NO INSTALLATION and just works exactly as designed. Opened it up and then selected the log file I wanted to view, and it started to open the file. At the bottom of the screen is a status display which was showing where I was and the total lines in the file… yes that’s 94 million lines and still loading.
Anyway – I left it while I had lunch to load and when I came back, found out that it ended up at over 329 Million lines.
I was able to use the controls at the top of the screen top quickly flick through the file as you can skip forward and backwards either by 10,000 lines or 1,000,000 lines at a time
I was able to quickly skip to the end and then find the most current errors, and see that it was consistently logging these. I then wanted to know – were these errors a result of the upgrade from last night or not.
I used the search function to begin a search for the specific error from the top and it showed me this great dialog box… yes 442 hours to wait.
Given I didn’t have time to wait, I then used the fast paging function to skip forward and backward by 1,000,000 lines at a time and pretty quickly I was able to ascertain that the first instance of this new error happened around 18:29:07,245 last night – yeah – love the precision huh.
Once I had the error – I was able to give this back to the developers at ConnectWise so they could dig into it. Long story short though – I’d never needed to dig into such a massive text based log file before and found this tool that worked well. Adding this to my toolkit for next time. I know there’s likely even better tools out there to view large text based log files, but this worked for me, and I love that it’s dead portable too.