Troubleshooting and Diagnosis of Hardware and Software Faults
December 5th, 2007 Posted in TechOne of my duties at work is repairing PC and Laptop computers with everything from spyware and virus infections to broken or defective hardware.
Over time I’ve came to a way of working which aims to rule out the various issues which occur repeatedly and follows a logical pattern where each likely factor that could influence the rest of the system is discounted in order of severity.
By following a methodology I’ve achieved a high degree of success and it has also enabled me to work on more than one system at a time which helps productivity. It also allows me to focus on other tasks while carrying out this, for the most part, mundane type of work.
Introduction
While I’ve developed the approach based on practical experience and have tried to make things as thorough as possible; it doesn’t always pan out in every case.
The correct course to follow isn’t always the next step on the list, in some cases steps can be skipped, in others you might actively harm the system you’re working on following a prescribed list of steps.
It takes time and experience to get a good grasp of effective troubleshooting skills. A high-level understanding of how all the components work and how they relate to one another in the context of the system is essential.
This guide is aimed at resolving issues with Windows XP SP2 computers. Although it may be effective with other Windows systems, as ever, mileage may vary.
Note: For the simplicity of this guide I will assume all procedures in each step result in a satisfactory outcome. To detail the steps to follow each possible course of action upon detecting a failure requires a level of detail I currently do not have time for but hope to develop in future posts. For now it is left to the reader to deduce the appropriate steps by trial and error.
Software Required
Most steps in this guide make use of the extremely useful Ultimate Boot CD for Windows. A guide to help make the version and customisations I use can be found here.
The UBCD4Win contains all the utilities used in this article including the Ultimate Boot CD (for DOS) and other useful CD boot menu options for Memtest86+ and the Windows Recovery Console among others.
Verify the Issue
A lot of the time when there is a problem with a system there isn’t a clear description from the user pinpointing what exactly is wrong. Therefore the most important thing to do before any troubleshooting or diagnosis takes place is to observe the issue first hand.
Sometimes the fix will be quite obvious or some indication of the area affected will help solve the problem more quickly. It is advised to check Event Viewer for errors and warnings in event logs as well as observe the output of utilities such as the netstat command as a matter of course.
It’s prudent not to plug potentially infected machines into a network, especially with internet access.
Backup
The first thing is to have a backup of the target computer. This is a safety net in case it all goes horribly wrong. Use of disk imaging software is definitely recommended so all data can be restored to the state it was in before any work was carried out.
Create the backup by booting the imaging software from some form of removable media on the target computer and imaging to a network share or locally attached storage (eg: USB external hard drive). If the computer has one or more hardware faults (or there’s any doubt) remove the hard drive(s) and use another computer to image the drive(s).
This is a prime example of a situation where the correct course of action is of paramount importance as it could result in corrupt backups. Always verify backup images upon creation as finding out they’re useless when needed is unacceptable and could prove costly if the data was of particular importance.
Note: If the target computer is going to be used for imaging then perform basic hardware diagnostics first.
Basic Hardware Diagnostics
The initial testing steps to perform are aimed at verifying the general working order of the basic core of the system. ie: CPU, RAM and the hard disk drive the system boots from.
This will help to rule out (or expose) common problems such as defective memory or hard drive as well as overheating CPU, power supply issues and possible motherboard problems.
Memory diagnostic
The general rule of thumb is one pass of all the tests will show up most major defects. Memtest86+ can be ran from the boot menu on the UBCD4Win. Other memory diagnostic programs are available on the UBCD for DOS if a second opinion needs to be sought.
Note: 24 hours of continuous testing is the much touted test of stability, however this is impractical for initial diagnosis of systems. Reserve lengthy testing for cases where these is no obvious fault after basic diagnostics of the whole system have been completed.
Hard disk diagnostic
To ensure the basic operation of the system drive in the system, it is recommended to run a quick test on the hard disk drive using the manufacturer’s diagnostic utility.
The UBCD for DOS has diagnostic utilities from most manufacturers and for quick testing the IBM Drive Fitness Test can be used on any drive.
Note: This may prove the interface and disk are working or, if errors are reported, may point to a defective drive or cable or motherboard interface or combination of all three.
As with memory diagnostics, the full or long test options should be reserved for systems displaying faults consistent with defective hard disks (these tests may cause data loss in some cases).
CPU stress test
Running a utility that will test known good results against those obtained whilst running on the target system will help determine any defects in the CPU and memory subsystem as well as flag any issues with overheating as high CPU usage will create a lot of heat which may result in the system locking up (due to poor airflow or an improperly fitted heat sink on the CPU, for example).
The UBCD for DOS has CPU stress testing utilities available such as Prime95.
Note: Memory diagnostics partially do this by comparing the results of patterns read into and out of RAM, but a more intensive application is recommended to more fully test a system.
It is acceptable to skip this step and return to it if the system exhibits associated symptoms further into testing (such as freezing, running sluggishly or programs crashing randomly, etc).
Summary
These initial tests give a degree of confidence to proceed with diagnosis of software issues. Bear in mind that proceeding further will require tasks to be performed which will affect the data currently on the target computer and in some cases may result in data loss, so it is important to have verified backups.
Software Diagnostics
UBCD4Win can be used to diagnose and repair various issues with the target computer’s Operating System (OS) and data.
The aim here is to verify the file system is intact and error-free, remove extraneous temporary files (in order to decrease scan times) and run several malware and virus scanning programs.
The advantage is this takes place from read only media and testing of the OS and data is done while it’s inert, which gives a better chance of detecting and removing hard to clean infections.
File system check
Start > Programs > Disk Tools > Diagnostic > Check Disk
This will run a script which prompts for options to be chosen. The “C:” drive is normally the system drive and it’s best to run the test in read-only mode (option 1) initially to detect any issues with the file system on the disk, it’s also faster but potentially less reliable using the /f or /r switches.
If many errors are reported it is often best to run a full (AKA long) test on the hard drive using the drive manufacturer’s diagnostic utility and, if the drive fails, restore the backup image taken at the outset to a new hard drive and then proceed with software diagnostics. In the case of few errors it’s usually safe to let chkdsk fix them.
Remove temporary files
Start > Programs > Anti-spyware > EZPCFix
Click Load Hives in the main program window (ensure the correct paths are selected for the system drive and user profiles location in the lower right) and double click Delete Temp Files in the top left.
Select the areas to clean in the child window that opens. Choosing Temp Folder, Temporary Internet Files and _Restore / System Volume Information\_Restore (System Restore files) will reduce the time required for malware and virus scans.
Note: Deleting the System Restore data can be a double edged sword. It performs a useful function by removing a refuge for malware to reappear from, but once gone the ability to restore the system to a previous state via restore points created by the OS is lost. It’s best to be sure the target computer is beyond help from this feature.
Anti-spyware scan
Start > Programs > Anti-Spyware Tools > SpyBot SD
When the program is loading, click through the Legal Stuff, then enter Proxy settings (click Cancel if none are required).
Once loaded, if a working internet connection is available, update the program via Search for Updates. Click Search & Destroy and Check for problems to run the scan.
Once the scan has completed any detected malware will be listed in the white pane. It is selected by default so ensure all items are unwanted before removal.
Note: From past experience not all ‘dialler’ software is unwanted. The target computer’s internet connection software from the user’s ISP may be falsely flagged.
Anti-virus scan
Start > Programs > Anti-Virus Tools > AVG v7.5
A script runs allowing the program to be updated before scanning. Press “Y” to update and follow the on-screen instructions. Before the program completes its launch the location of its temporary files must be chosen, if possible chose option 1 (the C: drive).
When the main program window opens click Scan Selected Areas and tick the C: drive (or the target computer’s system drive, if different).
On completion the program will prompt for further action as required.
Boot into Windows
The above steps should take care of most malware and viruses on the target computer’s system drive. If all the tests and scans have completed without any unexpected results (very poor performance, programs crashing, random reboots, etc), it’s time to boot into the OS for real and run a few more scans in order to give the machine a clean bill of health or instigate more extreme measures.
Safe Mode with Networking
Remove any CDs in the optical drive(s) and start the computer booting normally but press F8 (tap repeatedly to avoid missing the window of opportunity) after POST just as the OS starts to load. From the menu choose the “Safe Mode with Networking” option so an online anti-virus scanner can be used from a fairly safe environment. In this example F-Secure’s Online Scanner will be used but there are various others out there with varying degrees of functionality.
Log on as a user with admin rights and run Internet Explorer (required - most scanners use ActiveX, although some use Java) and enter the following URL:
http://support.f-secure.com/enu/home/ols.shtml
Scroll down and click the “Start scanning” button, install the ActiveX control and accept the EULA, then click the “Full System Scan” button to begin downloading the scanning engine and definition updates. The scan will proceed once all downloads are complete.
This scan should only find trace items such as tracking cookies or remnants of unwanted browser addons and the like. It will report its findings and offer to clean items once complete.
At this point if there are still significant viruses and other malware items being detected it’s time to consider how worthwhile it is to proceed with attempts to remove the infection(s).
Backing up specific user data using a utility such as the File and Settings Transfer Wizard and installing Windows (including drivers, programs and updates) can seem daunting but there is a point of diminishing returns when it comes to how compromised a system is and efforts to clean it.
Note: If the scan fails for any reason (the scanner reports an error or the browser crashes, etc) it may be due to malware infecting IE which may need eradicated before the scan may be successfully completed.
Boot into Windows for real
Restart the computer and let Windows start normally. Note any odd behaviour during startup and logon (excessively slow loading, warning popups for startup items failing to load and the system generally acting erratically).
Sometimes removal of malware can have quite an adverse impact on the stability of a system. In these cases a repair install could be performed on the existing installation but backing up the user data and clean installing Windows is the optimal solution.
If the system seems to be performing as expected the next step is to run a final spyware scan with a program such as Windows Defender. After installation it will update itself and perform a Quick Scan if the default options have been selected.
Windows Defender also includes a component called Software Explorer which can be found under the Tools menu. Use this to verify no unwanted programs are active or allowed to run on Startup.
Note: Internet Explorer is the point where most malware enters the system often infecting the browser itself. The browser’s settings may also need restored to their defaults for each user on the system.
Summary
If the target system is no longer detecting any malware or exhibiting signs of problems it can be declared stable and clean.
Conclusion
This article on Microsoft’s website prompted me to write this post. While their article is aimed at a less technical end user scenario I feel the advice is slightly backward in some of their recommendations. ie: They suggest scanning in the OS, then Safe Mode, then from a PE environment. It also does nothing to address potential hardware issues. I still feel it’s a decent guide and some people may find it useful in implementing such procedures in their organisation.
My guide is aimed more at techs who have the ongoing chore of dealing with users essentially doing themselves over. While it may be a constant stream of customers, it’s good to have effective and structured methods of dealing with the mess. I’d especially like to thank the people behind the software that makes this a lot easier than in the past. If you use their tools, consider donating to help sustain their efforts.