Common Maintenance Tasks (Clusters)

Edit

The following items should be completed to maintain the health of your Linux cluster. For servers and workstations, please see <a title="Common Maintenance Tasks (Workstations and Servers)" href="https://www.microway.com/knowledge-center-articles/common-maintenance-tasks-workstations-and-servers/">Common Maintenance Tasks (Workstations and Servers)</a>.
<h2>Backup non-replaceable data</h2>
Remember that RAID is not a replacement for backups. If your system is stolen, hacked or started on fire, your data will be gone forever. Automate this task or you will forget.

Compute clusters are built from a large group of computers, so there are many different places for data to hide. Make users aware of your backup policies and be certain they aren't storing vital data on the compute nodes. Let them know which areas are scratch space (for temporary files) and which areas are regularly backed up and designed for user data.

<strong>Strongly consider keeping a backup image of the entire head node installation (including a copy of the compute node software image).</strong> Bare-metal recovery software is available if you're not certain how to do this yourself.

As for the user data:
<ul>
	<li>For many groups, a weekly or monthly cron job is fine. Write a script calling <code>rsync</code> or <code>tar</code> which writes the files to a separate server, NAS or SAN. Place the script in <code>/etc/cron.weekly/</code> or <code>/etc/cron.monthly/</code></li>
	<li>Users with more complex requirements should look at <a title="Amanda Open Source Backup Software" href="http://www.amanda.org/" target="_blank" rel="noopener noreferrer">AMANDA</a> or <a href="https://www.bacula.org/blog/" target="_blank" rel="noopener noreferrer">Bacula</a></li>
	<li>Tape backup systems are still available for those who prefer them. <a title="Contact Microway" href="https://www.microway.com/contact/" target="_blank" rel="noopener noreferrer">Contact us</a>.</li>
</ul>
<h2>Verify the health of your Storage</h2>
Drive sectors can go bad silently. Scheduling regular verifies will weed out any issues before they occur. Automate them or you will forget.
<ul>
	<li>Linux Software RAID (mdadm) arrays can be easily kicked into verify mode. Many distributions (Red Hat, CentOS, Ubuntu) come with their own utilities. To manually start a verify, run this line for each RAID (as root):
<code>echo check &gt; /sys/block/md#/md/sync_action</code>
Watch the text file <code>/proc/mdstat</code> and the output of <code>dmesg</code> to watch the status of each verify.</li>
	<li>Hardware RAID controllers provide their own methods for automated verifies and alert notification. Reference the controller's manual.</li>
	<li>Enterprise and parallel storage systems typically provide their own management interfaces (separate from your cluster management software). Familiarize yourself with these interfaces and enable e-mail alerts.</li>
</ul>
<h2>Monitor system alarms and system health</h2>
If Microway provided you with a preconfigured cluster, then we performed the software integration before the cluster arrived at your site. The cluster can monitor its own health (via MCMS™ or Bright Cluster Manager), but you should familiarize yourself with the user interface and double-check that e-mail alerts are being sent to the correct e-mail address.

Each system in the cluster also supports traditional monitoring and management features:
<ul>
	<li><em>Preferred</em>: learn how to use the IPMI capability for remote monitoring and management. You'll spend a lot less time trekking to the datacenter.</li>
	<li><em>Alternative</em>: listen for system alarms and check for warning LEDs.</li>
</ul>
<strong>Don't ignore alarms! If you put it off, you'll soon find that something else is wrong and your cluster needs to be rebuilt from scratch.</strong>
<h2>Schedule and Test System Software Updates</h2>
Although modern Linux distributions have made it very easy to keep software packages up-to-date, there are some pitfalls an administrator might encounter when updating software on a compute cluster.

Cluster software packages are usually not managed from the same software repository as the standard Linux packages, so the updater may unknowingly break compatibility. In particular, upgrading or changing the Linux kernel on your cluster may require manual re-configuration – particularly for systems with large/parallel storage, InfiniBand and/or GPU compute processor components. These types of systems usually require that kernel modules or other packages be recompiled against the new kernel. <strong>Test updates on a single system before making such changes on the entire cluster!</strong>

Please keep in mind that updating the software on your cluster may break existing functionality, so don’t update just for the sake of updating! Plan an update schedule and notify users in case there is downtime from unexpected snags.

Cancel