The Working of node_monitor.tcl
Advantages of using node_monitor.tcl
The Working of node_monitor.tcl
The main program called node_monitor.tcl is an Tcl/Tk script that first pings all the hosts (other than the localhost) in the /root/.rhosts file of the master (front-end) machine of the cluster. After checking that all the hosts are accessible and that rsh works on all the computers in the cluster including the front-end or master it imports a file called node.status.old from all the computers in the cluster if running in non real time mode. (NOTE: The front-end or master must have localhost root entry in its /root/.rhosts file and from the front-end or master one must be able to rsh as root to all the nodes in the cluster. This program "node_monitor.tcl" must be run as root or super user.) Also note that the hostnames in the .rhosts file must be single words without white spaces . If you use two words as hostname in the .rhosts file you will need to change this. The program was written by me to enable debugging failing nodes in a cluster primarily due to TCP/IP buffer overflows and related errors. The program can also analyse statistically the various parameters being monitored. The requirement that localhost be set in the .rhosts file is because this program must be run from a node in the cluster that is listed in the .rhosts file. The program has been tested on RedHat Linux 6.2, 7.0. 7.1 and will work with "wish" 8 or higher and perl 5 or higher. The parameters that are monitored on each host are the iteration number, number of users, the load average, the RX and TX buffer usage and whether RX/TX Err/Drp/Ovr are non-zero or not, the memory usage, the buffer usage, the cache usage, the swap file usage, the number of processes running, and the number of TCP network connections requested. You can to chose to monitor all or any combination of these.
The file node.status.old imported is stored in the /root directory of the master with the $host extension when node_monitor.tcl is run in non real time mode (ie: as node.status.$host). Here $host is the value returned when the hostname command issued to the node. (NOTE: If the hostname and the entries in the /root/.rhosts file is not the same the program generates a warning message and this can be ignored. However it is better to have the two in sync.) The node.status.old is a file generated by a perl script node_status that is run on all the nodes including the front-end. Figure 1 shows a part of this file, where its size can be set by changing the value assigned to $filelimit variable. This file (node.status.old) is a collection of system calls made and stored in file node.status, and is delimited by an ITERATION NUMBER. The file node.status is rewritten every ITERATION and the file node.status.old is rewritten everytime its size is greater than that set for $filelimit variable. The calls logged in node.status.old include the results from system calls like uptime, netstat, /proc/meminfo. The program node_status and node_monitor.tcl can be easily modified for other system calls.
Figure 1: Parts of the node_status file showing the variable $filelimit that must be set before running it on all the nodes in the cluster. The number in system("sleep 1"), can also be changed to change the interval in issuing the system calls. If left as is this is done every second.
Before the program node_monitor.tcl is run this program must be running on all the nodes. In order that the node.status.old files on the nodes be of the same size when imported into the master for analysis, a script of the type shown in Figure 2 is advised to be used. This step is only necessary if you wish to run in non realtime mode.
Figure 2: A typical script to run node_status on all nodes in the cluster.
Once this part is done node_monitor can be run by issuing ./node_monitor.tcl in the current directory. A dialog like that shown in Figure 3 is generated. Remember some version of X-windows compatible with "wish" 8 or higher must be running before running node_monitor.tcl.
Figure 3: Dialog window when node_monitor.tcl is run. Note that by clicking on "Select Parameters to Monitor" and then the horizontal line above "Select display type" from the menu that pops up you can detach the window and drag it to the side. The same is true for the "Select variables to display" window. It is very useful to have these windows open as shown in the figure whenever running in Non-Real Time mode.
When run initally you will be asked a series of questions about the mode in which to run this utility. Select Yes if you want a graphic display or no for textual display. The next screen prompts for real-time. Select no if you are using the node_status program. If you select graphical and non real time display you will be prompted if you want to manipulate the graphic display. Also you will need to select the variables to display and the display type before you click on run. After a series of questions you will see the list of hosts that are up and from which the node.status.old files will be imported and renamed. Figure 4 shows an example of the list of hosts for a four node cluster on which this program was developed.
Figure 4: List of hosts found when the program node_monitor.tcl is run.
In the case when a node must be specified as a master (when display type MAX/MIN/AVG for all slave nodes or ALL SLAVE NODES or STANDARD DEVIATION ALL SLAVE NODES is selected) a prompt appears as shown below in Figure 5. Fill in the name of the host and if this matches one in the .rhosts file it is disacarded from being monitored, otherwise all nodes are monitored.
Figure 5: Prompt that appears asking for the hostname of the master. It the hostname specified is not present in the .rhosts file the name is discarded and all nodes are monitored.
If you selected text node you will see a window similar to that shown in Figure 6. All the information presented here for all the iterations using the master as a reference can be saved in a file like a .txt file for viewing later. Figure 6 shows this. An help system is also availble for more information. This is shown in Figure 7 and uses *.html pages. Note for the help system to work two other programs : html_library.tcl and html_display.tcl with help file help.html must be available in the same directory as node_monitor.tcl. On pressing RERUN the program reimports the node.status.old files from the nodes and displays the results.
Figure 6: In text mode the information for each iteration is presented in a convenient scrollabe format that can also be saved for detailed anaylsis and plotting at a later time. A typical save dialog where the results of running in text mode can be saved is also shown.
The help file is shown below in Figure 7.
Figure 7: The help file help.html is displayed when the HELP button is pressed.
Figure 7 gives the display when the GRAPHICAL mode is selected. This mode is useful when a general picture of the status of the cluster is needed for a overview of the various parameters and for a quick check of the node(s) where a problem may exist. From this one or more non-zero RX/TX Err/Drp/Ovr with the node and the corresponding ethernet card can also be identified. This is very importnant information for diagnosing TCP/IP problems. Note the figure below is for the mode graphical non real time and manipulate graphic display. Note that green is for MAX, blue for AVG and red for MIN.
Figure 8: The display of node information in GRAPHICAL format for easy "at-a-glance" cluster overview and critical TCP/IP stack information.
Now on clicking Full Display you should see a screen as shown in Figure 9 below, that has all the information.
Figure 9: The display of node information in GRAPHICAL format for easy "at-a-glance" cluster overview and critical TCP/IP stack information in FULL DISPLAY mode.
To zoom in use the mouse left click button and hold and drag in the region in the rectangle and release within the rectangle. The region between the left click and release will be redrawn when you reclick the FULL DISPLAY button. If you do not move the mouse on clicking within a rectangle or if you click or release outside a rectangle the original FULL DISPLAY screen will be drawn. The order in which the left and right region limits is selected by the mouse does not matter. Only the region selected is important. The selected region must be within a rectangle. This is shown in Figure 10.
Figure 10: The display of node information when zoom in using mouse is done. The region selected here is the right portion of Load Avg in Figure 9 where a hump in the maximim value (green) is seen.
Another feature in this mode is the fact that if you right click on any point in the rectangles you will see the current values displayed in a window. This is shown in Figure 11.
Figure 11: Effect of right clicking the mouse button when the cursor is in a rectangle. Note this feature works only for this mode.
Another way to zoom is to use the slider to move the focus in x or y direction and magnification and then click either RERUN or FULL DISPLAY.
Finally if you run in the graphical real time mode or the graphical non real time no manipulate of graphic display you will see a screen typical to that as shown in Figure 12.
Figure 12: Typical screen seen when running in the graphical real time mode or the graphical non real time no manipulate of graphic display.
One important feature that is very useful is the statistical analysis aspect of this utility. When the display type "STANDARD DEVIATION ALL NODES" or "STANDARD DEVIATION ALL SLAVE NODES" is selected besides the average value the standard deviation (SD), and coefficient of variance (CV) is also displayed. Additionally in the Non Real Time Graphic Manipulate Display mode nodes are ranked based on their fluctuations from the maximum value for each or the parameters being monitored. One can select the nodes outside some standard deviation multiple of the average for each parameter and use the concepts embodied in Tchebysheff's Theorem to screen out well-behaved nodes for the selected parameters. Similarly by using ranks a selection of worst nodes can be displayed. Figure 13 gives a snapshot when running in this mode with the prompt to set the range for the nodes to be monitored for each parameter selected.
Figure 13: Prompt (top) for entering a value to set a range for the nodes to be monitored for the given parameters. Snapshot (bottom) of the display as the program runs.
When FULL DISPLAY is clicked the information about the nodes is seen as shown in Figure 14. Figure 15 shown the results of zooming in on a selected region using the mouse and then clicking FULL DISPLAY. Note that after a certain zoom in individual data point can be easily identified by the boundaries or bars. Note that the number for the worst nodes is set at 4. If this is changed to 3, node2 for the case of network connections is dropped and the three worst nodes now based on the fluctuations are seen. The value of the fluctuations is also conveniently displayed helping to identify bad nodes quickly.
Figure 14: Display about SD, CV, AVG and individual node ranks. Green values on the left are the maximum for the given parameter.
Figure 15: Zooming into network connections parameter. By selecting NETWORK CONNECTIONS in the "Select variables to display" window or menu and selecting a region using the mouse left button and clicking FULL DISPLAY one can concentrate on a single parameter. The right click on a box (single data point) using the mouse results in specific information about point that being displayed in a window. This allows for easy scanning of rogue points.
Figure 16: When slider below is set to 3 nodes, the three worst nodes (node2 is now excluded) are displayed. Here the zoom was reset by left clicking outside the display rectangles or left clicking without moving the mouse inside a display rectangle. Also the parameter selected to be displayed was set as NETWORK CONNECTIONS only rather than ALL.
Advantages of using node_monitor.tcl
The advantages of using this program are many. The basic information after a node failure remains logged in the /root/node.status.old file and can be recovered after the node is brought up. Further as opposed to real time cluster monitoring this method relies very little on the state of the network except when the files are being imported. The program can be easily modified to incorporate any type of system call and can be made to run at any preset intervals to check the health of the system. For large clusters this is very useful and time saving. Finally if any node(s) is in danger of running out of importnant resources it can be identified and load balancing can be done. As discussed extensive statistical analysis of one or more parameters being monitored can be simultaneously undertaken.
To obtain this program please use the link below to send me an email and I will send it to you as a .tar.gz file or an RPM for x86 and Alpha architectures. Further running and installation instructions can be got from the help or this Application Note.