Secondary NameNode

In the series of Big Data Tips, we are in 5th post of the series.  Today, we are going to talk about Secondary NameNode.

Secondary NameNode IS NOT A BACKUP FOR THE NameNode! The name is horrible, but it is what it is.

The secondary Namenode job is to periodically merge the image file with edit log file. This merge operation is to avoid the edit log file from growing into a large file. The secondary namenode runs on a separate machine and copies the image and edit log files periodically. The secondary namenode requires as much memory as the primary namenode. If the namenode crashes, then you can use the copied image and edit log files from secondary namenode and bring the primary namenode up. However, the state of secondary namenode lags from the primary namenode.

NameNode may not have the available resources (CPU or RAM) to do this while continuing to provide service to the cluster so the secondary NameNode applies the updates from the edits file to the fsimage file and sends it back to the primary. This is known as the checkpointing process.  The checkpoint file is never changed by the NameNode only the Secondary NameNode.

This application of the updates (checkpointing) to the fsimage file occurs every 60 mins by default or whenever the NameNodes edits file reaches 64 MB. Which ever happens first. Newer versions of Hadoop use a defined number of transactions rather than file size to determine when to perform a checkpoint.

If the secondary NameNode is not running at all, the edit log will grow significantly and it will slow the system down. Also, the system will go into safemode for an extended time since the NameNode needs to combine the edit log and the current filesystem checkpoint image.