Day 10: Effective Log Analysis and Reporting Techniques

Day 10: Effective Log Analysis and Reporting Techniques

Scenario

ยท

3 min read

You are a system administrator responsible for managing a network of servers. Every day, a log file is generated on each server containing important system events and error messages. As part of your daily tasks, you need to analyze these log files, identify specific events, and generate a summary report.

Task

Write a Bash script that automates the process of analyzing log files and generating a daily summary report. The script should perform the following steps:

  1. Input: The script should take the path to the log file as a command-line argument.

  2. Error Count: Analyze the log file and count the number of error messages. An error message can be identified by a specific keyword (e.g., "ERROR" or "Failed"). Print the total error count.

  3. Critical Events: Search for lines containing the keyword "CRITICAL" and print those lines along with the line number.

  4. Top Error Messages: Identify the top 5 most common error messages and display them along with their occurrence count.

  5. Summary Report: Generate a summary report in a separate text file. The report should include:

    • Date of analysis

    • Log file name

    • Total lines processed

    • Total error count

    • Top 5 error messages with their occurrence count

    • List of critical events with line numbers

  6. Create a new directory and log the file

     ubuntu@ip-172-31-45-42:~$ mkdir logs
     ubuntu@ip-172-31-45-42:~$ cd logs
     ubuntu@ip-172-31-45-42:~/logs$ vim log_file.log
    
  7. Create shell script log_analyzer.sh

     #!/bin/bash
     #following funtion is display usage
     usage(){
            echo "Usage : $0 /home/ubuntu/logs/log_file.log"
            exit 1
     }
     #following if statment  will check log file path is provided or not
     if [ $# -ne 1 ]; then
        usage
     fi
    
     LOG_FILE=$1
    
     #Check if  the log file exist or not
     if [ ! -f "$LOG_FILE" ]; then
        echo "Error: Log file $LOG_FILE does not exist!"
        exit 1
     fi
     #Varible declaration
     ERROR_KEYWORD="ERROR"
     CRITICAL_KEYWORD="CRITICAL"
     DATE=$(date +"%Y-%m-%d")
     SUMMARY_REPORT="Summary_report_$DATE.txt"
     ARCHIVE_DIR="Processed_logs"
    
     #Summary report
     {
             echo "Date of analysis : $DATE"
             echo "Log file name :$LOG_FILE"
     } > "$SUMMARY_REPORT"
    
     #Total lines processed
     TOTAL_LINES=$(wc -l < "$LOG_FILE")
     echo "Total lines processed: $TOTAL_LINES" >> "$SUMMARY_REPORT"
    
     #Count the number of Error messages
     ERROR_COUNT=$(grep -c "$ERROR_KEYWORD" "$LOG_FILE")
     echo "Total error count: $ERROR_COUNT" >> "$SUMMARY_REPORT"
    
     #List of Critical events with line numbers
     echo "List of critical events with lin numbers:" >> "$SUMMARY_REPORT"
     grep -n "$CRITICAL_KEYWORD" "$LOG_FILE" >> "$SUMMARY_REPORT"
    
     #Identify the top 5 most common error massages
     declare -A error_messages
     while IFS= read -r line; do
         if [[ "$line" == *"$ERROR_KEYWORD"* ]]; then
             message=$(echo "$line" | awk -F"$ERROR_KEYWORD" '{print $2}')
             ((error_messages["$message"]++))
         fi
     done < "$LOG_FILE"
     #Sort and display to 5 Error messges
     echo "Top 5 Error messages with their occurrence count:" >> "$SUMMARY_REPORT"
     for message in "${!error_messages[@]}"; do
         echo "${error_messages[$message]} $message"
     done | sort -rn | head -n 5 >> "$SUMMARY_REPORT"
    
     #Optional: Archive or move processed log files
     if [ ! -d "$ARCHIVE_DIR" ]; then
         mkdir -p "$ARCHIVE_DIR"
     fi
    
     mv "$LOG_FILE" "$ARCHIVE_DIR/"
     echo "Log file has been moved to $ARCHIVE_DIR.!!"
    
     #Print the summary report
     cat "$SUMMARY_REPORT"
    
  8. Output

    This script automates the process of analysing log files, generates a detailed summary report, and optionally archives the processed log file for future reference.

ย