Employee attrition is one of the major problems that every organization faces, regardless of the sector. It refers to the lifecycle of an organization’s workforce and is an inevitable part of any business. It is defined as the process by which employees leave the workforce for personal or professional reasons. However, it becomes a cause of concern when attrition crosses a particular boundary. For example, attrition among female employee groups could be a significant issue that needs special attention. The different reasons for employee attrition include relocation, lack of professional growth, dissatisfaction, number of accidents, stress level due to overload, conflict, improper clarity, supervisor dissatisfaction, demographic reasons due to marital status, kinship responsibilities, children, age, and tenure, job dissatisfaction due to job involvement, job met expectations, job discontent, monotonous job, less promotional chances, instrumental communication, alternate job opportunities, improper work group cohesion and coworker dissatisfaction, lack of compensation, dissatisfaction in pay and distributive justice, personal reasons, pursuing another better opportunity, pursue higher studies, relocation because of a spouse, organization not delivering on the promise made during recruitment and many more.

Organizations must frame strategies and ideas to control the growing employee attrition rate. The primary responsibility of HR analytics is to help organizations understand what is most important to their employees. The goal is to improve understanding and design of interventions, thereby increasing employee engagement and productivity and reducing unwanted attrition risk. Predictive analytics capabilities enable the design of an employee retention model to keep these valuable employees engaged on board.

Organizations are better off when they can retain good employees and the organizational knowledge they possess. Due to attrition, organizations suffer productivity losses, revenue generation opportunities, and lost profits precisely due to changes in senior management. It becomes a significant cause of concern if employees leave the workforce faster than they are hired; these problems include financial considerations because of the cost incurred due to recruiting, hiring, onboarding, and training new employees. Due to these reasons, companies should use various metrics to measure the reasons for attrition. However, it should be clear that zero attrition (no one leaves the organization) is not possible and not suitable for the organization.

It is critical to understand the reasons that drive unwanted attrition in organizations. The most important reason can be dissatisfaction related to pay. In such a scenario, organizations should follow competitors regarding salary levels, pay increases, and retention schemes to check whether attrition rates align with other local companies in the industry. However, this is not enough; the organization must also check other reasons. Some other observations shared by HR personnel show that the attrition among female employees is generally higher and is lower among new recruiters from top educational institutions. However, a systematic approach will be to analyze the data after developing a comprehensive list of drivers of attrition through interviews with managers, employees, and HR staff. However, a deep insight can be gained by evaluating the results of exit interviews.

It is possible to dig deeper to determine the compelling reasons for attrition by using practical tools and, accordingly, determining strategies. For designing strategies to prevent attrition, different parameters that can be measured related to attrition include involuntary attrition rate, new hire attrition contribution, retention rate, attrition breakdown by performance rating, attrition reason breakdown, employee engagement by employee commitment index, employee engagement index, employee retention index, market opportunity index, offer fit index, voluntary attrition rate, cost of employee attrition, average attrition value, average voluntary attrition value, attrition value per FTA, attrition cost rate, etc. Strategies related to recruiting profiles, manager and employee training programs, career planning and development plans, and pay policies will be better to reduce attrition.

Supervised Classification Machine Learning

Classification is a machine learning technique for solving problems predicting a dependent categorical variable. The classification problem occurs when the dependent variable has two or multiple categories and is predicted by a set of independent variables. Most of the data in real-world problems require the concept of classification. It should be specified that any number of dependent variable categories can exist in classification. However, logistic regression applies only to two categories (binary classification). Logistic regression predicts binary values, such as whether the customer will buy or not, Pass/Fail, Yes/No, 0/1. It measures the probability of event=success and event=failure. The different steps that can be used in prediction using classification techniques include the following:

1. Data exploration: This step requires preparing data to develop an efficient model. It includes appropriately understanding data, handling missing values, feature engineering, etc.

2. Model development: Predictive analytics is extracting information from existing datasets to determine patterns and predict future outcomes and trends. We divide the dataset into training and test datasets to assess the patterns and predict outcomes. It is better always to split the data using a particular seed value to have the same data split. This further means that the result generated will always be the same if we split the data. This will help the system generate the same random numbers as specified in the function; otherwise, we would not be able to have uniformity across the different executions of the same program. This will help us always fetch the same result since the same observations will be in the training and test datasets. The model is developed according to the required independent variables on the training dataset. Many supervised machine learning classification techniques are available in KNIME based on different algorithms, such as Naïve Bayes, k-nearest neighbors (k-NN), support vector machines (SVM), decision tree, random forest, bagging, and gradient boosting.

3. Feature selection: Feature selection is critical in building predictive machine learning models. It involves selecting the most relevant features (variables, predictors) from your data that contribute significantly to the target variable's prediction. Here’s a breakdown of the process and its importance:

   Why Feature Selection?

   • Improves Model Performance: Reducing the number of irrelevant or redundant features can enhance model accuracy and reduce overfitting.

   • Reduces Complexity: Simplifies the model, making it more interpretable and faster to train.

   • Enhances Generalization: Focuses on the most critical features, leading to better generalization of new data.

Methods of Feature Selection

1. Filter Methods: These methods apply statistical techniques to evaluate the importance of each feature independently of the model.

   a) Correlation Coefficient: Measures the correlation between each feature and the target variable.

   b) Chi-Square Test: Evaluates the independence of categorical features concerning the target variable.

   c) ANOVA: Assesses the relationship between continuous features and the target variable.

2. Wrapper Methods: These methods evaluate the performance of a subset of features by training and testing a model. They are more computationally intensive but often provide better results.

   a) Recursive Feature Elimination (RFE): Iteratively removes the least important features and evaluates model performance.

   b) Forward/Backward Selection: This method adds or removes features one at a time based on their contribution to the model's performance.

3. Embedded Methods: These methods perform feature selection as part of the model training process. They are model-specific and incorporate feature selection into the model's construction.

   a) LASSO (Least Absolute Shrinkage and Selection Operator): Adds a penalty term to the model that shrinks the coefficients of less important features to zero.

   b) Tree-Based Methods: Decision trees and random forests naturally perform feature selection based on feature importance scores.

4. Predicting the model: The developed model predicts the values of the user-defined input of the test dataset. It stores the predicted value of the dependent variable for the given values of the input independent variables.

5. Determining the model's accuracy: This is the most crucial step because it finally reports the accuracy of the created model. This step shows the final result of the algorithm used on the data. The primary focus is on increasing the model's accuracy with meaningful interpretations. The model's accuracy is determined by comparing the test dataset's predicted and original values. The difference in the preceding values shows the model’s inaccuracy. Different techniques such as accuracy score, classification report, confusion matrix, receiver operating characteristics (ROC), and area under the curve (AUC) are used for measuring the results of classification problems.

   a) Accuracy score: The accuracy is determined using the two inputs: the dependent variable's predicted values and the dependent variable's original values from the test data. The accuracy score is calculated between 0% and 100%.

   b) Confusion matrix: This is used to determine to what extent prediction has been done appropriately. A confusion matrix is a table often used to describe the performance of a classification model on a set of test data for which the actual values are known. To understand the confusion matrix, let us assume the results of a binary classifier. Two possible predicted classes are “yes” and “no”. If we were predicting employee attrition, “yes” would mean they have left the company, and “no” would mean they have not. Suppose the model made 100 predictions (100 employees being analyzed for leaving the company). The model predicted “yes” 40 times and “no” 60 times. However, 45 people left the company in the original dataset, and 55 did not. The confusion matrix will be created as follows:

   

   • True Positives (TP): These are cases in which the model predicted “yes,” and the employees have left the company.

   • True Negatives (TN): These are cases in which the model predicted “no,” and the employees have not left the company.

   • False Positives (FP) (“Type I error”): We predicted “yes”, but they have not left the company.

   • False Negatives (FN) (“Type II error”): We predicted “no”, and they left the company.

   • Accuracy: Accuracy is determined by (TP+TN)/Total = (30+45)/100 = 0.75 (75%)

Two other terms frequently used to evaluate a model are Sensitivity and Specificity.

• Sensitivity (True Positive Rate): % of correctly predicted “yes” is determined by TP/(TP+FN) = 30/(30+15) = 0.67 (67%)

• Specificity (True Negative Rate): % of correctly predicted “no” is determined by TN/(TN+FP) = 45/(45+10) = 0.82 (82%)

c) Classification report: The report displays different values, precision, recall, sensitivity, specificity, F1-score (F-measure), and Cohen’s kappa. Precision is the True Positives ratio to the True and False Positives sum. This further means the percentage of correct for all the positives. Recall is the ratio of True Positives to the sum of the True Positives and False Negatives. This displays the percentage of records that were classified correctly. The F1 score is a weighted harmonic mean of precision and recall; the highest score is 1.0, and the lowest is 0.0. It is suggested that the F1 score be considered when comparing models and classifiers because they consider precision and recall scores in their computation. It should be noted that the F1-score is always lower than accuracy measures. Cohen's Kappa is a statistic that measures inter-rater agreement for categorical items. It considers the possibility of the agreement occurring by chance.

d) ROC curve and AUC: The ROC curve is a commonly used graph that summarizes the performance of a classifier over all possible thresholds. It is generated by plotting the True Positive Rate (y-axis) against the False Positive Rate (x-axis). The ROC curve is usually a good graph to summarize the quality of our classifier. The higher the curve above the diagonal baseline, the better the predictions. The model's performance is determined by looking at the area under the ROC curve, and the value is stored in AUC. The highest value of AUC is 1, while the least is 0.5, which depicts the 45-degree random line. It should be noted that if there is any value less than 0.5, we should do the exact opposite of the recommendation of the model to get a value more than 0.5. It should be noted that the ROC curve is a graphical plot that illustrates the diagnostic ability of a binary classifier system as its discrimination threshold is varied.

6. Creating a better model: The main objective of any analyst is to create a better model. Different approaches, such as hyper-parameter tuning and feature engineering, are adopted to achieve this.

Employee Attrition Data

The EmployeeAttrition.csv file helps us understand KNIME usage. This dataset can be downloaded from the “Data” folder on the KNIME Community Hub.

The Predicting Employee Attrition – Supervised Classification Machine Learning workflow can be downloaded from the KNIME Community Hub.

After downloading the CSV file from the Data folder, you are ready.

Reading the CSV-based dataset

The first step in our analysis will be to load the data for our exploratory analyses. We will do this first step using the CSV Reader node before we persist our analysis in a KNIME table.

The KNIME table is created by loading the EmployeeAttrition.csv CSV dataset. The above table shows that the employee dataset has 14999 observations and 10 columns.

Data Exploration

The Statistics node primarily determines the descriptive summary statistics of the columns in the dataset. This node calculates statistical moments such as minimum, maximum, mean, standard deviation, variance, median, overall sum, number of missing values, and row count across all numeric columns. It counts all nominal values together with their occurrences. The node provides the following three output tables

a) Statistics Table: All statistic moments for all numeric columns,

b) Nominal Histogram Table: Nominal values for all selected categorical columns and

c) Occurrences Table: The most frequent/infrequent values from the categorical columns (Top/bottom)

The variable LeftCompany is related to employees who have left the organization (employee attrition, yes – left the company & no – not left the company), and the rest of the columns are independent variables. We can observe from the statistics table output that an employee's SatisfactionLevel and LastEvaluation are measured between 0 and 1. The average satisfaction level is 0.613, and the average last evaluation score is 0.716. The variables WorkAccident, LeftCompany, and PromotionLast5years have binary values (yes and no). The highest number of projects an employee has undertaken is 7, and the maximum time spent in the organization by an employee is 10 years. The average time spent by an employee is 3.5 years, and the average monthly hours spent by an employee is 201 hours. The Salary is represented in the form of low, medium, and high. Similarly, the Department column represents the department to which the employee belongs. The number of employees who left the company is 3571 (23.8%), and 11428 (76.2%) are in the organization.

Data Visualization

Displaying Department-wise Employees Count

We will use the Bar Chart node to generate a bar plot that displays the department-wise employee count.

The chart shows that the Sales department had the most employees, followed by the technical and support departments. The Accounting, HR, and Management departments had the fewest employees.

Displaying Attrition by Department

Say we wanted to count the attrition aggregated across departments. We can use the Pivot node to perform this aggregation using column settings: group by LeftCompany, pivot by Department, and count by Salary. Subsequently, we will use the Bar Chart node to display the attrition count for each department.

The chart shows that people have left from all the departments. However, fewer people have left the Sales department, and proportionately more people have left from the technical, IT, product management, and marketing departments.

Displaying Attrition Rate by Department

The chart shows that the HR department has the highest attrition rate, followed by the accounting department. However, the attrition rate is lower in the research and development and management departments.

Visualizing Employee Details

We will use the Python View node to construct a count plot displaying the number of employees in the LeftCompany, Salary, TimeSpendCompany, PromotionLast5years, NumberProject, and WorkAccident columns.

import knime.scripting.io as knio

# Visualization of employee details
# Importing libraries
import seaborn as sns
import matplotlib.pyplot as plt
from io import BytesIO

# Plotting 
fig = plt.figure(figsize=(12, 10)) 

#Count plot for displaying number of employees who left/stayed
plt.subplot(231) 
sns.countplot(x='LeftCompany', data=knio.input_tables[0].to_pandas(), hue='LeftCompany', legend=False, palette="viridis") 
plt.title("Left Company")

#Count plot for displaying number of employees with different salaries
plt.subplot(232)
sns.countplot(x='Salary', data=knio.input_tables[0].to_pandas(), hue='Salary', legend=False, palette="mako")
plt.title("Salary-wise Employees")

#Count plot according to number of years spent in organization
plt.subplot(233)
sns.countplot(x='TimeSpendCompany', data=knio.input_tables[0].to_pandas(), hue='TimeSpendCompany', legend=False, palette="vlag")
plt.title("Time Spent")

#Count plot for displaying number of employees according to promotion
plt.subplot(234)
sns.countplot(x='PromotionLast5years', data=knio.input_tables[0].to_pandas(), hue='PromotionLast5years', legend=False, palette="rainbow")
plt.title("Employees Promotions")

#Count plot for displaying number of employees for number of projects
plt.subplot(235)
sns.countplot(x='NumberProject', data=knio.input_tables[0].to_pandas(), hue='NumberProject', legend=False, palette="terrain")
plt.title("Employees by Number of Projects")

#Count plot for displaying number of employees for work accident
plt.subplot(236)
sns.countplot(x='WorkAccident', data=knio.input_tables[0].to_pandas(), hue='WorkAccident', legend=False, palette="prism")
plt.title("Work Accident")

# Create buffer to write into
buffer = BytesIO()

# Create plot and write it into the buffer
fig.savefig(buffer, format='svg')

# The output is the content of the buffer
output_image = buffer.getvalue()

# Assign the figure to the output_view variable
knio.output_view = knio.view(fig)

We can observe from the chart that employees left, promotion, and work accidents are binary variables. Nearly 23.8% of the employees have left the organization; very few employees have been given promotions in the organization in the last 5 years, and accidents also rarely happen in the workplace. More employees belong to the low-salary group, more have spent 3 years in the organization, and a maximum number of employees have three or four projects.

Determining Correlation Between Variables

We will use the Linear Correlation node to generate a correlation matrix heatmap based on a selected set of columns.

The squared table shows the pair-wise correlation values of all columns. The color range varies from dark red (strong negative correlation) to dark blue (strong positive correlation). If a correlation value for a pair of columns is unavailable, the corresponding cell contains a missing value (shown as a cross in the color view). Hovering the mouse over each box in the views window will display the corresponding pair-wise correlation value. It is clear from the chart that there is a good correlation between the last evaluation, the number of projects, and average monthly hours. This is evident because the good performers might be given a lot of projects to work with. Surprisingly, a negative correlation exists between satisfaction level and number of projects.

Association Between Attrition and Categorical Variables

We will use the Python View node to generate multiple count plots between the LeftCompany column and the NumberProject, Salary*,* WorkAccident*,* and PromotionLast5years categorical columns.

import knime.scripting.io as knio

# Count plot between employee attrition and categorical variables
# Importing libraries
import seaborn as sns
import matplotlib.pyplot as plt
from io import BytesIO

# Plotting 
fig = plt.figure(figsize=(15,8))
plt.suptitle('Association between Attrition and Categorical Variables',
fontsize=20, fontweight='bold')

# Count plot for displaying number of employees for Number of Projects
plt.subplot(221)
sns.countplot(x='NumberProject', data=knio.input_tables[0].to_pandas(), hue='LeftCompany', palette="Set1_r", legend='auto')
plt.xlabel('Number of Projects')
plt.ylabel('Number of Employees')

# Count plot for displaying number of employees for Salary
plt.subplot(222)
sns.countplot(x='Salary', data=knio.input_tables[0].to_pandas(), hue='LeftCompany', palette="Set2_r", legend='auto')
plt.xlabel('Salary')
plt.ylabel('Number of Employees')

# Count plot for displaying number of employees for Work Accident
plt.subplot(223)
sns.countplot(x='WorkAccident', data=knio.input_tables[0].to_pandas(), hue='LeftCompany', palette="Set3_r", legend='auto')
plt.xlabel('Work Accident')
plt.ylabel('Number of Employees')

# Count plot for displaying number of employees for Promotion in Last 5 Years
plt.subplot(224)
sns.countplot(x='PromotionLast5years', data=knio.input_tables[0].to_pandas(), hue='LeftCompany', palette="Set1", legend='auto')
plt.xlabel('Promotion in Last 5 years')
plt.ylabel('Number of Employees')

# Create buffer to write into
buffer = BytesIO()

# Create plot and write it into the buffer
fig.savefig(buffer, format='svg')

# The output is the content of the buffer
output_image = buffer.getvalue()

# Assign the figure to the output_view variable
knio.output_view = knio.view(fig)

From the chart of the number of projects and number of employees, we can see that people who had three projects did not leave the organization. But people with 2, 4, 5, and 6 projects left the organization. This means there is no association between the “number of projects” and “number of employees.” It can be observed from the chart that more employees who were offered low to medium-salary packages have left the organization compared to those offered high salaries. It is evident from the chart that many employees left the organization regardless of the work accident. So, work accidents are not an essential factor in leaving the organization. Nearly 4000 employees might have left the organization unsatisfied with their promotion process. From the preceding four categorical data, we can reasonably assume that “promotion” and “salary” may be why employees leave the organization.

Association Between Attrition and Continuous Variables

We will use the Python View node to generate multiple count plots between the LeftCompany column and the LastEvaluation, SatisfactionLevel, and AverageMonthlyHours continuous columns.

import knime.scripting.io as knio

# Load your data (example) 
df = knio.input_tables[0].to_pandas() 

# Count plot between employee absenteeism and categorical variables
# Importing libraries
import seaborn as sns
import matplotlib.pyplot as plt
from io import BytesIO

# Association between employees attrition and continuous variables
absent_no=df[df["LeftCompany"]=='No']
absent_yes=df[df["LeftCompany"]=='Yes']

# Creating and setting the new palette
new_palette = ["#861B04", "#F86C4D", "#E5F803", "#4DB5F8", "#C4FC05", "#F30BF0", "#05FC93", "#FC0593", "#FC8F05"]
sns.set_palette(palette=new_palette)

fig, (ax1, ax2, ax3) = plt.subplots(nrows=3, figsize=(13,9))

# Kernel density plot for employee attrition and last evaluation

sns.kdeplot(data=absent_no, x="LastEvaluation", y=None, hue=None, weights=None, palette=None, hue_order=None, hue_norm=None, color='#861B04', fill=True, 
multiple="layer", common_norm=True, common_grid=False, cumulative=False, bw_method="scott", bw_adjust=1, warn_singular=True, 
log_scale=None, levels=10, thresh=.05, gridsize=200, cut=3, clip=None, legend=True, cbar=False, cbar_ax=None, cbar_kws=None, ax=ax1)

sns.kdeplot(data=absent_yes, x="LastEvaluation", y=None, hue=None, weights=None, palette=None, hue_order=None, hue_norm=None, color='#F86C4D', fill=True, 
multiple="layer", common_norm=True, common_grid=False, cumulative=False, bw_method="scott", bw_adjust=1, warn_singular=True, 
log_scale=None, levels=10, thresh=.05, gridsize=200, cut=3, clip=None, legend=True, cbar=False, cbar_ax=None, cbar_kws=None, ax=ax1)

ax1.set_title('Last Evaluation & Left Company')
ax1.legend(['No','Yes'])

# Kernel density plot for employee attrition and satisfaction level
sns.kdeplot(data=absent_no, x="SatisfactionLevel", y=None, hue=None, weights=None, palette=None, hue_order=None, hue_norm=None, color='#4DB5F8', fill=True, 
multiple="layer", common_norm=True, common_grid=False, cumulative=False, bw_method="scott", bw_adjust=1, warn_singular=True, 
log_scale=None, levels=10, thresh=.05, gridsize=200, cut=3, clip=None, legend=True, cbar=False, cbar_ax=None, cbar_kws=None, ax=ax2)

sns.kdeplot(data=absent_yes, x="SatisfactionLevel", y=None, hue=None, weights=None, palette=None, hue_order=None, hue_norm=None, color='#C4FC05', fill=True, 
multiple="layer", common_norm=True, common_grid=False, cumulative=False, bw_method="scott", bw_adjust=1, warn_singular=True, 
log_scale=None, levels=10, thresh=.05, gridsize=200, cut=3, clip=None, legend=True, cbar=False, cbar_ax=None, cbar_kws=None, ax=ax2)

ax2.set_title('Satisfaction Level & Left Company')
ax2.legend(['No','Yes'])

# Kernel density plot for employee attrition and average monthly hours
sns.kdeplot(data=absent_no, x="AverageMonthlyHours", y=None, hue=None, weights=None, palette=None, hue_order=None, hue_norm=None, color='#05FC93', fill=True, 
multiple="layer", common_norm=True, common_grid=False, cumulative=False, bw_method="scott", bw_adjust=1, warn_singular=True, 
log_scale=None, levels=10, thresh=.05, gridsize=200, cut=3, clip=None, legend=True, cbar=False, cbar_ax=None, cbar_kws=None, ax=ax3)

sns.kdeplot(data=absent_yes, x="AverageMonthlyHours", y=None, hue=None, weights=None, palette=None, hue_order=None, hue_norm=None, color='#FC0593', fill=True, 
multiple="layer", common_norm=True, common_grid=False, cumulative=False, bw_method="scott", bw_adjust=1, warn_singular=True, 
log_scale=None, levels=10, thresh=.05, gridsize=200, cut=3, clip=None, legend=True, cbar=False, cbar_ax=None, cbar_kws=None, ax=ax3)

ax3.set_title('Average Monthly Hours & Left Company')
ax3.legend(['No','Yes'])

plt.tight_layout()

# Create buffer to write into
buffer = BytesIO()

# Create plot and write it into the buffer
fig.savefig(buffer, format='svg')

# The output is the content of the buffer
output_image = buffer.getvalue()

# Assign the figure to the output_view variable
knio.output_view = knio.view(fig)

The chart shows that people who left the organization had evaluation scores less than 0.6 and greater than 0.8. This means that either high performers or low performers left the organization. Employees who performed average stayed in the organization to a great level. This is the same scenario with monthly hours. Employees with low or high average monthly hours left the organization, while those with average hours stayed back. However, more employees who had lower satisfaction levels left the organization.

Scatter Plot Matrix for Continuous Columns by Attrition

We will first use the Color Manager node to assign specific colors for the categories (Yes and No) in the LeftCompany column. Subsequently, we will use the Scatter Plot Matrix node to create a pair-wise scatter plot matrix of SatisfactionLevel, LastEvaluation, and AverageMonthlyHours columns with LeftCompany (Yes and No) as the color dimension for employees.

It is clear from the chart that employees with higher last evaluation and satisfaction levels, higher last evaluation and lower satisfaction levels, and medium last evaluation and medium satisfaction levels tend to leave. However, employees with a lower last evaluation, irrespective of satisfaction level, and employees with the highest satisfaction level and highest last evaluation also do not leave. This is because these employees might be getting higher salaries, and they do not leave. Similarly, employees with more monthly hours tend to leave, and employees with fewer monthly hours, irrespective of LastEvaluation and SatisfactionLevel, do not leave. Thus, AverageMonthlyHours is a significant reason behind leaving the organization. However, employees with higher satisfaction levels do not leave, irrespective of the monthly hours.

Dummy Encoding

As part of feature engineering, converting each category into different columns represented by binary values (0, 1) is essential. We will use the One to Many node to transform all possible category values in the selected columns into a new column. It is worth noting that the number of new columns created for a particular column is the same as the number of categories in the column.

The One to Many node creates new columns according to the number of categories for each variable. Ten new columns have been made since the Department has ten categories. Similarly, three new columns are designed for Salary, and two are for WorkAccident and PromotionLast5years. The result clearly shows that there are 14999 observations with 27 columns. Among these, 17 new columns have binary integer values.

In classification prediction, one category for each categorical variable is considered a reference, and others are considered dummy variables. For this process, the following dummy and reference variables are created for each attribute:

• For department, “management_Department” is the reference variable.

• For salary, “low_Salary” is a reference variable.

• For work accident, “No_WorkAccident” is a reference variable.

• For promotion in the last five years, “Yes_PromotionLast5years” is the reference variable.

Dropping Reference Variables

It is essential to retain only requirement columns for the analysis. Hence, the reference and categorical variables are removed using the Column Filter node before creating the classification model. LeftCompany is considered the dependent variable.

After dropping the unwanted columns, the table has 14999 rows and 19 columns.

Data Partitioning

The complete dataset clearly shows a much higher number of loan rejections than approvals. As a result, we would need to utilize stratified sampling based on the LeftCompany column in the Partitioning node. The random seed option helps to divide the dataset exactly according to the seed value. This is done intentionally to produce the same results for ease of replication. For this partitioning, the seed value is 30000. The dataset is split in the ratio of 70:30 for the training and test datasets, respectively.

The first partition training table has 10499 rows and 19 columns; the second partition test table has 4500 rows and 19 columns.

Predicting Employee Attrition Using Logistic Regression (All Attributes)

Predict employee attrition using the logistic regression model with all the independent variables. The characteristic of logistic regression helps to solve many real-world problems. Most of the data occurring in real-world problems require the concept of classification. It is essential to mention here that most of the necessary assumptions for regression need not be fulfilled before applying the logistic regression. First, logistic regression does not require a linear relationship between the dependent and independent variables. Second, the error terms do not need to be normally distributed. Third, homoscedasticity is not required. However, logistic regression requires the data to be normal.

We will run the logistic regression model on the training dataset using the Logistic Regression Learner node, with LeftCompany as the target column and ‘No’ as the reference category. We will include all the independent variables that we had filtered earlier. Subsequently, with the Logistic Regression Predictor node, we will predict the response on the test dataset. The resultant output contains three new columns, one containing the prediction for each row and two more containing the predicted probabilities for the target column's two categories (yes and no). They represent the probability that a row in the input data falls into a specific category.

We will use the Scorer node to compare the actual (LeftCompany) and predicted (LeftCompPredLogistic) columns by their attribute value pairs and show the confusion matrix. Additionally, the node provides several accuracy statistics such as True-Positives, False-Positives, True-Negatives, False-Negatives, Recall, Precision, Sensitivity, Specificity, Cohen’s Kappa, and F-measures, as well as the overall accuracy.

We will also use the ROC Curve node to visualize and find the Area Under the Curve (AUC) for the model based on the predicted probabilities column (LeftCompany=’Yes’).

The logistic regression model's accuracy for the test data is 0.797. The confusion matrix shows that True Positives are 404 and True Negatives are 3181. Similarly, False Positives are 248 and False Negatives are 667. Hence, accuracy is 0.797, and misclassification is 0.203. This further means that Sensitivity is 0.26 and Specificity is 0.92.

The results of the scorer report can be explained as follows:

Precision is the ratio of True Positives to the sum of True and False Positives. This further means that the percentage of correct positives. Our results show that the precision for ‘Yes’ is 0.62 and for ‘No’ is 0.827. This means that the records having ‘Yes’ are correctly predicted as 62%, and records having ‘No’ are correctly predicted as 82.7%. Recall is the ratio of True Positives to the sum of True Positives and False Negatives. This displays the percentage of records that are classified correctly. Our result shows that recall values are 0.377 and 0.928 for ‘Yes’ and ‘No’, respectively. This further suggests that the accuracy is lower at 0.377 for the categorical value of “Yes”. The F1 score is the weighted harmonic mean of precision and recall; the highest score is 1.0, and the lowest is 0.0. Our test data's F1-scores are 0.469 and 0.874 for ‘Yes’ and ‘No’, respectively. Cohen’s Kappa value is 0.352 (fair agreement), which is low in our case. From the ROC curve, we can observe that the AUC of 0.826 is excellent.

While all these metrics are helpful, each serves a specific purpose in fine-tuning the model and ensuring it meets the desired performance criteria for its particular applications.

Let's explore how these metrics can be used effectively in predicting employee attrition:

Accuracy

• Utility: Provides a general sense of the model’s overall performance by showing the proportion of correct predictions.

• Context: Useful when the cost of false positives and false negatives is similar. Attrition prediction shows how well the model is doing overall, but it might not capture the nuances of the data if there’s a class imbalance.

Precision

• Utility: Focuses on the accuracy of positive predictions (predicting an employee will leave).

• Context: Important when the cost of a false positive is high (e.g., unnecessary interventions). In predicting attrition, high precision means when the model predicts an employee will leave, it’s likely correct.

Recall

• Utility: Measures the model’s ability to capture all actual positives.

• Context: Crucial when the cost of missing a true positive is high (e.g., failing to identify an at-risk employee). High recall ensures that most employees at risk of leaving are identified.

F1-Score (F-Measure)

• Utility: Balances precision and recall by providing a single metric. It’s the harmonic mean of precision and recall.

• Context: Useful when there’s a need to balance the trade-offs between precision and recall in attrition prediction, ensuring neither false positives nor false negatives are disproportionately high.

Cohen's Kappa

• Utility: Adjusts for agreement occurring by chance, measuring how much better the model is compared to random guessing.

• Context: Useful for evaluating the model’s performance beyond what would be expected by chance, especially in imbalanced datasets. It provides a more nuanced understanding of the model’s predictive power.

False Positive Rate (FPR)

• Utility: Measures the proportion of negative instances incorrectly classified as positive (employees predicted to leave but stay).

• Context: Important in scenarios where the cost of false positives is high. In attrition prediction, a high FPR might lead to unnecessary interventions for employees not at risk.

In predicting employee attrition:

• Accuracy gives an overall idea but may not be sufficient if the classes are imbalanced.

• Precision ensures that identified at-risk employees are likely to leave.

• Recall ensures that most at-risk employees are flagged.

• F1-Score balances the trade-offs between precision and recall.

• Cohen’s Kappa provides a measure that accounts for chance agreement.

• FPR helps minimize unnecessary interventions for employees who are not at risk.

Using these metrics together helps you comprehensively evaluate and fine-tune the model, ensuring it’s effective and reliable in predicting employee attrition.

Feature Selection

Now that we have established metrics for a classification model that predicts whether an employee will leave the company or not using all of the features (attributes/independent variables) available in the dataset. The subsequent step is to analyze whether we can achieve better prediction accuracy using fewer features. We shall use the feature reduction nodes in KNIME to do the same.

We will apply the ‘Forward Feature Selection’ strategy to the Feature Selection Loop Start node. This iterative approach starts with no feature selected. Each iteration adds the feature that improves the model the most to the feature set. The Feature Selection Loop End node is used at the end of the feature selection loop to select features according to the feature selection strategy. In our case, we use the ‘Accuracy’ score variable as our objective is to improve/maximize the model's accuracy. Alternatively, we can use other score metrics like ‘Cohen’s Kappa’ or ‘Error’ depending on our model performance objective. Subsequently, we use the Feature Selection Filter node to choose the set of reduced features we want to take forward to build the classification model. In our case, we select the subset of six features with an accuracy of 0.824. The rationale is that it is the subset with the least number of features among the options with the highest accuracy of 0.824. Therefore, we can achieve the highest accuracy possible with the least features. Check the ‘include static columns’ option to include the target variable LeftCompany in the dataset.

Our chosen features subset has the following six independent variables: SatisfactionLevel, NumberProject, marketing_Department, medium_Salary, high_Salary, and Yes_WorkAccident, along with the target variable LeftCompany.

Predicting Employee Attrition Using the Subset of Attributes

We will now predict employee attrition on the reduced subset of the attributes dataset by applying different classification machine learning algorithms. After that, we will collate all the accuracy statistics derived by using these models to the test data to compare the performance of various models and choose the best model to serve the desired objective.

Predicting Employee Attrition Using Logistic Regression (Selected Attributes Subset)

We will run the logistic regression model on the training dataset using the Logistic Regression Learner node, with LeftCompany as the target column and ‘No’ as the reference category. We will include only the six independent variables that we had filtered earlier. Subsequently, with the Logistic Regression Predictor node, we will predict the response on the test dataset. The resultant output contains three new columns, one containing the prediction for each row and two more containing the predicted probabilities for the target column's two categories (yes and no). They represent the probability that a row in the input data falls into a specific category.

We will use the Scorer node to compare the actual (LeftCompany) and predicted columns by their attribute value pairs and show the confusion matrix. Additionally, the node provides several accuracy statistics such as True-Positives, False-Positives, True-Negatives, False-Negatives, Recall, Precision, Sensitivity, Specificity, Cohen’s Kappa, and F-measures, as well as the overall accuracy. We will also use the ROC Curve node to visualize and find the Area Under the Curve (AUC) for the model based on the predicted probabilities column (LeftCompany=’Yes’).

Predicting Employee Attrition Using Decision Tree (Selected Attributes Subset)

Decision trees are primarily used in machine learning and data mining applications. They are graphs that represent choices and their outcomes in a tree structure. The nodes in the graph represent an event or choice, and the edges represent the decision rules or conditions. The decision tree classifier is a supervised learning algorithm that can be used for both classification (categorical dependent variable) and regression (continuous dependent variable). However, the factors that help us to decide which algorithm to use are discussed below:

Situations to Use Decision Tree Model

1. A decision tree is preferred in cases of high nonlinearity and a complex relationship between dependent and independent variables.

2. A decision tree model is a simple and easy-to-understand graphical representation, even for people from nonanalytical backgrounds. No statistical knowledge is required to understand and interpret the results.

3. A decision tree is one of the fastest ways to identify the most significant variables and the relation between two or more variables. Decision trees can help us create new variables/features with better power to predict the target variable.

4. The decision tree model is a nonparametric method unaffected by outliers and missing values. Hence, we do not need to check assumptions. Less data cleaning is required, and imputation is not needed.

Situations When the Decision Tree Model Should Not Be Used

1. The linear regression algorithm should be adopted if a linear model well approximates the relationship between the dependent and independent variables.

2. Overfitting is one of the most real challenges in decision tree models. Setting constraints on model parameters and pruning solves this problem.

3. A decision tree should not be adopted while predicting a dependent continuous variable because while working with continuous numerical variables, it generally loses information when categorizing numerical variables into different categories.

Decision trees are typically drawn upside down such that the terminal node (leaf) is at the bottom and the root node is at the top. The root node represents the entire population or sample, divided into two or more homogeneous sets. Splitting is a process of dividing a node into two or more sub-nodes. When a sub-node splits into further sub-nodes, it is called a decision node. A node divided into sub-nodes is called the parent node of the sub-nodes; sub-nodes are the children of the parent node. The decision to make strategic splits heavily affects a tree’s accuracy. The creation of sub-nodes increases the homogeneity of the resultant sub-nodes. It then selects the split that results in the most homogeneous sub-nodes.

We will run the decision tree model on the training dataset using the Decision Tree Learner node, with LeftCompany as the target column. We will include only the six independent variables that we had filtered earlier. Subsequently, we will predict the response on the test dataset with the Decision Tree Predictor node. The Scorer and ROC Curve nodes are used to calculate the performance statistics for this model.

Predicting Employee Attrition Using Random Forest (Selected Attributes Subset)

Random forest is an extension of bagging. This algorithm also takes a random selection of features rather than using all the features to grow trees along with a random subset of data, as in the bagging algorithm. Decision trees, prone to overfit, have been transformed into random forests by training many trees over various subsamples of the data (in terms of observations and predictors used to train them). Hence, a lot of random trees are generated. Since there are many random trees, it is called a random forest.

We know that error occurs due to two main reasons: bias and variance. A too-complex model has a low bias but a significant variance, while a too-simple model has a low variance but a considerable bias. Hence, we need different ways to solve the problem. We need a variance reduction algorithm for a complex model and a bias reduction algorithm for a simple model. Random forest and boosting algorithms help reduce bias to a great extent in simple models. Random forest also reduces the variance of many complex models with low bias. The trees are large in number, and they are selected randomly. Additional random variable selection makes them even more independent and performs better than the bagging algorithm.

The parameters that control model complexity in decision trees are the pre-pruning parameters that stop the building of the tree before it is fully developed. The main advantage of decision trees is that they can be easily visualized and understood by nonexperts since the algorithm is entirely invariant to data scaling. However, the main drawback of decision trees is that they tend to overfit even with pre-pruning and provide more generalization performance. Hence, most applications use a random forest instead of a single decision tree.

Random forest is mighty and often works well without heavy tuning of the parameters and does not require data scaling. Random forest shares all the benefits of a decision tree. However, decision trees are used only if we need a compact representation of the decision-making process. Random forest helps interpret hundreds of trees in detail, and the trees in random forests tend to be deeper than decision trees. Thus, if a nonexpert wants to make a visual prediction, a single decision tree might be better. Random forest works well on large datasets and training but might be time-consuming and can be parallelized across multicore processors. Random forest performs poorly on high dimensional, sparse data such as text.

Random forest is considered to be a solution to all data science problems. Like a decision tree, random forest can also perform both regression and classification tasks. This method combines a group of weak models to form a robust model. In a random forest, we grow multiple trees instead of a single tree in the decision tree model. In a classification problem, each tree gives a classification, interpreted as the tree that voted for that class. The forest chooses the classification with the most votes (trees in the forest). In regression problems, the algorithm takes the average of outputs by different trees. In the random forest approach, many decision trees are created. Every observation is fed into every decision tree. The most common outcome for each observation is used as the final output. A new observation is fed into all the trees, and a majority vote is taken for each classification model.

The major drawback of random forest is that, like a decision tree, they are appropriate for classification problems but not equally good for regression problems as they do not give precise continuous-nature predictions. Regression does not predict beyond the range in the training data, and it sometimes overfits the datasets. However, because of their high prediction power, a data analyst generally uses random forest for prediction.

We will run the random forest model on the training dataset using the Random Forest Learner node, with LeftCompany as the target column. We will include only the six independent variables that we had filtered earlier. Subsequently, we will predict the response on the test dataset with the Random Forest Predictor node. The Scorer and ROC Curve nodes are used to calculate the performance statistics for this model.

Predicting Employee Attrition Using Gradient Boosted Trees (Selected Attributes Subset)

Boosting is another ensemble technique to create a collection of predictors, and gradient boosting is an extension of the boosting method. An ensemble of trees is built one by one, and individual trees are summed sequentially. In this algorithm, a new tree learns sequentially from the previous tree, which fits relatively simple models to the data. We fit consecutive trees, and at every step, the goal is to reduce errors from the previous tree. If an input is wrongly interpreted, its weight is increased so that the next step classifies it correctly. Combining the whole set at the end converts weak learners into a better-performing mode. The model works in a chain or nested iterative model. Thus, the new models are not independent parallel models, but each model is built based on all the previous small models by weighting. Hence, the latest model gets boosted from the earlier model. This helps in reducing the bias of a large number of small models with low variance. This algorithm supports different loss functions and works well with interactions, but it is prone to overfitting and requires careful tuning of different hyper-parameters.

Unlike bagging, booting does not focus on reducing the variance of learners. Still, it focuses on reducing the high variance of learners by averaging lots of models fitted on bootstrapped data samples generated with replacement from the training data to avoid overfitting. Boosting is a sequential technique that works on the principle of ensemble. It combines a set of weak learners and delivers improved prediction accuracy. At any instant t, the model outcomes are weighted based on the outcomes of the previous instant t-1. The outcomes predicted correctly are given a lower weight, and the ones misclassified are weighted higher. This technique is followed for a classification problem, while a similar method is used for regression.

Another major difference between the two techniques is that in bagging, the various models generated are independent of each other and have equal weightage, whereas boosting is a sequential process in which each new model generated is added to improve the performance of the previous collection of models.

We will run the gradient-boosted trees model on the training dataset using the Gradient Boosted Trees Learner node, with LeftCompany as the target column. We will include only the six independent variables that we had filtered earlier. Subsequently, we will predict the response on the test dataset with the Gradient Boosted Trees Predictor node. The Scorer and ROC Curve nodes are used to calculate the performance statistics for this model.

Collating the Performance Metrics for Comparison

Now that we have developed multiple models. The next logical step is to choose the optimum classification model to predict employee attrition based on the performance metrics. To do so, we will collate all the performance statistics from the various classification models and then conduct a comparative analysis.

import knime.scripting.io as knio

# This example script plots a histogram using matplotlib and assigns it to the output view
import matplotlib.pyplot as plt
import seaborn as sns
import matplotlib

# Set the backend to 'Agg' to ensure compatibility with non-GUI environments
matplotlib.use('Agg')

# Only use numeric columns
data = knio.input_tables[0].to_pandas()

# Determine the feature selection category with the highest F-score for each "Measure Names"
highlight_data = data.loc[data.groupby("Measure Name")["Measure Value"].idxmax()]

# Create FacetGrid
g = sns.FacetGrid(data, col="Measure Name", col_wrap=3, height=4, aspect=1.5, sharey=False)

# Add the plots
g.map_dataframe(sns.barplot, x="Classification Model", y="Measure Value", hue="Classification Model", palette="muted", legend=False)

# Highlight the bars with the highest F-score
for ax, (measure_name, group) in zip(g.axes.flat, data.groupby("Measure Name")):  # Use .flat to ensure compatibility
    highlight = highlight_data[highlight_data["Measure Name"] == measure_name]
    for p in ax.patches:
        if p.get_x() + p.get_width() / 2. in highlight["Classification Model"].values:
            p.set_edgecolor('red')
            p.set_linewidth(2)

# Rotate x-axis labels for better readability
for ax in g.axes.flat:  # Use .flat to ensure compatibility
    for label in ax.get_xticklabels():
        label.set_rotation(45)
        label.set_ha('right')

# Adjust layout to prevent label overlap
plt.tight_layout()

# Show data labels on each of the bars for 3 decimal places
for ax in g.axes.flat:  # Use .flat to ensure compatibility
    for p in ax.patches:
        if p.get_height() > 0:  # Avoid warnings for bars with height 0
            ax.text(p.get_x() + p.get_width() / 2., p.get_height(), '{:.3f}'.format(p.get_height()), 
                    fontsize=9, color='black', ha='center', va='bottom')

# Increase the padding between chart titles and the charts
plt.subplots_adjust(top=0.9)

# Assign the figure to the output_view variable
knio.output_view = knio.view(plt.gcf())  # Use plt.gcf() to get the current figure

Our comparative analysis indicates that we can effectively use a selected subset of attributes rather than the complete dataset for future predictions while maintaining model accuracy. Based on the above comparative table and graph, it is safe to conclude that the Decision Tree model strikes the optimum balance across the performance parameters for predicting employee attrition with a selected subset of attributes.

Best Practices for Using these Metrics Effectively

Here are some best practices for using Accuracy, Cohen's Kappa, and False Positive Rate effectively in your machine learning projects:

Model Accuracy

• Combine with Other Metrics: Always use accuracy with other metrics like precision, recall, and F1-score, especially when dealing with imbalanced datasets.

• Cross-Validation: Use cross-validation to get a more reliable estimate of your model's accuracy.

• Monitor Over Time: Track accuracy to detect performance drift or data changes.

Cohen's Kappa

• Interpret Correctly: Understand the range and meaning of Kappa values. Remember that values close to 1 indicate strong agreement, while values close to 0 indicate agreement similar to chance.

• Class Distribution Check: Be mindful of class distributions, as highly imbalanced data can affect the interpretation of Kappa.

• Compare with Other Metrics: Use Cohen's Kappa alongside other metrics to get a fuller picture of model performance.

False Positive Rate (FPR)

• Threshold Tuning: Adjust the decision threshold to balance FPR with other metrics, such as False Negative Rate (FNR) and Precision.

• Context Awareness: Consider the specific context of your application. In scenarios like medical diagnosis, prioritize minimizing FPR to avoid false alarms.

• ROC Curve: Use the Receiver Operating Characteristic (ROC) curve to analyze the trade-off between True Positive Rate (TPR) and FPR at different thresholds.

General Best Practices

• Understand Context: Tailor the choice of metrics to the specific application and the consequences of different types of errors.

• Data Quality: Ensure high-quality data with accurate labels. Poor data quality can skew all metrics.

• Continuous Monitoring: Implement continuous monitoring to detect any changes in model performance and take corrective actions promptly.

• Transparency: Document and communicate the chosen metrics and their implications to stakeholders.

• Iterative Evaluation: Regularly revisit and refine the metrics as the model evolves and the application context changes.

By following these best practices, you can make more informed decisions about model performance and ensure your machine-learning models are reliable and effective.

Summary

In conclusion, predicting employee attrition using supervised classification machine learning techniques offers organizations a powerful tool to understand and mitigate the factors leading to employee turnover. By leveraging data-driven insights, companies can identify at-risk employees and implement targeted interventions to enhance retention. The process involves data exploration, model development, feature selection, and accuracy assessment, ensuring the predictive model is robust and reliable. By focusing on key metrics such as precision, recall, and F1-score, organizations can fine-tune their models to balance the trade-offs between false positives and false negatives, ultimately leading to more effective human resource strategies. This approach helps retain valuable talent and contributes to maintaining organizational knowledge and reducing the costs associated with high attrition rates.

Performance appraisal is a regular review of an employee’s job performance and overall contribution to a company. It evaluates an employee’s attitude, knowledge, skills, commitment, achievements, etc. Organizations use performance appraisals to justify pay increases, bonuses, and termination decisions. They can be conducted at any given time but are annual, semi-annual, or quarterly. Organizations need to provide employees with big-picture feedback on their work and consider many dimensions that contribute to measuring performance appraisal. However, since the number of dimensions is large, grouping similar dimensions is necessary. This article focuses on determining important factors that emerge by grouping similar dimensions using two essential techniques: exploratory factor analysis and principal component analysis. The methods discussed can be used to determine factors related to different aspects of the HR department.

Unsupervised Machine Learning

Unsupervised machine learning algorithms are used when the output is unknown, and no predefined instructions are available to the learning algorithms. In unsupervised learning, the learning algorithm only has input data, and knowledge is extracted from these data. These algorithms create a new representation of the data that is easier to comprehend than the original data and help improve the accuracy of advanced algorithms by consuming less time and reducing memory. Standard unsupervised machine learning algorithms include association rule mining, dimensionality reduction algorithms, and clustering.

Dimension Reduction Techniques

The dimension reduction algorithms take a high dimensional representation of data as input, which consists of many features, and produce an output that summarizes the data by grouping essential characteristics and results into fewer factors. The two standard dimensionality reduction algorithms include principal component analysis and factor analysis. The principal component analysis replaces a large number of correlated variables with a smaller number of correlated variables. It is used to understand the data's structure, shape, and covariance, which is impossible with simple scatter plots. It is a method that rotates the dataset so that the rotated features are statistically uncorrelated. This method is used to summarize the data and reduce their dimensionality. In contrast, exploratory factor analysis can be used as a hypothesis-generating helpful tool for understanding the relationship between large numbers of variables.

When we deal with vast amounts of data, we are not sure about the usefulness of the information collected, and deriving helpful information becomes tedious. However, we cannot drop some variables to derive useful information, assuming they are not beneficial. For example, suppose the number of variables is more. Applying some tests, creating scatter plots, finding correlations between variables, and interpreting the data is not easy in that case. There would be too many pair-wise correlations between the variables to consider. It will also be challenging to comprehend the data through the graphical display. Hence, it is necessary to club these variables together and reduce the number of variables to a few interpretable linear combinations of the data to interpret the data in a more meaningful form. Each linear combination represents a principal component or a factor. Thus, dimensionality reduction is helpful when we have many variables in our dataset and we need to reduce this number or when responses for many questions tend to be highly correlated. This technique is generally used before performing t-test, ANOVA, regression, or cluster analysis on a dataset with correlated variables.

Performance Appraisal Data

The PerformanceAppraisal.csv file helps us understand KNIME usage. This dataset can be downloaded from the “Data” folder on the KNIME Community Hub.

The Identify Core Factors in Performance Appraisal – Principal Component Analysis workflow can be downloaded from the KNIME Community Hub.

After downloading the CSV file from the Data folder, you are ready.

Reading the CSV-based dataset

The first step in our analysis will be to load the data for our exploratory analyses. We will do this first step using the CSV Reader node before we persist our analysis in a KNIME table.

The KNIME table is created by loading the PerformanceAppraisal.csv CSV dataset. The above table shows that the employee dataset has 322 observations and 29 columns. Each column has values between one and five based on the Likert scale.

Data Exploration

The Statistics node primarily determines the descriptive summary statistics of the columns in the dataset. This node calculates statistical moments such as minimum, maximum, mean, standard deviation, variance, median, overall sum, number of missing values, and row count across all numeric columns. It counts all nominal values together with their occurrences. The node provides the following three output tables.

1. Statistics Table: All statistic moments for all numeric columns,

2. Nominal Histogram Table: Nominal values for all selected categorical columns and

3. Occurrences Table: The most frequent/infrequent values from the categorical columns (Top/bottom)

The above table shows that each column has values between one and five based on the Likert scale.

Data Visualization

We will use the Linear Correlation node to generate a correlation matrix heatmap based on a selected set of columns.

The squared table shows the pair-wise correlation values of all columns. The color range varies from dark red (strong negative correlation) to dark blue (strong positive correlation). If a correlation value for a pair of columns is unavailable, the corresponding cell contains a missing value (shown as a cross in the color view). Hovering the mouse over each box in the views window will display the corresponding pair-wise correlation value. It is clear from the chart that there is a correlation between many groups of variables.

Determine Factors of Performance Appraisal System Using Exploratory Factor Analysis

Objective: To determine factors of performance appraisal system using exploratory factor analysis

Factor analysis is an exploratory data analysis method used to search for critical underlying factors or latent variables from a set of observed variables. It helps in data interpretation by reducing the number of variables. It is widely utilized in nearly all specializations where we need to reduce the number of existing features, like market research, advertising, finance, etc. Factor analysis is a linear statistical model. It explains the variance among the observed variables and reduces a set of observed variables to unobserved variables called factors.

Factors are latent, hidden, unobserved, or hypothetical variables. A factor describes the association among the number of observed variables. The maximum number of factors is equal to the number of observed variables. Every factor explains a specific variance in observed variables with common patterns or responses. Each factor explains a particular amount of variance in the observed variables. It is a method for investigating whether the variables N1, N2,…, Nn are linearly related to a small number of unobserved factors F1, F2,…, Fn.

Some common assumptions that need to be fulfilled before applying factor analysis include the following: The data should not contain outliers, the sample size should be greater than the factor, and there should not be homoscedasticity among the variables.

There are two types of factor analysis:

1. Exploratory Factor Analysis: EFA is a commonly used approach for reducing the number of features. The basic assumption is that any observed variable is directly associated with any factor. EFA is a statistical technique for identifying latent relationships among sets of observed variables in a dataset. In particular, EFA seeks to model a large set of observed variables as linear combinations of some smaller unobserved latent factors.

2. Confirmatory Factor Analysis: In CFA, the basic assumption is that each factor is associated with a particular set of observed variables.

For our analysis, we will apply EFA using the “factor_analyzer” Python library in the Python Script node. Before applying factor analysis, it is critical to evaluate whether determining the factors in the dataset is possible. There are two methods to check the factorability or sampling adequacy:

1. Bartlett’s Test: Bartletts’ test of sphericity checks if the observed variables intercorrelate at all using the observed correlation matrix against the identity matrix. You should not use a factor analysis if the test is statistically insignificant. If the p-value is <0.05, the test will be considered statistically significant, indicating that the observed correlation matrix is not an identity matrix.

2. Kaiser-Meyer-Olkin (KMO) Test: The KMO test measures the suitability of data for factor analysis. It determines the adequacy for each variable and for the complete model. The KMO test estimates the proportion of variance among all the observed variables. A lower proportion is more suitable for factor analysis. The range of KMO values is between 0 and 1. A KMO value of less than 0.5 is considered inadequate for performing factor analysis.

import knime.scripting.io as knio
import pandas as pd

# Load the input data
df = knio.input_tables[0].to_pandas()

# Importing necessary libraries
from factor_analyzer.factor_analyzer import calculate_bartlett_sphericity

chisquare,p_value=calculate_bartlett_sphericity(df)

print("Chi Square value of Bartlett test: ",chisquare.round(3))
print("p value of Brtlett test: ",p_value.round(3))

# Applying KMO test for determining adequacy
from factor_analyzer.factor_analyzer import calculate_kmo

kmo_values,kmo_model=calculate_kmo(df)

print("KMO model:",kmo_model.round(3))
print("KMO values:\n",kmo_values.round(3))

# Convert datalist to a DataFrame for output
output_df = pd.DataFrame(kmo_values)

# Output the datalist as a table
knio.output_tables[0] = knio.Table.from_pandas(output_df)

The p-value of Bartlett’s test is 0, meaning it is statistically significant. The overall KMO value for our data is 0.786, which is excellent. The value of both tests indicates that factor analysis can be executed since the condition of adequacy is met.

Exploratory Factor Analysis

The primary objective of factor analysis is to reduce the number of observed variables and find unobservable variables. The factors with the lowest amount of variance should be dropped. Rotation, which can be orthogonal or oblique, is a tool for better interpretation of factor analysis. It redistributes the commonalities with a clear pattern of loadings.

We will apply EFA using the “factor_analyzer” Python library in the Python Script node. The eigenvalues are determined using the get_eigenvalues() function, which returns the eigenvalues for each variable. It tries to determine the optimum number of factors.

import knime.scripting.io as knio
import pandas as pd
from factor_analyzer import FactorAnalyzer

# Load the input data
df = knio.input_tables[0].to_pandas()

# Perform factor analysis
fa = FactorAnalyzer()
fa.fit(df)

# Determining eigenvalues
eigen_values, vectors = fa.get_eigenvalues()  # Corrected unpacking of the returned values
print("Eigen values are: \n", eigen_values.round(3))

# Convert eigenvalues to a DataFrame for output
EigenValues_df = pd.DataFrame(eigen_values, columns=["EigenValues"])

# Output the eigenvalues as a table
knio.output_tables[0] = knio.Table.from_pandas(EigenValues_df)

# Get factor loadings
factor_loadings = fa.loadings_.round(4)

# Convert factor loadings to a DataFrame for output
FactorLoadings_df = pd.DataFrame(factor_loadings, index=df.columns, columns=[f"Factor {i+1}" for i in range(factor_loadings.shape[1])])

# Output the factor loadings as a table
knio.output_tables[1] = knio.Table.from_pandas(FactorLoadings_df)

# Get factor variance (eigenvalues, proportion of variance, cumulative variance)
fa_variance = fa.get_factor_variance()

# Convert factor variance to a DataFrame for output
FactorVariance_df = pd.DataFrame({
    "Eigenvalues": fa_variance[0],
    "Proportion of Variance": fa_variance[1],
    "Cumulative Variance": fa_variance[2]
})

# Output the eigenvalues as a table
knio.output_tables[2] = knio.Table.from_pandas(FactorVariance_df)

One of the difficult things to determine when conducting a factor analysis is to determine the number of factors. Different metrics such as eigenvalue, total percent variance explained, factor loadings, and line plots are used to decide the number of factors. The Kaiser criterion is an analytical approach based on selecting the factor with a more significant proportion of explained variance. The eigenvalue is a good criterion for determining the optimum number of factors. Eigenvalues represent variance explained by each factor from the total variance. Generally, an eigenvalue greater than one will be considered the feature's selection criteria. Eigenvalues are determined by using the get_eigenvalues() function. A line plot is a graphical approach based on visualizing factors and eigenvalues. It helps us to determine the number of factors where the curve makes an elbow.

One of the criteria is to decide the number of factors based on an eigenvalue greater than or equal to one. This is because a factor with an eigenvalue of 1 accounts for as much variance as a single variance, and the logic is that only factors that explain at least the same amount of variance as a single variable are worth keeping. However, it is observed that this sometimes results in more factors. From the results, we can observe that eigenvalues are more significant than one for eight variables, and for two variables, it is nearly 1.0. We will try to understand using another option, a line plot. A line plot shows the eigenvalues on the y-axis and the number of factors on the x-axis. It always displays a downward curve. The point where the slope of the curve is leveling off (the elbow) indicates the number of factors the analysis should generate. A chart depicting the number of factors and the eigenvalues ranges from 1 to the maximum number of variables. The curve shows a straight line for each factor and its eigenvalues. However, the optimum number of factors for this dataset might be six.

It is clear from the results that both line plot and eigenvalues are yielding an unreasonably high number of factors. In our example, a cutoff of an eigenvalue greater than or equal to one would result in eight factors. And the line plot suggests six factors due to how the slope levels off. It is important to remember that one of the reasons for running a factor analysis is to reduce the large number of variables that describe a complex concept, such as a performance appraisal system, to a few interpretable latent variables. In other words, we would like to find fewer interpretable factors that explain the maximum variability in the data. Therefore, another essential metric to remember is the total variability of the original variables explained by each factor solution. For example, if three factors explain most of the variability in the original 12 variables, then those factors are a good, more straightforward substitute for all 12 variables. However, if it takes six factors to explain most of the variance in those 12 variables, then the purpose of factor analysis is not solved.

The matrix of weights, or factor loadings, generated from an EFA model describes the underlying relationships between each variable and the latent factors. Factor loadings are similar to standardized regression coefficients, and variables with higher loadings on a particular factor can be interpreted as explaining a more significant proportion of the variation in that factor. Factor loading matrices are often rotated after the factor analysis model is estimated to produce a more straightforward, more interpretable structure to identify which variables are loading on a particular factor. The factor loading is a matrix that shows the relationship of each variable to the underlying factor. It shows the correlation coefficient for the observed variable and factor. It is essential to determine the variance explained by each factor. The factor loadings are determined by loadings_, and the variance explained by the factors is determined by the get_factor_variances() function.

It should be noted that each identified factor should have at least three variables with high factor loadings and that each variable should load highly on only one factor. The preceding result displays the factor loadings and variances of each factor. From the factor loadings matrix, we can observe the loading of each variable on each factor. We need to determine the highest loading of the variable in a particular factor. From the matrix, we can observe that there are six factors. The bold letters clearly show the highest loading on the respective factor. For this dataset, the optimum number of factors is six.

Factor analysis has some limitations. Its interpretation can be debatable because multiple interpretations of the same data factors can be made. Besides, factor identification and naming require domain knowledge.

Determine Factors of Performance Appraisal System Using Principal Component Analysis

Objective: To determine factors of performance appraisal system using PCA

Most problems of interest to organizations are multivariate. They contain multiple dimensions that must be looked at simultaneously. Many statistical analysis techniques, such as machine learning algorithms, are sensitive to the number of dimensions in a problem. High dimensionality can render a problem computationally intractable in the significant data era. Hence, dimensionality reduction aims to replace a more extensive set of correlated variables with a smaller set of derived variables and lose the minimum amount of information. The best way to minimize information loss is to preserve variance. Principal Component Analysis (PCA) is a data reduction technique that transforms more correlated variables into a much smaller set of correlated variables called principal components. Simply put, PCA is a method of extracting essential variables (components) from a large set of variables available in a dataset. It extracts a low-dimensional set of features from a high-dimensional dataset to capture as much information as possible. It is always performed on a symmetric correlation and covariance matrix. This means the matrix should be numeric and have standardized data. The main idea of PCA is to reduce the dimensionality of a dataset consisting of many variables correlated with each other while retaining maximum information (variance) in the dataset. Transforming original variables to a new set of variables is called PCA.

PCA is used to overcome the redundancy of features in a dataset. These features are low-dimensional. The components aim to capture as much information as possible with high explained variance. The first component has the highest variance, followed by the second, third, and so on. The components must be uncorrelated. PCA is applied to a dataset with numeric variables. It is a tool that helps produce better visualizations of high-dimensional data.

The first principal component is a linear combination of original predictor variables, which captures the maximum variance in the data set. It determines the direction of the highest variability in the data. The larger the variability captured in the first component, the larger the information captured by the component. No other component can have variability higher than the first principal component. The first principal component results in a line closest to the data; that is, it minimizes the sum of the squared distance between a data point and the line.

The second principal component is also a linear combination of original predictors, which captures the remaining variance in the dataset and is uncorrelated with the first principal component. In other words, the correlation between the first and second components should be zero. All succeeding principal components follow a similar concept; they capture the remaining variation without correlating with the previous component. For n x p dimensional data, min (n-1,p) principal components can be constructed. The directions of these components are identified in an unsupervised way because the response variable is not used to determine the component direction. Therefore, it is an unsupervised approach.

It is a linear orthogonal transformation that transforms the data to a new coordinate system such that the most significant variance by any data projection lies on the first coordinate, the second most considerable variance on the next coordinate, and so on. The analysis uses an orthogonal projection of highly correlated variables to a set of values of linearly uncorrelated variables called principal components. The number of principal components is less than or equal to the number of original variables, and our objective is to maximize all the variance on the first principal component, then the second, and so on.

PCA components explain the maximum variance, while factor analysis explains the covariance in data. PCA components are entirely orthogonal to each other, whereas factor analysis does not require factors to be orthogonal. A PCA component is a linear combination of the observed variable, while in factor analysis, the observed variables are linear combinations of the unobserved variable or factor. PCA components are uninterpretable. In factor analysis, the underlying factors can be labeled and are interpretable.

PCA is a statistical procedure that transforms a dataset into a new dataset containing linearly uncorrelated variables, known as principal components. The basic idea is that the dataset is transformed into a set of components, each attempting to capture as much of the variance in data as possible. In the Python Script node, the PCA() function in the “sklearn.decomposition” library performs PCA by specifying the number of components.

import knime.scripting.io as knio
import pandas as pd
from sklearn.decomposition import PCA

# Load the input data
df = knio.input_tables[0].to_pandas()

# Ensure the input data contains only numeric values
df_numeric = df.select_dtypes(include=['number'])

# Perform Principal Component Analysis
pca = PCA(n_components=6)
pca.fit(df_numeric)
df_transformed = pca.transform(df_numeric)

# Convert the transformed data to a DataFrame
df_transformed_df = pd.DataFrame(df_transformed, columns=[f"PC{i+1}" for i in range(df_transformed.shape[1])])

# Output the transformed data as a table
knio.output_tables[0] = knio.Table.from_pandas(df_transformed_df)

# Get principal components
pca_components = pca.components_.round(3)

# Convert principal components to a DataFrame for output
pca_components_df = pd.DataFrame(pca_components.T, index=df_numeric.columns, columns=[f"Factor {i+1}" for i in range(pca_components.shape[0])])

# Output the factor loadings as a table
knio.output_tables[1] = knio.Table.from_pandas(pca_components_df)

# Get principal component wise explained variance
pca_variance = pca.explained_variance_ratio_.round(3)

# Convert factor variance to a DataFrame for output
pca_variance_df = pd.DataFrame(pca_variance)

# Output the eigenvalues as a table
knio.output_tables[2] = knio.Table.from_pandas(pca_variance_df)

Applying principal component analysis, 29 variables are reduced to six principal components using the command PCA(n_components=6) and pca.fit(PerformanceAppraisal) functions.

Thus, the reduced dataset has 322 rows and six columns, and the principal components dataset has 29 rows and six columns. The details of all components can be found using pca.components_. The resulting table displays the relationship of variables with the factors.

The variance explained by each component is determined using explained_variance_ratio_.

Thus, we can observe that the variance explained by the first component is 20.4%. The second principal component explains less variance than the first (10%), the third component explains 9% of the variance, the fourth component explains 6.3%, and the fifth component explains 5%. The sixth component explains 4.2% of the variance. Thus, the total variance explained by all the six components is nearly 55%.

It should be noted that since the principal component analysis and exploratory factor analysis derive their solutions from the correlation among the observed variables, we need to decide on the factor model that is the best fit for our research goals. We also need to determine how many component factors are required to extract. Then, we extract the components or factors and interpret the results. Finally, we compute the component or factor scores. We use the Python View node to create a color map depicting the components.

import knime.scripting.io as knio

# This example script plots a heatmap using matplotlib and assigns it to the output view
import matplotlib.pyplot as plt

# Only use numeric columns
data = knio.input_tables[0].to_pandas().select_dtypes('number')

# Convert the data to a numpy array
values = data.to_numpy()

# Plot the heatmap
fig, ax = plt.subplots(figsize=(15, 12))  # Adjusted figure size for better visualization
cax = ax.matshow(values, cmap='viridis')  # Use a colormap for better visualization
plt.colorbar(cax)

# Set x-ticks and y-ticks
ax.set_xticks(range(data.shape[1]))
ax.set_xticklabels(data.columns, rotation=45, ha='left')  # Rotate x-tick labels for better readability
ax.set_yticks(range(data.shape[0]))
ax.set_yticklabels(data.index)

# Annotate each cell with the numeric value
for i in range(values.shape[0]):
    for j in range(values.shape[1]):
        ax.text(j, i, f"{values[i, j]:.3f}", ha='center', va='center', color='white', fontsize=5)

# Set labels
ax.set_ylabel("Features")
ax.set_xlabel("Principal Components")

# Assign the figure to the output_view variable
knio.output_view = knio.view(fig)

From the table and chart, the highest loading of each variable on a particular factor is determined. It is clear from the table that.

1. Factor 1: Job-based Attitude: Efficiency, Commitment, Job Knowledge, Completeness, Quality, Quantity, Creativity, Appropriateness, Timeliness, and Flexibility contribute to the first factor. All these characteristics are related to job performance. Hence, this factor can be called a Job-based attitude.

2. Factor 2: Behavioral Aspects: Attitude, Availability, and Communication. All these characteristics pertain to an employee's behavior. Hence, this factor can be called behavioral aspects.

3. Factor 3: Leadership Quality: Responsibility, Focus, Decision Making, and Stress Tolerance. Since all these traits are traits of a leader, this factor can be called Leadership quality.

4. Factor 4: Team-based Attitude: Interpersonal Relations, Share Ideas, Resourcefulness, Confidence, and Acceptability. These characteristics depict an employee's performance when working in a team. Hence, this factor can be called a team-based attitude.

5. Factor 5: Planning: Receptive, Initiative, Strategic Approach, and Appropriateness. These qualities are required for planning, which is why this factor can be termed planning.

6. Factor 6: Problem-solving Attitude: Solution-oriented, Logical Approach, Judgment, and Dependability. These characteristics emphasize a problem-solving attitude; hence, this factor can be termed a problem-solving attitude.

Thus, 29 variables considered in the study related to performance appraisal can be reduced to six factors: problem-solving attitude, leadership quality, planning, behavioral aspects, team-based attitude, and job-based attitude.

Summary

In conclusion, performance appraisal is crucial for evaluating an employee's organizational contributions and effectiveness. Organizations can identify core factors that influence performance using techniques such as exploratory factor analysis and principal component analysis. These methods help reduce data’s complexity by grouping similar dimensions, making it easier to interpret and act upon. The study of performance appraisal data can reveal key factors such as job-based attitude, behavioral aspects, leadership quality, team-based attitude, planning, and problem-solving attitude. Understanding these factors allows organizations to provide more targeted feedback, improve decision-making regarding employee development, and enhance overall organizational performance.

Absenteeism is a habitual pattern of absence from a duty or obligation. Employee absenteeism is expensive and incurs a significant loss to an organization. It is essential to identify employees' characteristics according to their level of absenteeism. This is done using cluster analysis techniques to group employees based on similar characteristics. There are two main techniques for doing cluster analysis: k-means clustering and hierarchical clustering.

Unsupervised Machine Learning

Unsupervised machine learning algorithms are used when the output is unknown, and no predefined instructions are available to the learning algorithms. In unsupervised learning, the learning algorithm only has input data, and knowledge is extracted from these data. These algorithms create a new representation of the data that is easier to comprehend than the original data and help improve the accuracy of advanced algorithms by consuming less time and reducing memory. Standard unsupervised machine learning algorithms include association rule mining, dimensionality reduction algorithms, and clustering.

Clustering Techniques

Clustering is the task of partitioning the data into groups called clusters in which members are similar in some way. A cluster is a collection of observations that are similar among themselves but are different from the observations belonging to other clusters. K-means clustering and hierarchical clustering are commonly used clustering algorithms. Clustering tries to find cluster centers that represent some areas of the data.

Clustering deals with finding a structure in a collection of unlabeled data. Data points inside a cluster are homogeneous and heterogeneous to other groups. The correct number of clusters is an essential issue because it becomes noise beyond the correct number, and you are not capturing any observations below the number. Generally, two forms of clustering, k-means and hierarchical clustering, are used for grouping employees or observations.

Employee Absenteeism Data

The EmployeeAbsenteeism.csv file helps us understand KNIME usage. This dataset can be downloaded from the “Data” folder on the KNIME Community Hub.

The Identify Absenteeism Patterns – Cluster Analysis workflow can be downloaded from the KNIME Community Hub.

After downloading the CSV file from the Data folder, you are ready.

Reading the CSV-based dataset

The first step in our analysis will be to load the data for our exploratory analyses. We will do this first step using the CSV Reader node before we persist our analysis in a KNIME table.

The KNIME table is created by loading the EmployeeAbsenteeism.csv CSV dataset. The above table shows that the employee dataset has 740 observations and 21 columns.

Converting Continuous Column to Categorical Using Binning

Since the clustering works on categorical columns, converting the AbsenteeismTimeInHours column into a categorical one is essential. We will use the Auto-Binner node to convert the AbsenteeismTimeInHours column into a categorical column by splitting it into three equal bins based on the frequency of the observations in the column. Subsequently, we will use the Rule Engine node to assign meaningful category values (low, avg, high) to the Absenteeism column for easy analysis and interpretation.

Data Visualization

We will use the Python View node to generate multiple count plots between the Absenteeism column and the SocialDrinker, SocialSmoker*,* Son, Seasons, Pet, and DayOfTheWeek categorical columns.

import knime.scripting.io as knio

# Load your data (example) 
# df = knio.input_tables[0].to_pandas() 

# Count plot between employee absenteeism and categorical variables
# Importing libraries
import seaborn as sns
import matplotlib.pyplot as plt
from io import BytesIO

# Plotting 
fig = plt.figure(figsize=(15,8))
plt.suptitle('Association between Absenteeism and Categorical Variables',
fontsize=20, fontweight='bold')

# Count plot for displaying number of employees for Social drinker
plt.subplot(231)
sns.countplot(x='SocialDrinker', data=knio.input_tables[0].to_pandas(), hue='Absenteeism', palette="Set1_r", legend='auto')
plt.xlabel('Social Drinker')
plt.ylabel('Number of employees')

# Count plot for displaying number of employees for Social smoker
plt.subplot(232)
sns.countplot(x='SocialSmoker', data=knio.input_tables[0].to_pandas(), hue='Absenteeism', palette="Set2_r", legend='auto')
plt.xlabel('Social Smoker')
plt.ylabel('Number of employees')

# Count plot for displaying number of employees for Son
plt.subplot(233)
sns.countplot(x='Son', data=knio.input_tables[0].to_pandas(), hue='Absenteeism', palette="Set3_r", legend='auto')
plt.xlabel('Son')
plt.ylabel('Number of employees')

# Count plot for displaying number of employees for Seasons
plt.subplot(234)
sns.countplot(x='Seasons', data=knio.input_tables[0].to_pandas(), hue='Absenteeism', palette="Set1", legend='auto')
plt.xlabel('Seasons')
plt.ylabel('Number of employees')

# Count plot for displaying number of employees for Pet
plt.subplot(235)
sns.countplot(x='Pet', data=knio.input_tables[0].to_pandas(), hue='Absenteeism', palette="Set2", legend='auto')
plt.xlabel('Pet')
plt.ylabel('Number of employees')

# Count plot for displaying number of employees for Day of the week
plt.subplot(236)
sns.countplot(x='DayOfTheWeek', data=knio.input_tables[0].to_pandas(), hue='Absenteeism', palette="Set3", legend='auto')
plt.xlabel('Day of the week')
plt.ylabel('Number of employees')

# Create buffer to write into
buffer = BytesIO()

# Create plot and write it into the buffer
fig.savefig(buffer, format='svg')

# The output is the content of the buffer
output_image = buffer.getvalue()

# Assign the figure to the output_view variable
knio.output_view = knio.view(fig)

The chart clearly shows that employees generally have average absenteeism. A more significant number of employees have no pets. The day and season are not crucial for absenteeism because the number of absenteeism is uniform according to the day of the week and season. It can be observed that most of the absent employees do not smoke. Thus, smoking is not the reason for absenteeism. For the absentees, the number of employees who drink is the same as those who do not drink. It is also clear that more employees have no or one child.

We will use the Python View node to generate multiple count plots between the Absenteeism column and the BodyMassIndex, Age, and TransportationExpense continuous columns. Alternatively, we can use the Density Plot node to create an interactive plot with the Absenteeism column as the condition column and the BodyMassIndex, Age, and TransportationExpense as the dimension columns.

import knime.scripting.io as knio

# Load your data (example) 
df = knio.input_tables[0].to_pandas() 

# Count plot between employee absenteeism and categorical variables
# Importing libraries
import seaborn as sns
import matplotlib.pyplot as plt
from io import BytesIO

# Association between employees absenteeism and continuous variables
absent_low=df[df["Absenteeism"]=='Low']
absent_medium=df[df["Absenteeism"]=='Medium']
absent_high=df[df["Absenteeism"]=='High']

# Creating and setting the new palette
new_palette = ["#861B04", "#F86C4D", "#E5F803", "#4DB5F8", "#C4FC05", "#F30BF0", "#05FC93", "#FC0593", "#FC8F05"]
sns.set_palette(palette=new_palette)

fig, (ax1, ax2, ax3) = plt.subplots(nrows=3, figsize=(13,9))

# Kernel density plot for employee attrition for body mass index

sns.kdeplot(data=absent_low, x="BodyMassIndex", y=None, hue=None, weights=None, palette=None, hue_order=None, hue_norm=None, color='#861B04', fill=True, 
multiple="layer", common_norm=True, common_grid=False, cumulative=False, bw_method="scott", bw_adjust=1, warn_singular=True, 
log_scale=None, levels=10, thresh=.05, gridsize=200, cut=3, clip=None, legend=True, cbar=False, cbar_ax=None, cbar_kws=None, ax=ax1)

sns.kdeplot(data=absent_medium, x="BodyMassIndex", y=None, hue=None, weights=None, palette=None, hue_order=None, hue_norm=None, color='#F86C4D', fill=True, 
multiple="layer", common_norm=True, common_grid=False, cumulative=False, bw_method="scott", bw_adjust=1, warn_singular=True, 
log_scale=None, levels=10, thresh=.05, gridsize=200, cut=3, clip=None, legend=True, cbar=False, cbar_ax=None, cbar_kws=None, ax=ax1)

sns.kdeplot(data=absent_high, x="BodyMassIndex", y=None, hue=None, weights=None, palette=None, hue_order=None, hue_norm=None, color='#E5F803', fill=True, 
multiple="layer", common_norm=True, common_grid=False, cumulative=False, bw_method="scott", bw_adjust=1, warn_singular=True, 
log_scale=None, levels=10, thresh=.05, gridsize=200, cut=3, clip=None, legend=True, cbar=False, cbar_ax=None, cbar_kws=None, ax=ax1)
ax1.set_title('Body Mass Index & Absenteeism')
ax1.legend(['Low','Medium','High'])

# Kernel density plot for employee attrition for age
sns.kdeplot(data=absent_low, x="Age", y=None, hue=None, weights=None, palette=None, hue_order=None, hue_norm=None, color='#4DB5F8', fill=True, 
multiple="layer", common_norm=True, common_grid=False, cumulative=False, bw_method="scott", bw_adjust=1, warn_singular=True, 
log_scale=None, levels=10, thresh=.05, gridsize=200, cut=3, clip=None, legend=True, cbar=False, cbar_ax=None, cbar_kws=None, ax=ax2)

sns.kdeplot(data=absent_medium, x="Age", y=None, hue=None, weights=None, palette=None, hue_order=None, hue_norm=None, color='#C4FC05', fill=True, 
multiple="layer", common_norm=True, common_grid=False, cumulative=False, bw_method="scott", bw_adjust=1, warn_singular=True, 
log_scale=None, levels=10, thresh=.05, gridsize=200, cut=3, clip=None, legend=True, cbar=False, cbar_ax=None, cbar_kws=None, ax=ax2)

sns.kdeplot(data=absent_high, x="Age", y=None, hue=None, weights=None, palette=None, hue_order=None, hue_norm=None, color='#F30BF0', fill=True, 
multiple="layer", common_norm=True, common_grid=False, cumulative=False, bw_method="scott", bw_adjust=1, warn_singular=True, 
log_scale=None, levels=10, thresh=.05, gridsize=200, cut=3, clip=None, legend=True, cbar=False, cbar_ax=None, cbar_kws=None, ax=ax2)
ax2.set_title('Age & Absenteeism')
ax2.legend(['Low','Medium','High'])

# Kernel density plot for employee attrition for transportation expense
sns.kdeplot(data=absent_low, x="TransportationExpense", y=None, hue=None, weights=None, palette=None, hue_order=None, hue_norm=None, color='#05FC93', fill=True, 
multiple="layer", common_norm=True, common_grid=False, cumulative=False, bw_method="scott", bw_adjust=1, warn_singular=True, 
log_scale=None, levels=10, thresh=.05, gridsize=200, cut=3, clip=None, legend=True, cbar=False, cbar_ax=None, cbar_kws=None, ax=ax3)

sns.kdeplot(data=absent_medium, x="TransportationExpense", y=None, hue=None, weights=None, palette=None, hue_order=None, hue_norm=None, color='#FC0593', fill=True, 
multiple="layer", common_norm=True, common_grid=False, cumulative=False, bw_method="scott", bw_adjust=1, warn_singular=True, 
log_scale=None, levels=10, thresh=.05, gridsize=200, cut=3, clip=None, legend=True, cbar=False, cbar_ax=None, cbar_kws=None, ax=ax3)

sns.kdeplot(data=absent_high, x="TransportationExpense", y=None, hue=None, weights=None, palette=None, hue_order=None, hue_norm=None, color='#FC8F05', fill=True, 
multiple="layer", common_norm=True, common_grid=False, cumulative=False, bw_method="scott", bw_adjust=1, warn_singular=True, 
log_scale=None, levels=10, thresh=.05, gridsize=200, cut=3, clip=None, legend=True, cbar=False, cbar_ax=None, cbar_kws=None, ax=ax3)
ax3.set_title('Transportation Expense & Absenteeism')
ax3.legend(['Low','Medium','High'])

plt.tight_layout()

# Create buffer to write into
buffer = BytesIO()

# Create plot and write it into the buffer
fig.savefig(buffer, format='svg')

# The output is the content of the buffer
output_image = buffer.getvalue()

# Assign the figure to the output_view variable
knio.output_view = knio.view(fig)

Since the different categories of absenteeism in the chart overlap considerably, it is clear that the level of absenteeism is not related to transportation expense, age, or body mass index.

Assess Employee Absenteeism Using K-Means Clustering

Objective: To cluster the employee records for absenteeism using K-means cluster analysis

K-means clustering is an unsupervised algorithm that solves the clustering problem by following a simple and easy way to classify a given dataset through a certain number of clusters. We must determine the number of clusters for practical analysis before applying the K-means algorithm. The number of clusters can differ depending on the initialization and the distance function. A better machine-learning algorithm can be created if effective initialization and better distance functions are defined.

The K-means picks K number f points for each cluster known as centroids. Each data point forms a cluster with the closest centroids, K clusters. We find the centroid of each cluster based on the existing cluster members. Here we have new centroids. As we have new centroids, we repeat the steps. We find the closest distance for each data point from the new centroids associated with new K clusters. We repeat this process until convergence occurs; centroids do not change. We need to determine the distances between data points without having any information on the features of these points. Clustering aims to minimize the distance between the points and their representatives. However, even if K-means clustering is carried out many times, we are not sure we will get a globally optimal solution. Hence, we need to try different initialization points to find the optima and consider the best among those local optima.

In K-means, we have clusters, each with its centroid. The sum of the squares differences between the centroid and the data points within a cluster constitutes the sum of the square values for that cluster. Also, when the sum of square values for all the clusters is done, it becomes the total within the sum of the square values for the cluster solution. We know that as the number of clusters increases, this value keeps on decreasing, but if we plot the result, you may see that the sum of squared distance decreases sharply up to some value of k and then much more slowly after that. This determines the optimum number of clusters.

Using the Elbow Method

Objective: To determine the optimal number of clusters using the elbow method.

Determining the optimum number of clusters is always recommended before applying the K-means algorithm. We will use the sklearn.cluster library in the Python Script node to iterate and determine the optimal K value. An empty data list is created. A for loop is made from 1 to 11, and the K-means algorithm is executed considering the values of K from 1 to 10. The command KMeans(n_clusters=I,random_state=42) applies the K-means algorithm, and the value of inertia is stored in the list corresponding to each value of K. The command datalist.append(kmeans.inertia_) adds the value of inertia to datalist. Subsequently, we will use the Line Plot (Plotly) node to plot the different values of the Number of Clusters and the corresponding value of Inertia.

import knime.scripting.io as knio

# This example script simply outputs the node's input table.
# Using the elbow method to find the optimal number of clusters
from sklearn.cluster import KMeans
import pandas as pd

# Load the input data
df = knio.input_tables[0].to_pandas()

# Prepare the data for clustering (assuming all columns are used for clustering)
AbsenteeismData = df.values

# Initialize an empty list to store inertia values
datalist = []

# Loop through different numbers of clusters to calculate inertia
for i in range(1, 11):
    kmeans = KMeans(n_clusters=i, random_state=42)
    kmeans.fit(AbsenteeismData)
    datalist.append(kmeans.inertia_)

# Convert datalist to a DataFrame for output
output_df = pd.DataFrame({'Number of Clusters': range(1, 11), 'Inertia': datalist})

# Output the datalist as a table
knio.output_tables[0] = knio.Table.from_pandas(output_df)

From the line plot, we can observe that the optimum number of clusters is 3. So, we will start cluster analysis considering three clusters.

Applying K-means Clustering

Before applying the K-means clustering, we will use the Normalizer node to normalize the continuous columns using the Z-score normalization method. We will then use the k-means node to apply the K-means algorithm to predict the cluster for each observation in the dataset. We will use the Rule Engine node to assign the clusters to the corresponding categories (low, avg, and high) of absenteeism. Finally, we will use the Scorer node to compare the actual (Absenteeism) and predicted cluster (AbsenteeismPred) columns by their attribute value pairs and show the confusion matrix. Additionally, the node provides several accuracy statistics such as True-Positives, False-Positives, True-Negatives, False-Negatives, Recall, Precision, Sensitivity, Specificity, and F-measures, as well as the overall accuracy.

The confusion matrix result shows that 77 observations belonging to the average cluster are grouped similarly in the predicted cluster. Similarly, the low cluster had 118 correctly classified employees, and the high cluster had 129 correctly classified employees. The accuracy score is 43.784%. However, forming proper categories of absenteeism after closely examining the data will help improve the accuracy.

Assessing Employee Absenteeism Using Hierarchical Clustering

Objective: To cluster the employee records for absenteeism using hierarchical cluster analysis

Hierarchical clustering uses two approaches: top-down and bottom-up.

• Top-down or divisive: The algorithm starts with all data points in one huge cluster. The most dissimilar data points are divided into subclusters until each cluster has exactly one data point.

• Bottom-up or agglomerative: The algorithm starts with every data point as one single cluster and tries to combine the most similar ones into superclusters until it reaches one huge cluster containing all subclusters.

In the case of small observations, bottom-up or agglomerative hierarchical clustering provides better results since it helps us build clusters from n to 1 by merging the clusters bottom-up, obtaining all possible clusters between 1 and n.

A measure must be defined to determine the distance between clusters. There are three methods for comparing two clusters.

• Single Linkage: This defines the distance between two clusters, c1, and c2, as the minimal distance between any two points, x, and y, with x in c1 and y in c2.

• Complete Linkage: Defines the distance between two clusters, c1 and c2, as the maximal distance between any two points x, y with x in c1 and y in c2.

• Average Linkage: Defines the distance between two clusters, c1, and c2, as the mean distance between all points in c1 and c2.

A distance measure is necessary to measure the distance between two points. You can choose between the Manhattan and Euclidean distances, corresponding to the L1 and L2 norms.

The output of the Hierarchical Clustering node is the same data as the input, with one additional column containing the cluster name to which the data point is assigned. Since the hierarchical clustering algorithm produces a series of cluster results, the number of clusters for the output is defined as “three,” “Euclidean” as the distance function, and “complete” as the linkage type in the dialog. The production of the agglomerative clustering is depicted using a dendrogram. A dendrogram is a diagram that shows the hierarchical relationship between objects. A dendrogram's primary use is to find the best way to allocate objects to clusters.

We will use the Rule Engine node to assign the clusters to the corresponding categories (low, avg, and high) of absenteeism. Finally, we will use the Scorer node to compare the actual (Absenteeism) and predicted cluster (AbsenteeismPred) columns by their attribute value pairs and show the confusion matrix. Additionally, the node provides several accuracy statistics such as True-Positives, False-Positives, True-Negatives, False-Negatives, Recall, Precision, Sensitivity, Specificity, and F-measures, as well as the overall accuracy.

The confusion matrix result shows that 35 observations belonging to the average cluster are grouped similarly in the predicted cluster. Similarly, the low cluster had 207 correctly classified employees, and the high cluster had 35 correctly classified employees. The accuracy score is 37.432%. However, forming proper categories of absenteeism after closely examining the data will help improve the accuracy.

Summary

In conclusion, identifying absenteeism patterns through cluster analysis is a valuable approach for organizations to understand and address employee absenteeism. By utilizing unsupervised machine learning techniques such as k-means and hierarchical clustering, organizations can group employees based on similar characteristics and identify patterns that may contribute to absenteeism. Analyzing employee absenteeism data, including converting continuous data to categorical data and visualization tools, provides insights into factors that may or may not influence absenteeism. While the accuracy of k-means and hierarchical clustering can vary, they offer a structured way to analyze absenteeism data. They can be improved by refining categories and examining data more closely. Ultimately, these techniques help organizations make informed decisions to reduce absenteeism and improve productivity.

Job or employee satisfaction, an unquantifiable metric, is a positive emotional response. It is a measure of how contented workers are with their jobs. It is important to remember that job satisfaction varies from employee to employee. In the same workplace under the same conditions, the factors that help one employee feel good about their job may not apply to another employee. It is essential to have a multidimensional approach for measuring employee satisfaction. The parameters that should be considered for measuring job satisfaction include the challenging nature of work, flexible hours, regular appreciation, competitive pay, nature of the job, supervisor attitude, etc. This article focuses on determining the association of different levels of job satisfaction with different parameters. Association rule mining is adopted using the apriori algorithm to assess job satisfaction associations. The techniques discussed can be used to determine the association of different parameters according to the requirements of the HR department, such as attrition, performance levels, promotion, quality of work, employee attitude, etc.

Unsupervised Machine Learning

Unsupervised machine learning algorithms are used when the output is unknown, and no predefined instructions are available to the learning algorithms. In unsupervised learning, the learning algorithm only has input data, and knowledge is extracted from these data. These algorithms create a new representation of the data that is easier to comprehend than the original data and help improve the accuracy of advanced algorithms by consuming less time and reducing memory. Common unsupervised machine learning algorithms include association rule mining, dimensionality reduction algorithms, and clustering.

Association Rule

Association rule mining is a data-mining technique that originated in the field of marketing and, more recently, has been used effectively in other fields, such as human resource management, organizational behavior, entrepreneurship, strategic management, bioinformatics, nuclear science, geophysics, etc. The goal is to identify relationships (i.e., association rules) between groups of products, items, factors, or categories, leading to insights into substantive management domains. The association rule mining is predominantly done using the apriori algorithm.

Association rules can be considered an IF-THEN relationship. Suppose factor A is the employee’s choice, and then the chances of factor B being considered by the employee are determined. These are two elements of these rules:

• Antecedent (IF): This factor is generally related to employee information, such as evaluation score, number of projects, training programs, work accident, department, etc.

• Consequent (THEN): This comes along as a factor with an antecedent/group of antecedents. For example, job satisfaction is the consequence considering the number of projects, evaluation score, etc., as antecedents. Thus, many factors can be considered to be associated with the consequent (job satisfaction).

Job Satisfaction Data

The JobSatisfaction.csv file helps us understand KNIME usage. This dataset can be downloaded from the “Data” folder on the KNIME Community Hub.

The Evaluating Job Satisfaction – Association Rule Mining workflow can be downloaded from the KNIME Community Hub.

You are ready after downloading the CSV file from the Data folder.

Reading the CSV-based dataset

The first step in our analysis will be to load the data for our exploratory analyses. We will do this first step using the CSV Reader node before we persist our analysis in a KNIME table.

The KNIME table is created by loading the JobSatisfaction.csv CSV dataset. The above table shows that the employee dataset has 14999 observations and 10 columns.

Data Visualization

We will use the Python View node to generate multiple count plots for the LeftCompany, Salary, TimeSpendCompany, PromotionLast5years, NumberProject, and WorkAccident categorical columns.

import knime.scripting.io as knio

# Visualization of employee details
# Importing libraries
import seaborn as sns
import matplotlib.pyplot as plt
from io import BytesIO

# Plotting 
fig = plt.figure(figsize=(12, 10)) 

#Count plot for displaying number of employees who left/stayed
plt.subplot(231) 
sns.countplot(x='LeftCompany', data=knio.input_tables[0].to_pandas(), hue='LeftCompany', legend=False, palette="viridis") 
plt.title("Left Company")

#Count plot for displaying number of employees with different salaries
plt.subplot(232)
sns.countplot(x='Salary', data=knio.input_tables[0].to_pandas(), hue='Salary', legend=False, palette="mako")
plt.title("Salary-wise Employees")

#Count plot according to number of years spent in organization
plt.subplot(233)
sns.countplot(x='TimeSpendCompany', data=knio.input_tables[0].to_pandas(), hue='TimeSpendCompany', legend=False, palette="vlag")
plt.title("Time Spent")

#Count plot for displaying number of employees according to promotion
plt.subplot(234)
sns.countplot(x='PromotionLast5years', data=knio.input_tables[0].to_pandas(), hue='PromotionLast5years', legend=False, palette="rainbow")
plt.title("Employees Promotions")

#Count plot for displaying number of employees for number of projects
plt.subplot(235)
sns.countplot(x='NumberProject', data=knio.input_tables[0].to_pandas(), hue='NumberProject', legend=False, palette="terrain")
plt.title("Employees by Number of Projects")

#Count plot for displaying number of employees for work accident
plt.subplot(236)
sns.countplot(x='WorkAccident', data=knio.input_tables[0].to_pandas(), hue='WorkAccident', legend=False, palette="prism")
plt.title("Work Accident")

# Create buffer to write into
buffer = BytesIO()

# Create plot and write it into the buffer
fig.savefig(buffer, format='svg')

# The output is the content of the buffer
output_image = buffer.getvalue()

# Assign the figure to the output_view variable
knio.output_view = knio.view(fig)  # alternative: knio.view_matplotlib()

We can observe from the chart that employees leaving, promotion, and work accidents are binary variables. Nearly 30% of the employees have left the organization; very few employees have been given promotions in the organization in the last 5 years, and accidents have rarely happened at the workplace. More employees belonged to the low-salary group, more employees spent 3 years in the organization, and a maximum number of employees had three or four projects.

We will use the Bar Chart node to generate the occurrence count plot for the categories in the Department column.

The chart shows that the sales department had the most employees, followed by the technical and support departments. The accounting, HR, and management departments had the fewest employees.

From the output of the Statistics node, we can observe that

• Two columns SatisfactionLevel (92 unique values) and LastEvaluation (65 unique values), are continuous,

• Three columns, WorkAccident, LeftCompany, and PromotionLast5years, are binary,

• Two columns, Department (10 categories) and Salary (3 categories), are categorical,

• Three columns, AverageMonthlyHours (215 unique values), NumberProject (6 unique values), and TimeSpendCompany (8 unique values) are integers.

Converting Continuous Columns to Categorical Using Binning

Since the apriori algorithm works on categorical columns, converting each column into a categorical one is essential. We will use the Auto-Binner node to convert the SatisfactionLevel, LastEvaluation, and AverageMonthlyHours columns into categorical columns by splitting them into three equal bins based on the frequency of the observations in the column. Subsequently, we will use the Rule Engine node to assign meaningful category values (low, avg, high) to columns for easy analysis and interpretation.

Dummy Encoding

Note that the apriori algorithm only accepts Boolean values. Hence, converting each category into different columns represented by binary values (0, 1) is essential. We will use the One to Many node to transform all possible category values in the selected columns into a new column. It is worth noting that the number of new columns created for a particular column is the same as the number of categories in the column. Subsequently, we will use the Column Filter node to retain only the necessary columns for further analysis.

The resultant dataset now has 14999 observations and 40 columns; all added columns have Boolean (0,1) values.

Apriori Algorithm for Association Rule Mining

Using the Python Script node, the apriori algorithm is executed considering the apriori() and association_rules () functions available in the “mlxtrend” library. The apriori algorithm helps to determine the frequent factors based on their minimum support and maximum length. These factors are considered input to the association_rules() function with some conditions, and the data are filtered. The support and length of the factors play an essential role in the apriori algorithm.

Syntax of apriori()

Apriori(data, use_colnames=, min_support=, max_len=)

where

• data represents the data on which the algorithm is executed

• use_colnames has a Boolean value representing whether colnames are considered or not

• min_support is the minimum support required and has a value between 0 and 1

• max_len specifies the maximum number of factors that will be considered

Syntax of association_rules()

Association_rules(freq_fact, metric=, min_threshold=)

where

• freq_fact are frequent factors that are determined

• metric specifies the measurement that needs to be considered

• min_threshold specifies the threshold value that should be considered

The code apriori(df, use_colnames=True, min_support=0.1, max_len=3) considers minimum support as 0.1 and maximum length of factors as 3. This means that until and unless the frequency factors are not 10%, these factors will not be considered for further analysis. This is a correct approach because considering factors that do not have an association is meaningless. At the same time, it is not preferred to have 500 factors left after filtering based on support. The lift metric is used as the filter criterion to avoid such cases. This is done using the association_rules() function, considering metric as a lift and minimum threshold as 1. Subsequently, we will use the Sorter node to order the resultant association rules in descending order of “confidence”.

Identify the Association of Low Satisfaction Using Apriori Algorithm

Objective: To identify the association of low satisfaction level with different factors

We will determine the association between low satisfaction level and the factors of association. It is worth noting that the rule has more impact if there is more lift. The higher the value of the lift, the more the association. We will use the Row Filter node with the consequents column equal to “Low_SatisfactionLevel” as the filter criterion.

A lift value greater than one means that all the antecedents displayed in different records are associated with the consequent (Low_SatisfactionLevel). Also, a confidence greater than 0.5 indicates that the occurrence of the records is greater than 50%. The first record indicates that the combination of a number of projects equal to 2 and low average monthly hours is associated with a low satisfaction level. The results also suggest that the number of projects is equal to 2, the time spent in the company is 3 years, and the last evaluation is low, which are some critical factors for low satisfaction. Thus, to increase satisfaction, the organization should change the number of projects and design unique strategies for employees who have completed 3 years. Steps should also be considered for employees whose last evaluation score is low.

Identify the Association of Average Satisfaction Using Apriori Algorithm

Objective: To identify the association of average satisfaction level with different factors

We will determine the association between average satisfaction level and the factors of association. As the filter criterion, we will use the Row Filter node with the consequents column equal to “Avg_SatisfactionLevel.”

A lift value greater than one means that all the antecedents displayed in different records are associated with the consequent (Avg_SatisfactionLevel). Also, a confidence greater than 0.4 indicates that the occurrence of the records is greater than 40%. The first record indicates that the number of projects equal to 3 is highly associated with an average satisfaction level. The results also suggest that the number of projects is equal to 3, there has been no promotion in the last five years, and the monthly hours belong to the average category, which are some critical factors for average satisfaction. Thus, to increase satisfaction, the organization should change the number of projects and design unique strategies for employees who have not been promoted. Steps should also be considered for employees whose monthly hours are average.

Identify the Association of High Satisfaction Using Apriori Algorithm

Objective: To identify the association of high satisfaction level with different factors

We will determine the association between high satisfaction level and the factors of association. As the filter criterion, we will use the Row Filter node with the consequents column equal to “High_SatisfactionLevel.”

A lift value greater than one means that all the antecedents displayed in different records are associated with the consequent (High_SatisfactionLevel). Also, a confidence greater than 0.38 indicates that the occurrence of the records is greater than 38%. The first record indicates that the number of projects equals four and that no workplace accidents are highly associated with high satisfaction. The results also suggest that the number of projects is equal to 4, the last evaluation score is high, and there are no workplace accidents, which are some critical factors for high satisfaction. Thus, to increase satisfaction, the organization should change the number of projects to four and formulate strategies to ensure the last evaluation score is high and no workplace accidents.

Summary

In conclusion, evaluating job satisfaction through association rule mining provides valuable insights into the factors influencing employee contentment. By utilizing the apriori algorithm, organizations can identify key associations between job satisfaction levels and various workplace parameters. This analysis helps in understanding the critical factors that contribute to low, average, and high satisfaction levels among employees. By addressing these factors, organizations can implement targeted strategies to enhance job satisfaction, leading to improved employee retention, performance, and overall workplace morale. The approach outlined in this article offers a systematic method for HR departments to assess and improve job satisfaction, ultimately contributing to a more positive and productive work environment.

Cost in an organization is always a significant concern, and the management always wants to predict the cost before deciding on a future course of action. This article focuses on predicting cost using different time series models. Different models are used for predicting cost, such as autoregressive, moving average, and autoregressive integrated moving average (ARIMA) techniques, which are explained with examples. These techniques are used for any time series data existing in the organization. However, the methods discussed in this article can be used to extract different information according to an organization's requirements from any dataset in the human resource department.

HR Cost

HR cost is a key component of HR accounting. HR costs are mainly incurred from expenses related to salary, incentives, reimbursements, insurance, and fixed and miscellaneous costs. This includes total sot per hired employee across an entire company, the total compensation per employee, the total recruitment expenditures for every new hire across a particular time frame, expenses on job boards to draw valuable conclusions, the salary of recruiters, funds required to establish an employer brand like attending recruiting events, content writing, designing posters, and videos, social media, cost for partnerships with universities and institutions, cost of external recruiting agencies, recruiting technology costs like video interviewing tools, coding assessment tools.

Cost optimization continues to be a critical concern for many HR leaders. However, the HR professional should use different metrics that can be useful for determining an accurate assessment of total cost. For example, the quality of hire can be determined by comparing cost per hire and long-term performance evaluation. Different measures that an organization can take to reduce cost include restructuring the service delivery mechanism, incorporating flexibility for ensuring targeted utilization of capacities, redesigning processes to balance service delivery efficiency and effectiveness, developing the existing capacities to optimize services, balancing elements of the rewards model for effective cost optimization; structuring the workforce to align short-term business needs with long-term value; and achieving cost optimization effects through thoughtful layoff planning and execution.

Measure and forecast costs based on previous values are essential for smooth organizational functioning and effective strategy formulation.

Time Series Modeling

Forecasting helps formulate strategies in various businesses and is a basic need for managerial decision-making. Managers have to make decisions in uncertainty without knowing what will happen. Different methods, including qualitative and quantitative models, can obtain forecasting. Quantitative models include time series models and causal models. Causal models are used when one variable is dependent on the values of other variables. Time series data provide essential information related to time, and time series models attempt to predict the forecast demand (future values) using the past demand values (historical data). A time series is a series of data points in which each data point is associated with a timestamp. They play a significant role in understanding many details on specific factors over time. For example, stock price at different points of time on a given day, amount of sales in a region to varying months of the year.

Time series analysis can be used in many business applications to forecast a quantity into the future and explain its historical patterns. The ARIMA model is commonly used for time series forecasting. Exponential smoothing models were based on the data's description of trend and seasonality. At the same time, ARIMA models describe the autocorrelations in the data. Notably, every model is nonstationary, while ARIMA models can be stationary. ARIMA models use historical information to make predictions, which is a basis for complex models. ARIMA models support either no exponential or linear exponential smoothing but not nonlinear exponential smoothing. ARIMA is like a linear regression equation where the predictors depend on some parameters of the ARIMA model.

ARIMA models are a class of statistical models for analyzing and forecasting time series data. They explicitly cater to a suite of standard structures in time series data and, as such, provide a simple yet powerful method for making skillful time series forecasts. ARIMA generalizes the more straightforward autoregressive moving average and adds integration.

HR Cost Data

The HRCost file helps us understand KNIME usage. This dataset can be downloaded from the “Data” folder on the KNIME Community Hub.

The Forecast HR Cost – Time Series Modeling workflow can be downloaded from the KNIME Community Hub.

You are ready after downloading the CSV file from the Data folder.

Reading the CSV-based Dataset

The first step in our analysis will be to load the data for our exploratory analyses. We will do this first step using the CSV Reader node before we persist our analysis in a KNIME table.

The KNIME table is created by loading the HRCost.csv CSV dataset. The above table shows that the employee dataset has 144 observations and 3 columns.

Filter / Subset observations for the HR Department

We will consider a subset of the dataset for our preliminary data exploration and visualization. Based on the MonthDate column, we can use the Date&TimeBased Row Filter node to filter out the Jan 2016 to Dec 2019 observations. After that, we will use the Line Plot node to display the monthly expenses from the Cost column in a line chart.

Testing Stationarity of Time Series Data

It is essential to check the stationarity of the series before doing forecasting. A time series is considered stationary if it does not have a trend or seasonal effects and this has consistent statistical properties over time. These properties include mean, variance, and autocovariance. Before applying statistical modeling methods, the time series is required in stationery. This is necessary for easy modeling and proper series forecasts because it should be the same if we take a specific behavior over time. Observations from a nonstationary time series show seasonal effects, trends, and other structures that depend on the time index. We can determine whether the tied series is stationary or nonstationary by looking at plots and checking trends or seasonality or by using statistical tests such as the Dickey-Fuller test.

We will use the statsmodels.tsa.stattools package in the Python Script node to apply the Dickey-Fuller test. The test's null hypothesis is that a unit root can represent the time series. This means that it is nonstationary and has some time-dependent structure. The alternative hypothesis (rejecting the null hypothesis) is that the time series is stationary. It suggests the time series does not have a unit root and does not have a time-dependent structure. We interpret this result using the p-value from the test. If the p-value is less than or equal to 0.05, it suggests we reject the null hypothesis. This means the data are stationary and do not have a unit root. If the p-value is more significant than 0.05, we fail to reject the null hypothesis, and the data have a unit root and are nonstationary. For practical analysis, it is also recommended that the rolling mean of the series be created and plotted.

import knime.scripting.io as knio

# Importing necessary libraries
from statsmodels.tsa.stattools import adfuller
import pandas as pd

# Load the input table as a pandas DataFrame
df = knio.input_tables[0].to_pandas()

# Ensure the series is extracted correctly
if "Cost" in df.columns:
    series = df["Cost"]
else:
    raise ValueError("The column 'Cost' is not found in the input table.")

# Applying Dickey-Fuller Test
# Null hypothesis: The time series is nonstationary
result = adfuller(series)

# Prepare the results of the Dickey-Fuller test
output_data = {
    "ADF Statistic": [result[0]],
    "p-value": [result[1]],
    "Critical Value 1%": [result[4]["1%"]],
    "Critical Value 5%": [result[4]["5%"]],
    "Critical Value 10%": [result[4]["10%"]]
}

# Convert the results to a pandas DataFrame
output_df = pd.DataFrame(output_data)

# Save the output DataFrame to the KNIME output table
knio.output_tables[0] = knio.Table.from_pandas(output_df)

We will use the Moving Average node to create a backward simple moving average, with 12 windows, of the Cost column. Subsequently, we use the Line Plot node to plot the Cost and MA(Cost) columns.

In our data, the p-value is 0.992, meaning the null hypothesis is accepted. This further implies that our series is nonstationary. This can be further interpreted by looking at the figure, which shows that the mean is increasing. It should be noted that for creating models, it is essential to make the series stationary before developing a model and doing forecasting.

Smoothing the Time Series Data

When the time series data have significant irregular components, we want a smooth curve to reduce these fluctuations. Curve smoothing is generally achieved using a simple moving average method, which is the best method to accomplish this using the Moving Average node. To roll the mean, we first take some (k) consecutive values and define a parameter k value. Generally, as the value of k increases, the plot becomes increasingly smoothed. The major challenge is to find the appropriate value of k that highlights the significant patterns in the data without under or over-smoothing. In our data, the value of k depends on the frequency. Since it is one year, we will consider the value of k as 12. For our Cost data, we will use natural logarithmic transformation using the ln() function of the Math Formula node to get the LnCost column because there is a strong positive trend. Subsequently, we use the Line Plot node to plot the LnCost and MA(LnCost) columns.

We can observe that the MA(LnCost) starts from Dec 2009. This is because we take the average of the last 12 values, and the rolling mean is not defined for the first 11 months. Thus, the first 11 months will have missing values. Since we cannot analyze missing values, dropping these rows using the Row Filter node is essential. Subsequently, we will find the difference between the two columns, LnCost and MA(LnCost), using the Math Formula node to create the Ln_MA_Diff column.

We will now determine the series' stationarity by performing the Dickey-Fuller test on the Ln_MA_Diff column and plotting its moving average series using the Line Plot node.

import knime.scripting.io as knio

# Importing necessary libraries
from statsmodels.tsa.stattools import adfuller
import pandas as pd

# Load the input table as a pandas DataFrame
df = knio.input_tables[0].to_pandas()

# Ensure the series is extracted correctly
if "Ln_MA_Diff" in df.columns:
    series = df["Ln_MA_Diff"]
else:
    raise ValueError("The column 'Ln_MA_Diff' is not found in the input table.")

# Applying Dickey-Fuller Test
# Null hypothesis: The time series is nonstationary
result = adfuller(series)

# Prepare the results of the Dickey-Fuller test
output_data = {
    "ADF Statistic": [result[0]],
    "p-value": [result[1]],
    "Critical Value 1%": [result[4]["1%"]],
    "Critical Value 5%": [result[4]["5%"]],
    "Critical Value 10%": [result[4]["10%"]]
}

# Convert the results to a pandas DataFrame
output_df = pd.DataFrame(output_data)

# Save the output DataFrame to the KNIME output table
knio.output_tables[0] = knio.Table.from_pandas(output_df)

The p-value of 0.009 is less than 0.05. This means that the series is stationary at the 95% confidence interval. We can also observe from the visual representation that the rolling values vary slightly, but there is no specific trend. Hence, we can assume the series' stationarity. Thus, in the following sections, we will consider the data after the natural logarithmic transformation LnCost for developing the ARIMA model.

Forecast HR Cost Using Auto-Regressive (AR) Model

Objective: To forecast the HR cost using the Auto-Regressive (AR) model

The AR model is created by considering the ARIMA function's third argument (moving average) to be zero. However, the value of the first argument denotes the lag value. For example, the ARIMA (5,1,0) creates an AR model, sets the lag value to 5 for autoregression, uses a difference order of 1 to make the time series stationary, and uses a moving average value of 0. Since the moving average component is 0, it can be considered as only an AR model.

Using the maximum likelihood estimation method, we will use the ARIMA Learner node to develop an AR (2,1,0) model of lag two on the LnCost column. We will then use the ARIMA Predictor node to apply the regression model and obtain fitted values in the “In-sample predictions” output table. Based on this model, we can get the subsequent five-month forecasted cost CostForecast column by using the forecast option in the ARIMA Predictor node. The “Forecast” output shows the result in standard error and forecast values. Considering the forecasted values, the management of the organization can, thus, frame strategies and take decisions accordingly.

It should be noted that the fitted/forecast values need to be brought back to their original scale to predict actual values. The exp() function of the Math Formula node helps to determine the correct CostFitted values from the natural logarithmic LnCost(fitted values) values. It is essential to decide on the error of the developed model. This can be determined by observing the difference between the original and predicted values. The root mean squared function is one of the methods that can be used to determine the error (RMSE). Using multiple data manipulation nodes, the error in the developed AR model is 31.387. This means that there is a difference in the original and predicted values. It should be noted that the lower the value, the less the difference between the original and predicted values.

A visual display of the original Cost vs predicted CostFitted values uses the Line Plot node.

Forecast HR Cost Using Moving Average (MA) Model

Objective: To forecast the HR cost using the Moving Average (MA) model

The MA model is created by considering the ARIMA function's first argument (autoregressive) to be zero. However, the value of the first argument denotes the lag value. For example, the ARIMA (0,1,3) creates an MA model and sets the lag value to 3 for the moving average, uses a difference order of 1 to make the time series stationary, and uses an autoregressive value of 0. Since the autoregressive component is 0, it can be considered as only an MA model.

Using the maximum likelihood estimation method, we will use the ARIMA Learner node to develop an MA (0,1,2) on the LnCost column. We will then use the ARIMA Predictor node to apply the regression model and obtain fitted values in the “In-sample predictions” output table. Based on this model, we can get the subsequent five-month forecasted cost CostForecast column by using the forecast option in the ARIMA Predictor node. The “Forecast” output shows the result in standard error and forecast values. Considering the forecasted values, the management of the organization can, thus, frame strategies and take decisions accordingly.

It should be noted that the fitted/forecast values need to be brought back to their original scale to predict actual values. The exp() function of the Math Formula node helps to determine the correct CostFitted values from the natural logarithmic LnCost(fitted values) values. It is essential to decide on the error of the developed model. This can be determined by observing the difference between the original and predicted values. The root mean squared function is one of the methods that can be used to determine the error (RMSE). Using multiple data manipulation nodes, the error in the developed AR model is 33.149. This means that there is a difference in the original and predicted values. It should be noted that the lower the value, the less the difference between the original and predicted values.

A visual display of the original Cost vs predicted CostFitted values uses the Line Plot node.

Forecast HR Cost Using ARIMA Model

Objective: To forecast the HR cost using the ARIMA model.

The ARIMA model is created by considering the first and third arguments as nonzero. For example, the ARIMA (3,1,3) creates an ARIMA model, sets the lag value to 3, and uses a difference order of 1 to make the time series stationary.

Using the maximum likelihood estimation method, we will use the ARIMA Learner node to develop an ARIMA (2,1,2) on the LnCost column. We will then use the ARIMA Predictor node to apply the regression model and obtain fitted values in the “In-sample predictions” output table. Based on this model, we can get the subsequent five-month forecasted cost CostForecast column by using the forecast option in the ARIMA Predictor node. The “Forecast” output shows the result in standard error and forecast values. Considering the forecasted values, the management of the organization can, thus, frame strategies and take decisions accordingly.

It should be noted that the fitted/forecast values need to be brought back to their original scale to predict actual values. The exp() function of the Math Formula node helps to determine the correct CostFitted values from the natural logarithmic LnCost(fitted values) values. It is essential to decide on the error of the developed model. This can be determined by observing the difference between the original and predicted values. The root mean squared function is one of the methods that can be used to determine the error (RMSE). Using multiple data manipulation nodes, the error in the developed AR model is 34.5. This means that there is a difference in the original and predicted values. It should be noted that the lower the value, the less the difference between the original and predicted values.

A visual display of the original Cost vs predicted CostFitted values uses the Line Plot node.

Summary

In conclusion, forecasting HR costs using time series modeling is valuable for organizations aiming to optimize their financial planning and decision-making processes. By employing models such as autoregressive (AR), moving average (MA), and autoregressive integrated moving average (ARIMA), organizations can effectively predict future HR expenses based on historical data. This enables management to make informed strategic decisions, align resources with business objectives, and implement cost-saving measures. The process involves ensuring data stationarity, applying appropriate smoothing techniques, and selecting the best-fitting model to achieve accurate forecasts. Ultimately, these predictive insights contribute to more efficient HR operations and overall organizational success.

The HR department organizes training and development programs for employees. These programs instill a global mindset in budding talent and help them acquire comprehensive business knowledge to enhance their performance and deliver better values to clients. Moreover, development programs prepare future business leaders to develop leadership in thoughts and actions, besides grooming them to lead big and small teams efficiently at different levels.

The compare means technique is generally used to compare the means of continuous variables for different categories of categorical variables. The compare means techniques can be used to compare the means of continuous variables such as salary, performance score, recruitment expenses, satisfaction level, and general expenses of employees with different categorical variables such as demographic variables (gender, age, income group) and other variables such as departments, region, manager name, and recruitment sources. Specific examples include a comparison of expenses from different recruitment sources, a comparison of salary from different departments, regions, age groups, etc.; a comparison of performance scores from different age and gender groups; a comparison of general expenses from the different areas, a comparison of employee satisfaction level concerning different managers, etc. This article, however, focuses on measuring the effectiveness of training programs according to different demographic variables; besides, compare means techniques like independent sample t-test, ANOVA, etc., are also used.

Compare Means

A common problem in research is comparing the central tendency of one group to a value or another group or groups. A test compares sample means to see if sufficient proof exists to predict that the means of the corresponding population distributions also differ. Standard statistical tools for assessing these comparisons are t-tests, analysis of variance, and general linear models. Parametric techniques are used when some assumptions are not met. The general goal for most of these tools is to use the estimate of the means (or another central measure), assess the variation based on sample estimates, and use this information to provide the amount of evidence of a difference in means or central tendency.

T-test and analysis of variance (abbreviated as ANOVA) are two parametric statistical techniques used to test the hypothesis when the dependent variable is continuous and the independent variable is categorical. The sample is taken from different populations. The different samples are measured on some variable of interest. For example, a t-test will determine if the means of the two sample distributions differ significantly. At the same time, ANOVA will decide if the means of more than two sample distributions vary considerably from each other. Both these tests are based on the three common assumptions:

1. The sample drawn from the population should be normally distributed,

2. There should be homogeneity of variances and

3. The independence of observations should exist.

There is a thin line of demarcation between the t-test and ANOVA. When the categorical variable has two groups, the t-test is used, while ANOVA is preferred if there are more than two groups. Thus, when the population means of only two groups are to be compared, the t-test is used, but when the means of more than two groups are to be compared, ANOVA is preferred. Three common assumptions need to be fulfilled before applying a t-test or ANOVA. However, it should be noted that a one-sample t-test should fulfill only the normality assumption since only one sample is involved.

1. Normally distributed: Non-normally distributed variables (highly skewed or kurtotic variables or with substantial outliers) can distort relationships and significance tests. Before applying a t-test or ANOVA, it is assumed that variables are normally distributed (symmetric bell-shaped distribution).

2. Independent samples: If we randomly sample each set of items separately, under different conditions, the samples are independent. The measurements in one sample have no bearing on the measurements in the other sample. For example, consider randomly sampling two groups of people based on gender to test their online shopping behavior. Let's take one random sample from a group of males and record their perception and another sample from a group of females and record their perception. We know that the measurement in one sample does not affect the other sample.

3. Homogeneity of variances: The assumption of homogeneity of variance is an assumption of the independent samples t-test and ANOVA, which states that the variance within each population is equal. However, the independent samples t-test and ANOVA can handle this assumption if the group sizes are equal. Equal group sizes may be defined by the ratio of the largest to the smallest group being less than 1.5. If group sizes are vastly unequal, the homogeneity of variance is violated, then the result will be biased when significant sample variances are associated with small group sizes. When this occurs, the significance level will be underestimated, which can cause the null hypothesis to be falsely rejected. The result will also be biased in the opposite direction if significant variances are associated with large group sizes. This would mean that the significance level will be overestimated. This does not cause the same problems as falsely rejecting the null hypothesis; however, it can cause a decrease in the power of the test.

A test is a process for comparing sample means of different groups. While comparing the means of different groups, we frame a hypothesis in which the dependent variable is continuous, and the independent variable is categorical. For example, in determining the effect of gender on job satisfaction, gender (a categorical variable) will be considered the independent variable, and job satisfaction will be viewed as the dependent variable.

Training Score Data

The TrainingScore.csv file helps us understand KNIME usage. This dataset can be downloaded from the “Data” folder on the KNIME Community Hub.

The Evaluate Training Effectiveness - Comparative Statistics workflow can be downloaded from the KNIME Community Hub.

You are ready after downloading the CSV file from the Data folder.

Reading the CSV-based Dataset

The first step in our analysis will be to load the data for our exploratory analyses. We will do this first step using the CSV Reader node before we persist our analysis in a KNIME table.

The KNIME table is created by loading the TrainingScore.csv CSV dataset. The above table shows that the employee dataset has 220 observations and 5 columns.

Data Exploration and Visualization

We will now try to understand the dataset related to organizational training. The training dataset is created based on surveys conducted before and after training is conducted in an organization.

There are 220 observations and five columns. PostTrainingScore and PreTrainingScore are continuous columns, while Gender, Age, and Department are categorical.

The occurrences table from the Statistics node shows four unique categories for the Department column: ‘Finance’, ‘Human Resource’, ‘Marketing’, and ‘Operations’. Similarly, there are three Age groups: ’21-30’, ’41-60’, and ’31-40’, and two gender groups: ‘Male’ and ‘Female’.

Visualizing Categorical Columns

We will use the Bar Chart node to generate a count plot for the Department, Age, and Gender columns. The results clearly show that there are more female employees than male employees, most of the employees belong to the 41-60 age group, and all the departments have approximately the same number of employees.

Visualizing Continuous Columns

We will use the Line Chart node to display the PostTrainingScore and PreTrainingScore columns. The results show that the minimum scores of an employee pretraining and post-training are 10 and 12, respectively. The maximum scores of an employee pretraining and post-training are 17 and 20, respectively. The mean scores of employees pretraining and post-training are 14.13 and 16.31, respectively.

Compare Training Effectiveness Using One Sample T-Test

Objective: To determine the difference between the training score of employees and the desired score

The t-test is a statistical test that compares the sample mean of one group with a standard value. Since only one sample is involved, only the normality assumption must be fulfilled for this test.

We consider a situation in which an organization has conducted a training program for its employees. It records the scores of employees before and after training. The maximum score is considered to be 20. The organization wants its employees to score 20 after the training. The organization wants to know if there is any significant difference between the actual score of the employees after training and the desired maximum score of 20.

We will first display the observations of the PostTrainingScore column using the Histogram node with 10 bins.

The chart shows that the score after training ranges mainly between 12 and 20. The maximum number of employees achieves a score of 16-18. Skewness and kurtosis functions measure the normality of the data. The data are considered normal if skewness and kurtosis coefficients lie between -1 and +1. Since the skewness value is -0.242 and that of kurtosis is -0.63, as obtained from the Statistics node, we can say that the data are normal.

The probability plot is generated for observations in the PostTrainingScore column using the Continuous Probability Distribution component or Python View node using the “scipy.probplot” package. The chart shows that the data can be assumed normal since nearly all the dots lie on the line.

We will use the Single-sample t-test node to perform a one-sample t-test on the PostTrainingScore column with a test value of 20.

Since the normality assumption has been met, we can apply the test to the data. To determine whether the two distributions differ significantly from each other, the test that measures the probability associated with the difference between the groups may be either a one-tailed test or a two-tailed test of significance. The test examines whether the mean of one distribution differs significantly from the standard value. We reject the null hypothesis since the p-value equals 0.000, less than 0.05. This means that the value does vary substantially from the value 20.

This means that the employees' scores are significantly different from the 20. Thus, the training could not show a 100% score for all the employees. Based on the results, the organization can design strategies and act accordingly.

Compare Training Effectiveness Using Independent T-Test

Objective: To determine the difference between training scores of employees of different genders.

The t-test helps compare the population means of only two groups. Thus, it can be conducted to examine whether the population means of the two samples significantly differ. We will evaluate whether there is any difference between the scores of male and female employees.

We will use the Independent group's t-test node to perform the test on the PostTrainingScore column with Gender as the grouping column. We will also use the Density Plot node to create an interactive plot with PostTrainingScore as the dimension column and Gender (male and female) as the condition column.

We assume the data we have taken in the example above are independent. Thus, the assumption of independent samples is fulfilled. We use Levene’s test to check the homogeneity of variances. If the p-value of the test is greater than 0.05, the assumption of homogeneity of variance is met. From the preceding output, we can see that the p-value is more than the significance level of 0.05 using Levene’s test. This means that the variance across groups is statistically insignificant. Therefore, we assume the homogeneity of variances (equal variances assumed) in the different groups.

A null hypothesis is rejected if the p-value is less than 0.05 at a 5% significance level. Since, in our example, the p-value (0.1842) is greater than 0.05, the null hypothesis is not rejected. This means that the actual difference in the means of male and female groups is equal to 0, which implies that there is no significant difference in the scores of male and female groups.

Compare Training Effectiveness Using Dependent T-Test

Objective: To determine the difference between the scores of employees obtained before and after training.

A dependent (paired) t-test is used for dependent samples. If we collect two measurements on each item, person, or experimental unit, each pair of observations is closely related or matched. In this scenario, we apply the paired t-test using the Paired t-test node to compare the pre-training and post-training scores.

In our use case, we try to understand whether any significant difference occurred after the training was given. The columns PreTrainingScore and PostTrainingScore represent the scores of employees before and after the training. Since the p-value is 0.000, which is less than 0.05, we failed to reject the null hypothesis, which means that there is a significant difference in the scores of the employees after the training. This further implies that training impacts employees and can be considered helpful.

Compare Training Effectiveness Using One-Way ANOVA

Objective: To determine the difference between the training scores of employees belonging to different departments

ANOVA (analysis of variance) is a statistical method commonly used in situations where a comparison is made between more than two population means. In ANOVA, the total variation in a dataset is split into two types: the amount allocated to chance and the amount assigned to particular causes. Its basic principle is to test the variances among population means by assessing the variation within group items proportionate to the variation between groups. It should be noted that within the sample, the variance exists because of the random, unexplained disturbance. However, different reasons may cause a variance between the samples. Using this technique, we test the null hypothesis (H0), wherein all population means are the same, or the alternative hypothesis (H1), wherein at least one population mean is different.

We will now evaluate whether there is any significant difference between employees' scores in different departments.

We will first use the Color Manager node to assign specific colors for the categories in the Department column. Subsequently, we will use the Violin Plot (Plotly) node to create an interactive plot based on the PostTrainingScore grouped by Department.

We will use the One-way ANOVA node to test the PostTrainingScore column with Department as the grouping column.

The violin chart shows that the score is nearly the same for employees belonging to all departments. For the ANOVA test, the data are filtered according to the categorical variable from the primary data set and stored in different groups. According to various departments, different groups, namely ‘Finance’, ‘Human Resource’, ‘Marketing’, and ‘Operations’, are formed. The one-way ANOVA result shows that the p-value is 0.2112; hence, we failed to reject the null hypothesis. This means there is no significant difference between the scores of employees belonging to different departments after training.

Compare Training Effectiveness Using ANOVA with Tukey’s Test

Objective: To determine the difference between the training scores of employees belonging to different age groups

From the previous results of the ANOVA test, we can observe that a significant p-value indicates that some of the group means are different, but we do not know which pairs of groups are different. Multiple pairwise comparisons can also determine if the mean difference between specific pairs of groups is statistically significant. We can apply Tukey HSD (Honest Significant Difference) to perform multiple pair-wise comparisons between the means of groups from the “statsmodel.stats” package.

We will evaluate whether there is any difference between the scores of employees of different age groups. We will use the Python Script node with “statsmodels.stats.multicomp” package to test the PostTrainingScore column with Age as the grouping column. We will also use the Density Plot node to create an interactive plot with PostTrainingScore as the dimension column and Age as the condition column.

For the ANOVA test, the data are filtered according to the categorical variable from the primary dataset and stored in different groups. For example, according to different age groups, different groups, namely, first, second, and third, are formed for various age groups. The plot shows that most of the employees belonging to the 21-30 age group had a score of 18. The one-way ANOVA result from the One-way ANOVA node shows that the p-value is 0.0058; hence, we reject the null hypothesis. This means that there is a significant difference between the three groups.

Tukey’s test determines individual differences between the different groups formed. For Tukey’s test, the various groups are formed automatically after the information on categorical and continuous variables is collected. The function itself forms groups depending on the values of the categorical variable. For example, the dataset has only 21-30, 31-40, and 41-60 age groups. Tukey’s test shows that since there are three groups representing different age groups, three pairs are formed: (21-30 and 31-40), (21-30 and 41-60), and (31-40 and 41-60). The mean difference values for the three groups are -1.207, -0.957, and 0.25, respectively. The result shows that the null hypothesis is rejected for the last group but not for the first two groups.

This means there is no significant difference in the training scores of the people in age groups 31-40 and 41-60. Thus, the employees of these age groups show similar scores after training. However, the employees in age groups 21-30 and 31-40 show different scores.

Summary

In conclusion, evaluating the effectiveness of training programs through comparative statistics is crucial for organizations aiming to enhance employee performance and development. By utilizing statistical techniques such as t-tests and ANOVA, organizations can assess the impact of training across various demographic variables, including gender, age, and department. The analysis of training scores before and after training and comparisons across different groups provides valuable insights into the success of training initiatives. These insights enable organizations to tailor their training strategies to meet desired outcomes and effectively contribute to employee growth and organizational success.

Employee psychology is a branch involved in studying employees' behavior and the cognitive process behind how employees perceive various issues. Understanding employee psychology related to the compensation and benefits plans organizations provide is essential. The HR professional needs to understand the utility of different policies for health benefits, flexible schedules, retirement benefits, performance rewards, etc. The objective is to understand what employees consider the most critical benefits elements by individual, group, or segment. A key outcome is determining the relative importance of these benefits, as they impact attraction, retention, and performance levels, and identifying which elements can be cut back or minimized in building benefit programs.

Conjoint analysis is a statistical technique used in research to determine how people value different attributes (features, functions, benefits) related to any concept. It is used to understand the weight of the various attributes related to any plan. Conjoint analysis is generally used to determine which attributes are essential to a person (employee in HR, customer in marketing) and the importance of each attribute.

Conjoint analysis is a technique used to understand the preference or relative importance of various proposed plans related to any particular area. Four important attributes are considered when designing an adequate compensation and benefits plan: work-life balance, performance reward, retirement benefits, and health care benefits. In this article, we will see how conjoint analysis can be used to answer the following questions:

1. Which type of retirement benefits is preferred by employees?

2. What is employees' preference for flexibility in working hours?

3. Which type of health care benefit is employees’ preference?

4. Which performance reward system is preferred by employees?

Efforts are being made to understand the weightage given by the employees to each of the attributes and the level related to the compensation and benefits plan. An in-depth analysis of the utility of every combination of compensation and benefits is carried out to determine the importance of each combination.

Conjoint Analysis

Conjoint analysis is a valuable approach to measure the value that individuals place on features of a plan or policy. It combines real-life scenarios and statistical techniques with modeling actual decisions. Companies can ask employees to evaluate various factors such as wages, health benefits, flexible schedules, retirement benefits, wellness programs, life insurance, health savings accounts, and medical insurance to implement this approach in rewards.

Conjoint analysis is an excellent tool for answering these types of questions. However, it is essential to understand that conjoint analysis differs from surveys. A survey provides answers related to percentage analysis, such as “47% of employees have a preference for plan A,” “42% want plan B,” and “11% want plan C.”

The objective is to understand what employees consider the most critical benefits elements by individual, group, or segment. A key outcome is determining the relative importance of these benefits, as they impact attraction, retention, and performance levels, and identifying which elements can be cut back or minimized in building benefit programs. These are four types of conjoint analysis:

• Rating-based conjoint: The respondents are asked to rate the alternatives they are shown. This can be on a scale of 0 to 100. They may be required to allocate scores so that the scores sum to a certain number (e.g., all scores in each question must add up to 100).

• Best-worst conjoint: In each question, the respondents are asked to indicate which option is best and which is the worst among three or more alternatives.

• Ranking-based conjoint: The respondents are asked to rank alternatives from best to worst. It is similar to the best-worst scaling, but the respondents must also allocate rankings to the intermediate options.

• Choice-based conjoint: The respondents are asked which option they will choose. However, this is the most theoretically sound, practical, and popular method of conjoint analysis in the marketing domain.

Data Exploration and Visualization

All the employees' data regarding their preference for compensation and benefits were collected. Each employee was asked to give his or her preference for work-life balance for flexible and inflexible hours; for the type of performance rewards related to incentive system, pay plan, and commission; for retirement benefits related to benefits plan and contribution plan; and for health care benefits related to life insurance, medical insurance, and health savings account. Since there were two categories for work-life balance, three for performance rewards, two for retirement benefits, and three for medical benefits, 36 (2x3x2x3) combinations were created.

A conjoint model exposes respondents to combinations described in terms of the same attributes (with varying levels). The employees were asked to rank 36 different compensation and benefits schemes based on their preferences on an ordinal scale from 1 to 10. The rating column corresponds to the average score of all the employees. It should be noted that since there were only 36 different combinations, the employees were requested to evaluate all 36 combinations. However, if there had been more combinations, only the selected and preferred combinations would have been considered for getting the employees' choice.

Compensation Benefits Data

The CompensationBenefits.csv file helps us understand KNIME usage. This dataset can be downloaded from the “Data” folder on the KNIME Community Hub.

The Structuring Employee Rewards Package - Conjoint Analysis workflow can be downloaded from the KNIME Community Hub.

You are ready after downloading the CSV file from the Data folder.

Reading the CSV-based dataset

The first step in our analysis will be to load the data for our exploratory analyses. We will do this first step using the CSV Reader node before we persist our analysis in a KNIME table.

The KNIME table is created by loading the CompensationBenefits.csv CSV dataset. The above table shows that the employee dataset has 36 observations and 5 columns. The names of the five columns are ‘Rating’, ‘WorkLifeBalance’, ‘PerformanceReward’, ‘RetirementBenefits’, and ‘HealthCareBenefits’. It should be understood that the experimental design consists of four attributes: work-life balance, performance reward, retirement benefits, and health benefits. The occurrences table from the Statistics node shows that there are two categories in work-life balance: ‘InflexibleHours’, ‘FlexibleHours’; three categories in performance reward: ‘IncentiveSystem’, ‘PayPlan’, ‘Commission’; two categories in retirement benefits: ‘BenefitPlan, ‘ContributionPlan’; and three categories in health benefits: ‘LifeInsurance’, ‘MedicalInsurance’, ‘HealthSavingAccount’.

The dataset has 36 combinations, showing the rating for every combination. The highest rating is given to the scheme that has flexible hours and commission for performance rewards. However, retirement benefits can be either a benefit plan or a contribution plan, and health care benefits can be a health savings account or medical insurance.

Measure Part-Worth Utilities of Each Level Using Conjoint Analysis

Objective: To determine the importance of each level using conjoint analysis

Conjoint analysis is carrying out linear regression where the target variable could be binary (choice-based conjoint analysis) 1-10 Likert scale (rating conjoint analysis), or ranking (rank-based conjoint analysis). The part-worth utilities measure both attribute importance score and level preference score. The beta coefficient of the regression equation helps to calculate the part-worth utility (PWU) of the variants of variables for the attribute importance scores. It should be noted that large part-worth utilities are assigned to the most preferred levels, and small part-worth utilities are assigned to the least preferred levels. In this section, the importance of every predictor variable is determined using the ordinary least squares regression method considering rank-based conjoint analysis.

Part-worth utilities (conjoint analysis utilities) measure attribute importance and level values. The utilities are numerical scores that measure how much each feature influences the decision to make that choice. It should be noted here that the part-worth utilities allow us to have a deeper understanding of which specific features within an attribute influence the choice. Strongly preferred levels are assigned higher scores compared to the less preferred ones.

In this section, conjoint analysis is used to assign utility values for each sublevel of each attribute: performance reward, health care benefits, retirement benefits, and work-life balance. It should be noted that the sublevel getting the highest utility value is the one that the employee most favors for that attribute.

It is known that the ordinary least squares (OLS) model cannot be created on categorical variables. Also, the OLS model works well on binary data; hence, binary dummy data are made using the One to Many node. This will create new independent variables for all the categories of independent variables.

The One to Many node creates new columns according to the number of categories for each variable. Three columns are made since there are three categories for health care benefits (health savings account, life insurance, and medical insurance). Similarly, three new columns are designed for performance rewards, and two are for retirement benefits and work-life balance. The result clearly shows that there are 36 observations with 15 columns. Among these, 10 new columns have binary integer values.

In conjoint analysis, one category for each categorical variable is considered a reference, and others are considered dummy variables. For this process, the following dummy and reference variables are created for each attribute:

1. “InflexibleHours” is a dummy variable for work-life balance, and “FlexibleHours” is the reference variable.

2. For health care benefits, “LifeInsurance” and “MedicalInsurance” are dummy variables, and “HealthSavingsAccount” is a reference variable.

3. For performance reward, “IncentiveSystem” and “PayPlan” are dummy variables, and “Commission” is a reference variable.

4. “ContributionPlan” is a dummy variable for retirement benefits, and “BenefitPlan” is the reference variable.

Dropping Reference Variables

It is essential to retain only requirement columns for the analysis. Hence, the reference and categorical variables are removed using the Column Filter node before creating the regression model. Rating is considered the dependent variable.

After dropping the unwanted columns, the table has 36 rows and 7 columns.

Creating the Linear Regression Model

The regression model uses the Linear Regression Learner node with six independent variables and considers the Rating column the dependent variable.

Since we have created one variable as a reference for each category, the value of the coefficients will be considered concerning the base variable. Thus, the results can be interpreted as follows:

Coefficients of independent variables from the regression model and their interpretation

It should be understood that the predictor variable's importance depends on the coefficient's value. Large part-worth utilities are assigned to the most preferred levels, and small part-worth utilities are transferred to the least preferred levels. However, since these coefficients are for the base variable, the next step is determining the utilities of each level of the attributes. This is done by selecting the relative utility of each level separately for each attribute.

Utility of Levels of Work-Life Balance

It is clear from the model that the coefficient of InflexibleHours is -2.1984. Thus, the utilities for both the levels of the work-life balance will be:

• Utility of FlexibleHours = 0 – Average(0,-2.1984) = 0-(-1.0992) = 1.0992

• Utility of InflexibleHours = -2.1984 – Average(0,-2.1984) = -2.1984-(-1.0992) = -1.0992

Thus, it is clear that the utility of FlexibleHours + utility of InflexibleHours = 0.

Utility of Levels of Performance Reward

The model shows that the coefficients of IncentiveSystem and PayPlan are 0.7024 and 2.2024, respectively. Thus, the utilities for all the levels of performance reward will be:

• Utility of Commission: 0-Average(0.7024, 2.2024) = 0-0.968 = -0.968

• Utility of IncentiveSystem: 0.7024-Average(0, 0.7024, 2.2024) = 0.7024-0.968 = -0.265

• Utility of PayPlan: 2.2024-Average(0,0.7024,2.2024) = 2.2024-0.968 = 1.234

Thus, it is clear that the utility of PayPlan + the utility of IncentivePlan + the utility of Commission = 0.

Utility of Levels of Retirement Benefits

The model shows that the coefficients of BenefitPlan and ContributionPlan are 0 and 2.690, respectively. Thus, the equations that can be framed for determining the utilities for both the levels of retirement benefits will be:

• Utility of BenefitPlan: 0-Average(0,2.690) = 0-2.690 = -1.345

• Utility of ContributionPlan: 2.690-Average(0,2.690) = 0-2.690 = 1.345

Thus, it is clear that utility of BenefitPlan + utility of ContributionPlan = 0.

Utility of Levels of Health Care Benefits

The model shows that HealthSavingsAccount, LifeInsurance, and MedicalInsurance coefficients are 0, 3.369, and 3.7857, respectively. Thus, the utilities for all the levels of the health care benefits will be:

• Utility of HealthSavingsAccount: 0-Average(0,3.369,3.7857) = 0-2.3849 = -2.3849

• Utility of LifeInsurance: 3.369-Average(0,3.369,3.7857) = 3.369-2.3849 = 0.9841

• Utility of MedicalInsurance: 3.7857-Average(0,3.369,3.7857) = 3.7857-2.3849 = 1.4008

Thus, it is clear that the utility of HealthSavingsAccount + utility of LifeInsurance + utility of MedicalInsurance = 0.

The above calculations are performed using various data manipulation nodes, which are then combined into the Utility Value of Levels meta node.

The resultant table provides the Utility value of each Level of the Attribute.

Utility of All the Levels of Each Attribute

Displaying the Utilities of Each Category

We will use the Python View node to generate a subplot chart displaying the utility of different plans/levels of each attribute.

import knime.scripting.io as knio

# This script plots a histogram using matplotlib and assigns it to the output view
import matplotlib.pyplot as plt
from io import BytesIO

# Creating subplot chart for displaying the utilities of each category
fig = plt.figure(figsize=(20,20))
plt.subplot(221)
plt.bar(["Flexible Hours","Inflexible Hours"],[1.0992,-1.0992])
plt.title("Work Life Balance")
plt.subplot(222)
plt.bar(["Commission","Incentive System","Pay Plan"],[-0.968,-0.266,1.234])
plt.title("Performance Reward")
plt.subplot(223)
plt.bar(["Benefits Plan","Contribution Plan"],[-1.345,1.345])
plt.title("Retirement Benefits")
plt.subplot(224)
plt.bar(["Health Saving Account","Life Insurance","Medical Insurance"],[-2.385,0.984,1.401])
plt.title("Health Care")
plt.suptitle("Utility of Different Plans")

# Create buffer to write into
buffer = BytesIO()

# Create plot and write it into the buffer
fig.savefig(buffer, format='svg')

# The output is the content of the buffer
output_image = buffer.getvalue()

# Assign the figure to the output_view variable
knio.output_view = knio.view(fig)  # alternative: knio.view_matplotlib()

It is clear from the chart and summary of the model that the utility of flexible hours is more than that of inflexible hours; the utility pay plan is more than that of the commission and incentive system; the utility of the contribution plan is more than that of benefits plan; and the utility of medical insurance is more than that of life insurance and health savings account. Hence, these are considered preferred variables by the employees. The higher the value, the higher the importance of that attribute in employees' perception. Since, for the work-life balance attribute, the flexible hours level has the maximum utility, and for the performance reward attribute, the pay plan level has the maximum utility, this means that when selecting compensation and benefits plan by the employees, the highest preference is given to flexible hours and pay plan.

It should be noted that the HR professional will prefer a scheme that maximizes utility. This will lead to employee satisfaction with the organization. The figures show that performance rewards with incentive systems, pay plans, and health care benefits with health savings accounts and life insurance are the least preferred approaches.

Measure Part-Worth Utilities of Each Attribute Using Conjoint Analysis

Objective: To determine the importance of each attribute using conjoint analysis

We need to determine to what extent each component (attribute) contributes to the total utility of the plan. What is more important for the employees should be known- health care benefits, performance rewards, work-life balance, or retirement benefits- and how much weight should be given to these attributes. It provides us with an idea about which attribute is most important. In this section, we will first determine the utility values for each attribute (performance reward, retirement benefits, work-life balance, and health care benefits). The utility per attribute is the range of coefficients in a feature. The difference between the maximum and minimum utility of variants calculates the range. The total utility is computed by determining the sum of all partial utilities. The relative importance per feature is calculated by dividing the utility of each level by the total utilities. Thus,

• Range = Maximum(Utility of variants) – Minimum(Utility of variants)

• Importance of attribute = Range of attribute/Total range of all attributes

The above calculations are performed using various data manipulation nodes, which are then combined into the Attribute Importance meta-node.

The resultant table provides the Relative importance of all attributes as shown in the AttributeImportance column.

Pie Chart Depicting the Significance of Attributes

We will use the PieChart node to generate an interactive chart depicting the importance of the attributes.

The total range of all attributes is 10.877. Hence, the importance of work-life balance, performance rewards, retirement benefits, and health care benefits is 20%, 20%, 24.7%, and 34.8%, respectively. The results clearly show that the employees gave the maximum preference to health care benefits.

Compare Different Compensation and Benefits Plans

Objective: To determine the total utilities of different plans for compensation and benefits

To arrive at the total utility value of all 36 plans, we perform the calculations using various data manipulation nodes combined into the Total Utility meta-node.

This section determines the total utility, as represented by the TotalUtility column, of all 36 plans related to compensation and benefits that can be executed. It should be noted that the attribute and sublevel getting the highest utility value will be most favored by employees. The total utility of each plan is the sum of the utilities of each criterion. Thus, the total utility of the plan with inflexible hours, incentive system, benefit plan, and life insurance is the sum of their utilities.

Listing Top Five Plans

We will use the Top k-row Filter node to find the top 5 plans by TotalUtility.

It is clear from the results that the highest utility (5.079) is for flexible hours choice in work-life balance, pay plan in performance reward, contribution plan in retirement, and medical insurance in health care benefits. The second highest utility (4.663) is for flexible hours choice in work-life balance, pay plan in performance reward, contribution plan in retirement, and life insurance in health care benefits. The lowest utility (-5.798) is for inflexible hours, choice in work-life balance, commission in performance reward, benefit plan in retirement, and health savings account in health care benefits. Hence, the organization can decide strategies accordingly. This will help improve employees' satisfaction level and make them happy with the organization’s policies, thereby contributing to their better performance and finally fulfilling the organization’s objectives.

Summary

In conclusion, structuring an employee rewards package using conjoint analysis provides a comprehensive understanding of employee preferences regarding compensation and benefits. By evaluating attributes such as work-life balance, performance rewards, retirement benefits, and health care benefits, organizations can identify which elements are most valued by employees. This approach allows HR professionals to design benefit programs that maximize employee satisfaction and retention. The analysis reveals that employees highly prefer flexible working hours, pay plans, contribution plans, and medical insurance. By aligning reward packages with these preferences, organizations can effectively enhance employee satisfaction, improve performance, and achieve organizational objectives.

The HR department needs to frame strategies that benefit the employees and do not create complexity in understanding data. The HR professional must understand the data before making decisions or performing advanced analysis. Thus, employee data exploration starts with organizing data into a highly usable and valuable resource and extracting information to improve the functioning of the HR department. Through data visualization, the HR professional can better understand the dynamics of the employee data across multiple dimensions. This chapter discusses different visualization techniques in KNIME that help to analyze and visualize data according to the requirement.

Employee Data

To understand the usage of KNIME, the IBMdata.csv data for HR Analytics is considered. This dataset can be downloaded from the “Data” folder on the KNIME Community Hub.

The Reviewing Employee Details – Data Visualization workflow can be downloaded from the KNIME Community Hub.

You are ready after downloading the CSV file from the Data folder.

Reading the CSV-based dataset

The first step in our analysis will be to load the data for our exploratory analyses. We will do this first step using the CSV Reader node before we persist our analysis in a KNIME table.

The KNIME table is created by loading the IBMdata CSV dataset. The above table shows that the employee dataset has 1470 observations and 35 columns. Most columns are integers, and some, such as Gender, JobRole, MaritalStatus, etc., are string columns.

Distribution of Employees by Department

Objective: To create a pie chart corresponding to the department

Pie charts visualize the absolute and relative frequencies. A pie chart is a circle partitioned into segments where each of the segments represents a category. The size of each segment depends upon the relative frequency and is determined by the angle. A pie chart represents values as slices of a circle with different colors.

We will use the Pie Chart node to plot the number & percentage of employees in each Department.

The interactive pie chart shows the employee distribution of the three different departments.

Filter / Subset observations for the HR Department

For our data visualization, we will consider only employees belonging to the HR department. We can use the Row Filter node to filter out the observations for the “Human Resources” Department.

The filtered table shows that the employee dataset for the HR department has 63 observations and 35 columns.

Data Visualization using Bar Chart

Objective: To create an interactive bar chart to display category-wise count for the number of years since the last promotion

The count function of the bar chart is used to show the number of observations corresponding to the different values of the specified column. To begin with, we will use the Number to String node to convert the YearsSinceLastPromotion from integer to string. Then, we will use the Sorter node to order the table in ascending order of the YearsSinceLastPromotion column. Subsequently, we will use the Bar Chart node to create charts showing count (aggregation) values for YearsSinceLastPromotion for HR department employees.

The figure shows 24 observations corresponding to 0 years, 17 observations corresponding to 1 year, eight observations corresponding to 2 years, three observations corresponding to 3, 4, 5, and 7 years, and one observation for 10 and 12 years. The higher value of categorical columns corresponding to the maximum frequency of observations is displayed at the top. The chart shows that the organization has effective promotion policies because most employees are given promotions within 1 year.

Data Visualization using Line Plot

Objective: To create a line chart corresponding to the hourly rate

We will use the Line Plot node to create an interactive line chart based on the HourlyRate of employees in the HR department.

The line chart shows employees' hourly rates range from 30 to 100. However, it should be noted that the line chart is effective for time series data. Here, it is drawn based on row number, which does not signify any vital meaning. The intention is to familiarize the user with the concept of a line chart.

Data Visualization using Multiple Bar Chart

Objective: To create an interactive bar chart for different continuous columns according to gender

We will use the Bar Chart node to create multiple charts showing average (aggregation) values for Age, HourlyRate, TotalWorkingYears, and YearsAtCompany, with Gender as the category dimension column for HR department employees.

The output shows a pair of four-bar charts corresponding to four columns, one bar for each category. The chart shows that male and female employees have nearly the same average total experience and age. The average hourly rate of males is higher than that of female employees. Male employees have spent more years with the company than female employees.

Data Visualization using Correlation Matrix

Objective: To create a correlation matrix plot for multiple columns

We will use the Linear Correlation node to generate a correlation matrix heatmap based on a selected set of columns for HR department employees.

The squared table shows the pair-wise correlation values of all columns. The color range varies from dark red (strong negative correlation) to dark blue (strong positive correlation). If a correlation value for a pair of columns is unavailable, the corresponding cell contains a missing value (shown as a cross in the color view). Hovering the mouse over each box in the views window will display the corresponding pair-wise correlation value.

Data Visualization using Box Plot

Objective: To create an interactive box plot for daily rate by education field

We will use the Box Plot node to create an interactive plot based on theDailyRate, with EducationField as the condition column for HR department employees.

A box plot describes the distribution of a continuous variable by plotting the summary of the following statistical terms: The box itself goes from the lower quartile (Q1) to the upper quartile (Q3). The median is drawn as a horizontal bar inside the box. The distance between Q1 and Q3 is called interquartile range (IQR). Above and below the box are the so-called whiskers. They are drawn at the minimum and maximum value as horizontal bars and are connected with the box with a line.

The figure shows the general structure of an interactive box plot. The box plot is drawn for daily rate for education fields. There are five categories in education: medical, human resources, life sciences, technical degree, and others. Hence, five boxes are shown in the figure corresponding to these five categories. We can determine the minimum and maximum values along with the first and third quartiles. It is clear from the chart that the median value of the daily rate is highest for “other” while it is the least for “Technical Degree”. The interquartile range of daily rate is highest in “Life Sciences”. The lowest daily rate is for “Human Resources”.

Data Visualization using Violin Plot

Objective: To create an interactive violin plot for gender for years in the current role

We will first use the Color Manager node to assign specific colors for the categories (male and female) in the Gender column. Subsequently, we will use the Violin Plot (Plotly) node to create an interactive plot based on theYearsInCurrentRole grouped by Gender (male and female) for employees in the HR department.

The violin plot combines a box plot and a kernel density plot. The violin chart draws a shape depending on the frequency. Like a box plot, an interactive violin plot can produce multiple violins corresponding to different values of a categorical column. We can observe from the chart's shape that the maximum number of female employees spent 2 years in their current role, while the maximum number of male employees spent 3 years in their current role.

Data Visualization using Density Plot

Objective: To create an interactive density plot for years at the company by gender

We will use the Color Manager node configured earlier to assign specific colors for the categories (male and female) in the Gender column. Subsequently, we will use the Density Plot node to create an interactive plot with YearsAtCompany as the dimension column and Gender (male and female) as the condition column for employees in the HR department.

A kernel density estimate plot is a method for visualizing the distribution of observations in a dataset analogous to a histogram. KDE represents the data using a continuous probability density curve in one or more dimensions. The plot shows that male employees have spent more years in the HR department.

Data Visualization using Scatter Plot

Objective: To create an interactive scatter plot for age and total working years by gender

We will use the Color Manager node configured earlier to assign specific colors for the categories (male and female) in the Gender column. Subsequently, we will use the Scatter Plot node to create an interactive plot based on Age and TotalWorkingYears with Gender (male and female) as the color dimension for employees in the HR department.

The figure shows a scatter plot for age to total working years. The pattern indicates that a line covers the maximum points. The other points lie near the line. The chart shows that working years also increase with increasing age.

Data Visualization using Scatter Plot Matrix

Objective: To create a scatter plot matrix for age, hourly rate, daily rate, and distance from home.

We will use the Color Manager node configured earlier to assign specific colors for the categories (male and female) in the Gender column. Subsequently, we will use the Scatter Plot Matrix node to create a pair-wise scatter plot matrix of Age, DailyRate, HourlyRate, and DistanceFromHome columns with Gender (male and female) as the color dimension for employees in the HR department.

The figure shows a pair plot considering all the possible combinations for the four columns. Since there are four variables (n), a matrix of 3x3 images (n-1) is created. The scatter plot is drawn for every combination of two variables.

Data Visualization using Matplotlib

Objective: To create a shape chart corresponding to monthly income.

For this purpose, we will create a table with the Table Creator node containing four series of monthly income data. We will use the Matplotlib visualization library in the Python View node to create a custom chart. In the figure, we plot multiple lines with different shapes and colors. We can observe that for every x, y pair of arguments, there is an optional third argument, which is the format string that indicates the concatenation of a color string with a line-style string. Thus, the first line, Series1, is drawn using green triangles (g^); the second line, Series2, is drawn using blue squares (bs); the third line, Series3, is drawn using red dashes (r--); the fourth line, Series4, is drawn using magenta dots (mo).

import knime.scripting.io as knio

# Code to display multiple lines with different shapes and colors
import matplotlib.pyplot as plt
from io import BytesIO

df = knio.input_tables[0].to_pandas() 

# Plotting 
fig = plt.figure(figsize=(15,8))
plt.suptitle('Multiple Lines with Different Shapes and Colors',
fontsize=20, fontweight='bold')


# Creating a chart with different colors and effects for different series
plt.plot(df['List'], df['Series1'], 'g^', 
         df['List'], df['Series2'], 'bs',
         df['List'], df['Series3'], 'r--', 
         df['List'], df['Series4'], 'mo')

plt.xlabel('List')
plt.ylabel('Values')
plt.title('Multiple Series Plot')
plt.show()


# Setting limits of x-axis
plt.xlim(0,10)

# Setting limits for y-axis
plt.ylim(0,220)

# Displaying grid in chart
plt.grid(True,color='k')

# Create buffer to write into
buffer = BytesIO()

# Create plot and write it into the buffer
fig.savefig(buffer, format='svg')

# The output is the content of the buffer
output_image = buffer.getvalue()

# Assign the figure to the output_view variable
knio.output_view = knio.view(fig)  # alternative: knio.view_matplotlib()

Alternatively, we can plot the four series of monthly income using the Line Plot node, with each series in a different color line.

Data Visualization using Stacked Area Chart

Objective: To create an area chart for expenses of different months in four years.

For this purpose, we will create a table with the Table Creator node containing four years of monthly expense data. Subsequently, we will use the Stacked Area Chart node to create a stacked area chart visualization of the monthly expense data, with each year stacked one above the other. The difference between a stacked bar chart and an area chart is that the values are represented as cumulative values in a stacked bar chart. In contrast, an area chart defines them as individual values. These charts can track changes over time for two or more related groups that comprise one whole category.

The chart shows the total expenses for 12 months and each year. Thus, we can observe that the costs for 2013 are the highest since the area occupied is more than the expenses of any other year.

Summary

In this example, we explored using KNIME for hands-on data visualization. Working with the IBMdata.csv dataset and various KNIME charting and plotting nodes enabled us to interpret data for better understanding and extracting crucial insights.

Organizations have a vast amount of employee data. The HR personnel in these organizations must understand these data and use basic statistical techniques to formulate effective strategies. They can use statistical methods such as mean, median, correlation, etc.; mathematical techniques such as minimum, maximum, etc.; and string operations such as concatenating data, modifying/updating records, etc. These basic techniques help the HR professional to have a better understanding of the data. This chapter discusses different functions in KNIME that help to analyze and manipulate data according to the requirement.

Employee Data

To understand the usage of KNIME, the IBMdata.csv for HR Analytics is considered. This dataset can be downloaded from the “Data” folder from the KNIME Community Hub.

You are ready after downloading the CSV file from the Data folder.

You can download the 02 Analyzing Employee Details - Data Exploration & Manipulation workflow from the KNIME Community Hub

Reading the CSV-based dataset

The first step in our analysis will be to load the data for our exploratory analyses. We will do this first step using the CSV Reader node before we persist our analysis in a KNIME table.

The KNIME table is created by loading the IBMdata CSV dataset.

The above table shows that the employee dataset has 1470 observations and 35 columns. Most columns are integers, and some, such as Gender, JobRole, MaritalStatus, etc., are string columns.

Data Exploration and Descriptive Summary Statistics

The Statistics node primarily determines the descriptive summary statistics of the columns in the dataset. This node calculates statistical moments such as minimum, maximum, mean, standard deviation, variance, median, overall sum, number of missing values, and row count across all numeric columns. It counts all nominal values together with their occurrences. The node provides the following three output tables

1. Statistics Table: All statistic moments for all numeric columns,

2. Nominal Histogram Table: Nominal values for all selected categorical columns and

3. Occurrences Table: The most frequent/infrequent values from the categorical columns (Top/bottom)

The Data Explorer node offers various options for displaying the properties of the input data in an interactive view.

Data Manipulation of String Columns

Joining of Strings

Objective: To display the employee details using concatenation of Department and EducationField string columns

A new CombinedString column can be created by concatenation (combining) of different strings. The strings can be combined by using the Join function of the String Manipulation node.

Case Conversion of Strings

Objective: To display the employee details in capitalized, uppercase, and lowercase format.

In the String Manipulation node, the capitalize (str) function capitalizes all whitespace-separated words in a string so that each word comprises a title-case character and then a series of lowercase characters.

We apply the capitalize function on the CombinedString column to create the CapCombinedString column.

The uppercase(str) and lowercase(str) functions convert all the characters in a string into uppercase and lowercase, respectively.

Search and Replace of String

Objective: To search for a particular string and replace it with another

The replace(str, search, replace) function of the String Manipulation node replaces all occurrences of a String within another String. In our CombinedString column, we search for the word “Employee” and replace it with “Staff” in the appended RepCombinedString column.

Search and Match of String

Objective: To determine and indicate whether the employee belongs to the Sales department

We use the LIKE function of the Rule Engine node to search for “Sales” in the CombinedString column and create a new column, Sales, with “Yes” or “No,” to indicate whether the employee belongs to the Sales department.

The Like function helps locate the string “Sales” in the given string column. If the department has a “Sales” word, the corresponding Sales column is populated with a “Yes” value and a “No” value otherwise. The above results show that the first employee belongs to the sales department, and the subsequent four do not.

Aggregated Statistics using Group by Operation

The GroupBy node helps group observations based on a categorical column. After grouping observations, we can apply multiple aggregation functions, such as mean, median, minimum, maximum, count, sum, covariance, variance, standard deviation, etc.

We will use the GroupBy node to get the aggregated statistics for multiple columns across each Gender (male and female).

• Mean of DistanceFromHome

• Median of DistanceFromHome

• Minimum Age

• Maximum Age

• Count of EmployeeCount

• Sum of DailyRate

• Covariance of HourlyRate

• Variance of HourlyRate

• Standard deviation of HourlyRate

Group by with Sorting

Objective: To determine the employee count and percent for different marital status

We will use the GroupBy node to get the distinct employee count and percentage across each marital status (group by MaritalStatus, aggregate count, and percent by EmployeeNumber). Subsequently, we use the Sorter node to order the resultant table in descending order of count.

We can observe three distinct values of Marital Status: Single, Married, and Divorced. The number of observations of married employees is maximum (673, 45.782%) and displayed at the top; followed by the number of observations of single employees (470, 31.973%), and observations having divorced employees is minimum (327, 22.245%) and displayed at the bottom.

Data Filtering (subset) using AND and OR operation

Objective: To filter the data of female employees belonging to the sales department

We will use the Row Filter node and add the criterion1 filter column as Gender (value = “Female”) and the criterion2 filter column as Department (value = “Sales”). By selecting the “All criteria” option, the “AND” filter condition is applied (189 rows in the resultant filtered table). On the other hand, by choosing the “Any criteria” option, the “OR” filter condition is applied (845 rows in the resultant filtered table).

Determine Correlation Using Spearman Correlation Coefficient

Objective: To determine the correlation between hourly rate and job satisfaction

We will use the Rank Correlation node to determine the Spearman correlation coefficient of the HourlyRate and JobSatisfaction columns. The node calculates a correlation coefficient for each pair of selected columns, i.e., a measure of the correlation of the two variables. All measures are based on the rank of the cells. By executing the node, we get three outputs for statistical inference. Correlation measure contains correlation variables, p-values, and degrees of freedom. The correlation matrix displays the correlation variables in a matrix representation. The Rank table contains the fractional ranks of the columns where the rank corresponds to the position of the value in a sorted table.

The correlation between the two groups lies between -1 and +1. The value -1 denotes a perfect negative correlation, and +1 denotes a perfect positive correlation. The value 0 denotes no correlation. The value between 0 and 1 denoted the extent of correlation. A lesser value denotes a lower correlation, and a higher value denotes a higher correlation. From the output, we can see that the Spearman correlation is -0.06; this means a low correlation between the two groups. We know the null hypothesis is rejected if the p-value is <0.05. Our data shows that the p-value is less than 0.05, which means the null hypothesis is rejected. This means there exists a negative correlation between hourly rate and job satisfaction.

Determine Correlation Using Pearson Correlation Coefficient

Objective: To determine the correlation between monthly rate and job satisfaction

We will use the Linear Correlation node to determine the Spearman correlation coefficient of the MonthlyRate and JobSatisfaction columns. The node calculates a correlation coefficient for each pair of selected columns, i.e., a measure of the correlation of the two variables. By executing the node, we get two outputs for statistical inference. The correlation measure contains correlation variables, p-values, and degrees of freedom. The correlation matrix displays the correlation variables in a matrix representation.

The correlation coefficient is 0.001, which is very low. Also, the p-value (0.98) is more than 0.05, which means the null hypothesis failed to be rejected. This means no significant correlation exists between monthly rate and job satisfaction.

Summary

In conclusion, this article demonstrated the effective use of KNIME for data exploration and manipulation, specifically in HR analytics using the IBMdata.csv dataset. By leveraging various KNIME nodes, we performed a comprehensive exploratory data analysis, including reading and summarizing data, manipulating string columns, and conducting statistical analyses such as correlation. These techniques provide HR professionals with valuable insights into employee data, enabling them to make informed decisions and develop strategic initiatives. The hands-on approach with KNIME showcases its versatility and power in handling complex data tasks, making it an essential tool for data-driven decision-making in organizations.

What gets measured gets managed. What gets managed gets executed. -Peter Drucker

Global organizations with workforce analytics and planning outperform all other organizations by 30% more sales per employee. -CedarCrestone Research Survey

The corporate world is slowly evolving its practice of HR analytics toward a ‘model of predictive management’ for human resources because to be successful, HR analytics must extend beyond reporting what is (the present) or what was (the past) to predicting and analyzing what will be (the future). -Center for Advanced Human Resource Studies

The global HR analytics market will grow at a CAGR of around 12% during the 2019-2025 forecast period. -World HR Analytics Market

Companies using a portfolio of HR analytics solutions could realize an increase of 275 basis points in profit margins, on average, by 2025. -McKinsey Global Institute

HR Metrics and HR Analytics

The core responsibility of the HR department is to provide a high return on the business’s investment in its employees by effectively aligning with corporate culture, organizational values, vision, and mission. It provides organizations with insight and feedback on the contribution of HR initiatives to meeting operational goals and strategic business objectives.

HR Metrics are measurements used to determine the value and effectiveness of HR initiatives and help track key HR department activities by reporting numbers. These indicators allow businesses to measure performance, efficiency, and impact of business processes and change. While HR metrics act as a barometer for business performance, they can only track events but do not provide in-depth information about the event, which can be used for making strategic decisions and uncovering areas of improvement. On the other hand, HR analytics aims to provide insight into each process after gathering data and then use them to make relevant decisions about improving these processes. Thus, when used for HR analytics, HR metrics can pave the way toward effectively meeting organizational objectives.

What is HR Analytics?

HR analytics, also known as people analytics, talent analytics, or workforce analytics, can be defined as follows:

• The systematic identification and quantification of the people drivers of business outcomes.

• The process of collecting and analyzing human resources data to improve employee performance.

• Applying sophisticated data mining and business analytics techniques to human resources data.

The field of analytics refers to using analytics in an organization's HR department processes to improve workforce performance and result in a better return on investment.

HR analytics can be used to explore and examine the data and then transform their findings into insights that will help executives, managers, and operational employees of the HR department to make better and more informed decisions. These are three types of analytics:

• Descriptive analytics – What has happened?

• Predictive analytics – What could happen?

• Prescriptive analytics – What should happen?

The data-driven insights that HR analytics provides help identify existing and required outcomes. This will further improve productivity and efficiency, generating increased revenue for the organization.

Artificial Intelligence (AI) and machine learning help streamline the HR department's significant and essential functions. The latest developments in HR analytics use AI tools for text and image data analysis. AI helps measure employee behavior and hire the most suitable candidate for the job during recruitment by filtering resumes and recognizing and analyzing facial expressions and voice in the organization.

HR Analytics for Professional Challenges

To meet organizational objectives, HR professionals must recruit the right person, experience less attrition, utilize employees' best capabilities, improve the recruitment process, provide competitive salaries, launch successful employee training programs, etc. The organization's leaders can make consistently better decisions using facts supported by data. The key to finding the correct data for the organization is identifying the overall business needs, as every organization may differ in data collection, formulating strategies, and decision-making. According to the requirement, the data for doing HR analytics can be collected from the Human Resource Information System (HRIS) and through the customized survey portals.

Why KNIME for HR Analytics?

This series intends to encourage more ‘data-driven’ decisions in HR. HR Analytics is no longer a nice-to-have add-on; it is how HR practitioners should conduct HR decision-making in the future. Where applicable, human judgment is ‘added’ onto a rigorous data analysis done in the first place.

To achieve this ideal world, I need to equip you with some fundamental knowledge of KNIME, an open-source tool for data scientists. I am well aware that, on one side, you want to do something for your career in HR. However, you are most likely completely new to data analytics and coding.

KNIME is a data science platform for data analytics and machine learning. It is an open-source tool businesses can use to automate repetitive tasks, analyze data, and build AI models. The visual workflow representation of each node makes it easy to use and understand, even for people without a data analytics or programming background.

Topics Covered in HR Analytics Series

This series of articles on “HR Analytics with KNIME“ covers the role of HR analytics in the following functional areas.

Analyzing Employee Details – Data Exploration & Manipulation

It illuminates the available data exploration and manipulation nodes in KNIME, familiarizes them with their functions, and educates them on using them in different HR data analysis scenarios.

Reviewing Employee Details – Data Visualization

Educating on the various visualization techniques and their relevant nodes in the KNIME provides knowledge of charts for categorical and continuous variables, demonstrates the use of charts to solve real-world problems, and encourages effective decision-making regarding selecting a chart related to the data type.

HR Analytics Using Statistical Techniques

Structuring Employee Rewards Package – Conjoint Analysis

The article mentions the organization's flexible compensation and benefits plan and discusses how to perform conjoint analysis to determine the preference of different attributes and levels. It helps evaluate conjoint analysis results and determine the utility of varying compensation and benefits plans.

Evaluate Training Effectiveness – Comparative Statistics

It helps you understand the importance of employee training and development programs, provides knowledge of different comparison techniques, assesses the results of comparison techniques used for evaluating training programs, and elaborates on familiarity with applications of comparison techniques in other areas of HR analytics.

Forecasting HR Cost – Time Series Modeling

Educated on the importance of time series analysis, it helps determine the series’ stationarity and assists in constructing different models using ARIMA modeling techniques. It also mentions how to forecast using ARIMA modeling techniques.

HR Analytics Using Unsupervised Machine Learning

Unsupervised machine learning algorithms are used when the output is unknown, and no predefined instructions are available to the learning algorithms. In unsupervised learning, the learning algorithm only has input data, and knowledge is extracted from these data. These algorithms create a new representation of the data that is easier to comprehend than the original data and help improve the accuracy of advanced algorithms by consuming less time and reducing memory. Standard unsupervised machine learning algorithms include association rule mining, dimensionality reduction algorithms, and clustering.

Evaluating Job Satisfaction – Association Rule Mining

It helps determine the factors associated with job satisfaction, educates people to understand the methodology of association rule mining, and shares knowledge on implementing the apriori algorithm for determining factors impacting job satisfaction. It also facilitates the evaluation of the results of association rule mining for job satisfaction.

Identify Absenteeism Patterns – Cluster Analysis

It helps to understand the importance of clustering and mentions different clustering techniques. You will also be able to analyze the results of various clustering techniques and implement them in a real-world situation.

Identify Core Factors in Performance Appraisal – Principal Component Analysis

It helps to understand the factors of the performance appraisal system and helps in applying dimension reduction algorithms. It also educates on implementing algorithms to determine performance appraisal factors and analyze the results of these algorithms.

HR Analytics Using Supervised Classification Machine Learning

Classification is a machine learning technique for solving problems predicting a dependent categorical variable. The classification problem occurs when the dependent variable has two or multiple categories and is predicted by a set of independent variables. Most of the data in real-world problems require the concept of classification. It should be specified that any number of dependent variable categories can exist in classification. However, logistic regression applies only to two categories (binary classification). Logistic regression predicts binary values, such as whether the customer will buy or not, Pass/Fail, Yes/No, 0/1. It measures the probability of event=success and event=failure.

Predicting Employee Attrition - Supervised Classification Machine Learning

It helps to understand the importance of attrition for employees and gain knowledge of different classification techniques. You can use classification techniques to predict employee attrition and familiarize yourself with classification techniques applied in other areas of HR analytics.

Summary

In conclusion, HR analytics is a transformative approach that empowers organizations to make data-driven decisions, enhancing workforce performance and aligning HR initiatives with strategic business objectives. By leveraging tools like KNIME, HR professionals can efficiently analyze and visualize data, leading to improved recruitment processes, optimized employee training programs, and better overall organizational outcomes. Integrating advanced technologies such as AI and machine learning will further streamline HR functions as the field evolves, ensuring that companies remain competitive in a rapidly changing global market. Embracing HR analytics is a trend and a necessity for future-ready organizations aiming to maximize their human capital investment.

More specifically, in this article, we will cover the following topics:

• Loading our dataset from a CSV file and applying data-cleaning steps for our analysis.

• Using the Pivot & GroupBy for convenient aggregation operations.

• Using the data visualization nodes to create interactive data visualizations.

• Perform exploratory data analysis of pedestrian traffic counts using a public dataset in Melbourne City.

By the end of this article, you will be able to assemble workflows for practical data analysis with KNIME that work for your needs.

Downloading the dataset and workflow

We will be working with one of the pedestrian counting-system datasets produced and made public by the city of Melbourne. This dataset contains hourly pedestrian counts from sensors in and around Melbourne. We will use a historical snapshot of this dataset covering May 2009 to December 2022.

You are ready once you have extracted the CSV file from the ZIP archive.

You can download the Melbourne Pedestrian Sensor Data Analysis workflow from the KNIME Community Hub.

Reading and transforming the CSV-based dataset

The first step in our analysis will be to identify the data loading and transformation steps we will use to prepare the data for our exploratory analyses. We will do this first step using the CSV Reader node before we persist our transformation in a KNIME table.

The KNIME table is created by loading the pedestrian CSV dataset without cleaning.

As we can see, each record in this dataset (4,562,230 rows & 10 columns) contains the number of pedestrian counts passing through a given sensor for a specific hour, along with other information about the hourly reading, such as the sensor name and timestamp of the record and date and time components extracted from the timestamp.

The primary data issue you might have noticed is that the Date_Time field is of the String type. We must convert this column to a Date&Time type for some time series analysis and visualizations. Let us address this by using the String to Date&Time node. We can see that the Date_Time column now has the correct date & time format.

Given that we will be performing a range of time series analyses, we will almost certainly need these records ordered by Date, so the other transformation we will perform involves sorting the records by the Date_Time column.

The KNIME table that is produced after applying the transformations

Our data format is now suitable for performing our analysis. Of course, the nature of data analysis is iterative, and we may need to come back to tune our data processing steps after working with the dataset. But for now, we are ready to proceed with the data analysis.

Practical data analysis using KNIME

We will briefly cover two sets of tools that will assist us in performing our data analysis. The first is aggregation nodes like GroupBy and Pivot, which provide a convenient way to perform aggregate operations on the data. The second is comprehensive data visualization nodes (Views and JavaScript) that produce interactive visualizations. Let us get started.

Total number of pedestrians for every sensor across the years

Say we wanted to count the total number of pedestrian readings for every sensor aggregated across all years. We can use the Pivot node to perform this aggregation using the following column settings: group by Sensor_Name, pivot by Year, and sum by Hourly_Counts.

The resulting pivot table is produced after performing the aggregation. Note that some sensors have not captured data for some years, which are aptly represented as missing values.

Taking it further, say we wanted to count the total number of pedestrian readings for the Melbourne Central sensor alone. We can use the Row Filter node to filter out the aggregated count for this sensor.

Visualize the number of active sensors across each year

First, we will use the GroupBy node to get the distinct number of sensor names seen across each year (group by Year and unique count by Sensor_Name). Before we proceed with the visualization, we will convert the Year column from Number to String. Subsequently, we will use the Line Plot node to plot the unique count of sensors across each year.

The resultant interactive line plot shows the number of active sensors each year.

Top 10 sensors by traffic in 2022

First, we will use the Row Filter node to filter out the dataset for the year 2022. Then, we will use the GroupBy node on the resultant dataset to get the total number of pedestrians across each sensor (group by Sensor_Name and sum of Hourly_Counts). Before the visualization, we will use the Top k-row Filter node to find the top 10 sensors. Subsequently, we will use the Bar Chart node to get an interactive bar chart of the top 10 sensors by traffic in 2022.

Table with the top 10 sensors by total pedestrian counts for 2022

The resultant interactive bar chart shows the top 10 sensors by traffic in 2022.

Visualizing total pedestrian counts over time

Let us understand how the dataset's total number of pedestrian counts registered by the sensor network has changed over the years. We will use the GroupBy node to get the total number of pedestrians across all sensors by year (group by Year and sum by Hourly_Counts). Before we proceed with the visualization, we will convert the Year column from Number to String. Subsequently, we will use the Line Plot node to plot the total number of pedestrians yearly.

The resultant line chart shows the total pedestrian counts recorded annually across the entire pedestrian counting system.

This plot shows a steady increase over the years in the number of total pedestrians recorded moving across Melbourne city, with a rather pronounced crash in 2020, which is almost certainly due to the impact that COVID-19 lockdown restrictions had on the number of people walking through the city. However, we must be careful about the conclusions we draw from this plot. As we saw in the previous section, the number of active sensors in the system has increased, resulting in some pedestrians being counted more times over the same route than in previous years.

Pedestrian count over time for sensors active across all years

To address the confounding impact of the changing number of sensors over the years, let us filter down our dataset to only records from sensors that have readings across every year before repeating the same analysis. We will do this in two steps:

1. Extract the sensor names with readings for every year in the dataset.

2. Repeat our sum aggregation of pedestrian counts by year while retaining only those rows with sensor names in the above table (#1).

We can achieve the first step by grouping our dataset by sensor name and selecting only sensor names whose group of records includes at least one record for each year that our dataset covers. Our dataset contains data from 2009 to 2022, covering 14 years. We will use the GroupBy node (unique count by Year) to confirm that this is the distinct number of years seen in our dataset.

We will leverage the observation that each sensor needs to have 14 distinct Year values across its readings to be considered. We will use the GroupBy node to get the number of distinct years for every sensor (group by Sensor_Name and unique count by Year). Using the Table Row to Variable node, we will pass the distinct count value 14 as a flow variable for the Row Filter node. This gives us the list of 15 sensors with readings for all 14 years.

A distinct year count is passed to the row filter node as a flow variable.

We can perform the next step now that we know which sensors were active across all years in the dataset. Here, we will replicate the previous set of nodes we used to plot the total number of pedestrian counts for each year (yearly traffic across all sensors). However, this time, we will restrict pedestrian counts to only those active sensors for all years. We will use the Reference Row Filter node for this purpose.

A resultant line chart shows the total pedestrian counts by year across only active sensors yearly.

Comparing this with the previous plot, we can see that the trend regarding the increasing number of pedestrian counts over the years is less pronounced. This is what we expect to see when controlling the growing number of sensors. We can still see the dramatic drop in sensor counts due to the COVID-19 lockdowns, which, if anything, now looks even more pronounced compared to the previous years.

Monthly pedestrian count for sensors active in 2019, 2020, and 2021

To finish our investigation into pedestrian counts through the city over time, let us drill down a bit deeper into the impact of the lockdowns on Melbourne pedestrian traffic by looking at the month-to-month changes for 2019, 2020, and 2021. For this, we will need a new data extract, which we group by years and months and then aggregate the total counts for each combination of year and month. Continuing our previous analysis, we will filter this down to only readings from active sensors across all years.

First, we will convert the Year column from Number to String and use the Nominal Value Row Filter node to restrict the dataset for 2019, 2020, and 2021. We can then use the Pivot node to perform the monthly aggregation using the following column settings: group by Num_Month, pivot by Year, and sum by Hourly_Counts.

The resulting pivot table contains the aggregated pedestrian counts for each month and year combination.

Subsequently, we will use the Line Plot node to plot the monthly total number of pedestrians for sensors active in 2019, 2020, and 2021.

A resultant line chart shows the monthly traffic for sensors active in 2019, 2020, and 2021.

This visualization allows us to compare the month-by-month total counts for each target year readily. By hovering over the markers of different lines, we can gather that in April 2019, there were 8.2 million pedestrian counts, which dropped to 1.1 million for the same month in 2020, when the initial lockdown was in full effect.

Creating time series plots of sensor readings

We will now move on to a finer level of analysis and explore how the hourly pedestrian traffic changes over a range of days for specific sensors. Let us start by creating a plot of hourly traffic for the Flinders La-Swanston St (West) sensor across all of 2020. First, we will use the Row Filter node to extract only the records corresponding to our target sensor and year. Then, we will use the Line Plot node to plot the hourly pedestrian count.

A resultant line chart shows the hourly pedestrian counts for the Flinders La-Swanston St (West) sensor for 2020.

This is a very dense line plot since we have visualized every hour throughout the year. It provides a striking visualization of the impact the COVID-19 lockdowns had on pedestrian traffic across this sensor:

• Traffic dropped around the middle of March as restrictions were applied, leading up to the first complete lockdown on March 31.

• Traffic starts to increase in mid-May as restrictions are relaxed.

• A massive spike on June 6 coincided with a protest close to the central business district.

• Traffic drops off again, with a second lockdown being put in place on July 7

• We start seeing traffic increase again from October through to the end of the year, corresponding to the ongoing lifting of restrictions, which began in mid-September

This is a perfect time to take advantage of another of the interactive features the Line Plot node visualizations offer. We can zoom in on specific regions of the x and y axes by left-clicking and dragging over the desired area. Selecting the region from January 16 to 31 gives us the following time series:

A zoomed-in view of the same time series plot

We can now easily discern the periodic changes in pedestrian traffic for this sensor, corresponding to the daily rhythm of people’s activities. You can click on the zoom reset icon to reset the zoom level.

Comparing time series plots of three sensor readings

It might be interesting to compare the daily traffic rhythms across three different sensors to see whether there are variations. We can use a combination of Row Filter (for September 2019), and Rule-based Row Filter (for Sensor_Name in "Flinders St-Spark La", "Bourke Street Mall (North)", "Southern Cross Station"). We can thenuse the Pivot node to perform the aggregation using the following column settings: group by Date_Time, pivot by Sensor_Name, and sum by Hourly_Counts). Subsequently, we will use the Line Plot node to plot the hourly pedestrian count for the chosen three sensors in September 2019.

A resultant line chart shows the hourly pedestrian counts for three different sensors in Sep 2019.

By plotting the hourly traffic of these three sensors, we can readily compare pedestrian traffic patterns across them for September 2019. Comparing these three sensors, we can see that while they all have daily peaks and troughs, there is a noticeable variation:

• The Southern Cross Station sensor time series plot is striking. It reveals a pronounced difference in weekday traffic versus weekend traffic, with the weekend daytime peaks being much lower than those on the weekdays. This suggests that much pedestrian traffic comes from commuting to and from work.

• The daily peaks for the Bourke Street Mall (North) sensor look similar in range but with less pronounced variation between weekday and weekend traffic.

• Lastly, the Flinders St-Spark La sensor has a much lower range of daily traffic counts, with discernible drops over the weekend. We can also see a significant spike in pedestrian traffic on September 20.

In the context of exploratory data analysis, the particularly convenient interactive feature of these plots allows us to easily compare traffic through these sensors across narrower time windows using the zoom function without modifying the configuration.

Visualizing the distribution of daily pedestrian traffic for a sensor

To gain deeper insights into how pedestrian traffic varies across a particular sensor location, we must move away from visualizing individual data points (time series plots) and instead visualize aggregated statistics (box plots) across multiple data points. Let us create a box plot set that compares the distributions of daily pedestrian count for a single sensor across various years. We will look at traffic across the Bourke Street Mall (North) sensor, comparing 2019, 2020, and 2021 distributions.

We start by using a Rule-based Row Filter (for Sensor_Name in "Bourke Street Mall (North)" and Year in 2019, 2020, 2021). We will then use the GroupBy node to get the daily pedestrian count across each year (group by Year and Date and sum by Hourly_Counts) and convert the Year column from Number to String. After that, we will use the Box Plot node to plot the daily pedestrian count distribution across the Year dimension (Condition column).

A resultant interactive box plot with three box plots comparing the distributions of daily pedestrian counts from 2019 to 2021 for the Bourke Street Mall (North) sensor.

A resultant conditional box plot (JavaScript) with three box plots comparing the distributions of daily pedestrian counts from 2019 to 2021 for the Bourke Street Mall (North) sensor.

These three box plots provide us with a rich set of information about the distribution of daily pedestrian counts across the three years. Hovering over each box plot informs us of the value of key summary statistics, including that they include the median daily count (the horizontal line inside the box), the first and third quartiles (the bottom and top of the boxes, enclosing 50% of the data points), and the lower and upper fence values (the horizontal lines bounding the whiskers extending from the box), which indicate sentinel data points, below or above which any data points are considered outliers.

Unsurprisingly, the pre-COVID-19 distribution of daily counts for 2019 sits much higher than the next two years. We can also see that this year has a lower degree of spread than the other years, with the shorter box indicating a lower interquartile range – within which 50% of all the daily counts occur. 2020 and 2021 saw a greater spread of values, consistent with alternative lockdown periods in both years. We can also see that 2020 has a slight skew toward lower values, and 2021 has a more significant skew toward higher numbers. This, as well as the slightly lower interquartile range for 2021, is consistent with 2021 having fewer days of lockdown than 2020.

As you can see, this dataset provides a wealth of possible insights into changes and variability in pedestrian traffic through Melbourne. We will have to leave our exploratory analysis of the dataset here. Still, while going through these analyses, you might have thought about different lines of exploration and questions that we could have pursued. You can continue the analysis now that you have all the necessary ingredients.

Summary

In this example, we explored how to use KNIME in the context of hands-on data analysis. Working with the Melbourne Pedestrian Counting System dataset, various KNIME data manipulation and visualization features enabled us to perform exploratory data analysis, in which we uncovered a range of insights into pedestrian traffic through Melbourne City.

Introduction

In this short KNIME tutorial, we visualize the Olympic dataset derived from the Wikipedia entry on All-time Olympic games medals, as of the 2016 Summer Olympics and 2018 Winter Olympics. All changes in medal standings due to doping cases and medal redistributions, including 25-Nov-2020, are considered.

The KNIME workflow along with the data can be downloaded here

Using the CSV reader node, we read the CSV file containing the dataset into the KNIME table. Then perform data cleaning and preprocessing operations to get a more usable data table to meet the desired purpose.

Top ten countries

Get the top ten countries that have the most medals in the summer and winter games.

We use the Top k Selector node to select the top ten countries with the most medals

Bar chart

We use the JavaScript Bar Chart node to create an interactive stacked and grouped bar chart representing the top ten countries in terms of the total number of medals in the winter and summer games.

Bubble chart

We use JavaScript Bubble Chart (Plotly) node to create an interactive bubble chart showing the value of adjusted gold medals (no. of total gold / no. of total games) vs. the rank concerning the number of total medals won.

The size of the bubble corresponds to an adjusted value of total medals (no. of medals / no. of total games) won, and the color corresponds to the continent (North America, Europe, or Asia) or brown for NA – Not Applicable in case of Soviet Union.

The bubble chart shows that the United States has the most number of total medals in summer and winter games, as indicated by the x-axis (rank). Norway has the least number of combined medals among the top ten.

Based on the values of the adjusted gold medals represented by the y-axis, the Soviet Union won the most gold medals relative to the number of Olympic games in which they participated, followed by the US and Russia, respectively.

The sizes of the bubbles suggest that the top three countries that won the most medals relative to the number of games they were part of are respectively, the Soviet Union, the US, and Russia.

To put it in perspective, by comparing France and China, the latter won fewer total medals overall (position on the x-axis). But taking into account the number of Olympic games played, China did win more gold medals (position on the y-axis) and total medals (bubble-size).