Demographic, Statistical & Machine Learning Analysis
This project analyzes the termination and new policy datasets of Medibank Insurance Portfolio that will help to define interesting metrics in answering business questions as follow:
- Which group of customers has the higher cancellation rate?
- Which customer segment demonstrates the longest tenure and what strategies can be recommended to maintain their retention?
- Which Product has the high cancellation rate and what is the best marketing campaign should be applied?
The pre-processing steps involved in this project are as follow:
- Data Retrieve: 2 datasets (Termination & New Policy) provided by Medibank.
- Merge/combine the datasets into one and select useful variables to work with.
- Discover the data to provide description of each variable.
- Scan the data to find any missing values, errors, or special values, then find a suitable method to deal with it.
- Scan for any outliers using summary statistics, apply appropriate plots on the data, explain the methodology and transform it for better insight.
1. Data Explanatory
Both datasets have 5 main variables and described as follow:
New Policy dataset has 2 additional variables as follow:
Termination dataset has 3 additional variables as follow:
- Year: data collection year (consists of 3 years)
- State: state where the data was collected (e.g. VIC, NSW, ACT)
- Scale: customer demographic (e.g. Single, Couple, Family)
- Product: insurance products
- Age Band: group of age (e.g. less than 30 years, 30-39 years, 40-49 years, 50-59 years, 60+ years)
New Policy dataset has 2 additional variables as follow:
- Previous Health Fund: previous health fund type
- New Policies: total number of customers who joined
Termination dataset has 3 additional variables as follow:
- Tenure Band: time band with the company before termination (e.g. 0-2 years, 3-5 years)
- Next Health Fund: the next health fund product the customer switches to
- Terminations: total number of customers who cancelled
2. Demographic Analysis Dashboard
After merging and sanitizing the datasets, a dashboard was developed to summarise customer behaviour and cancellation patterns across products.
Based on a total customer of 432,620, the analysis revealed a significant cancellations rate of 49.61% for Product C, with 214,630 customers terminating their contracts. The highest number of cancellations was observed within the 0-2 years tenure band, indicating that customers are most likely to churn in the early stages of their lifecycle. In addition, customers aged under 30 represented the most affected age group, suggesting a higher churn tendency among younger demographics.
Further segmentation analysis identified that the highest-risk customer groups were those under 30 years old as well as those with 20+ years of engagement, highlighting distinct behavioural patterns across both early-stage and long-tenured customers.
Overall, the findings suggest that churn is primarily driven by early tenure customers, with additional risk concentrated in specific demographic segments, providing clear opportunities for targeted retention strategies and lifecycle-based engagement initiatives.
Based on a total customer of 432,620, the analysis revealed a significant cancellations rate of 49.61% for Product C, with 214,630 customers terminating their contracts. The highest number of cancellations was observed within the 0-2 years tenure band, indicating that customers are most likely to churn in the early stages of their lifecycle. In addition, customers aged under 30 represented the most affected age group, suggesting a higher churn tendency among younger demographics.
Further segmentation analysis identified that the highest-risk customer groups were those under 30 years old as well as those with 20+ years of engagement, highlighting distinct behavioural patterns across both early-stage and long-tenured customers.
Overall, the findings suggest that churn is primarily driven by early tenure customers, with additional risk concentrated in specific demographic segments, providing clear opportunities for targeted retention strategies and lifecycle-based engagement initiatives.
3. Descriptive Statistic
|
Descriptive Statistic were performed to analyse customer composition across different household segments including Single, Single Parent, Couple, and Family groups, to understand both distribution and behavioural differences within the portfolio.
The analysis revealed that Single customers represent the largest segment, with a significantly higher mean scale compared to other groups, indicating a concentration of customers within this category. However, variability across segments (as shown by standard deviation and range) suggests distinct behavioural patterns in engagement and utilisation across household types. Further examination of median and percentile values highlights that Couple and Family segments tend to exhibit more stable distribution patterns, whereas Single and Single Parent groups show greater dispersion in customer scale. These ingsights provide a clearer understanding of customer structure and support more targeted segmentation strategies for marketing, retention, and product positioning. Overall, the findings demonstrate that household composition is a meaningful segmentation driver, enabling more tailored customer engagement strategies based on behavioural and demographic differences across scale bands. |
4. Correlation Analysis
|
The correlation analysis between different customer scales was conducted to understand how strongly these segments move in relation to each other.
The results show several strong positive relationships between customer groups, indicating that certain segments tend to exhibit similar behavioural patterns. The strongest correlation is observed between Single Parent and Family customers (0.89), suggesting these groups share very similar underlying characteristics or respond similarity to external factors such as pricing, product offerings, or lifecycle changes. A strong correlation is also seen between Single and Couple customers (0.85), indicating overlapping behavioural trends between individual and dual-person households. This may reflect similarities in financial behaviour or product usage patterns across these segments. Moderate correlations across other combinations (ranging from 0.545 to 0.62) suggest that while all groups are somewhat related, there are still meaningful differences in behaviour, reinforcing the need for segment specific strategies rather than a one-size-fits-all approach. Overall, the analysis highlights that while customer scale segments are distinct, there are strong interdependencies between certain groups, particularly Family and Single Parent, and Single and Couple - providing valuable insight for targeted marketing, product design, and retention strategies. |
5. Machine Learning Analysis
Using machine learning, we can group similar customers together into distinct clusters to better inform marketing strategies and product roadmaps. For this dataset, we first used elbow method to identify the ideal number of customer segments (K). We then applied PCA which reduces dozens of customer attributes into just two dimensions to visualize these clusters. The resulting plot clearly show how different customer profiles naturally separate and group together.
|
|
To further validate and visualize the distinct characteristics of each cluster, we constructed the summary dashboard illustrated below, whcih maps demographic and behavioral metrics against the segmentation.
Cluster Distribution (Pie chart): this chart visualizes the overall proportion of the customer base assigned to each algorithmic cluster. The segmentation reveals an extreme population imbalance. Cluster 0 constitutes the vast majority of the dataset, accounting for 97.9% of all customers. Conversely, Cluster 1 represents a distinct minority, comprising only 2.1% of the total population. The machine learning model helps identified one highly specific, nice profile (Cluster 1). Given its size, Cluster 0 will likely drive the aggregate trend of business, whereas Cluster 1 requires isolated analysis to understand its unique behavioral drivers.
Scale Preferences by Cluster (Horizontal Stacked Bar chart): this chart breaks down the total policy count by household/family type (Single, Couple, Family, Single Parent) and maps the proportional contribution of both clusters to each category. The "Single" category represents the largest overall volume of policies by a significant margin, exceeding 1.2. million policies. While Cluster 0 (Red) dominates the total volume across all scale types, the "Single" category shows the highest representation of Cluster 1 (Teal). Cluster 1's presence is also visible, albeit smaller, in the "Couple" and "Family" brackets, but is virtually negligible in the "Single Parent" category. Cluster 1 is not evenly distributed across all houshold types; it is heavily concentrated within the single-adult demographic. This provides the first critical behavioral clue for defining the minority segment.
Age Distribution by Cluster (Left Grouped Bar chart): the chart displays the absolute count fo customers in each cluster across five age bands. The red bars indicate that Cluster 0 has a broad an relatively stable age distribution. The highest concentration falls within the "30-39 years", "40-49 years", and "50-59 years". Even the "60+ years" segment remains robust. Cluster 1 (Teal) show a starkly different pattern, almost entirely concentrated in the younger age bands. The highest concentration is in the "less than 30 years" and "30-39 years" categories, with minimal representation in the 40+ age brackets. The insight of this bar chart tells that age is defining variable for Cluster 1. While Cluster 0 represents a lifecycle-sanning customer base, Cluster 1 is demographically skewed toward young adults (under 40).
Product Distribution by Cluster (Right Grouped Bar chart): the chart illustrates the purchasing volume for various products (Other and Products A through H) broken down by cluster. Cluster 0 exhibits high purchasing volumes across the entire product portfolio. The highest volumes are seen in "Other" and "Product A" but significant adoption continues steadily down the line through Products B to H. The teal bars reveal a severe drop-off in product engagement for Cluster 1. Their purchasing is almost exclusively limited to "Other" and "Product A" categories. Penetration into Products C to H is negligible, with virtually no visible bars for those categories.
This is the most impactful visualization for business strategy. Cluster 1 shows strong initial engagement with introductory or entry-level offerings (Other, Product A) but completely fails to cross-sell into the rest of the portfolio. This indicates either a lack of product for their demographic (young singles) or a lack of targeted marketing to move them up the product ladder.
The four graphs paint a coherent picture:
The business is successfully retaining massive, older, multi-product buying majority (Cluster 0). However, there is a distinct, young, single demographic (Cluster 1) that is entering the funnel via introductory offers but failing to progress to the broader product suite. This presents a clear, data-backed opportunity for a tailored onboarding strategy.
Scale Preferences by Cluster (Horizontal Stacked Bar chart): this chart breaks down the total policy count by household/family type (Single, Couple, Family, Single Parent) and maps the proportional contribution of both clusters to each category. The "Single" category represents the largest overall volume of policies by a significant margin, exceeding 1.2. million policies. While Cluster 0 (Red) dominates the total volume across all scale types, the "Single" category shows the highest representation of Cluster 1 (Teal). Cluster 1's presence is also visible, albeit smaller, in the "Couple" and "Family" brackets, but is virtually negligible in the "Single Parent" category. Cluster 1 is not evenly distributed across all houshold types; it is heavily concentrated within the single-adult demographic. This provides the first critical behavioral clue for defining the minority segment.
Age Distribution by Cluster (Left Grouped Bar chart): the chart displays the absolute count fo customers in each cluster across five age bands. The red bars indicate that Cluster 0 has a broad an relatively stable age distribution. The highest concentration falls within the "30-39 years", "40-49 years", and "50-59 years". Even the "60+ years" segment remains robust. Cluster 1 (Teal) show a starkly different pattern, almost entirely concentrated in the younger age bands. The highest concentration is in the "less than 30 years" and "30-39 years" categories, with minimal representation in the 40+ age brackets. The insight of this bar chart tells that age is defining variable for Cluster 1. While Cluster 0 represents a lifecycle-sanning customer base, Cluster 1 is demographically skewed toward young adults (under 40).
Product Distribution by Cluster (Right Grouped Bar chart): the chart illustrates the purchasing volume for various products (Other and Products A through H) broken down by cluster. Cluster 0 exhibits high purchasing volumes across the entire product portfolio. The highest volumes are seen in "Other" and "Product A" but significant adoption continues steadily down the line through Products B to H. The teal bars reveal a severe drop-off in product engagement for Cluster 1. Their purchasing is almost exclusively limited to "Other" and "Product A" categories. Penetration into Products C to H is negligible, with virtually no visible bars for those categories.
This is the most impactful visualization for business strategy. Cluster 1 shows strong initial engagement with introductory or entry-level offerings (Other, Product A) but completely fails to cross-sell into the rest of the portfolio. This indicates either a lack of product for their demographic (young singles) or a lack of targeted marketing to move them up the product ladder.
The four graphs paint a coherent picture:
The business is successfully retaining massive, older, multi-product buying majority (Cluster 0). However, there is a distinct, young, single demographic (Cluster 1) that is entering the funnel via introductory offers but failing to progress to the broader product suite. This presents a clear, data-backed opportunity for a tailored onboarding strategy.
Key Performance Indicator & Recommendation
In conclusion, the analysis indicates that customers with a tenure of 0-2 years exhibit the highest cancellation rates, this suggests an opportunity to strengthen the onboarding process and enhance early-stage customer engagement to improve retention.
The findings also indicate a positive relationship between longer customer tenure and loyalty. Based on this observation, conducting regular satisfaction surveys with long-term customers could provide valuable insights into the factors that contribute to retention and help identify best practices that may be applied to newer customers.
From a product perspective, Products C, D and G demonstrate comparatively higher cancellation rates. It may therefore be worthwhile to review the performenace and value proposition of these products to determine whether portfolio optimization, product enhancement, or consolidation would improve overall customer outcomes.
In addition, customers under the age of 50 appear to exhibit more diverse behavioural patterns, suggesting a wider range of motivations for purchasing Private Health Insurance. As this segment represents a significant growth opportunity, a more personalized approach to product offerings and customer engagement may help better address their varying needs.
Finally, the analysis highlights the importance of recognizing differences across customer segments. Providing sales teams with training on segment-specific characteristics and adopting more tailored engagement strategies may improve customer experience and retention, acknowledging that the business serves multiple customer groups with distinct needs, behaviours, and value expectations rather than a single homogeneous market.
The findings also indicate a positive relationship between longer customer tenure and loyalty. Based on this observation, conducting regular satisfaction surveys with long-term customers could provide valuable insights into the factors that contribute to retention and help identify best practices that may be applied to newer customers.
From a product perspective, Products C, D and G demonstrate comparatively higher cancellation rates. It may therefore be worthwhile to review the performenace and value proposition of these products to determine whether portfolio optimization, product enhancement, or consolidation would improve overall customer outcomes.
In addition, customers under the age of 50 appear to exhibit more diverse behavioural patterns, suggesting a wider range of motivations for purchasing Private Health Insurance. As this segment represents a significant growth opportunity, a more personalized approach to product offerings and customer engagement may help better address their varying needs.
Finally, the analysis highlights the importance of recognizing differences across customer segments. Providing sales teams with training on segment-specific characteristics and adopting more tailored engagement strategies may improve customer experience and retention, acknowledging that the business serves multiple customer groups with distinct needs, behaviours, and value expectations rather than a single homogeneous market.