{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Demographic Plots\n",
    "This notebook is used to read the data from the pickle files and to create a dataframe with the demographic data.\n",
    "With this data we can create plots to show the distribution of the demographic data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from scipy.stats import chi2_contingency\n",
    "import sys\n",
    "import json\n",
    "\n",
    "sys.path.append('../scripts')\n",
    "import data_helper"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading GSVT\n",
      "Reading AFIB\n",
      "Reading SR\n",
      "Reading SB\n"
     ]
    }
   ],
   "source": [
    "data_demographic = data_helper.load_data(only_demographic=True)\n",
    "\n",
    "df_dgc = pd.DataFrame(data_demographic)\n",
    "\n",
    "# Change from group to category\n",
    "age_categories = [0, 10, 20, 30, 40, 50, 60, 70, 80, 90]\n",
    "df_dgc['age_group'] = pd.cut(df_dgc['age'], bins=age_categories)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 69556 characters]",
      "text/plain": [
       "<Figure size 640x480 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Correlation matrix\n",
    "corr_matrix_age_diag= pd.crosstab(df_dgc['age_group'], df_dgc['diag'])\n",
    "\n",
    "# Plot the correlation matrix\n",
    "sns.heatmap(corr_matrix_age_diag, annot=True, cmap='coolwarm', fmt='d')\n",
    "plt.title('Correlationmatrix of Age and Diagnostic Groups  ', fontsize=16)\n",
    "plt.xlabel('Diagnostic Group')\n",
    "plt.ylabel('Age Group')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chi-Square Statistc Test\n",
    "\n",
    "With this test we want to check if there is a significant difference between the groups."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Chi-Square Statistic: 11503.925605129169\n",
      "P-value: 0.0\n",
      "Chi-Square Statistic for SB in 60-70 vs others: 401.0271328859626\n",
      "P-value for SB in 60-70 vs others: 3.291102770159182e-89\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# Chi-square test\n",
    "chi2, p, _, _ = chi2_contingency(corr_matrix_age_diag)\n",
    "\n",
    "# Difference between observed and expected frequencies\n",
    "print(f\"Chi-Square Statistic: {chi2}\")\n",
    "print(f\"P-value: {p}\")\n",
    "\n",
    "# Check if SB (Sinusbradykardie) has a significantly higher frequency in the 60-70 age group\n",
    "sb_60_70 = corr_matrix_age_diag.loc[pd.Interval(60, 70, closed='right'), 'SB']\n",
    "sb_other = corr_matrix_age_diag.drop(pd.Interval(60, 70, closed='right')).sum()['SB']\n",
    "total_60_70 = corr_matrix_age_diag.loc[pd.Interval(60, 70, closed='right')].sum()\n",
    "total_other = corr_matrix_age_diag.drop(pd.Interval(60, 70, closed='right')).sum().sum()\n",
    "\n",
    "# Frequency table for the specific Chi-Square test\n",
    "observed = [[sb_60_70, total_60_70 - sb_60_70], [sb_other, total_other - sb_other]]\n",
    "chi2_sb, p_sb = chi2_contingency(observed)[:2]\n",
    "\n",
    "\n",
    "print(f\"Chi-Square Statistic for SB in 60-70 vs others: {chi2_sb}\")\n",
    "print(f\"P-value for SB in 60-70 vs others: {p_sb}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Correlation matrix of age groups and diagnoses.This matrix describes the four diagnosis groupings on the horizontal axis and the age groupings in decades steps on the vertical axis. The color scale blue (low) to red (high) describes the correlation of the two categorization types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 40696 characters]",
      "text/plain": [
       "<Figure size 640x480 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# cut out sex 'unknown' (because only one occurence)\n",
    "df_dgc_bineary = df_dgc[df_dgc['gender'] != 'Unknown']\n",
    "# Correlation matrix\n",
    "corr_matrix_sex_diag = pd.crosstab(df_dgc_bineary['gender'], df_dgc_bineary['diag'])\n",
    "\n",
    "# Plot the correlation matrix\n",
    "sns.heatmap(corr_matrix_sex_diag, annot=True, cmap='coolwarm', fmt='d')\n",
    "plt.title('Corleationmatrix of Gender and Diagnostic Group', fontsize=16)\n",
    "plt.xlabel('Diagnostic Group')\n",
    "plt.ylabel('Gender')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Correlation matrix of genders and diagnoses. This matrix describes the four diagnosis groupings on the horizontal axis and the gender in decades steps on the vertical axis. The color scale blue (low) to red (high) describes the correlation of the two categorization types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 46536 characters]",
      "text/plain": [
       "<Figure size 640x480 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "settings = json.load(open('../settings.json'))\n",
    "categories_dict = settings['labels']\n",
    "# 4 subplots for each diagnosis a histrgramm for the age\n",
    "fig, axs = plt.subplots(2, 2)\n",
    "fig.suptitle('Histogram of the Age Distribution', fontsize=16)\n",
    "for i, cat_name in enumerate(categories_dict.keys()):\n",
    "    ax = axs[i // 2, i % 2]\n",
    "    df_dgc[df_dgc['diag'] == cat_name]['age'].hist(ax=ax)\n",
    "    ax.set_title(cat_name)\n",
    "    ax.set_xlabel('Age')\n",
    "    ax.set_ylabel('Amount of Patients')\n",
    "    # add some space between the subplots\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Histogram of the age distribution. Breakdown of grouped diagnoses by age group and absolute incidence of diagnoses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 32200 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# a barplot for each category with the age grpuoped besides each other\n",
    "fig, ax = plt.subplots()\n",
    "sns.countplot(data=df_dgc_bineary, x='diag', hue='gender', ax=ax)\n",
    "plt.title('Amount of Patients per Diagnostic Group')\n",
    "plt.xlabel('Diagnostic Group')\n",
    "plt.ylabel('Amount of Patients')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Barplot to visualize the distribution of diagnoses by gender. Where blue is female and orange is male."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 37176 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# barplot how many diagnosis are in each age group\n",
    "fig, ax = plt.subplots()\n",
    "sns.countplot(data=df_dgc, x='age_group', hue='diag', ax=ax)\n",
    "plt.title('Amount of Patients per Age Group and Diagnostic Group')\n",
    "plt.xlabel('Age Group')\n",
    "plt.ylabel('Amount of Patients')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Barplot to show the distribution of diagnoses based on the age groupings. The difference in the incidence of the various diseases can be clearly seen here."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
