To get background_gradient plot for all data frames in the for loop

Question

I tried the below code to get the background_gradient plot for each diabetes_group. It is giving me only plot for the last group in the plot. How to get background_gradiant for all items (x) in diabetes_group with item name on the top ?

Subset of df before groupby looks like

Diabetes_Group  DM-1    DM-2    corr
Group1  56  56  1
Group1  56  58  0.611
Group1  56  61  0.448
Group1  56  63  0.3587
Group1  58  56  0.611
Group1  58  58  1
Group1  58  61  0.60113
Group1  58  63  0.534858
Group1  61  56  0.448
Group1  61  58  0.60113
Group1  61  61  1
Group1  61  63  0.622206
Group1  63  56  0.3587
Group1  63  58  0.534858
Group1  63  61  0.622206
Group1  63  63  1
Group2  78  78  1
Group2  78  85  0.622703
Group2  78  94  0.622244
Group2  78  23  0.687826
Group2  85  78  0.622703
Group2  85  85  1
Group2  85  94  0.697313
Group2  85  23  0.612089
Group2  94  78  0.622244
Group2  94  85  0.697313
Group2  94  94  1
Group2  94  23  0.603834
Group2  23  78  0.687826
Group2  23  85  0.612089
Group2  23  94  0.603834
Group2  23  23  1
Group3  27  27  1
Group3  27  80  0.118955
Group3  27  147 0.32038
Group3  27  9   0.335264
Group3  80  27  0.118955
Group3  80  80  1
Group3  80  147 0.430287
Group3  80  9   0.406426
Group3  147 27  0.32038
Group3  147 80  0.430287
Group3  147 147 1
Group3  147 9   0.546452
Group3  9   27  0.335264
Group3  9   80  0.406426
Group3  9   147 0.546452
Group3  9   9   1

My code

corr_matrix = []

for x, group in df.groupby('diabetes_group'):
    matrix = group[["diabetes_group", 'DM-1', 'DM-2', 'corr']]
    matrix_pivot = pd.pivot_table(matrix, values = 'corr', index=['DM-1'], columns=['DM-2'])
    d = matrix_pivot.style.background_gradient(cmap='Set3')

Answer 1

corr_matrix = []

for x, group in df.groupby('diabetes_group'):
    matrix = group[["diabetes_group", 'DM-1', 'DM-2', 'corr']]
    matrix_pivot = pd.pivot_table(matrix, values = 'corr', index=['DM-1'], columns=['DM-2'])
    d = matrix_pivot.style.background_gradient(cmap='Set3').set_caption(x)
    display(d)

Answer 2

Sorry I am really late answering, the set_caption was cool.

Background What we are talking about here is the style Class in pandas of which set_background is one method. Usually, it is for outputting a pandas dataframe as a webpage. If I was front-end developer it would be cool. In this example it is used to colour the different bits of the dataframe, presumably to make it easier to read a lot of data.

The thing I would have done is dump to Excel. This will preserve the colouration. The bits contained within # are my codes.

from Pathlib import Path
import pandas as pd

##########
path = Path('/path/todir/')
##########
for x, group in df.groupby('diabetes_group'):
    matrix = group[["diabetes_group", 'DM-1', 'DM-2', 'corr']]
    matrix_pivot = pd.pivot_table(matrix, values = 'corr', index=['DM-1'], columns=['DM-2'])
    d = matrix_pivot.style.background_gradient(cmap='Set3').set_caption(x)

    ##########
    pathout = Path(path, x + '.xlsx')
    d.to_excel(pathout)  
    ##########

What was happening originally was the 'loop' was overriding the diabetes Group in previous loop, hence appeared to be disappearing. What will happen now is an excel file will be produced separately for each group and the file name labeled as Group1.xlsx, Group2.xlsx etc ...

Here I dumped my test output to HTML simply via to_html and then to pdf.

Notes The variable corr_matrix = [] does not take part in this code. It may be important to other bits of code. I'm not sure matrix variable does much, but I've not checked.

The issue with IPython display is the output is not saved, so the code needs to be run each time the analyst wants to examine the correlation coefficients between DM-1 and DM-2.

---

My personal dataframe was

df = pd.DataFrame({'diabetes_group':['Group1','Group1','Group2','Group2', 'Group3','Group3'],'DM-1':[56,56,56,56,56,56], 'DM-2':[56,58,61,63,56,58], 'corr':[0.2,0.3,0.5,0.3,0.2,0.1]})