For most of the time, biomedical researchers have been dealing with ordinal outcome variable in multilevel models where patients are nested in doctors. We can justifiably apply multilevel cumulative logit model, where the outcome variable represents the mild, severe, and extremely severe intensity of diseases like malaria and typhoid in the form of ordered categories. Based on our simulation conditions, Maximum Likelihood (ML) method is better than Penalized Quasilikelihood (PQL) method in three-category ordinal outcome variable. PQL method, however, performs equally well as ML method where five-category ordinal outcome variable is used. Further, to achieve power more than 0.80, at least 50 groups are required for both ML and PQL methods of estimation. It may be pointed out that, for five-category ordinal response variable model, the power of PQL method is slightly higher than the power of ML method.
Data collected from hospitals and educational institutions are mostly multilevel or hierarchical data. This type of data is frequently used by researchers to construct statistical models such as multilevel models, hierarchical models, or mixed effects models [
It is really challenging to decide about an appropriate sample size for multilevel ordinal logistic models. In the contemporary literature, only [
A very popular concept is used in social sciences to develop a dichotomous multilevel logistic model through a latent continuous variable model [
That is why the squared sigma
Now
For identification purpose, it is common to set the first threshold to zero and to allow an intercept in the model [
The fixed effect parameters
The accuracy of different fixed effect and random effect parameters estimates was calculated through the relative parameter bias, that is,
Average relative bias of fixed effect estimates obtained as a function of groups, group size, and ICC, collapsing over the category distribution and number of categories.
Groups | Group size | ICC | ML method | PQL method | ||||
---|---|---|---|---|---|---|---|---|
|
|
|
|
|
| |||
30 | 5 | 0.1 | 0.0953 | 0.0960 | 0.0523 | −0.0925 | −0.0858 | −0.0810 |
0.3 | 0.0790 | 0.0823 | 0.0397 | −0.1061 | −0.0967 | −0.0899 | ||
0.5 | 0.0636 | 0.0675 | 0.0313 | −0.1186 | −0.1074 | −0.0999 | ||
|
||||||||
30 | 30 | 0.1 | 0.0708 | 0.0376 | 0.0469 | −0.0601 | −0.0595 | −0.0603 |
0.3 | 0.0577 | 0.0292 | 0.0496 | −0.0677 | −0.0714 | −0.0797 | ||
0.5 | 0.0506 | 0.0263 | 0.0347 | −0.0732 | −0.0816 | −0.0878 | ||
|
||||||||
30 | 50 | 0.1 | 0.0314 | 0.0562 | 0.0785 | −0.0427 | −0.0433 | −0.0452 |
0.3 | 0.0256 | 0.0507 | 0.0778 | −0.0539 | −0.0551 | −0.0720 | ||
0.5 | 0.0161 | 0.0466 | 0.0696 | −0.0642 | −0.0598 | −0.0829 | ||
|
||||||||
50 | 5 | 0.1 | 0.0750 | 0.0607 | 0.0656 | −0.0587 | −0.0616 | −0.0678 |
0.3 | 0.0577 | 0.0485 | 0.0558 | −0.0645 | −0.0669 | −0.0763 | ||
0.5 | 0.0471 | 0.0489 | 0.0423 | −0.0742 | −0.0690 | −0.0796 | ||
|
||||||||
50 | 30 | 0.1 | 0.0175 | 0.0196 | 0.0285 | −0.0410 | −0.0518 | −0.0506 |
0.3 | 0.0202 | 0.0154 | 0.0219 | −0.0474 | −0.0443 | −0.0559 | ||
0.5 | 0.0160 | 0.0154 | 0.0337 | −0.0536 | −0.0570 | −0.0611 | ||
|
||||||||
50 | 50 | 0.1 | 0.0281 | 0.0248 | 0.0501 | −0.0252 | −0.0384 | −0.0400 |
0.3 | 0.0340 | 0.0260 | 0.0333 | −0.0391 | −0.0370 | −0.0447 | ||
0.5 | 0.0265 | 0.0243 | 0.0307 | −0.0412 | −0.0402 | −0.0400 | ||
|
||||||||
100 | 5 | 0.1 | 0.0319 | 0.0357 | 0.0421 | −0.0520 | −0.0505 | −0.0472 |
0.3 | 0.0188 | 0.0169 | 0.0255 | −0.0549 | −0.0416 | −0.0357 | ||
0.5 | 0.0235 | 0.0188 | 0.0211 | −0.0518 | −0.0536 | −0.0303 | ||
|
||||||||
100 | 30 | 0.1 | −0.0025 | 0.0066 | 0.0092 | −0.0206 | −0.0270 | −0.0236 |
0.3 | −0.0033 | 0.0044 | 0.0060 | −0.0216 | −0.0324 | −0.0230 | ||
0.5 | −0.0037 | 0.0019 | 0.0041 | −0.0248 | −0.0172 | −0.0182 | ||
|
||||||||
100 | 50 | 0.1 | 0.0064 | 0.0069 | 0.0041 | −0.0102 | −0.0114 | −0.0084 |
0.3 | 0.0051 | 0.0048 | 0.0025 | −0.0105 | −0.0125 | −0.0243 | ||
0.5 | 0.0030 | 0.0031 | 0.0042 | −0.0148 | −0.0120 | −0.0205 |
Average relative bias of random effect estimates obtained as a function of groups, group size, and ICC, collapsed over the category distribution and number of categories.
Groups | Group size | ICC | ML method | PQL method | ||
---|---|---|---|---|---|---|
|
|
|
| |||
30 | 5 | 0.1 | −0.1106 | 0.2000 | −0.0812 | 0.1143 |
0.3 | −0.0701 | −0.0176 | −0.1158 | −0.1403 | ||
0.5 | −0.0326 | 0 .0127 | −0.1469 | −0.1725 | ||
|
||||||
30 | 30 | 0.1 | −0.0806 | −0.0572 | −0.0645 | −0.1429 |
0.3 | −0.0578 | −0.0702 | −0.0926 | −0.1053 | ||
0.5 | −0.0327 | −0.0633 | −0.1196 | −0.1311 | ||
|
||||||
30 | 50 | 0.1 | −0.0645 | −0.0858 | −0.0483 | −0.1066 |
0.3 | −0.0578 | −0.0351 | −0.0689 | −0.0878 | ||
0.5 | −0.0490 | −0.0620 | −0.0794 | −0.1079 | ||
|
||||||
50 | 5 | 0.1 | −0.0645 | 0.0286 | −0.0967 | 0.0282 |
0.3 | −0.0330 | −0.0702 | −0.1239 | −0.1555 | ||
0.5 | 0.0226 | −0.0507 | −0.1414 | −0.1809 | ||
|
||||||
50 | 30 | 0.1 | −0.0483 | −0.1143 | −0.0725 | −0.1525 |
0.3 | −0.0413 | −0.0527 | −0.0893 | −0.1209 | ||
0.5 | −0.0272 | −0.0380 | −0.1022 | −0.1461 | ||
|
||||||
50 | 50 | 0.1 | −0.0483 | −0.0572 | −0.0512 | −0.0882 |
0.3 | −0.0330 | −0.0176 | −0.0656 | −0.0933 | ||
0.5 | −0.0272 | −0.0291 | −0.0751 | −0.1131 | ||
|
||||||
100 | 5 | 0.1 | −0.0322 | 0.0000 | −0.0754 | −0.1220 |
0.3 | −0.0330 | −0.0179 | −0.1063 | −0.1425 | ||
0.5 | 0.0055 | −0.0254 | −0.1095 | −0.1520 | ||
|
||||||
100 | 30 | 0.1 | −0.0322 | −0.0286 | −0.0442 | −0.0832 |
0.3 | −0.0328 | −0.0176 | −0.0611 | −0.1051 | ||
0.5 | −0.0055 | 0.0000 | −0.0709 | −0.1165 | ||
|
||||||
100 | 50 | 0.1 | −0.0330 | 0.0000 | −0.0391 | −0.0595 |
0.3 | −0.0333 | −0.0180 | −0.0476 | −0.0682 | ||
0.5 | −0.0540 | 0.0000 | −0.0465 | −0.0729 |
Similarly, power was computed as
Empirical coverage rates of 95% confidence intervals were used to judge the accuracy of the standard errors of estimated parameters [
Tables
Tables
Power of fixed effect estimates of five-category response multilevel ordinal logistic model (method: ML and PQL, category distribution: symmetrical).
Groups | Group size | ICC | ML method | PQL method | ||||
---|---|---|---|---|---|---|---|---|
|
|
|
|
|
| |||
30 | 5 | 0.1 | 0.536 | 0.551 | 0.488 | 0.607 | 0.611 | 0.523 |
0.3 | 0.561 | 0.559 | 0.492 | 0.596 | 0.621 | 0.539 | ||
0.5 | 0.572 | 0.550 | 0.501 | 0.599 | 0.616 | 0.531 | ||
|
||||||||
30 | 30 | 0.1 | 0.624 | 0.661 | 0.569 | 0.661 | 0.659 | 0.591 |
0.3 | 0.639 | 0.639 | 0.589 | 0.650 | 0.672 | 0.602 | ||
0.5 | 0.635 | 0.669 | 0.594 | 0.658 | 0.649 | 0.611 | ||
|
||||||||
30 | 50 | 0.1 | 0.715 | 0.709 | 0.661 | 0.739 | 0.693 | 0.681 |
0.3 | 0.731 | 0.721 | 0.679 | 0.749 | 0.716 | 0.699 | ||
0.5 | 0.739 | 0.732 | 0.703 | 0.745 | 0.726 | 0.693 | ||
|
||||||||
50 | 5 | 0.1 | 0.850 | 0.835 | 0.731 | 0.886 | 0.849 | 0.756 |
0.3 | 0.861 | 0.849 | 0.729 | 0.897 | 0.863 | 0.769 | ||
0.5 | 0.873 | 0.858 | 0.740 | 0.897 | 0.869 | 0.763 | ||
|
||||||||
50 | 30 | 0.1 | 0.871 | 0.864 | 0.778 | 0.903 | 0.914 | 0.809 |
0.3 | 0.877 | 0.856 | 0.789 | 0.914 | 0.895 | 0.795 | ||
0.5 | 0.869 | 0.879 | 0.768 | 0.909 | 0.899 | 0.799 | ||
|
||||||||
50 | 50 | 0.1 | 0.891 | 0.859 | 0.815 | 0.956 | 0.916 | 0.849 |
0.3 | 0.882 | 0.880 | 0.826 | 0.969 | 0.931 | 0.871 | ||
0.5 | 0.896 | 0.878 | 0.831 | 0.972 | 0.939 | 0.878 | ||
|
||||||||
100 | 5 | 0.1 | 0.992 | 1.000 | 0.851 | 1.000 | 1.000 | 0.914 |
0.3 | 0.998 | 1.000 | 0.859 | 1.000 | 0.997 | 0.902 | ||
0.5 | 0.998 | 0.995 | 0.867 | 1.000 | 1.000 | 0.909 | ||
|
||||||||
100 | 30 | 0.1 | 0.995 | 0.989 | 0.890 | 1.000 | 1.000 | 0.929 |
0.3 | 0.997 | 0.998 | 0.897 | 1.000 | 1.000 | 0.936 | ||
0.5 | 0.992 | 0.998 | 0.879 | 1.000 | 1.000 | 0.939 | ||
|
||||||||
100 | 50 | 0.1 | 0.989 | 0.992 | 0.906 | 1.000 | 1.000 | 0.952 |
0.3 | 0.996 | 0.996 | 0.919 | 1.000 | 1.000 | 0.959 | ||
0.5 | 0.996 | 0.997 | 0.899 | 1.000 | 1.000 | 0.951 |
95% CI coverage rates for the estimates in a three-category ordinal response variable model by groups, group size, and intraclass correlation (method: ML, category distribution: symmetrical).
Parameters | Number of groups | Group size | ICC | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
30 | 100 | 120 |
|
5 | 30 | 50 |
|
0.1 | 0.3 | 0.5 |
|
|
|
0.942 | 0.949 | 0.953 | 0.0006 | 0.948 | 0.946 | 0.949 | 0.7886 | 0.945 | 0.948 | 0.951 | 0.0270 |
|
0.941 | 0.947 | 0.952 | 0.0008 | 0.946 | 0.946 | 0.950 | 0.3865 | 0.943 | 0.947 | 0.951 | 0.0165 |
|
0.939 | 0.946 | 0.951 | 0.0007 | 0.945 | 0.945 | 0.947 | 0.4895 | 0.946 | 0.946 | 0.946 | 0.9737 |
|
0.905 | 0.908 | 0.918 | 0.0041 | 0.908 | 0.910 | 0.913 | 0.3086 | 0.911 | 0.910 | 0.909 | 0.6199 |
|
0.907 | 0.915 | 0.927 | 0.0000 | 0.912 | 0.917 | 0.920 | 0.0632 | 0.918 | 0.913 | 0.918 | 0.8929 |
95% CI coverage rates for the estimates in a three-category ordinal response variable model by groups, group size, and intraclass correlation (method: ML, category distribution: skewed).
Parameters | Number of groups | Group size | ICC | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
30 | 100 | 120 |
|
5 | 30 | 50 |
|
0.1 | 0.3 | 0.5 |
|
|
|
0.941 | 0.948 | 0.951 | 0.0053 | 0.946 | 0.948 | 0.949 | 0.5283 | 0.943 | 0.948 | 0.949 | 0.0503 |
|
0.946 | 0.949 | 0.951 | 0.1780 | 0.947 | 0.948 | 0.950 | 0.4000 | 0.947 | 0.948 | 0.949 | 0.7360 |
|
0.942 | 0.946 | 0.955 | 0.0001 | 0.944 | 0.950 | 0.949 | 0.0821 | 0.948 | 0.947 | 0.948 | 0.9733 |
|
0.900 | 0.908 | 0.917 | 0.0000 | 0.904 | 0.908 | 0.913 | 0.0494 | 0.908 | 0.908 | 0.910 | 0.7760 |
|
0.909 | 0.913 | 0.922 | 0.0011 | 0.909 | 0.915 | 0.920 | 0.0077 | 0.918 | 0.913 | 0.918 | 0.8939 |
95% CI coverage rates for the estimates in a five-category ordinal response variable model by groups, group size, and intraclass correlation (method: ML, category distribution: symmetrical).
Parameters | Number of groups | Group size | ICC | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
30 | 100 | 120 |
|
5 | 30 | 50 |
|
0.1 | 0.3 | 0.5 |
|
|
|
0.946 | 0.948 | 0.950 | 0.2542 | 0.945 | 0.949 | 0.951 | 0.0702 | 0.947 | 0.946 | 0.951 | 0.3144 |
|
0.945 | 0.946 | 0.949 | 0.4840 | 0.945 | 0.947 | 0.948 | 0.2440 | 0.947 | 0.946 | 0.947 | 0.9730 |
|
0.943 | 0.947 | 0.949 | 0.0490 | 0.946 | 0.947 | 0.946 | 0.9470 | 0.948 | 0.944 | 0.948 | 0.9210 |
|
0.904 | 0.907 | 0.918 | 0.0001 | 0.909 | 0.910 | 0.912 | 0.4497 | 0.907 | 0.912 | 0.912 | 0.2305 |
|
0.903 | 0.915 | 0.926 | 0.0000 | 0.911 | 0.914 | 0.918 | 0.0585 | 0.914 | 0.915 | 0.914 | 1.0000 |
95% CI coverage rates for the estimates in a five-category ordinal response variable model by groups, group size, and intraclass correlation (method: ML, category distribution: skewed).
Parameters | Number of groups | Group size | ICC | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
30 | 100 | 120 |
|
5 | 30 | 50 |
|
0.1 | 0.3 | 0.5 |
|
|
|
0.939 | 0.944 | 0.946 | 0.0340 | 0.947 | 0.933 | 0.948 | 0.8470 | 0.944 | 0.943 | 0.941 | 0.3860 |
|
0.945 | 0.947 | 0.949 | 0.1940 | 0.951 | 0.937 | 0.953 | 0.6170 | 0.947 | 0.947 | 0.947 | 0.9470 |
|
0.944 | 0.947 | 0.948 | 0.1650 | 0.948 | 0.939 | 0.951 | 0.4080 | 0.944 | 0.947 | 0.947 | 0.3720 |
|
0.903 | 0.911 | 0.916 | 0.0024 | 0.910 | 0.904 | 0.917 | 0.0945 | 0.908 | 0.911 | 0.912 | 0.3885 |
|
0.907 | 0.916 | 0.922 | 0.0003 | 0.912 | 0.910 | 0.922 | 0.0328 | 0.913 | 0.915 | 0.916 | 0.5935 |
95% CI coverage rates for the estimates in a three-category ordinal response variable model by groups, group size, and intraclass correlation (method: PQL, category distribution: symmetrical).
Parameters | Number of groups | Group size | ICC | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
30 | 100 | 120 |
|
5 | 30 | 50 |
|
0.1 | 0.3 | 0.5 |
|
|
|
0.921 | 0.925 | 0.928 | 0.1135 | 0.918 | 0.926 | 0.932 | 0.0030 | 0.925 | 0.926 | 0.922 | 0.5156 |
|
0.918 | 0.922 | 0.925 | 0.0671 | 0.917 | 0.922 | 0.928 | 0.0170 | 0.928 | 0.924 | 0.912 | 0.0001 |
|
0.923 | 0.927 | 0.929 | 0.1099 | 0.920 | 0.926 | 0.933 | 0.0014 | 0.931 | 0.929 | 0.918 | 0.0009 |
|
0.897 | 0.900 | 0.907 | 0.0087 | 0.896 | 0.902 | 0.908 | 0.0044 | 0.905 | 0.903 | 0.895 | 0.0215 |
|
0.903 | 0.906 | 0.908 | 0.1327 | 0.898 | 0.906 | 0.912 | 0.0018 | 0.910 | 0.909 | 0.897 | 0.0055 |
95% CI coverage rates for the estimates in a three-category ordinal response variable model by groups, group size, and intraclass correlation (method: PQL, category distribution: skewed).
Parameters | Number of groups | Group size | ICC | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
30 | 100 | 120 |
|
5 | 30 | 50 |
|
0.1 | 0.3 | 0.5 |
|
|
|
0.918 | 0.920 | 0.924 | 0.1155 | 0.915 | 0.919 | 0.927 | 0.0053 | 0.928 | 0.923 | 0.912 | 0.0000 |
|
0.921 | 0.926 | 0.928 | 0.0245 | 0.920 | 0.927 | 0.931 | 0.0044 | 0.931 | 0.929 | 0.917 | 0.0002 |
|
0.921 | 0.924 | 0.928 | 0.0965 | 0.918 | 0.924 | 0.931 | 0.0031 | 0.929 | 0.926 | 0.918 | 0.0032 |
|
0.899 | 0.902 | 0.905 | 0.0567 | 0.897 | 0.902 | 0.907 | 0.0274 | 0.903 | 0.905 | 0.898 | 0.1459 |
|
0.905 | 0.908 | 0.911 | 0.1633 | 0.901 | 0.910 | 0.914 | 0.0015 | 0.916 | 0.909 | 0.900 | 0.0003 |
95% CI coverage rates for the estimates in a five-category ordinal response variable model by groups, group size, and intraclass correlation (method: PQL, category distribution: symmetrical).
Parameters | Number of groups | Group size | ICC | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
30 | 100 | 120 |
|
5 | 30 | 50 |
|
0.1 | 0.3 | 0.5 |
|
|
|
0.940 | 0.945 | 0.942 | 0.6312 | 0.936 | 0.942 | 0.948 | 0.0007 | 0.944 | 0.945 | 0.937 | 0.0590 |
|
0.940 | 0.945 | 0.946 | 0.0997 | 0.938 | 0.940 | 0.948 | 0.0027 | 0.945 | 0.945 | 0.940 | 0.1292 |
|
0.938 | 0.944 | 0.942 | 0.2029 | 0.937 | 0.942 | 0.946 | 0.0156 | 0.943 | 0.942 | 0.939 | 0.2029 |
|
0.906 | 0.910 | 0.911 | 0.2328 | 0.904 | 0.910 | 0.914 | 0.0458 | 0.913 | 0.911 | 0.903 | 0.0196 |
|
0.907 | 0.907 | 0.909 | 0.6620 | 0.904 | 0.909 | 0.910 | 0.2170 | 0.909 | 0.910 | 0.902 | 0.1050 |
95% CI coverage rates for the estimates in a five-category ordinal response variable model by groups, group size, and intraclass correlation (method: PQL, category distribution: skewed).
Parameters | Number of groups | Group size | ICC | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
30 | 100 | 120 |
|
5 | 30 | 50 |
|
0.1 | 0.3 | 0.5 |
|
|
|
0.941 | 0.941 | 0.943 | 0.5893 | 0.937 | 0.942 | 0.946 | 0.0047 | 0.944 | 0.943 | 0.937 | 0.0611 |
|
0.939 | 0.940 | 0.941 | 0.5301 | 0.935 | 0.940 | 0.946 | 0.0013 | 0.942 | 0.941 | 0.937 | 0.1578 |
|
0.942 | 0.943 | 0.944 | 0.6300 | 0.936 | 0.944 | 0.950 | 0.0000 | 0.946 | 0.944 | 0.938 | 0.0160 |
|
0.903 | 0.907 | 0.909 | 0.1598 | 0.901 | 0.908 | 0.910 | 0.0658 | 0.907 | 0.907 | 0.904 | 0.6272 |
|
0.909 | 0.911 | 0.914 | 0.3463 | 0.905 | 0.911 | 0.917 | 0.0060 | 0.915 | 0.915 | 0.903 | 0.0037 |
Table
Similarly, ML method random effects estimates were substantially biased when the number of groups was 30 as shown in Table
Table
Table
Table
Table
Table
Table
Table
Table
Table
In both symmetrical and skewed distribution shapes of category responses,
The accuracy of standard errors of the estimates (excluding threshold estimates) was judged through empirical coverage rates. The influence of the number of groups was significant on the accuracy of both three-category and five-category multilevel ordinal logistic models estimates standard errors under ML method of estimation. On the contrary, the group size factor and ICC had an insignificant effect on estimates standard errors under ML method of estimation. Estimates standard errors were least biased when number of groups was 100. On the other hand, the influence of group size factor was highly significant on the accuracy of estimates standard errors under PQL method of estimation. Furthermore, ICC also influenced estimates standard errors; that is, standard errors were substantially biased when population random effects were medium (
The power rates of PQL estimates were slightly higher than that of ML estimates when ordinal response variable had five categories, which indicate that PQL standard errors were least biased due to increases in the number of categories of ordinal response variable.
In general, ML method performed well in terms of estimates small bias, high coverage rates, and high power rates when ordinal response variable had three categories. However, in five-category ordinal response variable model, PQL method performances were comparable to those of ML method. PQL estimates were poor when population random effects (ICC) were medium and large while ML estimates were poor in small population random effects. In addition, ML estimates and estimates standard errors benefitted from larger number of groups while PQL estimates and estimates standard errors benefitted from larger group sizes. We recommend at least 100 groups and 30 units per group to achieve accurate multilevel ordinal logistic model estimates and estimates standard errors when method of estimation is ML. Furthermore, 100 groups and at least 50 units per group are essential for accurate multilevel ordinal logistic model estimates and estimates standard errors when method of estimation is PQL. Similarly, at least 50 groups are essential to achieve 0.80 or more power for both ML and PQL methods of estimation. On the basis of this study, it is recommended that PQL method may be avoided when group sizes are small, number of groups are large, random effects are medium and large, and outcome variable has three categories. In such conditions, ML method is the best option. However, when the outcome variable has five or more categories, random effects are small, group sizes are large, and number of groups is small, PQL method may be better option.
The authors declare that there is no conflict of interests regarding publication of this paper.