Evaluation of Psychometric Properties of a New Research Instrument for Measuring Collective Impact Based Cross-Sectoral Collaboration and Leadership: Oulu Collective Impact Study

,


Introduction
Te Finnish social and healthcare service system is based on public healthcare services to which everyone residing in the country is entitled.It is internationally considered to be both efective and efcient [1].However, over the past few years, it has not been able to respond satisfactorily to an increasing demand for services.One fundamental reason for this imbalance is stated to be the fragmented and isolated entities of service organizations [2][3][4].Furthermore, decentralized leadership and decision-making have been thought to have a deteriorating impact on the efciency of social and healthcare services [2].
Cross-sectoral collaboration is a widely used strategy with plenty of theories and models.Tese aim to solve the fragmentation of services and to improve dealing with complex and intertwined social and health problems.Te main purpose of this way of working is to bring together organizations and their expertise to solve defned signifcant societal and health problems [5,6].Despite various programs launched to enhance cross-sectoral collaboration, the service practices have remained somewhat isolated, and therefore, for example, the use of institutionalized social and healthcare services has remained at a high level [7][8][9][10][11].
Te collective impact (CI) approach is demonstrated to be a promising tool to improve cross-sectoral collaboration [12][13][14].Te CI approach is used to enhance cross-sectoral collaboration between public services to respond to the complex health and social problems of children, adolescents, and families [12,[15][16][17], such as nutrition, social and health problems, homelessness, and education-related challenges [15,18,19].Te structure of this approach is based on fve separate entities, termed key conditions by Kania & Kramer [12]: a common agenda, shared measurement system, mutually reinforcing activities, continuous communication, and backbone support organization.In addition, the role of leadership has been emphasized as an additional prerequisite for the successful implementation of CI-based work in everyday working practice [20][21][22].
In Finland, CI framework-based models have been implemented in some municipalities from the year 2013 onwards [13,14].Te models have aimed to enhance crosssectoral collaboration between daycare, education, and social and healthcare services.Te ultimate aim of these models has been to prevent the escalation of both health and social problems and, hence, to reduce the need for curative services for children.In the City of Oulu in Northern Finland, a cross-sectoral collaboration model based on the CI approach has been implemented in the welfare service areas since the year 2019 [14].During these implementation processes, the need for a research instrument to measure the realization of CI-based cross-sectoral collaboration and its leadership was emphasized.However, after reviewing previous literature to fnd an appropriate CI-based assessment tool, some CI-related surveys utilizing questionnaire-based data were found, e.g., [23][24][25][26][27], but those instruments did not appear to be comprehensive enough for the current CI research context.For example, a survey instrument of Salignac and workgroup [24] addressed only health-related collaboration but did not include other service sectors or CIrelated leadership.In summary, the prior instruments lacked child and family service context, leadership, or CI conditions-based items.
In order to evaluate the success of the implementation of the CI-based cross-sectoral collaboration model comprehensively in working practice, a new research instrument, named the Oulu cross-sectoral Collaboration and Leadership (OCL) scale, was developed by the research group.Tis instrument consists of seven domains; of these, six were based directly on CI conditions and one additional new domain addresses leadership as noted in discussions of earlier studies [20][21][22].Te preliminary OCL scale was thoroughly reviewed by a selected external group of professionals.After that, an online survey using this OCL scale was performed among the target group comprising professionals working in public health sectors for social care, healthcare, and education services in the city of Oulu.Te current study evaluates the psychometric properties of this new OCL scale.Suggestions for further development of the instrument are also presented.

Teoretical Background of the New Research Instrument.
Te OCL scale includes seven domains, six of which are based directly on the fve conditions of the CI approach [12].In addition, leadership was included in this scale as a seventh domain because the role of leadership is increasingly highlighted in the literature as an essential prerequisite for the successful realization of CI-related work [20][21][22]28].Te description of the domains and their theoretical context are presented in Table 1.

Development Process of the OCL Scale.
Te development process of the OCL scale is visualized in Figure 1.In the frst phase, fve conditions of the CI approach and the leadership were operationalized into the items by two members of the research group (TT and MN) with long-term experience and extensive knowledge of work using the CI approach.Te CI condition for the common agenda was assessed in the OCL scale using two separate domains.It was considered important to get distinct information on the strategy level (i.e., how the shared goal of the cross-sectoral collaboration was accepted) and operationalizing level (i.e., how the crosssectoral collaboration-related goal was carried out in everyday work).Tus, a total of seven separate domains (entities of items) were included in the OCL scale.Tese domains comprised a total of 47 items scored with a forced Likert-type scale from 1 (strongly disagree) to 4 (strongly agree).Te number of items in each domain is as follows: shared operating model, strategy level (8 items); shared operating model, action level (9 items); follow-up (6 items); practices that support (6 items); shared communication (5 items); backbone support structure (6 items); and leading (7 items).Te OCL scale is attached as supplementary material.
In the pilot phase, the preliminary OCL scale was tested and reviewed by a selected external group of fve professionals from social care, healthcare, and education services who had long-term experience in cross-sectoral collaboration work.Written feedback on the preliminary OCL scale was collected and checked thoroughly by the research group.Te external group of professionals considered the OCL scale appropriate for the intended purposes.Te target group of the online survey using the OCL scale comprised all professionals (n � 683) who were working in social, healthcare, and education services in the city of Oulu and were involved in concrete cross-sectoral collaboration work in their service sectors (Figure 2).Some of the potential participants working in the education sector were also in a leadership position (n � 97), and they had personal experience of concrete cross-sectoral collaboration, like arranging and participating in network meetings.Altogether 269 professionals responded to the online survey.Of them, 2 Health & Social Care in the Community Carrying out the implementation of all fve conditions of CI.Solving the feedback-based questions and starting new required initiatives [8,9,15] Te description of the domains of the new research instrument named the Oulu cross-sectoral Collaboration and Leadership (OCL) scale and their theoretical context [12,20].
Health & Social Care in the Community 3 15 were excluded from further analyses because their working unit had changed, and their current occupation did not meet the inclusion criteria of the initial target group of the study.Tus, the fnal number of respondents was 254, covering 37% of the target group.

Statistical Methods.
Data were mainly analyzed using the IBM SPSS Statistics, version 28; confrmatory factor analysis was conducted by utilizing R for statistical computing, version 4.2.3, and its lavaan package.In this study, the psychometric properties of the OCL scale comprised evaluation of the internal consistency of the items, assessment of the validity of the theoretical model in the dataset, and re-exploration of the factor structure of the OCL scale.Te detailed results of these analyses are summarized in Supplementary Tables 1 and 2.
First, the items of the seven domains of the OCL scale were evaluated for validity and reliability using the Cronbach's alpha statistics [30].Values above 0.9 are considered to describe excellent, 0.8 to 0.9 good, 0.7 to 0.8 acceptable, 0.6 to 0.7 questionable, 0.5 to 0.6 poor, and those below 0.5 unacceptable internal consistency of the items.Te evaluation of Cronbach's alpha if an item of a domain was deleted was used to assess whether an item should be removed to increase the internal consistency of a domain.
Sample adequacy was assessed using the Kaiser-Meyer-Olkin (KMO) statistic and Bartlett's sphericity test.Te KMO statistic was used to assess whether the sample size was sufcient for factor analysis.A KMO statistics value closer to 1.0 indicates better factor analysis results, while values below 0.6 suggest that factor analysis is not appropriate.Bartlett's sphericity test was used to assess whether the correlations of items in the data are strong enough to use a dimensionreduction technique.Te null hypothesis of Bartlett's sphericity test is that the items are not correlated.Furthermore, the adequacy of sample size was determined in terms of the study's subject to item ratio 5 : 1 and using the recommendation of N ≥ 200 for the theoretical model [31][32][33].In the current study, the sample size (N � 254) and the ratio of sample size to number of items (n � 47) of the scale were acceptable (5.4 : 1).
Confrmatory factor analysis (CFA) was used to test the validity of a theoretical model by analyzing the hypothesized relationships among the items of the seven domains of the OCL scale.Te following model ft statistics (and in parenthesis, criteria for acceptable level) were used to quantify the goodness of ft or the deviance from the perfect model ft in the dataset of our study: minimum discrepancy function by degrees of freedom divided (χ 2 /df) (≤3 � acceptable ft, ≤5 � reasonable ft), Comparative Fit Index (CFI) (>0.9 acceptable ft, value between 0.8 and 0.9 � moderate ft), Tucker-Lewis Index (TLI) (>0.9 � acceptable ft, value between 0.8 and 0.9 � moderate ft), root mean square error of approximation (RMSEA) (≤0.08 � acceptable ft, value between 0.08 and 0.10 � moderate ft), and standardized root mean squared residual (SRMR) (≤0.08 � acceptable ft) [34][35][36].However, it is recommended that the cut-of 4 Health & Social Care in the Community criteria of these ft indices should be interpreted as rules of thumb rather than golden rules for model ft [37].
Explorative factor analysis (EFA) with the varimax rotation method was used to assess the factor structure of the items in our dataset without a priori assumption of a certain structure or number of latent factors [38].A standardized factor loading of greater than 0.40 was considered to indicate a moderate correlation between the item and the underlying latent factor.Te factor structure based on the EFA was further evaluated with the same CFA using model ft and Cronbach's alpha statistics as was done in CFA for the theoretical model.
To visualize the relationship of the theoretical model built based on CFA to the EFA-based model, Sankey diagrams [39] were utilized for visualization.On the left side of the diagram, the seven domains with the number of items in each domain represent the theoretical factor structure of the OCL scale.Te width of each fow indicates the stability/ nonstability of factor structure, i.e., the proportion of items that "change place" or remain stable when comparing the theoretical CFA model to the factor structure of the EFAbased model (right side of the diagram) [40].Figure 3 is created using the open-source, online, and open-access software of SankeyMATIC [39].

Qualitative Analysis.
Te OCL scale included one openanswered question: "If you wish, you can comment on the operational models, practices, follow-up monitoring, and leading of cross-sectoral collaboration," providing respondent a possibility for additional comments.Content analysis was used to analyze these responses.Content analysis is defned as a systematic method describing and quantifying research phenomena.It is a report method for making valid conclusions from qualitative data with the purpose of providing knowledge, new insights, a representation of facts, and a practical guide to action [41].Content analysis includes both qualitative and quantitative approaches [42].
Based on the principles of the content analysis [41][42][43], the responses were thoroughly reviewed and categorized into three main categories: cross-sectoral collaboration, CIbased collaboration, and feedback on the use of research instrument (TT).In the current study, only the responses belonging to the use of the research instrument category were included for further analysis.To ensure the consistency of the content analysis [44], the other author (MN) reviewed the qualitative data and their categories.Finally, the data and the analysis process were presented to the research group to verify the consistency of the content analysis process.

Characteristics of the Study Participants.
Out of 254 survey participants, 45 (17.7%) were employed in leadership positions while the remaining 209 (82.3%) represented other professionals.More than two-thirds, i.e., 163 (64.2%) of all respondents had worked in their feld for more than fve years.Social and healthcare employees accounted for 34.3%, and daycare and school employees accounted for 65.7% of total respondents (Table 2). 3 shows the Cronbach's alphas calculated for each of the seven domains of the OCL scale.Te Cronbach's alpha of the total OCL was 0.968 and ranged in domains from 0.875 to 0.929, suggesting good to excellent internal consistency.

Sampling Adequacy.
Both the KMO statistic (KMO � 0.943) and Bartlett's sphericity test (χ 2 � 9741.8,p < 0.001) indicated that the data under analysis were adequate for a factor analytic approach.

Teoretical Model, Confrmatory Factor Analysis (CFA).
CFA was utilized to assess the validity of a factor structure of the theoretical model in the data under analysis.Te detailed results of CFA are reported in Supplementary Table 1.Generally, the model ft statistics for the seven-domain model suggested a moderate to acceptable ft of the data with the theoretical model: χ 2 /df � 2.856 (acceptable ft), CFI � 0.800 (moderate ft), TLI � 0.786 (poor ft), RMSEA � 0.085 (moderate ft), and SRMR � 0.078 (acceptable ft).

Explanatory Factor Analysis (EFA).
EFA was performed to explore the factor structure of the items in our dataset without a priori assumption of a certain structure or number of latent factors.Te results of EFA and subsequent CFA are reported in Supplementary Table 2.In the EFA model, when a limit of >0.40 for eligible factor loading was applied, there were eight items that loaded rather highly to two factors.Te EFA-based model was further evaluated with CFA, and the model ft statistics of this adjusted CFA model were as follows: χ 2 /df � 2.463 (acceptable ft), CFI � 0.842 (moderate ft), TLI � 0.831 (moderate ft), RMSEA � 0.076 (acceptable ft), and SRMR � 0.073 (acceptable ft).Te Cronbach's alphas of factors of the adjusted CFA model varied from 0.826 to 0.936 indicating good to excellent internal consistency.

Relationship between Teoretical CFA Model and EFA-
Based Model. Figure 3 visualizes the association between the theoretical factor structure (CFA) and the factor structure based on EFA.Te seven items of initial domain 7 for leadership remained unchanged in both CFA and EFA.In domain 5 for shared communication, the number of items after EFA modelling doubled, from 5 to 10 items, while the number of items in domain 5 for backbone support structures decreased from six to three.In other domains, there were moderate changes in items between the initial theoretical-based CFA model and the EFAbased model.

Qualitative Analysis.
Altogether 19 of a total of 105 responses were related to the actual use of the research instrument.Of them, 11 suggested that the research instrument should include neutral "don't know" or "cannot say" options in the scale:

"It would have been good to have the answer option "I don't know" in the survey. Many things were such that there is no information as to whether it works like this." (ID 244)
In addition, six respondents commented that the survey was difcult to answer.Tey responded that some of the items concerned issues with which they were not familiar.Tey recommended a neutral or "don't know" option because   Terefore, some respondents perceived that response options without the possibility to choose a "don't know" option did not refect their reality in working practice.
A more precise defnition of cross-sectoral collaboration and the focus group was desired by two respondents:

"At the beginning, it would be good to explain what the operating model means here and how it needs to work. Now, there is an item that is too open to evaluate." (ID 260) "When answering the survey, my answers may be distorted by which leader is meant in the questions: my own leader is not involved in cross-sectoral collaboration." (ID 204)
Based on the respondents' comments, the instructions for the research instrument need to be clarifed in the future.Te key concepts of the instrument also require more precise defnitions.Similarly, it is important to defne in a more detailed manner the target group of the subjects that are intended to be evaluated with the OCL scale.

Discussion
Dysfunctional cross-sectoral collaboration, including fragmentation and overlap of services, is widely regarded as fundamental causes of inefective utilization of resources, especially in dealing with complex social and health problems [2,3,[45][46][47].Tere have been attempts to solve this multifaceted problem with various approaches, of which the CI approach has been considered to be one of the most promising.Attention has also been focused on efective leadership, which has a key role in successful cross-sectoral collaboration.Terefore, leadership has been proposed as an eminent part of cross-sectoral collaboration-based work [48][49][50][51][52].Following the principles of the CI framework together with joint leadership, including leaders from all relevant service sectors, is a promising way to increase efciency in services [13,14,22,28].However, in terms of measuring comprehensively the success of CI-based cross-sectoral collaboration, an appropriate research instrument for evaluation purposes has been lacking.Evaluation of the implementation process and changes in the services after implementation of the CI-related approach in practices are also called for in previous literature [21,22].
In the City of Oulu, the CI approach was implemented to enhance collaboration over sectoral boundaries.Research on the realization of cross-sectoral collaboration and leadership was combined with this implementation process [9].Te necessity for the current study arose from a practical need to evaluate the fulfllment of collaboration across the social and healthcare, daycare, and education service sectors.For this purpose, a new research instrument named the Oulu crosssectoral Collaboration and Leadership (OCL) scale was developed based on the fve CI conditions of the CI approach and with leadership as an additional domain.Tis study evaluated the psychometric properties of this instrument from the perspective of with and without a priori theory of its factor structure.Suggestions for and discussion about the further development of the survey instrument are presented.
Te results of the psychometric analyses of the OCL scale indicated that the theoretical factor structure of the OCL scale with seven domains had moderate to acceptable ft with the data under analysis, whether internal consistency of the domains was assessed with confrmatory factor analysis (CFA) or Cronbach's alpha statistics.Further evaluation of the factor structure of the OCL scale was performed by utilizing exploratory factor analysis (EFA) without a priori theory of the latent factors of the scale and by performing subsequent CFA based on the EFA results.Te purpose of the latter analyses was to give further information on the items and their placement in the domains.Te main results of the analyses indicated that the items of the new domain for leadership were stable, while notable changes in items were observed in the domains for shared communication and backbone support structure.In other domains, some moderate changes in items were observed between the theoretical and the EFA-based models.
Te leadership domain as part of the CI-based crosssectoral collaboration was included in the OCL scale because the role and importance of efective leadership in collaboration have been increasingly emphasized in the previous literature [22,28] as well as in a published study of implementation work in the city of Oulu [14].In this new domain of leadership, internal consistency was shown to be high, and the stability of the items was evident.Tis fnding may indicate that from the point of view of the respondents, the leadership-related items were recognizable in everyday working practice.Te qualitative responses of the survey Health & Social Care in the Community participants also did not reveal any criticism towards leadership items as part of the research instrument.Furthermore, in our study, the items of the leadership domain of the OCL scale were shown to form their own and specifc entity, because none of the items of the other domains had the potential for it.In conclusion, the results suggested that the leadership domain is a useful addition to the OCL scale evaluating the implementation of CI-based cross-sectoral collaboration work in working practice.Despite the high internal consistency and promising results of the model ft statistics, further development of the OCL scale and its psychometric construct is worth considering in future research.For example, the results of EFA showed that eight items had rather high loading to two factors.Tis warrants further evaluation of the nature of the items that lead to this cross-loading.In addition, shortening the OCL scale by reducing the number of items included in the OCL scale, and its domains would be justifed to make the OCL scale more feasible to complete.Although the statistics for sampling adequacy and the ratio of sample size under analysis to the number of items showed satisfactory results, further testing of the OCL scale with large samples is called for.Some suggestions for further development of the OCL scale also arose from the qualitative responses of the instrument.Te answers to the open question of the OCL scale highlighted the need to clarify the key concepts used in the OCL scale.Clarifcation was called for concepts such as cross-sectoral collaboration, e.g., what kind of cross-sectoral collaboration was meant in the OCL scale.In addition, the respondents suggested that the fve-point Likert scale should also include a neutral (i.e., "don't know") option instead of the current forced four-point Likert scale used in the OCL scale.When developing the OCL scale, we considered that it was essential to get the respondents to express clear opinions on each item and, thus, a four-point Likert scale without a neutral option was chosen [53][54][55][56] However, the respondents' criticism was partly understandable; for example, some respondents claimed to have too little information of CI-based work to make a justifed choice between the four options of an item.
Previous research literature has documented that CIbased working models have been implemented and evaluated since 2011.Tese studies have been carried out mainly in the USA and Canada [15,18,19] as well as in Western Europe, e.g., in Ireland [57,58] and Scotland (e.g., 52).However, there is a lack of systematic, CI-based survey instruments measuring achievable proceedings in child and family-centered cross-sectoral collaboration.Te OCL scale, described and tested in the current study, appears to be promising as an evaluation tool in the child, adolescents, and family-related CI initiatives.Using an instrument such as the OCL scale in our study would also enable international assessments and comparisons, given that the characteristics of cross-sectoral collaboration are quite similar as described in this study, for example, how to carry out cross-sectoral leadership.In addition, it should be noted that the OCL scale was developed for evaluation of CI-based work in the Nordic welfare service systems and, thus, modifcations might be needed according to other service environments.In the Nordic countries, social and healthcare service systems have many similarities with each other and a need to enhance the cross-sectoral collaboration has been called for [4,11,[59][60][61].Te CI model applied in the Finnish service system [13,14] is a feasible way to proceed in improving cross-sectoral collaboration work.In the evaluation of the efectiveness of CI-based work, the OCL scale could be an appropriate tool to measure it.

Strengths and Limitations
Te strength of this study is that the respondents of the survey who completed the OCL scale worked by utilizing cross-sectoral collaboration in their everyday working practice, and thus, they were considered to form a homogeneous target group in terms of cross-sectoral collaboration work.Te overall response rate was good (37%) considering general response rates in corresponding surveys [55,56].Te limitation of this study was that only those leaders who had concrete psychosocial support responsibilities were included in the target group while leaders with no current practical experience concerning cross-sectoral collaboration were left out of the study.Tis may have caused bias while evaluating a leader's role in cross-sectoral collaboration.

Conclusion
Te OCL scale appears to be a promising new research instrument to measure comprehensively CI-related crosssectoral collaboration between social and healthcare, daycare, and school services.Te research instrument is based on all CI conditions and has leadership as an additional domain.Te OCL scale proved to be an appropriate tool for research purposes and for assessing the development work of the services, but some modifcations would improve its psychometric construct.Te results of this study indicated that leadership is a key part of collaboration, easy to recognize by the respondents, and, thus, is an important addition to the research instrument.A need to clarify concepts and shorten the survey is an essential goal for further revision of the scale.the data analyses.TT and MN wrote the frst draft of the manuscript.SR revised the manuscript and guided the process throughout the study.All authors participated in the critical drafting of the manuscript and approved the fnal version.

2. 3 .
Carrying Out the Survey.Te online survey utilizing the fnal version of the OCL scale was conducted using the Webropol survey and reporting tool [29].Te survey was open during two time periods: April 7th, 2021-May 10th, 2021, and September 24th, 2021-November 17th, 2021.

Figure 1 :not meet the inclusion criteria n= 15 Figure 2 :
Figure 1: Description of the development process of the Oulu cross-sectoral Collaboration and Leadership (OCL) scale.

Figure 3 :
Figure 3: Sankey-type fow diagram visualizing the theoretical factor structure of the seven domains of the Oulu cross-sectoral Collaboration and Leadership (OCL) scale in relation to factor structure produced by the exploratory factor analysis (EFA) [39].* On the left side, the theoretical factor structure of the Oulu cross-sectoral collaboration and leadership (OCL) scale is used as starting nodes.On the right side, the ending nodes represent factors suggested by the exploratory factor analysis.Te width of each fow is proportional to the number of items at starting and ending nodes.

Table 1 :
Te domains of the Oulu cross-sectoral Collaboration and Leadership (OCL) scale and their theoretical context.

Table 2 :
Background characteristics of survey participants from social, healthcare, and education services.

Table 3 :
Internal consistency of the items in each domain of the OCL scale.