Estimating Designers’ Performance considering Personal Characteristics and External Factors Together

Design team performance evaluation can occur in different ways, all of them requiring considerations on interactions among team members; in turn, these considerations should count on as many pieces of information as possible about individuals.'e literature already explains how personal characteristics and/or external factors influence designers’ performance; nevertheless, a way to evaluate performance considering several personal characteristics and external factors together is missing.'is research tries to fill the gap by developing the Designer’s Performance Estimator (DPE), a ready-to-use tool for researchers and practitioners who need to make information about team members as richer as possible.


Introduction
Due to advances in technology, higher complexity in product development processes, shortages in time and resources, etc., companies must base their design activities more and more on teams rather than individuals [1,2]. A team consists of two or more people who interact together to achieve a common and shared goal or mission [3]. Team performance is the extent to which the team accomplishes that goal or mission [4]. erefore, team performance evaluation becomes more and more important in modern design contexts both to tune up existing teams and to select the most suitable designers to generate new ones. is evaluation requires effective considerations on interactions among team members; in turn, these considerations should count on as many pieces of information as possible about the performance of the individuals belonging to the team or being candidate for it [5]. e literature already offers methods and tools for job performance evaluation focusing on individuals. ese methods and tools range from empirical studies to literature meta-analyses and formal methods.
ey take one or more personal characteristics (e.g., personality traits, skill, and knowledge) and/or external factors (e.g., contexts, types of design activities and representations of products, users, and environments) into consideration. Nevertheless, they barely address how mixes of personal characteristics and external factors together influence designers' performance. e research described in this paper develops the Designer's Performance Estimator (DPE), a tool to quantify designers' performance in terms of how varied, novel, and usable the design results are expected to be, strictly considering personal characteristics and external factors together.
is tool delivers immediate information to researchers and practitioners whose domains are already known to it; at the same time, the DPE knowledge base can be improved considering other design activities and this will make its application coverage wider. Specifically, this paper describes both the delivery of immediate information by considering shape-based design activities as already known to the DPE and the general procedure to improve the DPE knowledge base.
ere are several possible exploitations of the DPE, depending on design goals and resource availability. For example, the DPE can be used to build design teams very focused on novelty (design goal) in a design context where many designers are at disposal but just one representation (external factor) is present (resource availability) or the DPE can be used to select the best representations (external factors) to maximize the usefulness (design goal) of the outcome of the design effort of a small team whose composition cannot be changed. e paper runs as follows. Section 2 reports the background of the research, ranging from the fundamentals of individuals' job performance evaluation to some considerations about possible influences on performance. Section 3, describing the research activities, starts by clarifying the DPE role in team performance evaluations and carries on by reporting the DPE definition and an example of its adoption. Section 4 describes the early validation of the DPE by comparing foreseen designers' performance to actual one in the specific case of the shape-based design activities. Section 5 details possible exploitations of the results of the DPE adoption. Section 6 highlights and discusses the main results of the research and Section 7 closes the paper by summarizing the research and suggesting some perspectives. Finally, Appendix A contains the questionnaire used to collect data during the early DPE validation.

Job Performance Evaluation.
In general, job performance is a multidimensional concept that indicates how well employees perform their tasks, the initiatives they take, the extent to which they complete their tasks, the way they use the resources available, and the time and energy they spend on their tasks alone or in teams [4,6]. Evaluating job performance consists in judging the employees respect to several dimensions like quality, quantity, planning, and timeliness of work. [7,8].
e research described in this paper refers specifically to design activities and addresses individuals rather than teams; thus, the focus here is on single designers' performance. Designers' performance evaluation could exploit empirical studies, literature metaanalyses, and formal methods. Bakker et al. [9] conducted an empirical study to assess performance in terms of increments in structural and social job resources, energy, time, and dedication spent for the job, etc. Peeters et al. [7] did a meta-analysis of refereed journals to measure designers' performance in terms of results effectiveness. Salgado [10] developed a formal method to analyze job performance assessments where personality-related five-factor model-(FFM-) based inventories and non-FFM-based inventories were applied in different contexts. is analysis highlighted that FFM-based inventories are more reliable in assessing job performance compared to non-FFM-based inventories, especially when focusing on conscientiousness and neuroticism. Azadeh et al. [11] developed a tool to evaluate job performance focusing on stress, health, safety, environment, and ergonomics in petrochemical plants affected by noise and uncertainty. is tool considers seventeen well-ordered steps, from determining the reliability of the data collection procedure (questionnaire) and the definition of input and output to the achievement of the information needed to apply the algorithm that computes the efficiency of each operator.
e results allow evaluators to implement corrective actions on low scorers. Lee et al. [12] proposed an approach for evaluating job performance of IT departments of manufacturing industries in Taiwan based on fuzzy analytic hierarchy processes (FAHP) and balanced scorecards (BSC).
is approach has a well-ordered, rigorous structure and functioning; it starts by using questionnaires to define performance indices from the financial, customer, internal business, and learning and growth points of view; the measurement of these indices highlights strengths and weaknesses to focus enhancing/corrective actions on.

Possible Influences on Performance.
e many variables that could influence designers' performance can be classified as internal or external, referring to personal characteristics rather than external factors. e internal variables considered in this research are skill, knowledge, and personality; the external variables are design activities and representations.
is choice comes from the large literature highlighting that these variables influence job performance much more than others [4,[13][14][15].
Regarding the internal variables (skill, knowledge, and personality), there is much literature about the definition of skill and knowledge and about their influences on design [16][17][18][19][20]. Personality deserves more attention because this research refers directly to its components, the traits. Personality traits are characteristics of a person that account for consistent behavioral patterns over situations and time [21]. e well-known taxonomy of the big five [6] identifies the following personality traits: extroversion or surgency (from now on, Personal Trait 1-PT1), agreeableness (PT2), conscientiousness (PT3), neuroticism (PT4), and openness to experience/culture (PT5). Much research reports the influences of personality on designers' performance by exploiting the big five because these are considered as good predictors of job task and contextual performance [6,10,[22][23][24][25][26][27][28][29].
Regarding the external variables (design activities and representations), the research of Sim and Duffy [30], Filippi and Barattin [31], and Gero and Kannengiesser [32] addresses how design activities can be defined and classified. Design activities are sets of actions performed by different actors (e.g., designers and final users) in different contexts (development of electronic devices or mechanical CNC machines, furniture, clothes, etc.), starting from different sources (functions, user needs, shapes, etc.) and ending with the generation of concepts, prototypes, or products. Salas et al. [3] state that the type of design activities and their complexity play a crucial role in influencing designers' performance exactly as individual characteristics like personality traits, skills, task knowledge, motivation, and attitudes do. Sonnentag et al. [18] consider several studies about how different types of tasks can influence job performance and conclude that this influence exists but it is not as heavy as that of other variables like, e.g., cognitive abilities, past experiences, and personality traits. Representations deserve a deeper consideration because this research needs a univocal classification of them based on combinations of precise elements. In the classification of Filippi and Barattin [33], these elements are orthogonal, they cover both classic and more recent product development processes, and they are clearly stated thanks to discrete values assignable to them. ese elements are the environment-real if it corresponds to the physical one, virtual otherwise; the product-real or virtual, with the same meanings as the previous ones; the interaction between environment and product-aware if the product recognizes the environment and behaves accordingly, unaware otherwise; the user-real if the user is a human being, simulated otherwise; the interaction between product and user-direct if this interaction occurs in a natural way, exactly as the user expects it, indirect otherwise. Representations consist of combinations of the values these five elements can assume. e representations considered in the classification are virtual reality (VR)-virtual products and environments; augmented virtuality (AV)-virtual environments and real products; augmented reality (AR)-virtual products and real environments; pure reality (PR)-environments and products are both real; mixed reality (MR)-the combination of AR and AV where at least the environment or the product assumes both its values at the same time. Much research demonstrates the influence of representations on job performance [34][35][36][37][38].
e literature analysis created a solid background for the research described in this paper. On the one hand, it complies with the order and rigorousness suggested by the research approach of Lee et al. [12] and bases data collection on questionnaires as suggested by Azadeh et al. [11]; on the other hand, addressing and classifying what can influence designers' performance allowed defining precisely the set of variables (internal and external factors) used here. e discussion section of this paper will report in detail the relationships between the DPE and the literature described here in terms of affinities and differences.

Activities
is section opens by clarifying the role of the DPE in team performance evaluations. e descriptions of the DPE definition and an example of its adoption take place afterward.

3.1.
e DPE Role in Team Performance Evaluations. Figure 1 proposes an example of team performance evaluation exploiting the DPE (represented inside the dotted envelope). e DPE exploits DA (design activities) tables (A); these tables contain the relationships between internal and external variables referred to specific types of design activities (B). Each type of design activities leads to a different DA table. e collection of DA tables makes up the DPE knowledge base (C) (the generation of the DA tables can happen in different times and spaces and, currently, it occurs through user testing conducted by experts/practitioners (D)). Team performance evaluators (E) needing to measure a specific team (F) use the DPE to get as much information as possible about the designers belonging to the team. e DPE generates the pieces of information (G) in the record describing each designer by considering his/her personal characteristics (H), the specific type of design activities that the team will be called to perform (I), and the representations available (J). From that moment on, the evaluators can apply their knowledge, methods, and tools to compute (K) the performance of the team as a whole, considering all relationships and influences among the designers belonging to it.
is is just one type of DPE exploitation. e DPE can be of help as well when evaluators do not deal with an existing team but they need to select the best designers to build up a new one, when evaluators should have the team ready and the need is to select the best representations to work with, etc. Section 5 lists possible DPE involvements in different situations.

DPE Definition.
e definition of the DPE occurs by determining its main data structure named DA table, the procedure to fill it, the metrics to quantify the design results, the output data structure named designer record, and the procedure to fill it. Table. e DA table (please refer to Table 1, ignoring the values in it for the moment because they refer to a specific kind of design activities) puts into relationship internal and external variables using precise metrics. e rows (except for the last two) refer to the internal variables skill, knowledge, and personality traits; the columns correspond to the external variables; the columns correspond to the external variables representations (VR, AV, AR, PR, MR, all of them, or just some, due to the design activities considered), characterized in terms of quantity (Q), variety (V), novelty (N), and usefulness (U), the metrics used in this research, and described later. e whole table refers again to an external variable, a specific type of design activities. Each internal variable can assume five levels as suggested by Likert's scale. e skill (S), defined here as the ability in applying design methods and tools and problem solving techniques, goes from level 1 (no skill)-describing a designer unable to use design methods, tools, and problem solving techniques at all-to level 5 (high skill)-indicating a designer who uses design methods, tools, and problem solving techniques effectively, efficiently, and autonomously. e knowledge (K), defined as the quantity of information owned by designers about design theories, techniques, and processes, uses the same classification: level 1 (no knowledge) describes a designer without any knowledge about design theories, techniques, and processes although he/she basically knows the context; level 5 (high knowledge) indicates a designer who has deep and precise knowledge about design theories, techniques, and processes and about the context. Each personality trait develops through five levels as well, ranging from the opposite of the trait (level 1) to the trait itself (level 5). For example, level 1 of PT1 represents an introvert designer; an extrovert designer corresponds to level 5. e last two rows contain the performance of the best and worst designers for each representation and metric. ey are computed automatically and will be used as terms for comparison.

3.2.2.
e Procedure to Fill a DA Table. If the DPE knowledge base does not contain the DA table related to the specific type of design activities the DPE involvement is focusing on, the procedure to fill it is as follows. Some Advances in Human-Computer Interaction designers are selected with respect to their levels of internal variables (rows). e evaluators use a questionnaire to assess the characteristics of designers and classify them against the internal variables. e structure of the questionnaire is reported in Appendix A. It consists of three questions (Q1 to Q3) containing items that designers mark using values between 1 (strongly disagree) to 5 (strongly agree). e first two questions (Q1 and Q2) focus on skill and knowledge. ey consist of ten base items each, referring to design methods and tools, equipment to generate prototypes and produce objects, software packages for design, manufacturing, and/or demonstrations, etc., as well as notions of physics, thermodynamics, construction laws, human-machine interaction paradigms, etc. ese 20 items should be enough to characterize designers' skill and knowledge with the required precision for the downstream steps of the DPE adoption. Nevertheless, the evaluators can add further items to customize the questionnaire due to the specific type of design activities. Designers' skill and knowledge are assessed by considering mean values. For example, if the mean value of one designer's answers to the first question is around 4, that designer is assigned the fourth level of skill. e last question (Q3) focuses on personality and comes from the Big Five Inventory (BFI) consisting of 44 items whose marks lead to the computation of a 0 to 100 score for each trait [39]. Since the scores used here develop through five levels, BFI values in the interval [0‥20) correspond to level 1; those in the interval [20‥40) to level 2; [40‥60) to level 3; [60‥80) to level 4; and [80‥100] to level 5.
Once the designers have been classified based on the questionnaire outcomes, a simple algorithm is in charge of assigning all the representations available, aiming at covering as many different combinations of internal/external variables as possible. After that, the design activities to perform are described to the designers and they carry them on. Four metrics allow quantifying the results. Among all the possibilities offered in literature, the work of Shah and Vargas-Hernandez [40] suggested the first two metrics: quantity and variety; the second two, novelty and usefulness, come from the research of Sarkar and Chakrabarti, who claim that creativity can be measured based on them [41]. ese metrics are reputed as exhaustive to characterize the results because of their complementarity. Creativity, another metric quite common in these cases, does not appear explicitly because Sarkar and Chakrabarti claim that it can be easily derived from novelty and usefulness. e computation of the four metrics occurs as follows.
(i) Quantity (Q). It is the amount of results produced by each designer. e value can vary from 0 to ∞ and is considered for each of the designer's levels of internal variables for every representation. For example, designer George is quite extrovert (extroversion level equal to 4) and very low on conscientiousness (level equal to 1) and generates seven results using the VR representation; therefore, George's values of PT1(4) and PT3(1) are both equal to 7. (ii) Variety (V). It measures how much a result differs from those expressed by other designers. Each result has assigned a value ranging from 1 to 10. If all designers sharing the same levels of internal variables and exploiting a specific representation express that result, the value will be set to 1 (the lowest value of variety); if only one designer expresses it, the value will be 10 (the highest value). A simple formula allows assigning the other values in between.  Advances in Human-Computer Interaction computation occurs as follows. e value is equal to 0 if that result is already present in one or more existing products as it is. e value is in the range (0, 0.5) if that result is already present in some existing products as functions to perform, and the user and product behaviors during interaction are the same but the implementation (product structure) is different; the more the implementation is different, the higher the value is. e value is in the range [0.5, 1) if the result is already present in some existing product as functions to perform but the user and/or product behaviors are different. e more these behaviors are different, the higher the value is. Finally, the value is 1 if the result is not present in any existing product. After the assignment of the values to each result (to each designer, in the case of the quantity), the computation of mean values takes place, one for each cell of the DA table. For example, the mean value is computed for the variety of the results produced by all designers showing extroversion equal to 5 and having worked with VR. Finally, the best and worst designers' performance are computed. e best designer's performance   e values of the best and worst designers, 76 and 14, respectively, allow normalizing the performance of Robert and expressing it as a percentage using the formula des_perf_ % � 100 * ((des_perf-worst_perf )/(best_perf-worst_perf )). e result is equal to 32.2%. erefore, Robert is quite scarce in producing design results about prototype development when working with virtual reality representations. e filled designer records represent the outcome of the DPE adoption. By summarizing, the designer record of a specific individual foresees his/her performance about specific types of design activities, with specific representations available, due to his/her skill, knowledge, and personality traits, with all of this being quantified using the four metrics described before.

Example of DPE Adoption.
What follows describes how the DPE adoption can occur in a real context, from the filling of the DA table to that of the designer records. is real context considers shape-based design activities and the VR, AR, and PR representations. Shape-based design activities develop products by analyzing specific shapes and defining product behaviors and functions consequently [42]. ese design activities are becoming more and more important due to the role of User eXperience in design. is is why they have been selected for this example. One of the main goals of these activities is to arouse specific emotions in the people who will interact with those products. is type of design activities is used, for example, to develop deformable interfaces for mobile devices [43] or to produce iconic objects based on the analysis of shapes generated by fashion designers, as it happens for the Italian brand Alessi, specialized in developing home appliances [44]. Here, only the first part of these design activities-the analysis of specific shapes thanks to tests where interaction exploits the sight sense-is considered. is analysis suggests to the participants specific functions to perform with products shaped that way as well as personal and product behaviors meanwhile. ese suggestions are addressed in the DPE as F/B (function/behaviors) pairs. An example of F/B pair is "Contain and heat tea" (function)/"I put cold water inside the cup; what seems to be the resistor inside the cup, heats the water; when the water is hot, I put the tea bag" (behavior). Regarding the representations, this example of adoption considers only VR, AR, and PR because AV and MR require expensive tools and complex procedures not available. Table. ree evaluators carry on the procedure to fill the DA table. ey are experts of product development processes and shape-based design activities. e activities run from the setup of the material for the tests to their execution and to the collection and analysis of the resulting data. ese steps are described in the following.

Filling the DA
(1) Setup of the Material for the Tests. e material consists of the shapes used during the design activities and the documents that will help the participants meanwhile. Each participant will interact with the same two shapes, labelled as Sh1 and Sh2 in the following. ere are more than one shape in order to lower the bias due to specific shape characteristics. Shape definition occurs by obeying to precise rules [42]. Table 3 summarizes the characteristics of the two shapes selected in answering to those rules.
For what concerns the VR tests, the shapes and the desk where they are placed are modeled using the CAD software package Fusion 360 by AutoDesk [45].
anks to the Microsoft 3D Builder software package, participants can rotate the shapes to look at them from different points of Advances in Human-Computer Interaction view. For the AR tests, the shape models used in the VR tests are converted into holograms that the HoloLens device by Microsoft [46], worn by the participants, projects on a real desk. Finally, the physical models for the PR tests are built with the 3D printer Ultimaker 2 by Ultimaker [47]. Once finished, the models are placed on the same desk used for the AR tests. Figure 2 shows the three representations of Sh1 and environment as used during the VR, AR, and PR tests. e documents for the participants describe each design activity they must perform using nontechnical language.
(2) Execution of the Tests and Data Collection. Once the material is available, the questionnaire reported in Appendix A is sent by e-mail to 90 possible participants, designers who have been working for years in different companies where shape-based design activities are almost known and students of university courses in mechanical engineering who have been taught about the principles of design in general and on the shape-based design activities in particular. 78 people send back the answers and the collected data allow selecting 60 participants with different levels of skill, knowledge, and personality traits and distributing them in three tests as homogeneously as possible against these characteristics. In all, 25 participants perform the VR test, 19 the AR test, and 16 the PR test. Tests take place in a university lab, one participant at a time. At the beginning, the participant receives the document describing what the evaluators expect from him/her. After that, the first shape is unveiled; the participant has ten minutes to look at it moving around (without touching it) and to write down the F/B pairs that come to his/her mind. At the end, the evaluators unveil the second shape and the participant has ten minutes again to consider it and write down the F/B pairs. Finally, the participant returns the document to the evaluators.
(3) Data Analysis. Once the last test comes to the end, the evaluators apply the metrics to the results, separately for each shape. Finally, the mean values considering both shapes are calculated and become the content of the DA table shown as  Table 1.
Data undergo a statistical analysis using the t-test (test of Student). e t-test works by comparing two means [48]. Here, it verifies possible influences of internal and external variables on results. e computation does not appear here for space reason; nevertheless, a clear influence of both internal and external variables is detected in all the cases examined (for every shape, for every metrics). All P values range from 0.07 to 0.09; since the significance level here is set to 0.1 because of the low number of participants, all P values are lower than the significance level and the possible influence is confirmed.

Filling the Designer Records.
e DA table allows filling any designer record for what concerns shape-based design activities. For example, John is a skilled designer with diverse experience in design activities. erefore, his levels of skill and knowledge are high. He is extrovert in average, quite disagreeable, conscientious, quite neurotic, and open to experience; John's personal characteristics, summarized in the upper part of the designer record, correspond to the tuple (4,5,3,2,5,4,4). e design context where John could be involved has VR and PR representations available. Under these conditions, the DPE allows filling the lower part of the designer record, containing John's performance (Table 2). For example, the performance equal to 81% regarding V using PR means that John is very good, much more than the average (50%), in finding F/B pairs showing high variety when dealing with pure reality. e V value comes as

Early DPE Validation
e early validation of the DPE adopts its current release, containing the DA table about shape-based activities, to estimate the performance of designers interacting with different shapes than those used to generate the DA table.
en, the real performances of the same designers are measured through tests. e comparison of the results starts assessing the DPE applicability and reliability.

Adopting the DPE to Estimate Designers' Performance.
Four evaluators are involved. Again, all of them are expert in design processes and shape-based design activities. Nine designers are considered (Des1 to Des9); they never practiced shape-based activities; nevertheless, their variegate experiences as designers qualify them as good candidates to perform this validation. Table 4 summarizes their characteristics. Table 5 contains the performance of the nine designers as estimated thanks to the DPE adoption (indeed, the table collects only the lower parts of the records of the nine designers to represent them compactly).
is validation involves only the metrics whose computation assigns values to the F/B pairs considering individuals, i.e., Q and N. e values of the other two metrics (V and U) would be computed considering F/B pairs found by groups made by one designer only (only nine designers are present and each of them shows different personal characteristics); thus, V and U values would be meaningless.
According to the DPE results, it seems that, using VR, Des1 could find many more F/B pairs (Q � 80%) than all the other designers (maximum value is that of Des5, equal to 51%). Moreover, Des4, Des5, Des6, and Des7's F/B pairs seem to be showing more or less the same novelty when using AR. All of this suggests some hypotheses to verify in the field in order to start assessing the DPE applicability and reliability. e hypotheses considered here are as follows.
(i) Hyp1. Given VR and Q, Des1 should find more F/B pairs than Des3 and Des2, in this order, and the differences should be considerable.   Des1  80  64  50  89  47  29  Des2  11  26  33  17  52  65  Des3  39  31  64  32  23  62  Des4  48  60  44  62  58  38  Des5  51  57  47  57  26  29  Des6  31  26  47  57  66  65  Des7  45  60  51  60  21  32  Des8  43  45  41  53  41  38  Des9  45  50  51  79  21  pairs with AR than with VR and PR, in this order, and the differences should be considerable. Figure 3 shows the shapes used in the tests. Multiple shapes are used again to lower the bias as much as possible. ey are generated following the same rules as for the shapes used to fill the DA table. e four hypotheses lead the following associations between designers and representations. Des1, Des2, and Des3 consider all the shapes using only VR; Des4, Des5, Des6, and Des7 consider all the shapes as well but using only AR. Des8 considers Sh3 in VR, Sh4 in AR, and Sh5 in PR; Des9 does the same. Table 6 summarizes the results. e values of designers Des1 to Des7 are mean values of the results, since each of them considers the three shapes using the same representation.

Assessing the DPE Applicability and Reliability.
No problems arose from the DPE adoption throughout this early validation. erefore, regardless of the hypotheses verification, the DPE applicability appears as verified. For what concerns its reliability, data contained in Table 6 lead to the following considerations about the four hypotheses. Des8 found much more F/B pairs in the VR test (8) than in AR (4) and PR (3) tests; they are almost the double. is contradicts what the DPE foresaw, the independency from the representation. is misalignment could depend on the shapes used in the tests. e same designer cannot consider the same shape in the three representations because of the inevitable bias among them. erefore, three different shapes were considered. Although the shapes have been generated by strictly following the rules, Sh3 contains five elements to catch the attention while Sh4 and Sh5 have four elements only. is difference could be the main reason for the misalignment with the DPE estimate.
(iv) Hyp4: VERIFIED. Des9 found F/B pairs showing different N mean values using the three representations: 0.32 for VR, 0.55 for AR, and 0.2 for PR. As foreseen by the DPE, AR suggests more N in the F/B pairs than VR and PR (the former value is almost the double of the latter).
e verification of three hypotheses out of four starts giving positive indications about the DPE reliability.

Possible Exploitations of the DPE Results
ere are several ways team performance evaluators can exploit the designer records resulting from the DPE adoption. ree possibilities are as follows: (i) Situation A. e evaluators work in a design context where few representations are available; they are called to build a small team and there are precise expectations about the design results. In this case, the designer records can help in selecting the most promising people to build the team considering the expected characteristics of the results (e.g., novel design solutions) as leading criteria. (ii) Situation B. e evaluators work in a design context short in human resources from the design point of view and time-to-market is mandatory; nevertheless, they have all representations potentially available. In this case, the designer records can help in deciding the best representation(s) to use. More in detail, the designer records rank the representations; then, the company can select the most effective ones, depending on the time-to-market constraint. (iii) Situation C. e evaluators work in a design context where they are called to suggest the most promising design team to maximize specific characteristics of the design solutions and they have almost no limits about people to involve or representations to exploit. en, designer records of the candidates to be part of the team can help the selection of the most promising ones according to those characteristics of the design solutions; moreover, the records suggest also the best representations to use.
To go deeper in understanding these possible exploitations, the early DPE validation described in the previous section can be classified as corresponding to situation A. Consider a company having nine designers in the R&D department. e company size suggests teams of at most four people. Now, this company decides to exploit VR in design activities and aims at getting design results as novel as possible. e results of the DPE adoption, as in Table 5, help building up the required team. e situation makes the attention focus on the VR/N column. Its values allow ordering the nine designers against their performance and the result is as follows (best to worst): Des1 (64%), Des4 and Des7 (60%), Des5 (57%), Des9 (50%), Des8 (45%), Des3 (31%), and Des2 and Des6 (26%). erefore, looking for Advances in Human-Computer Interaction building up a team made by four designers, the DPE suggests selecting Des1, Des4, Des7, and Des5, the first four most performing individuals.

Results and Discussion
e main result of this research is the Designer's Performance Estimator (DPE), a ready-to-use tool for everyone who needs to characterize individuals and foresee their performance in specific types of design activities, all of this in order to evaluate existing or potential teams as effectively as possible.
Among the evaluation approaches considered in this research, empirical studies, literature meta-analyses, and formal methods, as summarized in Section 2.1, the DPE shows more affinity with the last ones. e comparison with those methods highlights its peculiarities and strong points. Salgado's research [10] considered different contexts as the DPE does; nevertheless, the DPE involves more personal characteristics and external factors (representations). e work of Azadeh et al. [11] showed clear data structures and procedures like the questionnaires, their generation, and the input/output definition; the DPE does the same but it offers higher versatility (it can be applied in different contexts) and more metrics to quantify the results. Finally, the DPE presents many analogies with the approach of Lee et al. [12] like the rigorous architecture and the exploitation of existing, well-known methods and tools; nevertheless, the DPE involves again more personal characteristics and manages results individually rather than in the aggregate form only.
Although the DPE appears overcoming some lacks of the evaluation methods and tools reported before, it has drawbacks to consider as well; these drawbacks are summarized here and recalled as subjects for future perspectives in the conclusions section. Current release of the DPE allows tests as the only way to collect data to fill new DA tables or update existing ones. Only four metrics are used now and they do not consider important topics like eco-sustainability, ergonomics, user experience, etc. e knowledge base management in the current release of the DPE considers the DA table structure as fixed; adding internal/external variables is not allowed. is is quite a limit since, for example, variables referring to team working like cooperation and communication would make the DPE even more answering to the evaluators' needs. Although the DPE has proven to be applicable, its usability is quite scarce. Data collection and analysis must be performed almost manually and this makes the DPE adoption time consuming. Finally, the knowledge base is bare indeed; it contains only the DA table related to the shape-based design activities. Moreover, this table misses data referring to skill and knowledge as well as to AV and MR representations. All of this limits the DPE coverage and applicability and makes the DPE scarcely readyto-use for practitioners now.

Conclusions
e research described in this paper aimed at helping team performance evaluators. As a result, it defined the Designer's Performance Estimator (DPE). e DPE is a ready-to-use tool Advances in Human-Computer Interaction ways to represent technical specifications laws and standard for simple structural mechanics manufacturing techniques thermodynamics laws usability and human-machine interaction paradigms and design and evaluation procedures ways to manufacture specific products (from technical drawings to real final products) procedures for thermal-mechanical analysis optimization techniques of costs and resources to generate products ergonomics standards energy saving techniques industrial design and fashion trends (Q3) I see myself as someone who (strongly disagree � 1; strongly agree � 5): is talkative tends to find fault with others is depressed, blue is reserved is helpful and unselfish with others can be somewhat careless is relaxed and handles stress well is curious about many different things is full of energy starts quarrels with others is a reliable worker is ingenious, a deep thinker generates a lot of enthusiasm has a forgiving nature tends to be disorganized worries a lot tends to be quiet is generally trusting tends to be lazy is inventive has an assertive personality can be cold and aloof perseveres until the task is finished can be moody values artistic, aesthetic experiences is sometimes shy, inhibited is considerate and kind to almost everyone remains calm in tense situations prefers work that is routine makes plans and follows through with them gets nervous easily likes to reflect and play with ideas has few artistic interests is easily distracted is sophisticated in art, music, or literature for researchers/practitioners that allows describing and quantifying designers' performance considering personal characteristics and external factors together. e computation exploits a knowledge base generated thanks to the analysis of different types of design activities in different situations. Some adoptions in the field already stated the DPE applicability and started demonstrating its reliability.
For what concerns possible research perspectives, some hints, corresponding to the drawbacks described in the results and discussion section, are as follows. Other types of data sources should be allowed to fill the DA tables like scientific literature and companies' history; moreover, the way to merge pieces of information coming from heterogeneous sources needs to be investigated. Other metrics than the current four need to be made available in order to widen the DPE coverage. For example, learnability, aesthetics, and enjoyment would allow orienting the DPE towards user experience; nevertheless, it must be pointed out once again the role of the DPE as estimator of single designers' performance. Other methods, tools, competencies, and knowledge are required to perform a complete design team performance evaluation. Procedures and/or suggestions should be introduced to allow evaluators adding internal and external variables. Automatisms must be introduced, especially to collect data and fill the DA tables and the designer records, in order to lower the time required by the DPE adoption. e author is working on making the DPE more usable by implementing Google forms and developing code in Microsoft Excel workbooks. All of this should make the DPE adoption almost automatic. Finally, the knowledge base should be more populated; more DA tables should be added and the existing one would need further tests to fill the empty rows and columns. Clearly, filling empty DA tables or empty cells of existing ones is not a problem; everybody can do this by simply following the indications described in this paper. On the contrary, if fresh data should affect nonempty cells of existing DA tables, the merging policy would have to be defined time-by-time, requiring competencies and expertise.

AR:
Augmented reality representation AV: Augmented virtuality representation DA table: It contains the relationships between internal and external variables that referred to a specific type of design activities Des1-Des9: e nine designers involved in the early DPE validation Designer record: It contains the result of the DPE adoption, the estimate of the specific designer performance DPE: Designer's Performance Estimator, the tool developed in this research F/B pair: Function/behavior pair. It consists of a function suggested by the shape of a product and the related product and/or user behavior Hyp1-Hyp4: e four hypotheses used in the early DPE validation K: Designers' knowledge MR: Mixed reality representation N: Novelty, the metrics representing how much a result does not resemble to anything known PR: Pure reality representation PT1-PT5: Personality traits (extroversion, agreeableness, conscientiousness, neuroticism, and openness) Q: Quantity, the metrics representing the amount of results S: Designers' skill Sh1-Sh5: e shapes used in the research U: Usefulness, the metrics representing the social value of a result V: Variety, the metrics representing how much a result differs from the others VR: Virtual reality representation. Table 7 in the following contains the questionnaire used to collect data about designers in order to assess their characteristics in the early DPE validation dealing with shapebased design activities.

Data Availability
Part of the data used to support the findings of this study are included within the article (please see the content of Tables 1  and 4). Other data, like the test results used to validate the DPE, have not been made available because a language different from English was used. Reporting these data in the original language as well as giving their translation and interpretation would be time consuming and almost useless. Nevertheless, the author thinks that the description of the DPE data structures as well as that of the procedures to fill them should be clear enough to allow the reader to replicate the research activities easily.

Conflicts of Interest
e author declares that there are no conflicts of interest.