Proposed Framework for the Evaluation of Standalone Corpora Processing Systems: An Application to Arabic Corpora

Despite the accessibility of numerous online corpora, students and researchers engaged in the fields of Natural Language Processing (NLP), corpus linguistics, and language learning and teaching may encounter situations in which they need to develop their own corpora. Several commercial and free standalone corpora processing systems are available to process such corpora. In this study, we first propose a framework for the evaluation of standalone corpora processing systems and then use it to evaluate seven freely available systems. The proposed framework considers the usability, functionality, and performance of the evaluated systems while taking into consideration their suitability for Arabic corpora. While the results show that most of the evaluated systems exhibited comparable usability scores, the scores for functionality and performance were substantially different with respect to support for the Arabic language and N-grams profile generation. The results of our evaluation will help potential users of the evaluated systems to choose the system that best meets their needs. More importantly, the results will help the developers of the evaluated systems to enhance their systems and developers of new corpora processing systems by providing them with a reference framework.


11
The path for any given task is a reasonable length (2-5 clicks) 12 When there are multiple steps in a task, the site displays all the steps that need to be completed and provides feedback on the user's current position in the workflow 13 Users of the tool do not need to remember information from place to place 14 The use of metaphors is easily understandable by the typical user 15 Details of the software's internal workings are not exposed to the user 16 The tool caters for users with little prior experience of the tool 17 The tool makes it easy for users to explore the tools and try out different options before committing themselves 18 A typical first-time user can do the most common tasks without assistance 19 When they use the tool again, users will remember how to carry out the key tasks 20 Action buttons (such as "Submit") are always invoked by the user, not automatically invoked by the system when the last field is completed 21 Command and action items are presented as buttons (not, for example, as hypertext links)

22
When a page presents a lot of information, the user can sort and filter the information 23 If there is an image on a button or icon, it is relevant to the task 24 The tool is robust and all the key features work (i.e. there are no javascript exceptions, CGI errors or broken links)

25
The tool supports novice and expert users by providing different levels of explanation (e.g. in help and error messages) There is a convenient and obvious way to move between related functions and sections and it is easy to return to the main window 2 The information that users are most likely to need is easy to navigate to from most windows 3 Navigation choices are ordered in the most logical or task-oriented manner 4 The navigation system is broad and shallow (many items on a menu) rather than deep (many menu levels) There are clearly marked exits on every window allowing the user to bale out of the current task without having to go through an extended dialog 17 If the tool spawns new windows, these will not confuse the user (e.g. they are dialog-box sized and can be easily closed)

18
Menu instructions, prompts and messages appear on the same place on each screen

Forms and Data Entry 16 criteria)
1 Field labels on forms clearly explain what entries are desired 2 Text boxes on forms are the right length for the expected answer 3 There is a clear distinction between "required" and "optional" fields on forms 4 Questions on forms are grouped logically, and each group has a heading 5 Fields on forms contain hints, examples or model answers to demonstrate the expected input 6 Pull-down menus, radio buttons and check boxes are used in preference to text entry fields on forms (i.e. text entry fields are not overused) 7 With data entry screens, the cursor is placed where the input is needed 8 Users can complete simple tasks by entering just essential information (with the system supplying the non-essential information by default) 9 Forms allow users to stay with a single interaction method for as long as possible (i.e. users do not need to make numerous shifts from keyboard to mouse to keyboard).
10 The user can change default values in form fields

11
Text entry fields indicate the amount and the format of data that needs to be entered 12 Forms are validated before the form is submitted 13 With data entry screens, the site carries out field-level checking and form-level checking at the appropriate time 14 The site makes it easy to correct errors (e.g. when a form is incomplete, positioning the cursor at the location where correction is required) 15 There is consistency between data entry and data display 16 Labels are close to the data entry fields (e.g. labels are right justified)

Trust and Credibility (8 criteria)
1 The tool is authoritative and trustworthy 2 The tool contains third-party support (e.g. citations, testimonials) to verify the accuracy of information.
3 It is clear that there is a real organisation behind the tool (e.g. there is a physical address or a photo of the office) 4 The company comprises acknowledged experts (look for credentials) 5 Each window is clearly branded so that the user knows he is still using the same tool 6 The tool is free of typographic errors and spelling mistakes 7 The visual design complements the brand and any offline marketing messages 8 There are real people behind the organisation and they are honest and trustworthy (look for bios)

Writing & Content Quality (14 criteria)
1 Text is concise, with no needless instructions or welcome notes 2 windows use bulleted and numbered lists in preference to narrative text 3 Lists are prefaced with a concise introduction (e.g. a word or phrase), helping users appreciate how the items are related to one another 4 The most important items in a list are placed at the top 5 Information is organised hierarchically, from the general to the specific, and the organisation is clear and logical 6 windows are quick to scan, with ample headings and sub-headings and short paragraphs 7 Each window is clearly labelled with a descriptive and useful title that makes sense as a bookmark 8 Buttons and button titles are descriptive and predictive, and there are no "Click here!" buttons 9 The tool avoids cute, clever, or cryptic headings 10 Button names match the title of destination window, so users will know when they have reached the intended window 11 Button labels and link labels start with action words 12 Headings and sub-headings are short, straightforward and descriptive 13 The words, phrases and concepts used will be familiar to the typical user 14 Acronyms and abbreviations are defined when first used

Page Layout and Visual Design (29 criteria)
1 The screen density is appropriate for the target users and their tasks The search box and its controls are clearly labelled (multiple search boxes can be confusing)

10
The scope of the search is made explicit on the search results window and users can restrict the scope (if relevant to the task)

11
The search results window displays useful meta-information, such as the size of the document, the date that the document was created and the file type (Word, pdf etc.)

12
The search engine provides automatic spell checking and looks for plurals and synonyms

Help, Feedback and Error Tolerance (24 criteria)
1 The FAQ or on-line help provides step-by-step instructions to help users carry out the most important tasks 2 It is easy to get help in the right form and at the right time There is sufficient space between targets to prevent the user from hitting multiple or incorrect targets 20 There is a line space of at least 2 pixels between clickable items 21 The tool makes it obvious when and where an error has occurred (e.g. when a form is incomplete, highlighting the missing fields)

22
The tool uses appropriate selection methods (e.g. pull-down menus) as an alternative to typing 23 The tool does a good job of preventing the user from making errors 24 The tool ensures that work is not lost (either by the user or tool error)

25
Error messages are written in plain language with sufficient explanation of the problem 26 When relevant, the user can defer fixing errors until later in the task 27 It is easy to "undo" (or "cancel") and "redo" actions