User Acceptance Testing

In the final phase of this project our developed application was assessed by users representing three user groups. The following describes the test users; tasks each user group performed with this application, and the feedback and outcome from the user acceptance test.

Test user (groups)

User 1 is a postdoc from the department of Computing and Information Systems at the University of Melbourne.  This user is a representative of the general Bioinformatician that has  both basic biological knowledge and adept programming skills.

User 2 is a postdoc from the University of Queensland, representing biologists and clinicians who want to easily integrate their own research data with datasets from multiple data sources and analyse the data. This user group are generally researchers who have very limited or no programming skills.

User 3 a system administrator from the University of Queensland. This user represents the GVL Galaxy site administrator who works primarily on the server side, managing the Galaxy workflow system, environment and data libraries.

Tests performed by users

User 1 - the main task for this user group is to design and generate scientific workflows/protocols using the Galaxy workflow system. Then publish these workflows with the   research community so that other researchers can apply it to their own data.

User 2 - the main task or this user group is to reuse existing workflows for their own research, modify existing workflows and create new workflows. Additional important features for this user group include the: 

• ability to publish and cite workflows.
• ability to assign a Digital Object Identifier (DOI) to workflows .
• ability to rerun shared workflows and cite them using the DOI in their publications.
• ability to generate important datasets for the community and share them through Galaxy.

User 3 - the main task or this user group is to manage and update the data libraries shared with the community via the Galaxy workflow system. This user works primarily on the server side on tasks such as configuring user, tool or data repository access.

Testing outcomes

The Collection Manager has been introduced to User 1 via video conference and in person to Users 2 and 3. All functionalities have been presented to all users.

User testing identified some issues from the Collection Manager, the main issues were:

• concurrent access to data records in the Collection manager result in database errors when saving the record
• email notification functionality

All issues have been now been fixed.

Individual User Experiences

User 1 found the Collection Manager interface very user friendly and easy to use. User 1 comments that the application will be another means for them to promote their Galaxy workflows in the near future. This will increase awareness of their workflows further in the research community. An account has been created for this user on the production server and they have published two workflows to RDA using the Collection Manager.

User 2 found the Collection Manager impressive. In their opinion it is a very useful tool to easily share and promote workflows with the research community. An account has been created for them on the production server and this user will spend a couple of days testing the Collection Manager to see how it could be merged with their current working style.

User 2 expressed further interest in the DOI minting option provided by the Collection Manager. This feature was identified as a major advantage to using the application, which is not provided by the Galaxy workflow system. Further discussion with User 2 needs to take place to evaluate: when should a workflow be assigned a DOI during the publishing process.

User 3 who has a background in teaching Web Interface design found the Collection Manager Interface very easy to understand and use. They also found value in using this interface to help the public identify which Galaxy instances are available on the web and what data libraries and workflows they host. Many public users do not know where to go to use specific tools with specific data. Having a general repository indexed by Google will ease this process. In addition, User 3 raised a few points for the web-interface that include providing more help text for fields such as ANZSRC and Rights when users are creating a Data record. These changes  are currently being addressed.

What does the application do? The bigger picture

Although cancers may look the same under the microscope, they behave quite differently at the genomic level. By identifying these genetic differences, researchers can start to understand what treatment works for particular patients and administer those first. Understanding which cancers don’t respond to treatment is also vital as this allows researchers to focus down on the molecular mechanisms that are responsible and design new drugs to attack those particular mechanisms that cause the cancer to be lethal [1].


In this context, the primary goal of the International Cancer Genome Consortium (ICGC) is to generate comprehensive catalogues of genomic abnormalities in tumours from 50 different cancer types and/or subtypes which are of clinical and societal importance across the globe. The Australian component of that Consortium, the Australian Pancreatic Cancer GenomeInitiative (APGI) is delivering the genomic data associated with pancreatic tumour samples. For this, the APGI uses the best clinical material available, with well-characterised and accurately annotated clinico-pathological, treatment and outcome data acquired prospectively. However, discovery of variations and similarities within the genomes of a given cancer collection and being able to compare these to other datasets remain a significant challenge for biologist and clinician researchers.  The effective re-use of datasets of international importance is limited by the ability of research biologists and clinicians to access and to use computational and data infrastructure. A researcher performing such an analysis is currently expected to be conversant with programming, command line scripting, data management, high performance computing, network-based communications, and visualisation at a minimum. Additionally, they must ensure that the steps in the analysis are recorded in sufficient detail for the results to be reproduced in-house at least, and ideally would document the analysis in such a way as to allow publication of the method(s).

The Cancer Genomics Linkage Application funded by ANDS enables the re-use and integration of data available from public repositories such as the ICGC variant database or the DrugBank database of drugs and drug targets by leveraging the Genomics Virtual Lab capability on the research cloud.  Researchers, such as Professor Andrew Biankin and colleagues from the Garvan Institute for Medical Research are now able to access genomic datasets of international importance and to integrate them with their own clinical and genomic datasets in order to explore, discover and validate key genomic abnormality that cause cancer using user friendly computational workflows. The project further provides the mechanism for such researchers to publish and to make available their analysis for re-use by the community.
The project enables the in-depth interrogation of cancer genomic datasets and allows the comparison to other genomic datasets by providing research biologists and clinicians with direct access to them through the Genomics Virtual Lab (GVL). The key benefits to the Australian genomics community gained from this project are

• Integration of analysis tools, public and private datasets, and visualisation platforms: streamlining research and reducing time from experiment to publication 
• Enhanced collaboration between researchers and across the community through shared datasets, workflows and customised toolsets 
• Reproducibility and research provenance: workflow engines record all aspects of an experiment allowing for confidence in repeatability and for the publication of workflows along with the resulting data. The application enables researchers to mint their workflows using Digital Object Identifier (DOI).

References

[1] http://www.youtube.com/watch?v=ctohpNzTFvg