Clinical Trials Operational Analytics (CTOA) provides an overview of operational data such as Enrollment, Query and Audit metrics, Adverse Event rates, and Protocol Deviations.
CTOA can be grouped by compound, therapeutic area, or program. If the CTOA app is configured to be grouped at the URL level, the name reflects the grouped app.
CTOA, like all Analytics apps, has several common features such as filtering, selection tools, bookmarks, etc. For details on all Analytics functionality, please refer to the Analytics section or the Analytics courses in the Learning Portal.
In this article, users can find information on all the available out-of-the-box CTOA sheets. Use the Table of Contents at the right to navigate to the sheets of interest.
Important: The list of sheets does not include custom sheets specific to the user's team.
CTOA Home Page
The Home Page displays key summary information and metrics across all studies.
- By default, the stacked bar chart located in the top left corner presents the Historical Subject Status tab. This chart displays the progression of subjects from status to status, allowing users to visually compare segments and observe how they evolve as the trial progresses.
- To explore current subject status, click the Current Subject Status tab. This chart displays where subjects are presently in the trial process.
Tip: Any applied filters (shown in the Selections toolbar) remain when navigating to another sheet. Remove a filter by clicking the 'x' next to the filter in the Selections toolbar.
Enrollment Summary
Use the Enrollment Summary sheet to view enrollment metrics and information by Study, Country, Site, Status, and Group Assignment.
- The Total Planned Enrollment KPI supports planned enrollment values stored at the study level, in addition to more granular levels such as country, cohort, or site.
- In the Enrollment Summary chart, click the x-axis drop-down arrow to select from Site Number (default), Country, or Study.
- In the Enrollment Details chart, click the y-axis drop-down arrow to select from Site Number (default), Study, or Country.
Recruitment and Retention
Use the Recruitment and Retention sheet to monitor subject recruitment and retention metrics with configurable thresholds that categorize sites as On Target, Under Target, or Action Required.
Planned enrollment values defined at the study level are fully supported, ensuring metrics accurately reflect data whether stored at the study, country, site, or cohort level.
- An instructional banner in the upper right corner of the sheet provides in-app guidance on how to use the threshold input boxes and interpret the resulting status indicators.
- Default target values are pre-populated for all threshold inputs, enabling users to start analysis right away. These values can be modified as needed to match study-specific expectations.
- Use the filters across the top of the sheet to refine the analysis by Study, Country, Site, and Group Assignment, enabling cohort-level review in addition to site-level analysis.
- Enter or adjust threshold values for the following metrics:
- Planned Enrollment: On Target % (>=) and Action Required % (<=)
- Screen Failed: On Target % (<=) and Action Required % (>=)
- Consented: On Target % (>=) and Action Required % (<=)
- Discontinued: On Target % (<=) and Action Required % (>=)
Enrollment Trend
Use the Enrollment Trend sheet to view actual, planned, and cumulative enrollment trends in various forms. Users can compare actual enrollment with the forecast at the study or site level, broken down by year and month.
Planned enrollment values defined at the study level are fully supported. This ensures accurate reporting whether data is at the study level or more granular levels such as country, site, or cohort. The charts and KPIs on the sheet fully accommodate this flexibility, even if detailed breakdowns are not included.
Planned enrollment numbers must be provided during configuration for the sheet to reflect accurate study details.
- Use the tabs across the top to see Cumulative Actual Enrollment (default), Cumulative Enrollment, or Cumulative Screen Failure.
- In the Planned vs Actual Totals by Study/Site/Country chart, click the x-axis drop-down arrow to select from Study (default), Site, or Country.
- In the Actual Study Enrollment vs Actual Site Enrollment chart, to see a site compared to the entire study, select only one site.
Subject Early Discontinuation
Use the Subject Early Discontinuation sheet to get detailed information for subjects who withdrew, dropped out, or discontinued from the study (or studies) for various reasons.
- In the Discontinued Rate by Study/Country/Site chart, click the x-axis drop-down arrow to select from Study (default), Site, or Country.
- In the # of Discontinued by Site / Reason chart, # of Discontinued By Site tab is displayed by default. Click the # of Discontinued By Reason tab to change the view.
- Click the y-axis drop-down arrow to select from # of Discontinued Subjects (default) or % of Discontinued of Enrolled.
Query Metrics
Use the Query Metrics sheet to find information regarding query metrics.
If multiple studies are available, select one Study.
- By default, 20% is the default Open Query Threshold rate. Update the value if desired. Sites that exceed the defined threshold are highlighted in yellow.
- Click the x-axis drop-down to display the chart by Site (default), Study, or Country.
- Use the tabs across the top to see Queries vs Subjects (default), Current Query Statuses, or # of Queries by Sender-Recipient/Query Status.
- Use the listing at the bottom of the sheet to view the data in tabular form with additional details.
Queries Duration
Use the Queries Duration sheet to view the duration of queries in the opened status, the average number of days it took for the opened queries to be answered, and the average number of days it took for the answered queries to be closed, along with other relevant metrics.
- Enter an unacceptable number of Average Days Outstanding (>=) in the threshold field to focus the data on the sites with average days outstanding greater than or equal to the number entered.
- In all charts, click the y-axis drop-down arrow to select from # of Queries (default) or % of Queries.
- In all but the first chart, click the x-axis drop-down arrow to display the % of Queries instead of the default value.
Query Details
Use the Query Details sheet to explore query activity across studies, countries, sites, and multiple dimensions such as folder, form, and data field. This sheet includes visual summaries, pivot-based breakdowns, and a table to review details.
- Use the top row filters to view all data on the sheet by Study, Country, Site, Subject, Query Type, Query Status, Query Sender, Query Recipient, and Sender-Recipient.
- The three-tab container on the left contains horizontal stacked bar charts for Queries by Sender and Dimension, Queries by Recipient and Dimension, and Queries by Sender-Recipient and Dimension. These charts reflect the selected dimension and display query counts.
- Use the dimension selector above the charts to view data by Study (default), Country, Site, Folder, Form, or Data Field. The selected dimension updates all visualizations on the sheet except the query table.
- The pivot table groups rows by the selected dimension and the user who opened the query. Columns display counts by Sender-Recipient and Query Status. Expand to see additional details.
- The table in the bottom right corner of the sheet includes key fields such as Study, Site, Query Text, Query Response, Data Field, and Field.
Queries by Open Date
Use the Queries by Open Date sheet to analyze query trends over time, monitor resolution progress, and compare query volume across sites, users, and sender-recipients.
- Use the top row filters to view data by Study, Country, Site, Subject, Folder, Query Status, and Sender-Recipient.
-
The Queries By Date & Site line chart shows cumulative queries (default) or the number of queries over time by site. The Queries By Date & User and Queries By Date & Sender-Recipient line charts are displayed in a tabbed container, showing cumulative queries (default) or the number of queries opened over time by user or sender-recipient. Use the y-axis to toggle between Cumulative # of Queries or # of Queries.
For all three line charts:
- Click a value on the x-axis to drill down from Year > Month > Week > Day.
- The pivot tables are organized into two tabs: Queries By Site & User and Queries By Subject.
- Each pivot table displays a count of queries, broken down by Sender-Recipient and Query Status.
- The Current Query Status Over Time stacked bar chart displays the number of queries opened each year as well as their current status, with color-coded counts for query statuses (Open, Answered, Closed, Cancelled). Click a bar to drill down by Year > Month > Week > Day.
Query Response
Use the Query Response sheet to view the details on query response time.
- For the Average Response in Days by Site Over Time chart, enter the graph's desired timeframe in months, 12 (months) is entered by default.
- The chart goes from the current day back to the number of months entered.
- In the Queries by Open to Answered Duration chart, click the y-axis drop-down arrows to select from Open to Answered Duration (default), Country, or Study, and Site (default), Country, or Study.
Protocol Deviations - Overview
Use the Protocol Deviations - Overview sheet to find details related to subjects who had protocol deviations.
- Use the Protocol Deviation Type filter in the top row to enable easier group analysis.
- The Level filter in the top row can be used to toggle the view of all charts and the table on the sheet by severity level.
- The Number of Protocol Deviations chart contains:
- A reference line for # of PDs/Subject that dynamically responds to the selection in the filters.
- Use the x-axis drop-down to view the chart by Site of Deviation (default), Country, PD Type, Current Site, Subject, Class, and Study.
- In the Protocol Deviation Details by Study, Country, or Site chart, use the y-axis drop-down arrows to select from Site of Deviation (default), Current Site, Country, Protocol Deviation, Study, Field, or Form, and then choose by Class (default), Current Site, Country, Protocol Deviation, Study, Field, Form.
- In the Protocol Deviations Over Time chart:
- Click the y-axis drop-down to select from # of PDs (default) or # of PDs/Subject.
- Click the x-axis drop-down to select Date (default) or Visit. If a protocol deviation happened that was unrelated to a visit, the visit shows as 'missing.'
- The Protocol Deviation table presents the data in a tabular format, including additional details.
Protocol Deviation Details
Use the Protocol Deviation Details sheet to view details regarding subjects with protocol deviations, as well as the number of eligible vs ineligible subjects who had protocol deviations. Also, identify the site of deviation details for subject transfers.
- Use the Protocol Deviation Type filter in the top row to enable easier group analysis.
- In the Eligible vs Ineligible Subjects (%) with PDs chart, click the y-axis down-down to select from Site of Deviation (default) or Current Site and Eligibility (default) or Current Site.
- In the Subjects with PDs by Study, Country, or Site chart, click the y-axis down-down to select from Site of Deviation (default), Current Site, Study, or Country.
- The Protocol Deviation table presents the data in a tabular format, including additional details.
AE and SAE Rate
Use the AE and SAE Rate sheet to view detailed information and metrics related to adverse events and serious adverse events (AEs and SAEs).
- Use the top row filters on the left to view data by Study, Country, Site, Group Assignment, Subject, and AE Outcomes.
- Select the AEs (default) or SAEs button at the top left to view the rate, sorting, and reference lines on the charts by AEs or SAEs.
- Enter a value in the # of AEs/Subject (≥) threshold field to highlight outliers.
- Use the buttons above the Subject Ratio chart to switch between Subject (default), Event, and Unresolved views. The chart title updates to reflect the selected ratio.
- See the details of the AEs and SAEs in the table at the bottom of the sheet.
Time to SDV and Details
Use the Time to SDV and Details sheet to view metrics related to Source Data Verification.
If multiple studies are available, select one Study.
Enter a threshold in the Average Days to SDV (>=) to quickly identify outliers.
Timeliness
Use the Timeliness sheet to analyze how quickly data is reviewed, verified, or queried after initial entry. This sheet offers multiple methods to explore timeliness metrics.
- All four charts on the sheet are controlled by a shared dimension selector in the upper right corner. Select from Study (default), Country, Site, Subject, Folder, Form, or Data Field to update all charts simultaneously. Picking a value adjusts each chart’s x-axis and title accordingly.
- The three combo charts display average timelines (blue bars) along with the number of enrolled subjects (red diamonds).
- The Audits by Audit Date and Dimension line chart displays the number of audits over time for the selected dimension, with the year on the x-axis. Click a year to drill down to the months.
- A drill-down pivot table at the bottom of the sheet displays all three average day metrics. Expand rows to view values by Study, Country, Site, and Subject.
Post SDV and Post Query Changes
Use the Post SDV and Post Query Changes sheet to review metrics related to data changes and query activity that happen after source data verification (SDV).
- The three horizontal bar charts are controlled by a shared dimension selector located in the upper right corner of the sheet. Select from Study (default), Country, Site, Subject, Folder, Form, or Data Field to update all three charts at once.
- The Number of Queries Resulting in a Data Change by Role chart shows how many queries caused data changes, grouped by user role. A secondary y-axis shows the percentage of queries that resulted in data changes for each role.
Data Changes
Use the Data Changes sheet to view data changes related to folders, forms, fields, and sites.
- For the Data Changes Over Time chart, enter the graph's desired timeframe in months, 12 (months) is entered by default.
- In the Studies or Sites with Changes chart, click the x-axis drop-down arrow to select from Study (default) or Site.
- In the Forms or Folders with Changes chart, click the x-axis drop-down arrow to select from Form (default) or Folder.
Audit Summary
Use the Audit Summary sheet to view metrics related to data changes by change reason and audits by user and action type.
- In the Data Changes by Change Reason chart, select a change reason (within the treemap), and / or a particular site (either within the treemap or the legend).
- In the Audits by Action Type chart, click the y-axis drop-down arrows on to select from Site (default), Folder, Form, Field, Subject, or Study, and Action Type (default), Folder, Form, Field, Subject, or Study.
Application Information
Use the Application Information sheet to view application Information, such as what's new in CTOA for the version being used, the last data refresh date (by domain), a glossary of terms used in CTOA, a link to the Standards module, a Data Model Profile chart which displays the Actual Cells (number of base data cells) vs the Target Cells (number of standard application cells), and eCS support contact information.
Download Sheet Descriptions
Click the links below to Download the PDF with full sheet descriptions.