Configure and Schedule Data Ingestion

This article is currently being updated. Please come back later for updates.

Permissions: When configuring Data Sources at the Study Level, user must have the Study Configure privilege. When Configuring System Data Sources, user must have the System Configure privilege. When using Tasks, users must have the Importer privilege for Data Import and Email Notifications.

Note: The User Interface was adjusted to what is seen in the video for versions 22.3 and later, but the steps for previous versions should be generally the same. System Data Sources are available with 23.1 and later. The option to cancel stalled Tasks was added to 24.1 and later, not seen in the video.

Instructions

Step One: Create a (System or Study) Data Source

  1. From the Platform Menu, choose Platform Administration > System Data Sources OR choose Study Management > Data Source > Specific Study. (This tutorial provides the steps for a Study Data Source.)
  2. When Data Sources for that Study opens, click the New button from the Master Header.
  3. New (Data Source) Connection window:
    1. Enter Name.
    2. Enter optional Description.
    3. Choose an existing System Data Source (to adjust as needed) OR choose a Data Source Type to create a new one. (This tutorial provides the steps for a Rave Type Data Source.)
      1. Enter the EDC Address/URL of the source.
      2. Enter the username and password used to access the source, which is a single username and password used to access the remote system that must have both Read Only and Architect access.
      3. Enter the Environment.
      4. Enter the name of the Study being remotely accessed.
  4. To create the Data Source, click Save.

Tip: With the System Configure Privilege, any configured Study Data Source can be promoted to a System Data Source at any time.

Step Two: Create a Data Import Definition

  1. Open Importer for a Study.
  2. To configure a New Import Definition, click the New button from the Imports toolbar.
  3. New Import Definition Details window:
    1. Enter Name.
    2. Select the Data Source.
    3. Choose the Datastore to Import To. (In this case, Clinical).
    4. Because this tutorial chose a Rave Biostat Import, the Data Source Details Tab was added, which requires additional information:
      1. Optionally, check for Incremental.
      2. Optionally, check to Include Lab View.
      3. Optionally, check to Use CSV Clinical View configurable datasets.
      4. Choose the Specific or Latest Metadata Version. (Required for Rave Biostat Imports)
    5. Click Save.
  4. To ensure the Data Source and Import are configured properly, click Run to manually import the data.
    1. Optionally, check to Integrate Automatically (if unchecked, the data will need to be integrated in an additional step).
    2. Optionally, check to Integrate upon Discrepancies (if unchecked and there are data discrepancies, the data will need to be integrated in an additional step).
    3. Optionally, defer Mapping.
    4. Optionally, skip Mapping.
    5. Optionally, skip Publishing.
  5. To run the import, click Yes. Domains will begin populating, and the status will say Complete once the Import has successfully completed.

Step Three: Schedule the Import

  1. Select the Import, and click the Schedule button.

    Note: The schedule button conveniently opens a New Task window with a Name entered, the task set to active, and actions for Import and Email already configured.

  2. Task Tab
    1. Optionally, adjust the Name.
    2. Optionally, add a Description.
    3. Optionally, check to cancel the task after a # of hours if it's stalled. (Added with 24.1+)
  3. Schedule Tab
    1. Enter Start Date, the date this Task will first run.
    2. Optionally, check to have the Task repeat.
      1. Repeat every # and interval (Minute, Hour, Day, Week, and Month).
      2. Optionally, check to ensure the repeat Ends on a specific date. Leave unchecked to have the Task repeat indefinitely until modified.
  4. Actions Tab
    1. Optionally, edit the default Import task.
    2. Optionally, edit the default Email notification task.
    3. Optionally, add additional Actions by clicking the Add button.

      Note: Only actions you have privileges for will appear in the Add list.

    4. To finalize and create the Schedule, click Save. Data will now be automatically imported on that schedule.
  5. To view and edit the Task, enter Tasks from the drop-down menu for that Study.

Was this article helpful?

0 out of 0 found this helpful

Have more questions? Submit a request