Introduction to Data Imports

elluminate's Importer enables users to bring clinical, operational, reference, audit data, and more into a study from almost any source by either defining and configuring an Import Definition or by a Manual File Import. 

Import definitions use a Data Source to connect and import data from:

• Vendor-specific EDC and lab portals - (e.g., Rave, Oracle InForm, PPD Preclarus, Clario eCOA, Veeva CDMS, SAS LSAF)
• Secure Cloud Storage - (e.g., Amazon S3, Box)
• SFTP / FTPS - (e.g., Covance FTP)
• ODBC - (e.g., SQL, Snowflake)
• Metadata repositories for import into Standards - (e.g., Nurocor MDR, Formedix MDR used in Global Data Store only)

Manual file imports support the following file types:

• Excel
• CSV
• Text
• SAS Datasets (.sas7bdat)
• XPT (SAS export file)
• ZIP (Deflate64 zipped files are not supported.)

Each import delivers to a target data store, such as a staging area, data mart, or Global Data Store. However, the Global Data Store can only accept FTP, S3, Box, Nurocor MDR, Formedix MDR, or a manual file as data sources.

Run a Manual File Import

1. Click the 9-dot icon to open the Platform Menu.
2. By default, the Studies tab is selected at the top of the window. If importing to a global data store, click the Global Data Stores tab.
3. Select Importer under Study Management or Global Data Stores Management from the left side of the Platform Menu.

4. Select the Study / Global Data Store from the list on the right side of the Platform Menu. The Imports window opens.

   

5. Click the Manual File Import button. The File Import window opens.

   

6. Select Integrate Automatically to integrate the imported data after the import completes. If not selected, the data is listed as 'Ready to integrate.'
   • If selected, select Integrate Upon Discrepancies to integrate data even when discrepancies are found.
   • If not selected, data with discrepancies generates an error.
7. Select Skip Mapping to skip running mappings on imported data. This option may be used when importing into a global data store with many studies.
8. Select Defer Mapping to defer mapping until a later import. This option is useful when multiple imports occur in the same study and mapping is required only after the final import.

9. Select Skip Publishing to skip publishing imported data for visualizations.

   Important: Skip Mapping and Skip Publishing can speed up imports but may affect Analytics and other elluminate tools. If mappings are not skipped or deferred, the import applies any mappings previously deferred from manual imports.

10. Select the Target Data Store from the drop-down. This is the Data Store where the data is imported.

11. Select the file type from the File Type drop-down. This is optional if the file has a supported extension.

    Tip: When importing an Excel spreadsheet, the Importer scans the first 10 rows to identify the column header row. If detection fails, an import definition can be used to specify the layout.

12. Click in Drop Files Here or click Click to Browse to select a file. The file is added to the bottom half of the Imports window and is ready for import and integration.

    

13. To add another file to the same import, select the Add File tab in the bottom half of the window.

14. When adding another file, the file must not share the same domain name. Each file imported within the same active import must have a unique domain name or a unique Excel tab name.

    Note: Selecting Manual File Import creates a new active import. Only one active import can exist at a time. Each file added to the same import must have a unique domain name or Excel tab name.

15. Use the Import? column to select the domains to integrate. All domains are selected by default.
16. Review the Status column to identify whether a domain is ready to integrate, has issues, or is a new domain not previously added to the study.
17. Click Integrate in the upper-right corner of the bottom half of the window. A confirmation window opens.

18. Click Yes to proceed with integration or No to cancel.

    When integration begins, the import status displays in the bottom half of the Imports window. Click the plus icon next to a domain to view the file. The bottom half of the window displays the most recent completed import.

    

Tip: If an import is already running when another import starts, the conflicting import is queued until the current import completes.

Create a New Import Definition

After a data source is defined, an import definition can be created. The definition includes details such as the import name, data source, staging area, file, and header information. The import definition appears in the Imports list and can be reused for future imports without reselecting these settings. Previously created import definitions display in the top half of the Imports window. Import definitions are typically scheduled to run at defined times.

Note: For details, see the Configure and Manage Data Sources article in the Data Sources section.

Note: A data source is not required for a Manual File Import.

1. From the Imports window, click New or New from Template on the right of the window. 
   • To view available templates, click the drop-down arrow and select Templates.
   • If New from Template is selected, the Select Template window opens.

2. Select the template to use and click Select.

   

3. The Import Definition Details window opens. Enter a name in the Name field. This field is required.
   
   • When an import definition is created from a template, all template details populate automatically except for the Name field.
4. Review the tabs across the top of the window. Tabs include Import Definition and may include Options, Data Source Details, and / or Data Source, depending on the selected Data Source Type. Fields within each tab vary based on the selected data source.
5. Select a data source from the Data Source drop-down.

   • Available options include previously defined Study Data Sources and System Data Sources.

   • Details for the selected data source display on the right side of the window.

Data Source Types

Click a Data Source from the list below for instructions specific to that Data Source:

• Amazon S3

  Import Definition Tab

  Import To: Select a Data Store (staging area or data mart) that receives the imported data.

  Additional Mapping (if available): Identify an additional mapping to associate with this import.

  Password (for zip file): Enter the password if the imported file is zipped.

  Zip File Folder: Enter the folder within the zip file to load. The default is the root folder.

  Encoding: The default is UTF8. Other options include UTF7, UTF32, ASCII, or a numeric code page.

  Delimiter: Enter the delimiter to apply to all included files.

  Append Only: Select the checkbox to append data only.

  Include Unstructured Files: Select the checkbox to include unstructured files during the import. Unstructured files are placed in the study eDrive.

  Options Tab 

  When the file is missing: Select an option from the drop-down to define how the system handles a missing file.

  • Fail when missing: Stops the import job.
  • Continue, but log: Continues the import when possible and logs the missing file.

  Archive: Select the checkbox to store a zip file containing the imported data. The archive is stored in the data source specified by the import.
  Archive Location: Enter the relative path used to store the archived copy of the imported data. This path is relative to the original URL defined in the Data Source and is available only when Archive is selected.
  Fail Import on failed archive: Select the checkbox to fail the import if the archive cannot be created on the data source.

  Data Source Tab 

  Details entered on the Import Definition tab display in each field. To edit Data Source details, click the pencil icon to the right of a field, make updates, and click the checkmark to save the changes.

• Box

  Import Definition Tab

  Import To: Select a Data Store (staging area or data mart) that receives the imported data.

  Additional Mapping (if available): Identify an additional mapping to be associated with this import.

  Password (for zip file): Enter the password if the file to be imported is zipped.

  Zip File Folder: Folder within zip to load (default is root folder)

  Delimiter: Enter a delimiter. The delimiter applies to all included files.

  Append Only: Select the checkbox to append data only.

  Include Unstructured Files: Select the checkbox to include unstructured files during the import. Unstructured files are placed in the study eDrive.

  Options Tab 

  • When the file is missing: Select an option from the drop-down to define how the system handles a missing file.
    • Fail when missing: Stops the import job.
    • Continue, but log: Continues the import when possible and logs the missing file.

  Archive: Select the checkbox to store a zip file containing the imported data. The archive is stored in the data source specified by the import.
  Archive Location: Enter the path used to store the archived copy of the imported data. The path is relative to the original URL defined in the Data Source and is available only when Archive is selected.
  Fail Import on failed archive: Select the checkbox to fail the import if the archive cannot be created on the data source.

  Data Source Tab 

  Details entered in the Import Definition tab display in each of the fields. To edit Data Source details, click the Pencil icon at the right of a field, make updates, and then click the checkmark.

  If a Study or System Data Source is not already defined, use the Data Source tab to define a study-level data source.

• Clario eCOA

  Import Definition Tab

  Import To: Select a Data Store (staging area or data mart) that receives the imported data.

  Data Source Details Tab

  Metadata Version: Select the appropriate metadata version for the import.

  • Specific Version: Select this option to choose a metadata version from the drop-down list. Ensure the selected version corresponds to the study defined in the data source.
  • Latest Version Containing Text: Select this option to enter a search term that identifies the most recent metadata version. The search term typically follows a naming convention pattern and ensures that the latest metadata version for the selected environment is used during import.
    Click Test to verify that the text search functions correctly and that an appropriate metadata version can be located.

  Data Source Tab

  Use this tab to directly access or define the Clario eCOA data source for import. To modify the data source for the study, click the Pencil icon next to a field to enable editing and apply updates.

  Note: Clario eCOA contains electronic patient-reported outcomes (ePRO) data, which includes domains different from typical EDC domains. This data is imported using both cumulative and incremental domains and is processed to create a complete import without pausing for integration. The following domains are imported as cumulative datasets:
     Sites
     Users
     MetadataVersions
  
  All other domains are received as incremental datasets.

• File

  Import Definition Tab

  Import To: Select a Data Store (staging area or data mart) that receives the imported data.

  ODM_Stage and ODM_Mapped are the required names for the ODM staging area and data mart and must not be changed. Imports of ODM data to a data store other than ODM_Stage fail.

  Additional Mapping (if available): Identify an additional mapping to be associated with this import.

  Choose File(s): Select the file(s) to be imported.

  Password (for zip file): Enter the password if the file to be imported is zipped.

  Delimiter: Enter a delimiter. The delimiter applies to all included files.

  Append Only: Select the checkbox to append data only.

  Include Unstructured Files: Select the checkbox to include unstructured files during the import. Unstructured files are placed in the study eDrive.

  Note: This setting can be applied to any files imported through storage areas like SFTP, S3, and Box as well.

  Data Source Tab

  Click the Data Source tab to define a different Data Source if appropriate. When the Data Source Type is File and saved, the Data Source tab is no longer visible.

• FTP (Including use for Covance)

  Import Definition Tab

  Import To: Select a Data Store (staging area or data mart) that receives the imported data.

  Additional Mapping (if available): Identify an additional mapping to be associated with this import.

  Choose File(s): Select the file(s) to be imported.

  Password (for zip file): Enter the password if the file to be imported is zipped.

  Zip File Folder: Enter the folder within zip to load (default is root folder).

  Delimiter: Enter a delimiter. The delimiter applies to all included files.

  Append Only: Select the checkbox to append data only.

  Include Unstructured Files: Select the checkbox to include unstructured files during the import. Unstructured files are placed in the study eDrive.

  Options Tab 

  • When the file is missing: Select an option from the drop-down to define how the system handles a missing file.
    • Fail when missing: Stops the import job.
    • Continue, but log: Continues the import when possible and logs the missing file.
  • Archive: Select the checkbox to store a zip file containing the imported data. The archive is stored in the data source specified by the import.
    Archive Location: Enter the path used to store the archived copy of the imported data. The path is relative to the original URL defined in the Data Source and is available only when Archive is selected.
    Fail Import on failed archive: Select the checkbox to fail the import if the archive cannot be created on the data source.

  Data Source Tab 

  Details entered in the Import Definition tab display in each of the fields. To edit Data Source details, click the Pencil icon at the right of a field, make updates, and then click the checkmark.

  If a Study or System Data Source is not already defined, use the Data Source tab to define a study-level data source.

• Medrio Web Service

  Import Definition Tab

  Import To: Select a Data Store (staging area or data mart) that receives the imported data.

  Additional Mapping (if available): Identify an additional mapping to be associated with this import.

  Export Options Tab 

  Coming Soon

  Data Source Tab 

  Details entered in the Import Definition tab display in each of the fields. To edit Data Source details, click the Pencil icon at the right of a field, make updates, and then click the checkmark.

  If a Study or System Data Source is not already defined, use the Data Source tab to define a study-level data source.

• Merge eClinicalOS Web Services (Zelta)

  Import Definition Tab

  Import To: Select a Data Store (staging area or data mart) that receives the imported data.

  Additional Mapping (if available): Identify an additional mapping to be associated with this import.

  Options Tab 

  Select the checkbox for additional configuration of the data source

  • Include Coded Values
  • Expand Categorical Variables
  • Include Extended Subject Information
  • Include Dictionary Coded Values
  • Include Only Locked Data
  • Include Code List Files
  • Merge Repeating and Non-Repeating Data on Log Forms
  • Include Form Timestamps
  • Include Cleared Data
  • Append Form Visit Names

  Data Source Tab 

  Details entered in the Import Definition tab display in each of the fields. To edit Data Source details, click the Pencil icon at the right of a field, make updates, and then click the checkmark.

  If a Study or System Data Source is not already defined, use the Data Source tab to define a study-level data source.

• ODBC

  Import Definition Tab

  Import To: Select a Data Store (staging area or data mart) that receives the imported data.

  Data Source Tab 

  Details entered in the Import Definition tab display in each of the fields. To edit Data Source details, click the drop-down arrow at the right of a field and update the selection.

  If a Study or System Data Source is not already defined, use the Data Source tab to define a study-level data source.

• Oracle Health Sciences InForm Discrepancy

  Import Definition Tab

  Import To: Auto-filled 'ODM_Stage'

  Additional Mapping (if available): Identify an additional mapping to be associated with this import.

  Data Source Tab 

  Details entered in the Import Definition tab display in each of the fields. To edit Data Source details, click the Pencil icon at the right of a field, make updates, and then click the checkmark.

  If a Study or System Data Source is not already defined, use the Data Source tab to define a study-level data source.

• PPD - Deprecated

  Import Definition Tab

  Import To: Select a Data Store (staging area or data mart) that receives the imported data.

  Data Source Tab 

  Details entered in the Import Definition tab display in each field. To edit Data Source details, click the Pencil icon at the right of a field, make updates, and then click the checkmark.

  If a Study or System Data Source is not already defined, use the Data Source tab to define a study-level data source.

• PPD Pleclarus - Labs

  Import Definition Tab

  Data Source: Select the PPD Preclarus - Labs study or system data source from the dropdown menu. The details display on the right side of the Import Definition window.

  Import To: Select a Data Store (staging area or data mart) that receives the imported data.

  Data Source Details: The details of the selected PPD Preclarus—Labs data source display on the right side of the Import Definition window.

  Data Source Tab 

  Use this tab to directly access or define the import's PPD Preclarus—Lab data source. If modifying a data source for the study, click the pencil icon next to the field to make it editable for the updates.

  Note: If the import definition is updated to change the target data store and has previously been executed, eCS advises that the data store to which it was once imported be cleared of all originally imported PPD data to support the incremental nature of the data source.

• PPD Pleclarus - Subjects

  Import Definition Tab

  Data Source: Select the PPD Preclarus—Subjects study or system data source from the dropdown menu. The details appear on the right side of the Import Definition window.

  Import To: Select a Data Store (staging area or data mart) that receives the imported data.

  Data Source Details: The details of the selected PPD Preclarus—Subjects data source display on the right side of the Import Definition window.

  Data Source Tab 

  Use this tab to directly access or define the import's PPD Preclarus—Subjects data source. If modifying a data source for the study, click the pencil icon next to the field to make it editable for the updates.

• Rave Biostats Gateway

  Import Definition Tab

  Data Source Type: Select the Data Source Type from the drop-down.

  Import To: Select a Data Store (staging area or data mart) that receives the imported data.

  Additional Mapping (if available): Identify an additional mapping to be associated with this import.

  Data Source Details 

  For a Rave Biostats gateway import, select the applicable checkboxes to modify import behavior:

  • Incremental: Select this checkbox to limit the import to New and Changed records only.
  • Include Lab View: Select this checkbox to import Rave LAB View data. This option should be selected only when the LAB View feature is configured in RAVE. Selecting this option may cause conflicts if an existing CRF is named LAB.
  • MetaData Version:
    • Specific Version: Select this option to choose the metadata version from the drop-down list.When a specific version is selected, the system does not automatically update if Rave data structures change. Ensure the selected version corresponds to the environment defined in the data source.
    • Latest Version Containing Text: Select this option to enter search text to to identify the most recent metadata version. A pattern derived from the naming convention is usually entered if this choice is selected (for example: REL, PROD, etc.). This ensures that the latest metadata version for the selected environment is used during import.
    • Click Test to verify that the text search functions correctly and that the system can find a metadata version to use.

  Data Source Tab 

  Data Source details display in each field. To edit Data Source details, click the Pencil icon at the right of a field, make updates, then click the checkmark.

  If a Study or System Data Source is not already defined, use the Data Source tab to define a study-level data source.

• Rave ODM Adaptor

  Import Definition Tab

  Data Source Type: Select the Data Source Type from the drop-down.

  Import To: There is no selection for Rave ODM Adaptor – it defaults to ODM Stage. There are no other data options when importing Rave ODM data.

  Data Source Details Tab 

  For a Rave ODM import, select the operational data to import.

  • Audits: Selected by default and selects all items in the list below. Clear the Audits checkbox to make individual selections.
    • Subjects and Statuses
    • Study Metadata
    • Signatures
    • Sites
    • Users
    • Version Folders

  Data Source Tab 

  Data Source details display in each field. To edit details, select a field and enter updated values.

  If a Study or System Data Source is not already defined, use the Data Source tab to define a study-level data source.

• Rave Web Services

  Import Definition Tab

  Data Source Type: Select the Data Source Type from the drop-down.

  Import To: Select a Data Store (staging area or data mart) that receives the imported data.

  Additional Mapping (if available): Identify an additional mapping to be associated with this import.

  Data Source Tab 

  Data Source details display in each field. To edit details, select a field and enter updated values.

  If a Study or System Data Source is not already defined, use the Data Source tab to define a study-level data source.

• SAS Life Science Analytics Framework (LSAF)

  Import Definition Tab

  Data Source Type: Select the SAS LSAF study or system data source from the dropdown menu. The data source details display on the right side of the Import Definition window.

  Import To: Select a Data Store (staging area or data mart) that receives the imported data.

  Data Source Tab 

  Data Source details display in each field. To edit details, select a field and enter updated values.

  If a Study or System Data Source is not already defined, use the Data Source tab to define a study-level data source.

• TrialMaster

  Import Definition Tab

  Import To: Select a Data Store (staging area or data mart) that receives the imported data.

  Additional Mapping (if available): Identify an additional mapping to be associated with this import.

  Data Source Tab 

  Data Source details display in each field. To edit details, select a field and enter updated values.

  If a Study or System Data Source is not already defined, use the Data Source tab to define a study-level data source.

• Veeva

  Import Definition Tab

  Import To: Select a Data Store (staging area or data mart) that receives the imported data.

  Data Source Tab 

  Data Source details display in each field. To edit details, select a field and enter updated values.

  If a Study or System Data Source is not already defined, use the Data Source tab to define a study-level data source.

• Veeva CDMS

  Import Definition Tab

  Import To: Auto-filled 'ODM_Stage'

  Data Source Tab 

  Data Source details display in each field. To edit details, select a field and enter updated values.

  If a Study or System Data Source is not already defined, use the Data Source tab to define a study-level data source.

4. Click Save.
    

Important: After the data source is created and the Import Definition Details are specified, the import does not occur unless it is scheduled or run manually.

Run an Import From a Defined Definition

Select Run to execute a currently defined import definition listed in the Imports window. Import definitions may be scheduled to run at set times; however, selecting Run allows data to be imported immediately before a scheduled run.

1. Click the 9-dot icon to open the Platform Menu, then select Importer under Study Management.
2. Select a study from the list on the right side of the Platform Menu.

3. From the study-level Imports window, select an import definition by clicking a row. The row highlights in blue.

   

4. Click the Run button. The Run Import confirmation window opens. Additional checkboxes may display depending on the import type.

   

5. Select Integrate Automatically to integrate the data into elluminate. Data is not available until integration completes.
6. If Integrate Automatically is selected, select Integrate Upon Discrepancies to integrate data even when discrepancies are found. If not selected, data with discrepancies generates an error.
7. If the study has configured Enable Changed Data Records Threshold, selecting Skip Data Change Threshold Check allows files that exceed the threshold percentage to import without changing the study configuration.
8. Select Defer Mapping to postpone mapping until a later import. This option is useful when multiple imports occur in the same study and mapping is required only after the final import.
9. Select Skip Mapping to skip running mappings on imported data. This option may be used when importing into a global data store with many studies.

10. Select Skip Publishing to skip publishing imported data for visualizations.

    Important: Skip Mapping and Skip Publishing can speed up imports but may affect Analytics and other elluminate tools. If mappings are not skipped or deferred, the import applies any mappings previously deferred from manual imports.

11. Click Yes to proceed or No to cancel.
    If Yes is selected, the import begins and results display in the bottom half of the window. The Status field in the upper-left corner indicates whether the import is successful or if an error occurs.

Tip: If an import is already running when another import starts, the conflicting import is queued until the current import completes.

Schedule an Import from a Defined Definition

Selecting Schedule configures the import to run automatically at a specified time and frequency. Scheduling can be performed directly from the Importer tab using the Schedule button or managed through the Tasks module. 

Note: For example, Study_A and Study_B can each schedule imports to their Clinical Staging Areas concurrently; however, a single study cannot schedule multiple imports to the same data store at the same time.

1. From the study-level Importer window, select an import definition by clicking a row. The row highlights in blue.
2. Click the Schedule button.
3. The New Task window opens with 3 tabs: Task, Schedule, and Actions.
4. From the Task tab, review the Name field. The default value combines 'Import,' the Study name, and the import definition. Update the name as needed.
5. In the Description field, enter additional details for the schedule.
6. The Active checkbox is selected by default. Clear the checkbox to prevent the schedule from running.
7. Click the Schedule tab.
8. Select the import timing. Use the calendar and clock icons to set the date and time.
9. Select Repeat and specify the repeat interval, if applicable.
10. Select Ends and use the calendar icon to define the end date, if applicable.
11. Click the Actions tab.
12. Review the predefined actions. When scheduled from Importer, the actions include an import and an email notification. By default, the email notifies when the import completes.
13. To modify an action, click the Edit Action icon for the import or email. Options include updating the import name, modifying email recipients, or limiting notifications to errors only.
14. Click Save. A task for the scheduled import is created.

Access and View Files from an Import

View Files in Importer

Files can be viewed and accessed by clicking the Domain name listed in the bottom half of the window.

1. Click the side arrow next to a Domain Name to expand the Domain.

   

2. Click the Download icon to download the file to the computer.

3. Click the Domain Name to open a window displaying the Domain Schema and Source Data. This window provides a preview of the Domain file structure. By default, the Schema data displays. Click the Source Data tab to view the source data.

   

4. Click OK to close the window.

View Files in eDrive

For any import, a Files in eDrive link displays in the bottom half of the window. Click the link to open eDrive for that import and download the files. Select an import from the top half of the Imports window to access files associated with that import.

Tip: If an import is unsuccessful, use this link to view any associated log files. For example, when an SAS import fails, the referenced log file can be found in eDrive under the sas_etl subfolder.

View Import History

The History option in Importer displays a record of all imports for the selected study or Global Data Store, including both successful and unsuccessful imports.

1. From the study-level Importer window, click History.

   

2. From the list, click an import row to view the domains in the bottom half of the window.

   

Best Practices Checklist

• Define the Data Source first. Configure the URL, credentials, remote folder, remote study, API keys, and other necessary properties before building the Import Definition; manual file imports are the only exception.
• Choose the simplest data source. Use out-of-the-box API connectors or S3 / Box pulls over custom SFTP scripts, as they require less setup and deliver data more quickly.
• Confirm column size and precision match data formats. Ensure all columns with numeric values match the set scale and precision when using a precise file format like SAS. Any cases where imported data exceeds the format assigned in metadata can cause truncation or other issues downstream in elluminate.
• Watch Study Import Control limits. If record-change thresholds are enabled and the import exceeds configured thresholds, the import completes with a warning indicating that elluminate does not import the data.
• Schedule imports to avoid contention. A study can run only one import into a data store at a time. For example, Study A and Study B may import into their respective Clinical data stores simultaneously, but Study A cannot schedule two imports to run concurrently in its Clinical data store. Subsequent imports queue automatically, so stagger schedules for faster throughput.