IDOL Ingest

Dynamics Connector is one of the components in IDOL Ingest, a collection of tools for data retrieval and enrichment. IDOL Ingest can prepare data for indexing into an IDOL Content engine, so that you can use IDOL to search and analyze your information, but you can also use IDOL Ingest as the first step in any data processing task and send the information to other systems.

IDOL Ingest includes connectors, which retrieve information from a specific type of repository. There are IDOL connectors for over 150 repositories, including:

  • Local and network file systems.
  • Web sites and social media feeds.
  • Microsoft 365 (Office 365) services such as OneDrive and Teams.
  • Document and content management systems such as Microsoft SharePoint.
  • E-mail servers such as Microsoft Exchange.
  • Communication tools such as Slack.
  • Database servers such as Microsoft SQL Server or MySQL.
  • Cloud services such as Amazon (AWS) S3, Dropbox, or Google Drive.

Connectors produce documents. A document represents a single item that exists within a repository - such as a file in a file system, a page from a web site, or a message from a chat application. The documents that are produced by connectors contain metadata that was extracted from the repository, such as the location of the item and an Access Control List (ACL) that describes who is permitted to view it. The presence of an ACL allows an IDOL Content engine to restrict access to information, maintaining the security permissions defined in the source repository without impacting query performance, even when you have millions of documents.

Each document produced by a connector can, optionally, have an associated file. For example, a document produced by an IDOL File System Connector to represent a file has an associated copy of the file. A document produced by an IDOL Web Connector to represent a web page can have an associated file containing the HTML source of the page. These binary files are not indexed into your IDOL Content engine, but the information they contain can be used by other IDOL Ingest components to enrich the document.

Connectors send documents to other IDOL components for enrichment and further processing. For example you can:

  • use IDOL KeyView to extract subfiles from containers, such as ZIP archives. Subfiles are files that are contained within other files. For example, an e-mail message might contain attachments, or a Microsoft Word document might contain an embedded image or spreadsheet.
  • use IDOL KeyView to filter the text from files, so that you can access the text without needing to process a file in its native format.
  • use IDOL Rich Media components to analyze media files, adding information from Optical Character Recognition, Speech-To-Text, or Face Recognition to a document.
  • use IDOL Eduction to locate or redact Personally Identifiable Information (PII).
  • run many other processing tasks, including running a custom Lua or Python script to process the data however you wish.

OpenText recommends that you deploy your IDOL connectors and IDOL Ingest components in Apache NiFi. Apache NiFi is an open-source framework that can help you visualize and configure the flow of data in a system. Many IDOL features can be deployed in Apache NiFi - including connectors, KeyView extraction and filtering, and rich media analysis. Using Apache NiFi makes all of these components easier to deploy, configure, and manage.

Alternatively, you can deploy IDOL connectors as standalone ACI servers. If you deploy connectors in this way you might also need to deploy an Apache NiFi system or an IDOL Connector Framework Server (CFS) to manage the enrichment and indexing tasks. If you use CFS and want to run media analysis, you would also need to install a standalone IDOL Media Server.

For more information about the IDOL platform, refer to the IDOL Getting Started Guide.