Retrieve Information from Google Cloud Storage

To automatically retrieve information from Google Cloud Storage, create a new fetch task by following these steps. The connector runs each fetch task automatically, based on the schedule that is configured in the configuration file.

To create a new Fetch Task

  1. Stop the connector.
  2. Open the configuration file in a text editor.
  3. In the [FetchTasks] section of the configuration file, specify the number of fetch tasks using the Number parameter. If you are configuring the first fetch task, type Number=1. If one or more fetch tasks have already been configured, increase the value of the Number parameter by one (1). Below the Number parameter, specify the names of the fetch tasks, starting from zero (0). For example:

    [FetchTasks]
    Number=1
    0=MyTask
  4. Below the [FetchTasks] section, create a new TaskName section. The name of the section must match the name of the new fetch task. For example:

    [FetchTasks]
    Number=1
    0=MyTask
    
    [MyTask]
  1. In the new section, set the parameters required to authenticate with Google Cloud Storage. The OAuth configuration tool creates a file that contains these parameters, and you can include them in the connector's configuration file using the following syntax:

    [MyTask] < "oauth.cfg" [OAUTH]

    For more information about including parameters from another file, see Include an External Configuration File

  2. To authenticate with Google Cloud Storage as an ordinary user, set the Username parameter. You do not need to set this parameter if you are using a service account.

    Username The user name to use to authenticate with Google Cloud Storage.
  3. (Optional) Set one of the following configuration parameters:

    ProjectIDs A comma-separated list of project IDs for the projects that you want to process.
    ProjectFilter A filter to choose the projects that you want to process. For information about how to construct the filter, refer to the Google Cloud Storage documentation.

    For example:

    [MyTask] < "oauth.cfg" [OAUTH]
    ProjectIDs=idol-123,idol-987

    Or alternatively:

    [MyTask] < "oauth.cfg" [OAUTH]
    ProjectFilter=name:idol*

    If you do not set either of these parameters, the connector ingests data from all available projects.

  4. (Optional) You can set additional parameters to choose the buckets and objects that you want to synchronize. For information about these parameters, refer to the Google Cloud Storage Connector Reference.

  5. Save and close the configuration file. You can now start the connector.