Retrieve Information from a Sampled Stream

To automatically start the synchronize action and retrieve tweets from a sampled stream, create a new fetch task using the following procedure.

NOTE: Twitter allows a user account to connect to only one stream at a time. If you run multiple fetch tasks concurrently (by setting the TaskThreads parameter) and these tasks connect to Twitter streams, Twitter might close the connections.

To retrieve information from a sampled stream

  1. Stop the connector.
  2. Open the connector’s configuration file in a text editor.
  3. In the [Ingestion] section of the configuration file, set the IngestBatchSize parameter:

    IngestBatchSize The number of documents to send to the ingestion server in each batch. If you are processing large numbers of streamed tweets and sending the information to CFS, OpenText recommends setting this parameter to 500 or greater. The Twitter Connector sends metadata-only documents to CFS, and CFS can process large batches more efficiently than small batches.
  4. In the [FetchTasks] section of the configuration file, specify the number of fetch tasks using the Number parameter. If you are configuring the first fetch task, type Number=1. If one or more fetch tasks have already been configured, increase the value of the Number parameter by one (1). Below the Number parameter, specify the names of the fetch tasks, starting from zero (0). For example:

    [FetchTasks]
    Number=1
    0=MyTask
  5. Below the [FetchTasks] section, create a new TaskName section. The name of the section must match the name of the new fetch task. For example:

    [FetchTasks]
    Number=1
    0=MyTask
       
    [MyTask]
  6. Ensure that the relevant OAuth parameters are included in the configuration file. For example, the following line imports the relevant parameters from a file named oauth.cfg. This file is generated by the OAuth configuration tool that is supplied with the connector. For more information about configuring OAuth authentication, see Configure OAuth Authentication.

    [MyTask] < "oauth.cfg" [OAUTH]

    For information about importing parameters into a configuration file, see Include an External Configuration File.

  7. In the new section, set the following parameters:

    ApiVersion Set ApiVersion=V2.
    StreamType The type of stream to ingest tweets from. To retrieve tweets from a sampled stream, set this parameter to statuses/sample.
  8. Save and close the configuration file. You can now start the connector.