Set up a NiFi Cluster

To create an Apache NiFi cluster, set up either three or five ZooKeeper nodes.

To set up ZooKeeper nodes for a NiFi cluster

  1. Install Apache NiFi on the first node.
  2. Open the file ./conf/nifi.properties and set or review the following properties:

    Property Description
    nifi.cluster.is.node Set this property to true.
    nifi.state.management.embedded.zookeeper.start Specifies whether this node runs an embedded ZooKeeper server. Set this property to true.
    nifi.zookeeper.connect.string

    A comma-separated list of host names and ports, for connecting to ZooKeeper on each node. The port number you set here must match the ZooKeeper client port that you set later (see step 4). For example:

    host1:4881,host2:4881,host3:4881

    nifi.web.http.port The NiFi HTTP Port. This is the port that you would use to open the Apache NiFi web interface.
    nifi.cluster.node.protocol.port Specify any free port above 1024.
    nifi.cluster.load.balance.port Specify any free port above 1024.
    nifi.state.management.embedded.zookeeper.properties This property specifies the path to the ZooKeeper configuration file. You can use the default value.
  3. Open the file ./conf/state-management.xml and set the Connect String property (in the ZooKeeper section) to the same value you set for nifi.zookeeper.connect.string in the nifi.properties file. For example:

    <property name="Connect String">host1:4881,host2:4881,host3:4881</property>
  4. Open the file ./conf/zookeeper.properties.

    • Apache NiFi 1.10.x - List the nodes that you have created. For example:

      server.1=host1:2881:3881;4881
      server.2=host2:2881:3881;4881
      server.3=host3:2881:3881;4881

      The first two port numbers on each row are used by ZooKeeper. Specify any free port above 1024. The last port number on each row must correspond to the port you specified in the property nifi.zookeeper.connect.string in the nifi.properties file.

    • Apache NiFi 1.9.x and earlier - Complete the following steps:

      1. Set the clientPort parameter to the port you specified for this node in nifi.zookeeper.connect.string in the nifi.properties file. For example:

        clientPort=4881
      2. List the nodes that you have created. For example:

        server.1=host1:2881:3881
        server.2=host2:2881:3881
        server.3=host3:2881:3881

        These port numbers are used by ZooKeeper. Specify any free ports above 1024.

  5. Create a text file ./state/zookeeper/myid that contains the server index. In the previous step you edited zookeeper.properties and specified that host1 has a server index of 1 (because it is listed as server.1), host2 has a server index of 2 (server.2), and host3 has a server index of 3 (server.3). The text file that you create should contain only the number and nothing else.

  6. Repeat the previous steps for the other nodes in your cluster.
  7. Start Apache NiFi on all of the nodes in the cluster.
  8. Open the NiFi user interface for any node in the cluster.

    • The user interface shows this icon, describing the number of nodes in the cluster:
    • To check the status of the nodes in the cluster, click followed by Cluster.