The connector can run custom JavaScript on web pages before it crawls and ingests them. This allows you to manipulate pages or interact with them before they are processed. For example, if a page has a button to load more content you can configure the connector to run a script that clicks the button. The ingested page will then include the additional content.
Tip: Running custom JavaScript on pages is usually only necessary in advanced cases, such as processing unconventional sites.
You must write the custom script and save it to a file that can be accessed by the connector.
HPE recommends that you enclose your script in a try...catch
block to prevent exceptions being thrown out of the script. For example:
try { // custom javascript true; } catch(e) { console.log("Javascript Error: " + e.message) false; }
Any messages that you write to the JavaScript console are written to the connector's synchronize log, but only when you set the parameter LogLevel
to FULL
and EnableJavascriptConsoleLogging
to TRUE
. To troubleshoot your script, set these parameters and include a reasonable amount of logging in your script. The connector also logs the return value of the script.
To run custom JavaScript on web pages
Stop the connector and open the configuration file.
Modify your fetch task by adding the parameter CustomJSSections
. This parameter specifies the names of the sections in the connector's configuration file that contain settings for running custom JavaScript on pages. For example:
[MyTask0]
...
CustomJSSections=MyCustomJS
Create a new section in the configuration file to contain the settings for running the script:
[MyCustomJS]
In the new section, set the following parameters:
CustomJSUrlRegex
|
A Perl-compatible regular expression to match the full URLs of the pages on which you want to run custom JavaScript. |
CustomJSScriptPath
|
The path of the file that contains the custom JavaScript to run. The path must be absolute, or relative to the HPE Web Connector executable file. |
CustomJSElementSelector
|
(Optional) The page elements on which to run the custom JavaScript. Specify the elements using CSS2 selectors. |
CustomJSAllSelected
|
(Optional) A Boolean value that specifies whether to run the custom JavaScript on all elements that match the selector specified by CustomJSElementSelector (the default value, false , means that the connector runs the JavaScript only on the first matching element). |
For example:
[MyCustomJS] CustomJSUrlRegex=http://www\.hpe\.com/.* CustomJSScriptPath=javascript/my_script.js CustomJSElementSelector=div#maincontent
|