Contact Us 1-800-596-4880
  1. Homepage
  2. MuleSoft WebCrawler Connector (1.0)

  3. Using Anypoint Studio to Configure MuleSoft WebCrawler Connector

All Connectors

Using Anypoint Studio to Configure MuleSoft WebCrawler Connector 1.0

Anypoint Studio (Studio) editors help you design and update your Mule applications, properties, and configuration files.

To add and configure a connector in Studio:

  1. Create a Mule project.

  2. Add the connector to your Mule project.

  3. Configure a source for the connector’s flow.

  4. Add a connector operation to the flow.

  5. Configure a global element for the connector.

  6. Configure the other connector fields.

When you run the connector, you can view the app log to check for problems in real time, as described in View the App Log.

If you are new to configuring connectors in Studio, see Using Anypoint Studio to Configure a Connector. If, after reading this topic, you need additional information about the connector fields, see the MuleSoft WebCrawler Connector Reference.

Create a Mule Project

In Studio, create a new Mule project in which to add and configure Anypoint Connector for MuleSoft WebCrawler (MuleSoft WebCrawler Connector):

  1. In Studio, select File > New > Mule Project.

  2. Enter a name for your Mule project and click Finish.

Add the Connector to Your Mule Project

Add MuleSoft WebCrawler Connector to your Mule project to automatically populate the XML code with the connector’s namespace and schema location and add the required dependencies to the project’s pom.xml file:

  1. In Mule Palette, click (X) Search in Exchange.

  2. In Add Dependencies to Project, type mulesoft webcrawler in the search field.

  3. Click MuleSoft WebCrawler Connector in Available modules.

  4. Click Add.

  5. Click Finish.

Adding a connector to a Mule project in Studio doesn’t make that connector available to other projects in your Studio workspace.

Configure a Source

A source initiates a flow when a specified condition is met. You can configure one of these sources to use with MuleSoft WebCrawler Connector:

  • On Content Crawl

    Crawls a website starting from a configured path, emitting each discovered page, image, or document as a Mule event. Supports deduplication via an object store and URL scheduling via a VM queue.

  • HTTP Listener

    Initiates a flow each time it receives a request on the configured host and port.

  • Scheduler

    Initiates a flow when a time-based condition is met.

For example, to configure an HTTP > Listener source, follow these steps:

  1. In Mule Palette, select HTTP > Listener.

  2. Drag Listener to the Studio canvas.

  3. On the Listener configuration screen, optionally change the value of the Display Name field.

  4. Specify a value for the Path field.

  5. Click the plus sign (+) next to the Connector configuration field to configure a global element that can be used by all instances of the HTTP > Listener source in the app.

  6. On the General tab, specify the connection information for the connector.

  7. On the TLS tab, optionally specify the TLS information for the connector.

  8. On the Advanced tab, optionally specify reconnection information, including a reconnection strategy.

  9. Click Test Connection to confirm that Mule can connect with the specified server.

  10. Click OK.

Add a Connector Operation to the Flow

When you add a connector operation to your flow, you are specifying an action for that connector to perform.

To add an operation for MuleSoft WebCrawler Connector:

  1. In Mule Palette, select MuleSoft WebCrawler Connector and then select the operation to add.

  2. Drag the operation onto the Studio canvas, next to the source.

Configure a Global Element for the Connector

When you configure a connector, configure a global element that all instances of that connector in the app can use. Configuring a global element requires you to provide the connection settings that the connector requires to access the target web resources.

To configure the global element for MuleSoft WebCrawler Connector:

  1. Select the operation in the Studio canvas.

  2. In the General configuration screen for the operation, click the Add icon to access the global element configuration fields.

  3. In the General tab, in Connection, select the connection type to configure:

    You can reference a configuration file that contains ANT-style property placeholders (recommended), or you can enter your connection settings in the global configuration properties. For information about the benefits of using property placeholders and how to configure them, refer to Anypoint Connector Configuration.

  4. In the Advanced tab, optionally specify reconnection information, including a reconnection strategy.

  5. Click Test Connection to confirm that Mule can connect with the specified server.

  6. Click OK.

HTTP

Use the HTTP connection type for crawling static web content.

Enter this information in the General tab of the Global Element Properties screen to configure the HTTP connection:

Field User Action

Name

Enter the configuration name.

Connection

Select HTTP.

Base URL

Enter the base URL against which relative paths are resolved.

Browser User Agent

Enter the browser user-agent advertised on outbound requests.

Proxy Configuration

Configure a proxy for outbound HTTP requests. Specify host, port, and optional credentials.

Response timeout

Enter the connect and read timeout for the connection’s HTTP client. Default is 30 seconds.

Response timeout unit

Select the time unit for the response timeout.

Optionally configure TLS settings for secure connections in the TLS tab.

Remote WebDriver

Use the Remote WebDriver connection type for crawling dynamic web content that requires JavaScript rendering.

Enter this information in the General tab of the Global Element Properties screen to configure the Remote WebDriver connection:

Field User Action

Name

Enter the configuration name.

Connection

Select Remote WebDriver.

Base URL

Enter the base URL against which relative paths are resolved.

Remote Browser URL

Enter the URL of your own remote Selenium WebDriver endpoint (for example, Selenium Grid, standalone ChromeDriver, or BrowserStack). Chromium-only (Chrome, Chromium, Edge).

Browser User Agent

Enter the browser user-agent advertised on outbound requests.

Browser Authentication

Select the authentication used to reach the remote browser endpoint: None, Basic, or API Access Token.

Proxy Configuration

Configure a proxy for outbound HTTP requests. Specify host, port, and optional credentials.

Response timeout

Enter the connect and read timeout for the connection’s HTTP client. Default is 30 seconds.

Response timeout unit

Select the time unit for the response timeout.

Optionally configure TLS settings for secure connections in the TLS tab. You can also configure a Pooling Profile in the Advanced tab to control connection pool behavior.

Configure Additional Connector Fields

After you configure a global element for the connector, configure the other required fields. The required fields vary depending on which connector operation you use.

For detailed configuration information for each operation type, see MuleSoft WebCrawler Connector - Additional Configuration Information.

View the App Log

To check for problems, you can view the app log as follows:

  • If you’re running the app from Anypoint Platform, the app log output is visible on the Anypoint Studio console window.

  • If you’re running the app using Mule from the command line, the app log output is visible in your operating system console.

Unless the log file path is customized in the app’s log file (log4j2.xml), you can also view the app log in the default location MULE_HOME/logs/<app-name>.log. You can configure the location of the log path in the app log file log4j2.xml.

See Also

Back to top