Selenium WebDriver is a framework for browser automation, but it was not originally designed to handle API responses. In real-world testing scenarios, web applications frequently fetch data using asynchronous calls that return JSON payloads. To validate such responses, we can extend Selenium by pairing it with BrowserMob Proxy, a tool that intercepts network traffic and makes responses available for inspection. In this article, we demonstrate how to capture and parse JSON responses in Selenium with a practical example.
1. What is BrowserMob Proxy
BrowserMob Proxy is a free and open-source tool that helps developers monitor and manipulate network traffic in web applications. Acting as an HTTP proxy server, it provides several key capabilities.
It supports traffic manipulation, allowing developers to modify HTTP requests and responses, whitelist or blacklist resources, simulate network conditions such as latency and bandwidth, and even rewrite headers or content. It can also capture performance data, exporting detailed information in the standard HAR (HTTP Archive) format, which records web page loading performance.
When integrated with automation frameworks like Selenium WebDriver, BrowserMob Proxy becomes especially valuable, enabling the capture of client-side performance metrics during automated testing and offering insights into application performance bottlenecks. Finally, it can be used either as a standalone proxy server or embedded directly into Java-based programs and unit tests, making it highly flexible for different use cases.
2. What is HAR (HTTP Archive)?
A HAR file is a JSON-based log of all network traffic generated by the browser. It includes:
- Request URLs and methods.
- Response status codes and headers.
- Full request and response payloads.
By enabling HAR capture in BrowserMob Proxy, we can extract the raw JSON responses that the application fetches in the background.
3. Getting Started: Prerequisites
Selenium WebDriver controls browsers, but it does not directly expose APIs for capturing network traffic. That’s where BrowserMob Proxy comes in. It acts as a middleman between Selenium and the website, recording all network traffic in the form of an HTTP Archive (HAR) file. From there, we can extract JSON data.
To follow along, you should have:
- Basic knowledge of Java and Maven
- Java 11+ installed
- Chrome browser installed
- ChromeDriver (managed automatically with WebDriverManager in this article)
3.1 Project Setup
We’ll use Maven to manage dependencies for Selenium, BrowserMob Proxy, and Jackson. Maven makes it easy to configure and build the project.
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>4.35.0</version>
</dependency>
<dependency>
<groupId>io.github.bonigarcia</groupId>
<artifactId>webdrivermanager</artifactId>
<version>6.3.2</version>
</dependency>
<dependency>
<groupId>net.lightbody.bmp</groupId>
<artifactId>browsermob-core</artifactId>
<version>2.1.5</version>
</dependency>
This configuration includes Selenium WebDriver for automation, BrowserMob Proxy for intercepting network traffic and WebDriverManager for managing browser drivers automatically.
4. Building a Sample HTML Page
Here’s a minimal HTML page that fetches a JSON response from the JSONPlaceholder API when the page loads. We will save this file in the src/main/resources folder.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Selenium JSON Capture Test</title>
<style>
body {
font-family: Arial, sans-serif;
margin: 20px;
}
pre {
background-color: #f4f4f4;
border: 1px solid #ddd;
padding: 10px;
border-radius: 6px;
white-space: pre-wrap;
word-wrap: break-word;
max-width: 600px;
}
</style>
<script>
async function fetchData() {
try {
const response = await fetch("https://jsonplaceholder.typicode.com/posts/3");
const data = await response.json();
console.log("Fetched JSON:", data);
// Display JSON in the <pre> tag
document.getElementById("json-output").textContent = JSON.stringify(data, null, 2);
} catch (error) {
console.error("Error fetching JSON:", error);
document.getElementById("json-output").textContent = "Failed to fetch JSON.";
}
}
window.onload = fetchData;
</script>
</head>
<body>
<h1>Selenium JSON Capture Example</h1>
<p>The JSON response from <code>jsonplaceholder.typicode.com</code> will appear below:</p>
<pre id="json-output">Loading...</pre>
</body>
</html>
This HTML page makes a JSON API call when the page loads. By opening this page in Selenium with BrowserMob Proxy enabled, we can capture and parse the JSON response. You can serve the HTML page locally using any lightweight server. For example:
Python
python3 -m http.server 8000
Node.js (http-server)
npx http-server -p 8000
5. Java Code to Capture JSON Responses
The following Java program demonstrates how to capture JSON responses with BrowserMob Proxy HAR.
public class SeleniumJsonResponse {
public static void main(String[] args) throws Exception {
// Start BrowserMob Proxy
BrowserMobProxy proxy = new BrowserMobProxyServer();
proxy.start(0);
// Get Selenium Proxy
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
// Enable request/response content capture
proxy.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT);
DesiredCapabilities cap = new DesiredCapabilities();
cap.setCapability(CapabilityType.PROXY, seleniumProxy);
// Configure ChromeDriver with proxy
ChromeOptions options = new ChromeOptions();
options.addArguments("--disable-web-security");
options.addArguments("--allow-insecure-localhost");
options.addArguments("--ignore-urlfetcher-cert-requests");
options.setProxy(seleniumProxy);
cap.setCapability(ChromeOptions.CAPABILITY, options);
WebDriver driver = new ChromeDriver(options);
try {
// Enable HAR capture
proxy.newHar("demo");
// Navigate to test page
driver.get("https://localhost:8000/index.html");
// Wait a bit for network activity
Thread.sleep(3000);
Har har = proxy.getHar();
File harFile = new File("json.har");
har.writeTo(harFile);
} finally {
driver.quit();
proxy.stop();
}
}
}
When you run this file, it starts a BrowserMob Proxy server and configures ChromeDriver to route all browser traffic through the proxy. Once the proxy is running, it begins recording a HAR (HTTP Archive) of network activity. The browser then opens the local test page (index.html), which makes a JSON API request.
During this time, BrowserMob Proxy captures both the request and response payloads. After waiting a few seconds for the network call to complete, the HAR data is retrieved and written into a file named json.har. Finally, the browser and proxy are shut down. The result is a complete HAR file on disk.
Capturing HAR Files with Chrome Developer Tools
To get started, open your application in Google Chrome and press F12 (or Cmd + Option + I on macOS) to launch Developer Tools. Next, navigate to the Network tab and check the Preserve log option. This step is important because it ensures that the log is not cleared whenever the page reloads or navigates. After enabling this option, you should reload the page and visit the test page so that the network requests you want to capture are recorded.
Once network activity is being recorded, we can also export the traffic as a HAR file by simply clicking on the download arrow button in the Network tab. This saves the HAR log to a .har file, preserving the complete request and response bodies. We can later share this file or analyze it in detail using HAR viewer.
6. Conclusion
In this article, we examined how to capture JSON responses in Selenium with the help of BrowserMob Proxy. We started by introducing BrowserMob Proxy, highlighting its key features, and explaining the importance of HAR files for recording web performance data. From there, we demonstrated how to capture network traffic using HAR files. Finally, we explored how Chrome Developer Tools can be leveraged to manually record and export HAR files for further analysis.
7. Download the Source Code
This article discussed the use of Selenium to capture and process JSON response data.
You can download the full source code of this example here: selenium json response
