I'm implementing an app to fetch requests from a website API that seems to be protected by Cloudfare. When I make requests to the API using Java HttpClient framework, I get the following response (I've piped the response to an html file, and opened it in browser)
It's worth noting that when I make the same request using NodeJS fetch method, I still get the same response, but when I make the request in Firefox console using Javascript fetch method, there's no error, the response is valid and there's no problem. I thought that maybe my app should somehow pretend that the request is being sent from Firefox (or any other browser). But I don't know how to do that. Here is a sample code of what I'm trying to do:
import java.net.*;
import java.net.http.*;
import java.time.Duration;
public class HttpClientTest1 {
public static void main(String[] args) throws Exception {
CookieManager cm = new CookieManager();
cm.setCookiePolicy(CookiePolicy.ACCEPT_ALL);
CookieHandler.setDefault(cm);
HttpClient client = HttpClient.newBuilder()
.connectTimeout(Duration.ofSeconds(10))
.cookieHandler(cm)
.followRedirects(HttpClient.Redirect.ALWAYS)
.build();
HttpRequest request = HttpRequest.newBuilder()
.GET()
.uri(new URI("https://api.doggy.market/listings/nfts/minidoges?sortBy=price&sortOrder=asc&offset=0&limit=30"))
.header("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8")
.header("Accept-Language","en-US,en;q=0.5")
.header("Sec-Fetch-Dest", "document")
.header("Sec-Fetch-Mode", "navigate")
.header("Sec-Fetch-Site", "none")
.header("Sec-Fetch-User", "?1")
.header("TE", "trailers")
.header("Upgrade-Insecure-Requests", "1")
.header("User-Agent", "Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0")
.build();
HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
System.out.println(response.statusCode());
System.out.println(response.body());
}
}
What's wrong with my request? What's missing that I need to consider? Any help would be appreciated.
