Get reason for failed LWP::Simple head request

Question

I have migrated my scrilpts from CentOS 7 to 8 and there's a new Perl version. I have the folowing snippet that uses head to check if a URL exists:

#!/bin/perl
use strict;
use warnings;
use LWP::Simple;

my $sitemapurl = "https://www.prosinger.net";
if (head($sitemapurl)) {
    ...
}
else {
    print "The $sitemapurl doesn't exist\n";
    exit(1);
}

It now always returns that the URL doesn't exist. I'm quite sure that this has to do something with https (I have perl-LWP-Protocol-https installed), but I'm not sure how to get any feedback information from head method to check what the error code is. Any ideas?

Answer 1

You can use LWP::UserAgent instead of LWP::Simple, which allows you to get an error message:

my $ua = LWP::UserAgent->new;
my $sitemapurl = "https://www.prosinger.net";
my $req = $ua->head($sitemapurl);
if ($req->is_success) {
    ...
} else {
    die "Could not head($sitemapurl): " . $req->status_line;
}

Running this code prints:

Could not head(https://www.prosinger.net): 403 Forbidden at head.pl line 15.

You can fix this (for this specific website; this will not work for all website) by setting a User-Agent in your LWP::UserAgent object:

my $ua = LWP::UserAgent->new( agent => 'anything seems to work' );

Of interest is the decoded_content method of HTTP::Response that allows you to get the content of the request (you don't need it in that case, but you might later):

my $req = $ua->get(...);
if ($req->is_success) {
    my $content = $req->decoded_content;
    ...
}

Answer 2

Your code that uses LWP::Simple and Dada's version that switches to LWP::UserAgent are basically doing the same thing, except that you can get details of the error when using LWP::UserAgent.

Running the LWP::UserAgent version gives this error:

Could not head(https://www.prosinger.net): 500 Can't connect to www.prosinger.net:443 (SSL connect attempt failed error:2707307E:OCSP routines:OCSP_check_validity:status not yet valid)

And Googling that error message gives this SO answer as the first result. Is it possible that the clocks on your your client machine and the server are out of sync?