Best way to randomly access and modify sequential data in Perl

Question

The input to my Perl program is a text file that contains one 'item' per line. For example, a file with below contents:

• item1 abc yyy anything blabla whatever
• item2 efg dod whatever blabla mmm
• item3 hij naa anything gagaga whatever gooo 1,2,3
• item4 fff ahh whatever blabla whatever
• item5 noo kir whatever bbbbbb hhhhhh
• item6 123 kkk anything blabla whatever
• item7 555 yyy anything ghghgh a,b,c
• item8 777 yxy whatever blabla whatever

This input file is read into an array (in reality is read into an array of hashes but for this example lets assume an array strings):

my @items;

Associated with each item is a file on the network. For each item in the input file, the program searches the network for its associated file:

my $associated_files_handle;

open ( $associated_files_filehandle, "> $associated_files_filename");

my $associated_file;

my @items_whose_associated_file_was_found;
my @items_whose_associated_file_was_not_found;

foreach my $item (@items) {
    # sub search_associated_file returns:
    #   - If associated file is  found: path to the associated file, or
    #   - If associated file not found: undef
    $associated_file = search_associated_file( $item );
    
    if (defined $associated_file) {
       print $associated_files_filehandle, "$associated_file\n";
       push @items_whose_associated_file_was_found, $item;
    } else {
       print $associated_files_filehandle, "\n";
       push @items_whose_associated_file_was_not_found, $item;
    }
}

Lets assume associated files for items 1, 2, 3, 6, and 8 were found. After above loop finishes, the output file will contains:

• <path to item1's associated file>
• <path to item2's associated file>
• <path to item3's associated file>
• <path to item6's associated file>
• <path to item8's associated file>

The program now invokes an external process to generate the associated files for items 4, 5, and 7:

my @missing_associated_files = generate_associated_files_of (\@items_whose_associated_file_was_not_found);

Now I have the associated files of all items and want to update the output file to contain:

• <path to item1's associated file>
• <path to item2's associated file>
• <path to item3's associated file>
• <path to item4's associated file>
• <path to item5's associated file>
• <path to item6's associated file>
• <path to item7's associated file>
• <path to item8's associated file>

And my question is what is the best way to do it?

One way is to discard the output file and re-run my program (since I generated the missing associated files at the end of my first run, I know all associated files will be found in the new run), but this is very undesirable due to long runtimes.

I am looking for a way to keep track of the index of the missing files so I can insert them in their correct position after they are generated.

My software skills are limited to what I learned in Programming 101 some 40 years ago so I would really appreciate an answer in the form of Perl code that I can integrate into my program despite my limited S/W literacy.

Thank you in advance.

Answer 1

Here is an example of how you can build the list of files in advance before you start writing to the output file:

use feature qw(say);
use strict;
use warnings;
use experimental qw(signatures);

{
    my @items = map { "item_" . $_ } 1..10;
    my @files; # all files
    my @items_not_found; # items whose associated file was not found
    my @idx_not_found; # indexes of items not found
    for my $i (0..$#items) {
        my $item = $items[$i];
        my $associated_file = search_associated_file( $item );
        if (defined $associated_file) {
            $files[$i] = $associated_file;
        } else {
            push @items_not_found, $item;
            push @idx_not_found, $i;
        }
    }

    # The program now invokes an external process to generate the associated files
    #   for the missing items:
    @files[@idx_not_found] = generate_associated_files_of ( \@items_not_found);

    my $fn = 'associated_files.txt';
    open ( my $fh, '>', $fn ) or die "Could not open file '$fn': $!";
    for my $file (@files) {
        say $fh "$file";
    }
    close $fh;
}

sub generate_associated_files_of($items) {
    my @files;
    for my $item (@$items) {
        my ( $idx ) = $item =~ /item_(\d+)/;
        die "Unexpected index" if !defined $idx;
        push @files, "file_$idx";
    }
    return @files;
}

sub search_associated_file($item) {
    my ( $idx ) = $item =~ /item_(\d+)/;
    my %not_ok_idx = map { $_ => 1} 4,5,7;
    return undef if $not_ok_idx{$idx};
    return "file_" . $idx;
}