How to remove a entire record conditionally using npm "csv" or "csvtojson" package?

Question

I wanted to store CSV values to an object variable using npm "csv" sync API by parsing it.

// Read the CSV file synchronously
const csvData = fs.readFileSync(csvFilePath, "utf-8");

// Parse the CSV data synchronously
const parsedData = parse(csvData, {
   delimiter: ",",
   skip_empty_lines: true,
   skip_records_with_error: true,
   skip_records_with_empty_values: true,
   cast: function (val, ctx) {
      if (ctx.header) {
         return val;
      }

      if (!val.length) {
         return undefined;
      }

      switch (ctx.index) {
         case 0:
            return new Date(val);
         default:
            return Number(val).toFixed(2);
      }
   },
   columns: true,
   trim: true,
});

Here the code

if (!val.length) {
         return undefined;
      }

Doesn't work the way I want. I'm trying to skip all the record for which any of the fields is empty eg: string,,,,. Using skip_records_with_empty_values don't work for me because it's not that any of that field is missing but the value is missing. I have tried returning null and undefined. nothing works...

So my question right now is how do I simply filter out conditionally without having to use the filter on the results?

Edit: I have tried to do this using both "csv" and "csvtojson" packages and I can't seem to figure out a simple way prebuilt to the package itself.

Answer 1

If you want to filter out records in the CSV file where any of the fields have missing values, you can modify the parsing logic to achieve that. Instead of using the skip_records_with_empty_values option, you can manually filter the parsed data before storing it in your object variable.

Here's an updated code snippet that includes the manual filtering step:

// Read the CSV file synchronously
const csvData = fs.readFileSync(csvFilePath, "utf-8");

// Parse the CSV data synchronously
const parsedData = parse(csvData, {
  delimiter: ",",
  skip_empty_lines: true,
  skip_records_with_error: true,
  cast: function (val, ctx) {
    if (ctx.header) {
      return val;
    }

    if (!val.length) {
      return undefined;
    }

    switch (ctx.index) {
      case 0:
        return new Date(val);
      default:
        return Number(val).toFixed(2);
    }
  },
  columns: true,
  trim: true,
});

// Filter out records with missing values
const filteredData = parsedData.filter(record => {
  return Object.values(record).every(value => value !== undefined);
});

// Store the filtered data in your object variable
const filteredObject = filteredData;

// Use the filteredObject as needed

after parsing the CSV data, the parsedData array is checked if any of the values in each record is undefined. If all the values are defined (i.e., no missing values), the record is included in the filteredData array. Finally, the filteredData is stored in your object variable filteredObject.

This way, you can manually filter out the records without having to rely solely on the skip_records_with_empty_values option.