FileHelpers WriteStream writes no data

Question

I need to read a fixed width file, map the records to another type and then write that data to a csv stream. I'm trying to use FileHelpers to do so but the WriteStream doesn't seem to write any data to the stream.

FixedWidthType.cs (the type to read)

[FixedLengthRecord, IgnoreFirst(2), IgnoreLast(1)]
public class FixedWidthType
{
    [FieldFixedLength(6), FieldTrim(TrimMode.Right)]
    public string BranchCode { get; set; }
    // ...
}

CsvType.cs (the type to write)

[DelimitedRecord("|")]
public class CsvType
{
    public string BranchCode { get; set; }
    // ...
}

Program.cs

var readerEngine = new FileHelperEngine<FixedWidthType>();
var writerEngine = new FileHelperEngine<CsvType>();

var file = File.ReadAllBytes("fixedWidthFile.txt");

using (var sr = new MemoryStream(file))
using (var reader = new StreamReader(sr))
using (var sw = new MemoryStream())
using (var writer = new StreamWriter(sw))
{
    // read fixedwidth data
    var fixedWidthRecords = readerEngine.ReadStream(reader);

    var csvRecords = fixedWidthRecords.Select(r => new CsvType
    {
        BranchCode = r.BranchCode,
        // ...
    }).ToList();

    Console.WriteLine(csvRecords.Count()); // output: 2

    // write csv data
    writerEngine.WriteStream(writer, csvRecords);

    Console.WriteLine(sw.ToArray().Count()); // output: 0

    File.WriteAllBytes("newfile.csv", sw.ToArray());
}

The above code writes a blank file. ps: I know I can use writerEngine.WriteFile to accomplish the above(which does work btw), but I specifically need a stream in my case. The purpose of writing the file above is only to view the output.

Answer 1

While closing a TextWriter and/or its underlying Stream (either an via an explicit .Close() or .Dispose() call - or implicitly via a using(){} block or using; statement will also flush it, if you intend to read back from a Stream after writing to it without closing it, then you'll need to flush the TextWriter because it has its own buffer, and you may also need to rewind the stream by setting .Position too - and take advantage of leaveOpen: true too.

var readerEngine = new FileHelperEngine<FixedWidthType>();
var writerEngine = new FileHelperEngine<CsvType>();

Byte[] inputBytes = File.ReadAllBytes("fixedWidthFile.txt");

UTF8Encoding utf8 = new UTF8Encoding( encoderShouldEmitUTF8Identifier: false, throwOnInvalidBytes: true ); // Using the default UTF8Encoding can lead to unexpected proplems: see https://stackoverflow.com/questions/5266069/streamwriter-and-utf-8-byte-order-marks

using (MemoryStream inputStream = new MemoryStream( inputBytes ))
using (StreamReader reader = new StreamReader( inputStream ) )
using (MemoryStream outputStream = new MemoryStream())
using (StreamWriter writer = new StreamWriter( outputStream, utf8, bufferSize: 4096, leaveOpen: true ) )
{
    var fixedWidthRecords = readerEngine.ReadStream(reader);

    var csvRecords = fixedWidthRecords
        .Select(r => new CsvType
        {
            BranchCode = r.BranchCode,
            // ...
        })
        .ToList();

    Console.WriteLine( "csvRecords.Count: {0}", csvRecords.Count ); // output: 2

    // write csv data
    writerEngine.WriteStream( writer, csvRecords );

    writer.Flush();
    outputStream.Flush(); // <-- This isn't necessary for a MemoryStream, but will be for a FileStream.

#if OPTION_1

    // Avoid `ToArray()` as that will do an allocation+copy.
    Console.WriteLine( outputStream.Position );
    Console.WriteLine( outputStream.GetBuffer().Length );

    ReadOnlySpan<Byte> msContents = outputStream.GetBuffer().AsSpan( 0, outputStream.Position );

    File.WriteAllBytes( "newfile.csv", msContents );

#elif OPTION_2

    // Close `writer` here (which will also flush it), because we set `leaveOpen: true` the MemoryStream can be reused:
    writer.Dispose();
    // Rewind the MemoryStream:
    outputStream.Position = 0;
    
    using( FileStream fileOutputStream = File.OpenWrite( "newfile.csv" ) )
    {
        await outputStream.CopyToAsync( fileOutputStream );
    }

#endif

}

---

But if you look at your program's application logic, you don't actually need to Flush anything, just change your code to this:

var readerEngine = new FileHelperEngine<FixedWidthType>();
var writerEngine = new FileHelperEngine<CsvType>();

UTF8Encoding utf8 = new UTF8Encoding( encoderShouldEmitUTF8Identifier: false, throwOnInvalidBytes: true ); // Using the default UTF8Encoding can lead to unexpected proplems: see https://stackoverflow.com/questions/5266069/streamwriter-and-utf-8-byte-order-marks

using( FileStream   fixedWidthInputFS     = new FileStream( "fixedWidthFile.txt", FileAccess.Read, FileShare.None, FileMode.Open ) )
using( StreamReader fixedWidthInputReader = new StreamReader( fixedWidthInputFS ) )
using( FileStream   outputFS              = new FileStream( "output.dat", FileAccess.Write, FileShare.None, FileMode.CreateNew ) )
using( StreamWriter outputWriter          = new StreamWriter( outputFS, utf8 ) ) // <-- Watch out for the default encoding possibly being UTF8 with BOM, 
{
    var csvRecords = readerEngine.ReadStream( fixedWidthInputReader )
        .Select(r => new CsvType
        {
            BranchCode = r.BranchCode,
            // ...
        });

    writerEngine.WriteStream( outputWriter, csvRecords );
}

• Protip: When a Stream or TextWriter/TextReader is being used only for input and/or output, you should put the word "input" or "output" (or "in" / "out") in their names to make things easier to follow.