F# seq behavior

Question

I'm a little baffled about the inner work of the sequence expression in F#.

Normally if we make a sequential file reader with seq with no intentional caching of data

 seq { 
       let mutable current = file.Read()
       while current <> -1 do
           yield current
     }

We will end up with some weird behavior if we try to do some re-iterate or backtracking, My Idea of this was, since Read() is a function calling some mutable value we can't expect the output to be correct if we re-iterate. But then this behaves nicely even on boundary reading?

let Read path =
    seq {
        use fp = System.IO.File.OpenRead path
        let buf = [| for _ in 0 .. 1024 -> 0uy |]
        let mutable pos = 1
        let mutable current = 0
        while pos <> 0 do
            if current = 0 then
                pos <- fp.Read(buf, 0, 1024)
            if pos > 0 && current < pos then
                yield buf.[current]
                current <- (current + 1) % 1024 
   } 

 let content = Read "some path"

We clearly use the same buffer to enhance performance, but assuming that we read the 1025 byte, it will trigger an update to the buffer, if we then try to read any byte with position < 1025 after we still get the correct output. How can that be and what are the difference?

FileStream is already buffered. From the docs: FileStream buffers input and output for better performance. If you have a stream that isn't buffered, you can wrap it in BufferedStream and it will be. But can you explain what you are trying to do here? We will end up with some weird behavior if we try to do some re-iterate or backtracking is confusing since sequences can't be rewound or accessed randomly. — dbc
@Sebastian, example let s = Read "some path"; let s_1024 = Seq.skip 1024 s; let s_1025 = Seq.tail s_1024 This above holds to different instance of the same sequence and they will belong to different buffers too. As mentioned below, the reason why I see some "weird" stuff when getting the same value, was because of the scope of the filestream. I'm implementing a 'persistent' file reader that can be passed as a sequence for ease. The use of it will read to some point, and then backtrack as seen above, but with a little bigger gab. — kam
with persistent meaning, that assuming that no other thread are writing to the file at the same time, we will get the same output at the same location of the sequence — kam

Arshia001 Arshia001 · Accepted Answer · 2020-12-05T13:58:24

Your question is a bit unclear, so I'll try to guess.

When you create a seq { }, you're essentially creating a state machine which will run only as far as it needs to. When you request the very first element from it, it'll start at the top and run until your first yield instruction. Then, when you request another value, it'll run from that point until the next yield, and so on.

Keep in mind that a seq { } produces an IEnumerable<'T>, which is like a "plan of execution". Each time you start to iterate the sequence (for example by calling Seq.head), a call to GetEnumerator is made behind the scenes, which causes a new IEnumerator<'T> to be created. It is the IEnumerator which does the actual providing of values. You can think of it in more classical terms as having an array over which you can iterate (an iterable or enumerable) and many pointers over that array, each of which are at different points in the array (many iterators or enumerators).

In your first code, file is most likely external to the seq block. This means that the file you are reading from is baked into the plan of execution; no matter how many times you start to iterate the sequence, you'll always be reading from the same file. This is obviously going to cause unpredictable behaviour.

However, in your second code, the file is opened as part of the seq block's definition. This means that you'll get a new file handle each time you iterate the sequence or, essentially, a new file handle per enumerator. The reason this code works is that you can't reverse an enumerator or iterate over it multiple times, not with a single thread at least.

(Now, if you were to manually get an enumerator and advance it over multiple threads, you'd probably run into problems very quickly. But that is a different topic.)

F# seq behavior

1 Answers