padding binary-block lazy sequences

Question

I have Clojure function that takes a sequence of numbers chops it into the appropriate number of bits and returns a lazy sequence of the chunks (lowest order bits first). It pads the high order bits of the last block to fill out the block size and I need advice on the "best way(tm)" to record the amount of padding while keeping it lazy and functional?

Words of wisdom much appreciated.

(defn block-seq
  ([block-size bytes]
    "reads a byte-seq into a sequence of block-size bits."
    (block-seq 8 block-size bytes))
  ([in-block-size out-block-size bytes]
    "converts a seq from in-block-size to out-block-size"
  ...

Parameters:

in-block-size is the number of significant bits in each number in the input sequence
out-block-size is the number of significant bits in each of the numbers in the lazy seq that is returned.
bytes is a lazy sequence of numbers from which to extract bits

Here is an example that takes a sequence of three bytes and breaks it up into a sequence of two twenty bit numbers (and then prints it as a binary string).

user> (map #(java.lang.Integer/toBinaryString %) (block-seq 20 [0xAA 0xAA 0xAA]))
("10101010101010101010" "1010")
user>

The second number in the sequence of 20 bit numbers has only four significant bits and has an effective 16 zeros added. If i then passed this sequence to another function that wanted to do something with the sequence and send it over a network; the code on the receiving end needs to know not to print/store/etc the last 16 bits.

PS: these can be chained. (block-seq 20 15 (block-seq 8 20 (read-bytes-from-file)))

It's hard to tell what you're doing without seeing more code, but could you use metadata to record the padding? — Brian Carper
What needs to read the information about the number of bits of padding, and what's a convenient way for it to read that? — cjs
Just curious, what is the advantage of converting these numbers to "binary strings" for a wire protocol when you could seemingly just encode the numbers using a network byte order protocol? On the surface, converting to/from strings seems wasteful, but would like a better feel for your use case before digging too deep. — Joe Holloway
its a variable size word. so it will be used to read ints of 32 bits, than process them in chunks of 67 bits for instance. — Arthur Ulfeldt

Mark Probst Mark Probst · Accepted Answer · 2009-09-23T12:08:11

It's still not quite clear what you want to do, but it seems that you want to know the best way for block-seq to return the number of padded bits in the last chunk. Of course that's not possible up front if you want to be properly lazy, so the number would have to be returned with or after the last chunk.

Without using metadata you could just return a list like

(1 2 3 :pad 12)

Using metadata, you could add that padding information to the last cons (Clojure cannot add metadata to Integers), so the last cons would be equivalent to

(with-meta '(3) {:pad 12})

For chaining to work, binary-block would have to be aware of that padding information, in both cases, so as to be able to un-pad and then re-pad the last chunk.

How to transmit the padding information over the wire is another question, though.

padding binary-block lazy sequences

1 Answers