streaming-pickle

What is it? streaming-pickle allows you to save/load a sequence of Python data structures to/from disk in a streaming (incremental) manner, thus using far less memory than regular pickle. When is it useful? streaming-pickle is useful for any ad-hoc data processing task involving a linear sequence of records. For example, let"s say you write a script A to perform some analysis and then dump 1 million records to disk in some textual format, with each record taking up one line. Then you write another script B that reads in those records one line at a time and performs some more analysis. This strategy is memory-efficient (you only need to store one record at a time in RAM) and provides the benefits of incremental stream processing. However, you need to write the parsing and unparsing code to convert between your plaintext format and Python data structures, which