5
votes

I wrote an IronPython package to do some data-crunching, and now I am wrapping it in a C# application. Part of the application's functionality is to save the state of a project, and then later to restore that saved state.

I am using the pickle module in IronPython to save an object of a class from my custom package. Before I did the wrapping in C#, that was no problem: I used the pickle.dump() function to serialize the object to a file. Now I want to use the pickle.dumps() function to serialize the object to a string, then pass that string to a C# object and serialize THAT object with an XmlSerializer.

Serialization seems to work, but deserialization breaks down: C# gets the deserialized string and passes it to IronPython, which should be able to reconstitute the original object with the pickle.loads() function, but instead raises this error:

System.Collections.Generic.KeyNotFoundException {"ô"}

Can you help me solve this problem? I have two theories:

  1. Perhaps there is a difference in the string encodings between IronPython and C#, or between C#, IronPython, and what is expected by the pickle module?

  2. Not the entire string is being serialized in the first place, so I am just passing nonsense to pickle.loads()

My evidence that leads me to these theories:

  1. The missing key in the error message (ô) looks like a bit of unicode-parsed-as-ASCII-text.

  2. If I break execution (in debug mode of Visual Studio 2010) and look at the string before it is passed to IronPython for unpicking, what I see is not long enough to represent the entire object. But it might just reach the limit of what the Visual Studio debugger will display.

Thanks in advance!

1
Could post some short code that exhibits this problem? - svick
I'm afraid that the amount of code necessary to produce my setup goes beyond what would be reasonable here. - Wesley
We don't need exactly your setup. But some (relatively) simple code that shows the same problem could help. - svick

1 Answers

0
votes

Short answer: pickling with protocol=-1 works for me where the default, human-readable protocol does not.

Long answer: It looks like theory #1 is substantially correct. I use a dictionary of large numpy arrays to store my data, and even using the default "human-readable" pickling protocol, they are pickled to a binary-like form. Notice the fourth line from the bottom (this is pulled from a pickled file that can be successfully unpickled, but it gives an idea of what I see in the files that can't):

(g10
(I0
tp25
g12
tp26
Rp27
(I1
(I2
tp28
g19
I00
Vq=\u000a×£°(@R¸\u2026ëQ.@
p29
tp30
bssb.

There is the potential for this to cause an error somewhere in the chain of

  1. pickling to a string in IronPython
  2. passing that string to a C# object
  3. serializing that object with an XmlSerializer
  4. unserializing the xml to get back a C# object
  5. passing the string that represents the pickled object back to IronPython
  6. unpicking the string in IronPython

Exactly where the error occurs, I don't yet know. I have worked around the problem by pickling the IronPython object with protocol=-1, which turns the object into a string of binary gibberish that can survive the process.