How to pickle in Client and Unpickle in Server?

Question

I have a client scala code like the below

import java.net._
import java.io._
import scala.io._
import scala.pickling._        
import scala.pickling.json._  
val sk = new Socket(InetAddress.getByName("localhost"), 13373)
val output = new PrintStream(sk.getOutputStream())
val textRDD = sc.textFile("some file");
output.println( #pickle the textRDD and pass it to server)
output.flush()
sk.close()

and python server.py like below,

import SocketServer
import json
import pickle
class MyTCPServer(SocketServer.ThreadingTCPServer):
allow_reuse_address = True
class MyTCPServerHandler(SocketServer.BaseRequestHandler):
def handle(self):
    try:
        data = self.request.recv(1024)       
      #unpickle the data received here and print it.
        print data
    except Exception, e:
        print "Exception wile receiving message: ", e
server = MyTCPServer(('127.0.0.1', 13373), MyTCPServerHandler)
server.serve_forever()

How to pickle the TextRDD file in the scala client and pass it to python server to unpickle it and print the received data?

Answer 1

I don't think you will be able to do this without rewriting scala-pickling in Python first. scala-pickling is Scala-specific serialization library, it is not even intended to be able to serialize/deserialize arbitrary formats; it is designed to be used as a substitute for Java serialization - fast and precise serialization for internal purposes.

If you need to send data across different languages, you should consider using portable protocols and serialization formats, for example, Protobuf, Cap'n'Proto, Thrift or MessagePack. For most of them there are multiple libraries in different languages, including Java/Scala and Python.