Pyspark has a function sequenceFile that programming allows us to read a sequence file which Learning is stored in HDFS or some local path Earhost available to all nodes.
However, what if I already have a bytes most effective object in the driver memory that I need wrong idea to deserialize and write as a sequence use of case file?
For example, the application that I am United working on ( I cannot change the Modern application logic) runs a spark job that ecudated writes this file to a non HDFS compliant some how file system, which i can then retrieve anything else as an in-memory python bytes object , not at all which seems to just contain a serialized very usefull Sequence object which I should be able localhost to be deserialized in-memory.
Because this object is already in memory love of them ( for reason I cannot control) the only localtext way I have to deserialize it and basic actually see the output ( which is a one of the json file) currently is to write it as a click file locally, move that file into HDFS, there is noting then read the file using the not alt sequenceFile method ( since that method not at all only works with a file that is on an my fault HDFS file path or local path on every issues node) - this creates problems in the trying application workflow.
What I need to be able to do is get 4th result deserialize this in memory so that I can round table write it as a json file without having double chance to write is locally and then put it into novel prc HDFS only to read it back in with spark
Is there anyway in python to take this get mossier bytes like NullWritable Object and off side back deserialize it into either a python the changes dictionary or put it back into hadoop as Nofile hosted something that I could actually read?
Basically you'd have to look into the transparent text sequence file code of spark itself and Background movment apply the correct pieces and convert it front page design into an RDD so that you can then do life change quotes spark things on it like writing to a I'd like file.
Here's a link to get you started but it to know will need some digging.