Spark context broadcast variable throwing java.io.NotSerializableException even though its Serializable

Question

I am running Spark 2.1 on Ubuntu 14.04 I am trying to broadcast a lookup variable in spark. The variable is of type scala.collection.immutable.Map[String, MyObject]

MyObject has following fields

'name' of type String
'address' of type String
'rangeSet' of type com.google.common.collect.{TreeRangeSet}

Exception in thread "main" java.io.NotSerializableException: com.google.common.collect.TreeRangeSet

Serialization stack:
    - object not serializable (class: com.google.common.collect.TreeRangeSet, value: {[/101.32.168.0‥/101.32.181.255][/4626:7800:4048:0:0:0:0:0‥/4626:7800:4048:ffff:ffff:ffff:ffff:ffff]})
    - field (class: com.test.MyObject, name: rangeSet, type: class com.google.common.collect.TreeRangeSet)
    - object (class com.test.MyObject, MyObject(Jack,Test,{[/101.32.168.0‥/101.32.181.255][/192.16.10.224‥/192.16.10.255][/4626:7800:4048:0:0:0:0:0‥/4626:7800:4048:ffff:ffff:ffff:ffff:ffff]}))
    - writeObject data (class: scala.collection.immutable.HashMap$SerializationProxy)
    - object (class scala.collection.immutable.HashMap$SerializationProxy, scala.collection.immutable.HashMap$SerializationProxy@708f7386)
    - writeReplace data (class: scala.collection.immutable.HashMap$SerializationProxy)

MyObject.scala

import com.google.common.collect.{TreeRangeSet}
@SerialVersionUID(123L)
case class MyObject(name:String, address:String,rangeSet:TreeRangeSet[CustomInetAddress]) {
}

CustomInetAddress.java

public class CustomInetAddress implements Comparable<CustomInetAddress>, Serializable {

    private InetAddress inetAddress;

    public CustomInetAddress(String ip) throws UnknownHostException {
        this.inetAddress = InetAddress.getByName(ip);
    }

    public CustomInetAddress(InetAddress address) throws UnknownHostException {
        this.inetAddress = address;
    }

    @Override
    public int compareTo(final CustomInetAddress address){
        byte[] ba1 = this.inetAddress.getAddress();
        byte[] ba2 = address.inetAddress.getAddress();

        if(ba1.length < ba2.length) return -1;
        if(ba1.length > ba2.length) return 1;

        for(int i = 0; i < ba1.length; i++) {
            int b1 = unsignedByteToInt(ba1[i]);
            int b2 = unsignedByteToInt(ba2[i]);
            if(b1 == b2)
                continue;
            if(b1 < b2)
                return -1;
            else
                return 1;
        }
        return 0;
    }

    @Override
    public String toString(){
        return this.inetAddress.toString();
    }

    private int unsignedByteToInt(byte b) {
        return (int) b & 0xFF;
    }
}

TreeRangeSet[CustomInetAddress] is the actual type of the object. CustomInetAddress has one field of type InetAddress. All of them are serializable. I am not sure why this is throwing exception.

zsxwing zsxwing · Accepted Answer · 2017-08-02T00:41:08

The message is clear: com.google.common.collect.TreeRangeSet doesn't implement Serializable.

Spark context broadcast variable throwing java.io.NotSerializableException even though its Serializable

1 Answers