Type mismatch with projection on a []byte property

Question

I have a struct as follows

  type MyEntity struct {
     PF []byte `json:"-" datastore:"_pf"`
  }

Querying without a Projection works fine. However, when I query with projection on "_pf" field, I get "type mismatch: string versus []uint8" error. I implemented the PropertyLoadSaver and examined the prop.Value for "_pf" property and found that some rows return []byte type and some return string. So, why is projected query failing with this error while non-projected queries are fine? At the moment I am resolving this by implementing PropertyLoadSaver interface and explicitly checking types and converting string type to []byte type to solve this problem.

Here is the complete test case. This is reproduced on cloud datastore emulator. Use appropriate value for datastoreProject variable below. Rest all should directly work. You can see the behavior by inserting both entities or one of the types of the entities. The error that is displayed is

panic: datastore: cannot load field "_pf" into a "tests.MyEntity": type mismatch: string versus []uint8 [recovered]
    panic: datastore: cannot load field "_pf" into a "tests.MyEntity": type mismatch: string versus []uint8

Following is the code.

type MyEntity struct {
    PF []byte `json:"-" datastore:"_pf"`
}

func TestPackedField(t *testing.T) {
    e1 := &MyEntity{PF: []byte{83, 0, 0, 0, 93, 150, 154, 206, 0, 0, 0, 0, 0, 0, 0, 10, 0, 0, 0, 0, 0, 0, 0, 3}} // returns []byte on projection
    e2 := &MyEntity{PF: []byte{83, 0, 0, 0, 93, 120, 79, 87, 0, 0, 0, 0, 0, 0, 0, 10, 0, 0, 0, 0, 0, 0, 0, 3}}   // returns string on projection

    ctx := context.Background()

    conn, err := datastore.NewClient(ctx, datastoreProject)
    if err != nil {
        panic(err)
    }

    bkey := datastore.NameKey("Bytes", "bytearray", nil)
    if true {
        conn.Put(ctx, bkey, e1)
    }
    skey := datastore.NameKey("Bytes", "string", nil)
    if true {
        conn.Put(ctx, skey, e2)
    }

    q1 := datastore.NewQuery("Bytes").Order("-_pf").Limit(2)
    var elfull []*MyEntity
    if _, err := conn.GetAll(ctx, q1, &elfull); err != nil {
        panic(err)
    }

    q2 := datastore.NewQuery("Bytes").Project("_pf").Order("-_pf").Limit(2)
    var elprojected []*MyEntity
    if _, err := conn.GetAll(ctx, q2, &elprojected); err != nil {
        conn.Delete(ctx, bkey)
        conn.Delete(ctx, skey)
        panic(err)
    }
}

Could you, please provide some code where are you doing the projection queries? Could you, also, give some data example? What is returned as []byte, what is returned as string? — Andrei Tigau
Thank you Andrei. Updated the post with the test case and sample data. — Siva

Andrei Tigau Andrei Tigau · Accepted Answer · 2019-10-10T07:21:37

In order to understand why this works fine on normal queries and not on projection queries you might want firstly to read about what is the actual difference between those two. As it is mentioned in this post:

Whereas "regular" (which I'm taking to mean SELECT * ...) queries against Cloud Datastore typically use indexes that only contain a sorted subset of the properties of the queried entities, plus pointers to the full entities, projection queries run against indexes that contain all the fields requested by the query. So it appears the significant latency gain comes from the elimination of the need to fetch the queried entities once the set of entities matching the query has been discerned via the index.

So, basically, when you do your projection query, your queried entities are not fetched.

As I was reading through this official documentation, I found some really interesting phrases related to your question :

A field of slice type corresponds to a Datastore array property, except for []byte, which corresponds to a Datastore blob. If a non-array value is loaded into a slice field, the result will be a slice with one element, containing the value.

Key Field If the struct contains a *datastore.Key field tagged with the name "key", its value will be ignored on Put. When reading the Entity back into the Go struct, the field will be populated with the *datastore.Key value used to query for the Entity.

What I understood from here is that maybe in your case, the fields are populated with the key value for that query ( string ).

Another interesting thing that I have discoverd is that according to the properties and values type documentation the type []byte is not indexed. And as it is told here, unindexed properties cannot be projected. So projected queries should not work for this particular use case at all.

Type mismatch with projection on a []byte property

1 Answers