Bulk-uploading fails from csv source file for fields with data length greater then 500 unicode characters

0
votes

I am using appcfg.py to bulkupload data into appengine from csv source file. Length of data value for one field in csv file is greater then 500 Unicode characters.

Execution of appcfg upload-command generates below logs :-

[INFO ] Opening database: bulkloader-progress-20121101.024343.sql3

[INFO ] Connecting to xxxxxxx.appspot.com/remote_api

[INFO ] Starting import; maximum 10 entities per post

.

[INFO ] 1 entities total, 0 previously transferred

[INFO ] 1 entities (3590 bytes) transferred in 15.5 seconds

[INFO ] All entities successfully transferred

After execution, Datastore Viewer on appengine dashboard does not show value for fields having values of length greater then 500 Unicode characters.

I googled a lot for bulk-uploading data with values of length greater then 500 Unicode characters from csv source file, but no success.

EDIT 1 :-

Found Text string (long) com.google.appengine.api.datastore.Text None 

Up to 1 megabyte Not indexed

on https://developers.google.com/appengine/docs/java/datastore/entities

But, not sure, how to use this for bulk-uploading data from csv source file.

2
google-app-enginegoogle-cloud-datastore

2 Answers

0
votes

Use the export_transform tag at the property in want to convert to Text in your bulkloader.yaml file.

From Uploading Data docs:

import_transform

A single-argument function that returns the correct value and type data based on the external_name or import_template strings. Examples include the built-in Python conversion operators (such as float), any of several helper functions provided in transform, such as get_date_time or generate_foreign_key, a function provided in your own library, or an in-line lambda function...

...

export_transform

Like import_transform, except performed on export.

With that you can setup a function that converts your data to Text before uploading, solving your problem.

0
votes

500 characters is the limit of string fields. Unless they are text or blob fields, anything over 500 will be truncated.