issue/bug using SelectTokens with JsonPath

Question

I have an issue using NewtonSoft v8.0.3 JToken.SelectTokens() with a specific JsonPath. To reproduce issue in C#:

        string json = "[{\"name\":\"string\",\"value\":\"aString\"},{\"name\":\"number\",\"value\":123},{\"name\":\"array\",\"value\":[1,2,3,4]},{\"name\":\"object\",\"value\":{\"1\":1}}]";
        string path = "$.[?(@.value!=null)]";
        string newJson = String.Empty;

        JToken jt = JToken.Parse(json);
        IEnumerable<JToken> tokens = jt.SelectTokens(path, false);
        newJson = JsonConvert.SerializeObject(tokens, Formatting.Indented);
        // Output is missing objects and arrays??
        // newJson = [{"name": "string","value": "aString"},{"name": "number","value": 123}]

Running above code will not return any objects or arrays?! Is there a bug in JToken.SelectTokens() of Json.Net?

To validate the path I have used http://goessner.net/articles/JsonPath/ which works fine, see this working fiddle

I hope someone can shine some light on this.

UPDATE: Thought this issue occured as we are trying to use "null" value, but no matter what inequality query I run, no {} / [] is returned from JSON.

Answer 1

I can make SelectTokens() return the results you want by using the existence operator rather than the inequality operator:

string path = "$.[?(@.value)]"; 

Now all 4 objects are returned. Sample fiddle.

As to whether this is a bug, the JSONPath article is ambiguous. It says:

[]   ?()     applies a filter (script) expression.
n/a   ()     script expression, using the underlying script engine. 

But what does using the underlying script engine actually mean? If I test your JSON and query expression at https://jsonpath.curiousconcept.com/, then an error is generated for both Flow Communications JSONPath 0.3.1 and Stefan Goessner JSONPath 0.8.3. If I use my existence query, then the former works correctly but the latter still throws an exception.

Should Json.NET behave as it does currently? The present implementation implicitly provides the ability to filter for properties that have primitive values. The proposed behavior would lose this capability. I.e., any object or array value would always match any inequality operator such as "$.[?(@.value!='aString')]", which might not be expected.

Update

Your clarified requirement is to run a query "$.[?(@.value!=null)]" to return all objects with a property named "value" that exists and has non-null value, regardless of whether that value is a primitive or a nested container.

One the one hand, this seems like a reasonable query that ought to be possible. On the other, the current capability of being able to run inequality queries on primitive values without also getting all container values seems useful and should also be possible. And either interpretation of that particular query string seems plausible; the JSONPath language definition is just too vague and leaves too much up to the interpretation of implementers.

Your options to get what you need include:

1. You could certainly report an issue about this. Newtonsoft could decide that the current behavior is wrong, and change it.

2. You could fork your own version of Json.NET and modify BooleanQueryExpression.IsMatch(JToken t) as follows:

               case QueryOperator.NotEquals:
                   if (v != null && !EqualsWithStringCoercion(v, Value))
                   {
                       return true;
                   }
                   else if (v == null && r != null)
                   {
                       return true;
                   }
                   break;
   

3. You could use the following workaround:

   var path = "$.[?(@.value)]";
   var tokens = jt.SelectTokens(path, false)
       .Where(t => !t["value"].IsNull());
   

   using the extension method:

   public static class JsonExtensions
   {
       public static bool IsNull(this JToken token)
       {
           return token == null || token.Type == JTokenType.Null;
       }
   }
   

Finally, you ask What I don't understand is why it works fine in javascript, but not in selectTokens of the Json.Net.. The reason that Json.NET is behaving differently than, say, http://jsonpath.com/, is that:

1. The standard is completely ambiguous as to what a script expression is, exactly, and

2. They are implemented using different technologies and code bases. Json.NET interprets (@.value!=null) to mean

   a value of type JValue named "value" that is not equal to the null json primitive.

   whereas the other parser interprets the query as

   a value named "value" that is not equal to the null json primitive.

   The fact that c# is strongly typed as compared to, say, javascript, may explain the difference.

Answer 2

There is no standard implementation of Jsonpath, so your results will vary depending upon which parser you are using. However, you should never need to check for nulls, as it doesn't really make sense with Jsonpath.

This is what you need:

var result = jt.SelectTokens("$.[?(@.value)]");

Answer 3

Finally figured out what was causing the .SelectTokens to ignore arrays and objects when using JSONPath. The issues occurs in QueryExpressions.cs line 78

JValue v = r as JValue;

This will cause any JToken that is JObject/JArray to become null and hence being ignored when compared. My way of fixing this was to replace line 78 with the following code that handles JObject and JArray:

            JValue v = null;
            switch (r.Type)
            {
                case JTokenType.Object:
                case JTokenType.Array:
                    v = new JValue(r.ToString());
                    break;
                default:
                    v = r as JValue;
                    break;
            }

This will ensure that JObject and JArray are returned when calling .SelectTokens. Please feel free to comment solution or suggest alternative and better aproaches.

UPDATE: Happy to inform that this issue been fixed with new releases. See GITHUB for more information.