Why the ExpressionLanguageScope in DBCPConnectionPool service is limited to only 'VARIABLE_REGISTRY' and not ' FLOWFILE_ATTRIBUTES'?

2
votes

The DBCPConnectionPool Service requires 5 connection parameters to establish connection to a database as shown in the picture below [Marked Yellow] enter image description here

I used UpdateAttribute Processor to manually add these 5 connection parameters and gave them their respective values as shown in the picture below [Marked Yellow] enter image description here

Now, when I was trying to read the values for the connection parameters in DBCPConnectionPool Service through these attributes (Shown in picture below) , I was unable to read them. enter image description here

To know the reason why the DBCPConnectionPool Service was unable to read the Flowfile attributes, I went ahead to check the source code for both DBCPConnectionPool Service and UpdateAttribute Processor.

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/main/java/org/apache/nifi/dbcp/DBCPConnectionPool.java

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-update-attribute-bundle/nifi-update-attribute-processor/src/main/java/org/apache/nifi/processors/attributes/UpdateAttribute.java

Souce code for DBCPConnectionPool Service : enter image description here

Souce code for UpdateAttribute Processor : enter image description here

Thus, I came to know the reason why it was unable to read the values from FlowFile attributes. This is because the ExpressionLanguageScope is limited to VARIABLE_REGISTRY and not FLOWFILE_ATTRIBUTES.

Now, My Question is that why the ExpressionLanguageScope for DBCPConnectionPool Service is limited to VARIABLE_REGISTRY. What is the reason for this limitation? The reason why I am asking this question is because I want to read the values for the connection parameters through FlowFile attributes.

1
apacheminifyapache-nifihortonworks-data-platformhortonworks-dataflow
because that's an idea of the connection pool. it creates a set of connections to the destination datasource in advance, before any flowfile is coming. so it has no access to flowfile attributes during connection. - daggett

1 Answers

5
votes

For the same question that has been asked in the NiFi dev mailing list, Andy had answered it in the best way possible. The reason why DBCPConnectionPool services or any controller services for that matter, uses ExpressionLanguageScope.VARIABLE_REGISTRY is that, the controller services have no access to the flowfiles so it won't read the flowfiles' attributes. And for the question, why it only supports VARIABLE_REGISTRY is:

  • Just because it doesn't read flowfile attributes doesn't mean that it should not use attributes from elsewhere.
  • One of the major reasons why VARIABLE_REGISTRY was introduced was to avoid exposing the sensitive values which is the case when we pass around such values as flowfile attributes. The controller services fit this case, because many of them use sensitive properties like Password.

And if you're assuming that you can make it work just changing the the scope for those properties to ExpressionLanguageScope.FLOWFILE_ATTRIBUTES, you're wrong. Changing them makes no sense and doesn't work, the reason is again the controller services never get to access the flowfiles.

If there is a specific requirement for you wherein you need to use different property values for different flowfiles, Andy in the original dev thread had shared some links which I'm posting again: