ZAP active scan works in desktop but fails in docker image with url_not_in_context error

Question

I am able to scan my API using ZAP desktop but fails with 'url_not_in_context' error in active scan from zap docker image. Context definition is exported from desktop and specified as argument to zap-api-scan.py.

I am using zap2docker-stable image to scan APIs. Custom scripts are loaded for authentication httpsender.

Error: 51660 [ZAP-ProxyThread-15] WARN org.zaproxy.zap.extension.api.API - Bad request to API endpoint [/JSON/ascan/action/scanAsUser/] from [127.0.0.1]: org.zaproxy.zap.extension.api.ApiException: url_not_in_context at org.zaproxy.zap.extension.ascan.ActiveScanAPI.scanURL(ActiveScanAPI.java:879) ~[zap-2.10.0.jar:2.10.0] at org.zaproxy.zap.extension.ascan.ActiveScanAPI.handleApiAction(ActiveScanAPI.java:370) ~[zap-2.10.0.jar:2.10.0] at org.zaproxy.zap.extension.api.API.handleApiRequest(API.java:507) [zap-2.10.0.jar:2.10.0]

I have already implemented suggestion mentioned at ZAP SCAN: Jenkins Job failed (url_not_in_context)

Docker command:

docker run -v D:/dev/cloud/zap/scripts:/zap/wrk/:rw -t owasp/zap2docker-stable zap-api-scan.py -d -t customer-api-docs.json -f openapi -r /zap/wrk/testreport.html -n customer-service.context.xml -U [email protected] --hook=load-script.py -z "-addoninstall jython"

incregexes configuration in context file:

<incregexes>http://dev.xyz.com/customer.*</incregexes>
<excregexes>http://dev.xyz.com/customer/v3*</excregexes>

Same configuration work in ZAP desktop.

Following logs are printed at start of execution but then it continues with starting up zap, loading plugins and eventually fails. Is it expected or does it point to some issue ?

Jun 11, 2021 6:58:40 AM java.util.prefs.FileSystemPreferences$1 run                                              
INFO: Created user preferences directory.                                                                        
zap_started(<zapv2.ZAPv2 object at 0x7f3750bf13d0>, customer-api-docs.json)                                      
load authentication script                                                                                       
load http sender script                                                                                          
2021-06-11 06:59:20,857 Number of Imported URLs: 9                                                               
Traceback (most recent call last):                                                                               
  File "/zap/zap-api-scan.py", line 484, in main                                                                 
    zap_active_scan(zap, target, scan_policy)                                                                    
  File "/zap/zap_common.py", line 104, in _wrap                                                                  
    return_data = func(*args_list, **kwargs)                                                                     
  File "/zap/zap_common.py", line 450, in zap_active_scan                                                        
    raise_scan_not_started()                                                                                     
  File "/zap/zap_common.py", line 399, in raise_scan_not_started                                                 
    raise ScanNotStartedException('Failed to start the scan, check the log/output for more details.')            
zap_common.ScanNotStartedException: Failed to start the scan, check the log/output for more details.             
Found Java version 11.0.9.1                                                                                      
Available memory: 3917 MB                                                                                        
Using JVM args: -Xmx979m                                                                                         
2381 [main] INFO  org.parosproxy.paros.Constant - Copying default configuration to /home/zap/.ZAP/config.xml

Do I have to set apiKey during scan ? How do I determine apiKey of docker instance ?

2021-06-11 10:33:20,894 http://localhost:46219 "GET http://zap/JSON/ascan/action/scanAsUser/?apikey=&url=http%3A%2F%2Fdev.xyz.com&contextId=1&userId=10&recurse=True&scanPolicyName=API-Minimal HTTP/1.1" 400 89

Context file:

<configuration>
    <context>
        <name>customer-service</name>
        <desc/>
        <inscope>true</inscope>
        <incregexes>http://dev.xyz.com/customer.*</incregexes>
        <excregexes>http://dev.xyz.com/customer/v3*</excregexes>
        <tech>
            <include>Db.IBM DB2</include>
            <include>Language.JSP/Servlet</include>
            <include>Language.Java</include>
            <include>Language.JavaScript</include>
            <include>OS.Linux</include>
            <include>WS.Tomcat</include>
            <exclude>Db</exclude>
            <exclude>Db.CouchDB</exclude>
            <exclude>Db.Firebird</exclude>
            <exclude>Db.HypersonicSQL</exclude>
            <exclude>Db.Microsoft Access</exclude>
            <exclude>Db.Microsoft SQL Server</exclude>
            <exclude>Db.MongoDB</exclude>
            <exclude>Db.MySQL</exclude>
            <exclude>Db.Oracle</exclude>
            <exclude>Db.PostgreSQL</exclude>
            <exclude>Db.SAP MaxDB</exclude>
            <exclude>Db.SQLite</exclude>
            <exclude>Db.Sybase</exclude>
            <exclude>Language</exclude>
            <exclude>Language.ASP</exclude>
            <exclude>Language.C</exclude>
            <exclude>Language.PHP</exclude>
            <exclude>Language.Python</exclude>
            <exclude>Language.Ruby</exclude>
            <exclude>Language.XML</exclude>
            <exclude>OS</exclude>
            <exclude>OS.MacOS</exclude>
            <exclude>OS.Windows</exclude>
            <exclude>SCM</exclude>
            <exclude>SCM.Git</exclude>
            <exclude>SCM.SVN</exclude>
            <exclude>WS</exclude>
            <exclude>WS.Apache</exclude>
            <exclude>WS.IIS</exclude>
        </tech>
        <urlparser>
            <class>org.zaproxy.zap.model.StandardParameterParser</class>
            <config>{"kvps":"&amp;","kvs":"=","struct":[]}</config>
        </urlparser>
        <postparser>
            <class>org.zaproxy.zap.model.StandardParameterParser</class>
            <config>{"kvps":"&amp;","kvs":"=","struct":[]}</config>
        </postparser>
        <authentication>
            <type>4</type>
            <strategy>EACH_RESP</strategy>
            <pollurl/>
            <polldata/>
            <pollheaders/>
            <pollfreq>60</pollfreq>
            <pollunits>REQUESTS</pollunits>
            <loggedin>HTTP\/1.1\s(200|404|400|500|403)</loggedin>
            <loggedout>HTTP\/1.1\s401</loggedout>
            <script>
                <name>oidc_ropc_script</name>
                <params>Y2xpZW50SWQ=:cnhub3Zh</params>
            </script>
        </authentication>
        <users>
            <user>10;true;Ym9iQHNzYy5jb20=;4;cGFzc3dvcmQ=:d2VsY29tZTE=&amp;dXNlcm5hbWU=:Ym9iQHNzYy5jb20=</user>
        </users>
        <forceduser>10</forceduser>
        <session>
            <type>1</type>
        </session>
        <authorization>
            <type>0</type>
            <basic>
                <header/>
                <body/>
                <logic>AND</logic>
                <code>-1</code>
            </basic>
        </authorization>
    </context>
</configuration>

What am I missing ?

Difficult to tell without knowing what you are scanning and what context you are using. Can you give more details, obfiscated if necessary? My working assumption is that the URL you are scanning is not in the context you have specified, as per the error message :) — Simon Bennetts
@SimonBennetts Added required details. ZAP version in desktop is 2.10.0 — Pradeep
What URL are you specifying as the starting point for the scan? — kingthorin
Do I have to ? It is an API scan. I have provided OpenAPI json file which has base url as "dev.xyz.com/customer" — Pradeep
That looks right :/ Are you sure theres not a typo in the json, or that its specifying https ? — Simon Bennetts

Pradeep Pradeep · Accepted Answer · 2021-06-11T13:03:29

Finally, identified the issue !!

zap-api-scan normalizes the target URL to remove context. So http://dev.xyz.com/customer was changed to http://dev.xyz.com.

Since this was not in part of allowed URLs it always failed. Changing incregex to remove context path /customer resolved the issue.

This is different behavior than desktop mode. Probably a bug.

ZAP active scan works in desktop but fails in docker image with url_not_in_context error

1 Answers