Firebase Test Lab throws FileNotFound for /jacoco.exec

0
votes

I am trying to set up Code Coverage using Android Espresso tests, and am running the tests on Firebase Test Lab.

The tests are executing on Firebase, so I think everything should "work". But, the coverage.ec file generated in Firebase Test Lab, doesn't contain any coverage info.

No matter what I try, I can't see to get it to actually generate coverage info.

When looking @ the Firebase test lab logs, I see this stacktrace:

10-02 23:55:37.746: W/System.err(7917): java.io.FileNotFoundException: /jacoco.exec (Read-only file system)
10-02 23:55:37.746: W/System.err(7917):     at java.io.FileOutputStream.open0(Native Method)
10-02 23:55:37.746: W/System.err(7917):     at java.io.FileOutputStream.open(FileOutputStream.java:287)
10-02 23:55:37.746: W/System.err(7917):     at java.io.FileOutputStream.<init>(FileOutputStream.java:223)
10-02 23:55:37.746: W/System.err(7917):     at org.jacoco.agent.rt.internal_8ff85ea.output.FileOutput.openFile(FileOutput.java:67)
10-02 23:55:37.746: W/System.err(7917):     at org.jacoco.agent.rt.internal_8ff85ea.output.FileOutput.startup(FileOutput.java:49)
10-02 23:55:37.746: W/System.err(7917):     at org.jacoco.agent.rt.internal_8ff85ea.Agent.startup(Agent.java:122)
10-02 23:55:37.746: W/System.err(7917):     at org.jacoco.agent.rt.internal_8ff85ea.Agent.getInstance(Agent.java:50)
10-02 23:55:37.746: W/System.err(7917):     at org.jacoco.agent.rt.internal_8ff85ea.Offline.<clinit>(Offline.java:31)
10-02 23:55:37.746: W/System.err(7917):     at org.jacoco.agent.rt.internal_8ff85ea.Offline.getProbes(Offline.java:51)
10-02 23:55:37.746: W/System.err(7917):     at com.example.idea.MainActivity.$jacocoInit(Unknown Source:12)
10-02 23:55:37.746: W/System.err(7917):     at com.example.idea.MainActivity.<init>(Unknown Source:0)
10-02 23:55:37.746: W/System.err(7917):     at java.lang.Class.newInstance(Native Method)
10-02 23:55:37.746: W/System.err(7917):     at android.app.Instrumentation.newActivity(Instrumentation.java:1174)
10-02 23:55:37.746: W/System.err(7917):     at android.support.test.runner.MonitoringInstrumentation.newActivity(MonitoringInstrumentation.java:754)
10-02 23:55:37.746: W/System.err(7917):     at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2669)
10-02 23:55:37.746: W/System.err(7917):     at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2856)
10-02 23:55:37.746: W/System.err(7917):     at android.app.ActivityThread.-wrap11(Unknown Source:0)
10-02 23:55:37.747: W/System.err(7917):     at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1589)
10-02 23:55:37.747: W/System.err(7917):     at android.os.Handler.dispatchMessage(Handler.java:106)
10-02 23:55:37.747: W/System.err(7917):     at android.os.Looper.loop(Looper.java:164)
10-02 23:55:37.747: W/System.err(7917):     at android.app.ActivityThread.main(ActivityThread.java:6494)
10-02 23:55:37.747: W/System.err(7917):     at java.lang.reflect.Method.invoke(Native Method)
10-02 23:55:37.747: W/System.err(7917):     at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:438)
10-02 23:55:37.747: W/System.err(7917):     at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:807)

Looking at a couple of other threads with somewhat similar errors, I see some suggestion adding:

<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>

To my AndroidManifest.xml, which I've already done.

As well as:

debug { testCoverageEnabled true }

In my app's build.gradle file.

You can also see the full jacoco configuration here:

apply plugin: 'jacoco'
jacoco {
    toolVersion = "0.8.4"
}
def coverageSourceDirs = [
        'src/main/java',
        'src/debug/java'
]

tasks.withType(Test) {
    jacoco.includeNoLocationClasses = true
    jacoco.destinationFile = file("$buildDir/jacoco/jacocoTest.exec")
}

task jacocoTestReport(type : JacocoReport, dependsOn : 'testDebugUnitTest') {
    group       = 'Reporting'
    description = 'Generate JaCoCo coverage reports'

    reports {
        xml.enabled  = true
        html.enabled = true
    }

    classDirectories = fileTree(
            dir      : 'build/intermediates/classes/debug',
            excludes : [
                    '**/R.class',
                    '**/R$*.class',
                    '**/*$ViewInjector*.*',
                    '**/*$ViewBinder*.*',
                    '**/BuildConfig.*',
                    '**/Manifest*.*',
                    '**/*RealmProxy.*',
                    '**/*ColumnInfo.*',
                    '**/*RealmModule*.*',
                    '**/AutoValue_*.*',
                    '**/Dagger*.*',
                    '**/*Module_Provide*Factory.*',
                    '**/*_Factory.*',
                    '**/*_MembersInjector.*',
                    '**/*_LifecycleAdapter.*'
            ]
    )

    sourceDirectories = files(coverageSourceDirs)
    executionData     = fileTree(
            dir     : "$buildDir",
            include : [ 'jacoco/testDebugUnitTest.exec', 'outputs/code-coverage/connected/*coverage.ec' ]
    )

    doFirst {
        files('build/intermediates/classes/debug').getFiles().each { file ->
            if (file.name.contains('$$')) {
                file.renameTo(file.path.replace('$$', '$'))
            }
        }
    }
}

task jacocoTestReportLocal(type : JacocoReport, dependsOn : ['testDebugUnitTest', 'createDebugCoverageReport']) {
    group       = 'Reporting'
    description = 'Generate JaCoCo coverage reports'

    reports {
        xml.enabled  = true
        html.enabled = true
    }

    classDirectories = fileTree(
            dir      : 'build/intermediates/classes/debug',
            excludes : [
                    '**/R.class',
                    '**/R$*.class',
                    '**/*$ViewInjector*.*',
                    '**/*$ViewBinder*.*',
                    '**/BuildConfig.*',
                    '**/Manifest*.*',
                    '**/*RealmProxy.*',
                    '**/*ColumnInfo.*',
                    '**/*RealmModule*.*',
                    '**/AutoValue_*.*',
                    '**/Dagger*.*',
                    '**/*Module_Provide*Factory.*',
                    '**/*_Factory.*',
                    '**/*_MembersInjector.*',
                    '**/*_LifecycleAdapter.*',
                    '**/models/**'
            ]
    )

    sourceDirectories = files(coverageSourceDirs)
    executionData     = fileTree(
            dir     : "$buildDir",
            include : [ 'jacoco/testDebugUnitTest.exec', 'outputs/code-coverage/connected/*coverage.ec' ]
    )

    doFirst {
        files('build/intermediates/classes/debug').getFiles().each { file ->
            if (file.name.contains('$$')) {
                file.renameTo(file.path.replace('$$', '$'))
            }
        }
    }
}

And the full app build.gradle here:

apply plugin: 'com.android.application'
apply from: '../jacoco.gradle'

android {
    compileSdkVersion 28
    defaultConfig {
        applicationId "com.example.idea"
        minSdkVersion 22
        targetSdkVersion 28
        versionCode 1
        versionName "1.0"
        testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
    }
    buildTypes {
        release {
            minifyEnabled false
            proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
        }
        debug { testCoverageEnabled true }
    }
}

dependencies {
    implementation fileTree(dir: 'libs', include: ['*.jar'])
    implementation 'com.android.support:appcompat-v7:28.0.0'
    implementation 'com.android.support:support-media-compat:28.0.0'
    implementation 'com.android.support:support-v4:28.0.0'
    implementation 'com.android.support.constraint:constraint-layout:1.1.3'
    implementation 'com.google.firebase:firebase-analytics:15.0.0'
//    implementation 'com.google.firebase:firebase-core:15.0.0'
    testImplementation 'junit:junit:4.12'
    androidTestImplementation 'com.android.support.test:runner:1.0.2'
    androidTestImplementation 'com.android.support.test.espresso:espresso-core:3.0.2'
    androidTestImplementation 'com.android.support.test:rules:1.0.2'
}

apply plugin: 'com.google.gms.google-services'

As well as the config.yml here:

version: 2
references:

  ## Cache

  cache_key: &cache_key
    key: cache-{{ checksum "gradle/wrapper/gradle-wrapper.properties" }}-{{ checksum "build.gradle" }}-{{ checksum "app/build.gradle" }}
  restore_cache: &restore_cache
    restore_cache:
      <<: *cache_key
  save_cache: &save_cache
    save_cache:
      <<: *cache_key
      paths:
        - ~/.gradle
        - ~/.m2

  ## Workspace

  workspace: &workspace
               ~/workspace
  attach_debug_workspace: &attach_debug_workspace
    attach_workspace:
      at: *workspace
  attach_release_workspace: &attach_release_workspace
    attach_workspace:
      at: *workspace
  persist_debug_workspace: &persist_debug_workspace
    persist_to_workspace:
      root: *workspace
      paths:
        - app/build/outputs/androidTest-results
        - app/build/outputs/apk
        - app/build/outputs/code-coverage
        - app/build/test-results
  persist_release_workspace: &persist_release_workspace
    persist_to_workspace:
      root: *workspace
      paths:
        - app/build
  attach_firebase_workspace: &attach_firebase_workspace
    attach_workspace:
      at: *workspace
  persist_firebase_workspace: &persist_firebase_workspace
    persist_to_workspace:
      root: *workspace
      paths:
        - firebase

  ## Docker image configuration

  android_config: &android_config
    working_directory: *workspace
    docker:
      - image: circleci/android:api-28-alpha
    environment:
      TERM: dumb
      _JAVA_OPTIONS: "-Xmx2048m -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap"
      GRADLE_OPTS: '-Dorg.gradle.jvmargs="-Xmx2048m"'
  gcloud_config: &gcloud_config
    working_directory: *workspace
    docker:
      - image: google/cloud-sdk:206.0.0
    environment:
      TERM: dumb

  # Google Services

  export_gservices_key: &export_gservices_key
    run:
      name: Export Google Services key environment variable
      command: echo 'export GOOGLE_SERVICES_KEY="$GOOGLE_SERVICES_KEY"' >> $BASH_ENV
  decode_gservices_key: &decode_gservices_key
    run:
      name: Decode Google Services key
      command: echo $GOOGLE_SERVICES_KEY | base64 -di > app/google-services.json

  # Google Cloud Service

  export_gcloud_key: &export_gcloud_key
    run:
      name: Export Google Cloud Service key environment variable
      command: echo 'export GCLOUD_SERVICE_KEY="$GCLOUD_SERVICE_KEY"' >> $BASH_ENV
  decode_gcloud_key: &decode_gcloud_key
    run:
      name: Decode Google Cloud credentials
      command: echo $GCLOUD_SERVICE_KEY | base64 -di > ${HOME}/client-secret.json

jobs:

  # Build debug APK for unit tests and an instrumented test APK

  build_debug:
    <<: *android_config
    steps:
      - checkout
      - *restore_cache
      - run:
          name: Download dependencies
          command: ./gradlew androidDependencies
      - *save_cache
      - *export_gservices_key
      - *decode_gservices_key
      - run:
          name: Gradle build (debug)
          command: ./gradlew -PciBuild=true :app:assembleDebug :app:assembleAndroidTest
      - *persist_debug_workspace
      - store_artifacts:
          path: app/build/outputs/apk/
          destination: /apk/

  # Build release APK

  build_release:
    <<: *android_config
    steps:
      - checkout
      - *restore_cache
      - run:
          name: Download dependencies
          command: ./gradlew androidDependencies
      - *save_cache
      - *export_gservices_key
      - *decode_gservices_key
      - run:
          name: Gradle build (release)
          command: ./gradlew -PciBuild=true :app:assembleRelease
      - *persist_release_workspace
      - store_artifacts:
          path: app/build/outputs/apk/
          destination: /apk/
      - store_artifacts:
          path: app/build/outputs/mapping/
          destination: /mapping/

  # Run unit tests

  test_unit:
    <<: *android_config
    steps:
      - checkout
      - *restore_cache
      - run:
          name: Download dependencies
          command: ./gradlew androidDependencies
      - *save_cache
      - *export_gservices_key
      - *decode_gservices_key
      - run:
          name: Run unit tests
          command: ./gradlew -PciBuild=true :app:testDebugUnitTest
      - *persist_debug_workspace
      - store_artifacts:
          path: app/build/reports/
          destination: /reports/
      - store_test_results:
          path: app/build/test-results/
          destination: /test-results/

  # Run instrumented tests

  test_instrumented:
    <<: *gcloud_config
    steps:
      - *attach_debug_workspace
      - *export_gcloud_key
      - *decode_gcloud_key
      - run:
          name: Set Google Cloud target project
          command: gcloud config set project i-de-a
      - run:
          name: Authenticate with Google Cloud
          command: gcloud auth activate-service-account [email protected] --key-file ${HOME}/client-secret.json
      - run:
          name: Echo sha1 variable for debugging
          command: echo ${CIRCLE_SHA1}
      - run:
          name: Run instrumented test on Firebase Test Lab
          command: gcloud firebase test android run --type instrumentation --app app/build/outputs/apk/debug/app-debug.apk --test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk --device model=Nexus6P,version=27,locale=en_US,orientation=portrait --no-use-orchestrator --environment-variables coverage=true,coverageFile="/sdcard/coverage.ec" --directories-to-pull=/sdcard --timeout 20m --results-dir=${CIRCLE_SHA1}
      - run:
          name: Create directory to store test results
          command: mkdir firebase
      - run:
          name: Download instrumented test results from Firebase Test Lab
          command: gsutil -m cp -r -U gs://test-lab-hzda62mwyy730-n7hcz7bxtxrx0/${CIRCLE_SHA1} /root/workspace/firebase/
      - *persist_firebase_workspace
      - store_artifacts:
          path: firebase/
          destination: /firebase/

  # Submit JaCoCo coverage report

  report_coverage:
    <<: *android_config
    steps:
      - checkout
      - *restore_cache
      - run:
          name: Download dependencies
          command: ./gradlew androidDependencies
      - *attach_debug_workspace
      - *attach_firebase_workspace
      - run:
          name: Move Firebase coverage report
          command: mkdir -p app/build/outputs/code-coverage/connected && cp firebase/${CIRCLE_SHA1}/Nexus6P-27-en_US-portrait/artifacts/coverage.ec app/build/outputs/code-coverage/connected/coverage.ec
      - *export_gservices_key
      - *decode_gservices_key
      - run:
          name: Generate JaCoCo report
          command: ./gradlew -PciBuild=true :app:jacocoTestReport
      - run:
          name: Upload coverage report to CodeCov
          command: bash <(curl -s https://codecov.io/bash)
      - store_artifacts:
          path: app/build/reports/
          destination: /reports/

workflows:
  version: 2
  workflow:
    jobs:
      - build_debug
      - build_release
      - test_unit
      - test_instrumented:
          requires:
            - build_debug
      - report_coverage:
          requires:
            - build_release
            - test_unit
            - test_instrumented

https://github.com/kyleo83/idea

You can see the full code on this GitHub.

Any help here would be greatly appreciated!

1
androidfirebasejacocofirebase-test-lab
Please edit the question and copy all relevant code and configuration into the question itself, instead of linking out to an external resource. This will ensure the question is always relevant to others even if your code changes. - Doug Stevenson
@DougStevenson edited to include all relevant configurations. - NuttGuy

1 Answers

0
votes

From https://github.com/jacoco/jacoco/issues/968

The solution is well-documented in jacoco but for Android people, what you need is to add the file in /src/androidTest/resources/jacoco-agent.properties with the contents output=none so jacoco can startup without failing, and coverage will be written as normal and transferred correctly later by the android gradle plugin coverage implementation.