5
votes

I am new to go and trying to make a simple webcrawler. I keep getting a "panic: runtime error: invalid memory address or nil pointer dereference" and have no idea how to fix the issue. I have an "advancedFetcher" function and a "basicFetcher" function and I get the same error under either one. This answer suggests checking each error (which I have I think) but I am still getting an error. Thanks!

package main

import (
    "crypto/tls"
    "fmt"
    "io/ioutil"
    "net"
    "net/http"
    "time"
)

var tr = &http.Transport{
    TLSClientConfig:     &tls.Config{InsecureSkipVerify: true},
    MaxIdleConnsPerHost: 25,
    DisableKeepAlives:   true,
    Proxy:               http.ProxyFromEnvironment,
    Dial: (&net.Dialer{
        Timeout:   10 * time.Second,
        KeepAlive: 10 * time.Second,
    }).Dial,
    TLSHandshakeTimeout: 5 * time.Second,
}
var client = &http.Client{Transport: tr}

func makeGetRequest(uri string) *http.Response {
    req, err := http.NewRequest("GET", uri, nil)
    if err != nil {
        fmt.Println(err)
    }
    req.Header.Add("User-Agent", "Hi it's me again")
    resp, err := client.Do(req)
    if err != nil {
        fmt.Println(err)
    }
    return resp
}

func advancedFetcher(uri string, c chan int) {
    fmt.Println("Getting: ", uri)
    resp := makeGetRequest(uri)
    defer resp.Body.Close()
    c <- resp.StatusCode
}

func basicFetcher(uri string, c chan int) {
    fmt.Println("Getting: ", uri)
    resp, err := http.Get(uri)
    if err != nil {
        fmt.Println(err)
    }
    defer resp.Body.Close()
    _, err = ioutil.ReadAll(resp.Body)
    c <- resp.StatusCode
}

func main() {
    c := make(chan int)
    rows := []string{"https://www.google.com", "http://www.fdicconnect.gov"}
    for _, row := range rows {
        //go basicFetcher(row, c)
        go advancedFetcher(row, c)
    }
    for _ = range rows {
        select {
        case result := <-c:
            fmt.Println(result)
        }
    }
    fmt.Println("All Done")
}

EDIT #1:

Getting:  https://www.google.com
Getting:  http://www.fdicconnect.gov
200
Get http://www.fdicconnect.gov: dial tcp 167.176.6.86:80: i/o timeout
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x400ff2]

goroutine 21 [running]:
runtime.panic(0x6594c0, 0x7e9b53)
    /usr/local/go/src/pkg/runtime/panic.c:279 +0xf5
main.advancedFetcher(0x6c3b70, 0x1a, 0xc208004180)
    /home/me/go/src/testfolder/tester/main.go:41 +0x1c2
created by main.main
    /home/me/go/src/testfolder/tester/main.go:61 +0x123

goroutine 16 [chan receive]:
main.main()
    /home/me/go/src/testfolder/tester/main.go:65 +0x198

goroutine 19 [finalizer wait]:
runtime.park(0x413bd0, 0x7ee660, 0x7ec8c9)
    /usr/local/go/src/pkg/runtime/proc.c:1369 +0x89
runtime.parkunlock(0x7ee660, 0x7ec8c9)
    /usr/local/go/src/pkg/runtime/proc.c:1385 +0x3b
runfinq()
    /usr/local/go/src/pkg/runtime/mgc0.c:2644 +0xcf
runtime.goexit()
    /usr/local/go/src/pkg/runtime/proc.c:1445

goroutine 17 [syscall]:
runtime.goexit()
    /usr/local/go/src/pkg/runtime/proc.c:1445
1
Is your program printing an error before panicking? If memory serves, if http.Get returns an error, resp.Body is nil which would cause your error.Jimmy Sawczuk
@JimmySawczuk: You mean something like Edit #1? If so, I still get the same error.user_78361084
What line is throwing the panic?Jimmy Sawczuk
Yeah, include the crashdump. Usually makes something like this much easier to debug.twotwotwo
@twotwotwo I've added the crashdump. Line #41 refers to "defer resp.Body.Close()" in the "advancedFetcher" function.user_78361084

1 Answers

4
votes

Is your program printing an error before panicking? If memory serves, if http.Get returns an error, then resp.Body == nil, which would cause your error because you can't call resp.Body.Close() on nil.

As twotwotwo suggests, you should write your function so that it returns a (result, error) pair so you can terminate it gracefully before running into problems.