Iterating website URLs from a text file into BeautifulSoup w/ Python

Question

I have a .txt file with a different link on each line that I want to iterate, and parse into BeautifulSoup(response.text, "html.parser"). I'm having a couple issues though.

I can see the lines iterating from the text file, but when I assign them to my requests.get(websitelink), my code that previously worked (without iteration) no longer prints any data that I scrape.

All I receive are some blank lines in the results.

I'm new to Python and BeautifulSoup, so I'm not quite sure what I'm doing wrong. I've tried parsing the lines as a string, but that didn't seem to work.

import requests
from bs4 import BeautifulSoup
filename = 'item_ids.txt'

with open(filename, "r") as fp:
    lines = fp.readlines()
    for line in lines:

        #Test to see if iteration for line to line works
        print(line)

        #Assign single line to websitelink
        websitelink = line

        #Parse websitelink into requests
        response = requests.get(websitelink)
        soup = BeautifulSoup(response.text, "html.parser")

        #initialize and reset vars for cd loop
        count = 0
        weapon = ''
        stats = ''

        #iterate through cdata on page, and parse wanted data
        for cd in soup.findAll(text=True):
            if isinstance(cd, CData):
                #print(cd)
                count += 1
                if count == 1:
                    weapon = cd
                if count == 6:
                    stats = cd

        #concatenate cdata info
        both = weapon + " " + stats
        print(both)

The code should follow these steps:

1. Read line (URL) from text file, and assign to variable to be used w/ request.get(websitelink)
2. BeautifulSoup scrapes that link for the CData and prints it
3. Repeat Step 1 & 2 until final line of the text file (last URL)

Any help would be greatly appreciated,

Thanks

Answer 1

I don't know this could help you or not but I've added a strip() to your link variable when you are assigning it to the websitelink and helped me to make your code work. You could try it.

websitelink = line.strip()