Scrapy output JSON or CSV

Question

I'm trying web scraping using this code settings.py

FEED_EXPORT_ENCODING = 'utf-8'

import datetime
now = datetime.datetime.now ()
formatted = now.strftime ("%Y%m%d_%H%M")
FEED_URI = f'\\C:\\Users\\Acer\\Desktop\\{formatted}.csv'
FEED_TYPE = 'csv'

with this special_offers.py

# -*- coding: utf-8 -*-
import scrapy
import datetime


class SpecialOffersSpider(scrapy.Spider):
    name = 'special_offers'
    allowed_domains = ['www.tinydeal.com']

    def start_requests(self):
        yield scrapy.Request(url='https://www.tinydeal.com/specials.html', callback=self.parse, headers={
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36'
        })

    def parse(self, response):
        for product in response.xpath("//ul[@class='productlisting-ul']/div/li"):
            yield {
                'title': product.xpath(".//a[@class='p_box_title']/text()").get(),
                'url': response.urljoin(product.xpath(".//a[@class='p_box_title']/@href").get()),
                'discounted_price': product.xpath(".//div[@class='p_box_price']/span[1]/text()").get(),
                'original_price': product.xpath(".//div[@class='p_box_price']/span[2]/text()").get(),
                'User-Agent': response.request.headers['User-Agent'].decode('utf-8'),
                'datetime': datetime.datetime.now().strftime("%Y%m%d %H%M")

            }

        next_page = response.xpath("//a[@class='nextPage']/@href").get()

        if next_page:
            yield scrapy.Request(url=next_page, callback=self.parse, headers={
                'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36'
            })

then I open terminal and use

scrapy crawl special_offers

the problem is, when I export JSON the data came without comma between }{. making my file not read by Power BI for example

when I export CSV the data came differente then I expect when I open using EXCEL

CSV data example {"title": "ABS Plastic Case for Raspberry Pi 3 Model B & Raspberry Pi 2 E-524988", "url": "https://www.tinydeal.com/abs-plastic-case-for-raspberry-pi-3-model-b-raspberry-pi-2-p-163950.html", "discounted_price": "R$12.74", "original_price": "R$13.66 ", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36", "datetime": "20200420 2330"} {"title": "3M 9001 KN90 Dust Masks Respirator Anti-dust PM2.5 Industrial Construction Polle RTH-562440", "url": "https://www.tinydeal.com/3m-9001-kn90-dust-masks-respirator-anti-dust-pm25-industrial-construction-polle-p-179487.html", "discounted_price": "R$10.29", "original_price": "R$12.40 ", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36", "datetime": "20200420 2330"} {"title": "2-in-1 Vintage Blue Rhinestone Necklace + Earring Jewelry Set DJA-562974", "url": "https://www.tinydeal.com/2-in-1-vintage-blue-rhinestone-necklace-earring-jewelry-set-p-180097.html", "discounted_price": "R$11.77", "original_price": "R$30.77 ", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36", "datetime": "20200420 2330"} {"title": "64GB USB 2.0 Flash Drive USB Pen Drive U Disk EFM-561923", "url": "https://www.tinydeal.com/64gb-usb-20-flash-drive-usb-pen-drive-u-disk-p-178875.html", "discounted_price": "R$34.83", "original_price": "R$99.43 ", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36", "datetime": "20200420 2330"}

JSON data example

{ "title": "ABS Plastic Case for Raspberry Pi 3 Model B & Raspberry Pi 2 E-524988", "url": "https://www.tinydeal.com/abs-plastic-case-for-raspberry-pi-3-model-b-raspberry-pi-2-p-163950.html", "discounted_price": "R$12.74", "original_price": "R$13.66 ", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36", "datetime": "20200420 2329" } { "title": "3M 9001 KN90 Dust Masks Respirator Anti-dust PM2.5 Industrial Construction Polle RTH-562440", "url": "https://www.tinydeal.com/3m-9001-kn90-dust-masks-respirator-anti-dust-pm25-industrial-construction-polle-p-179487.html", "discounted_price": "R$10.29", "original_price": "R$12.40 ", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36", "datetime": "20200420 2329" } { "title": "2-in-1 Vintage Blue Rhinestone Necklace + Earring Jewelry Set DJA-562974", "url": "https://www.tinydeal.com/2-in-1-vintage-blue-rhinestone-necklace-earring-jewelry-set-p-180097.html", "discounted_price": "R$11.77", "original_price": "R$30.77 ", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36", "datetime": "20200420 2329" }

Can anyone tell me where I'm going wrong in these outputs?

Karla Karla · Accepted Answer · 2020-04-21T16:19:23

How are you getting the scraped data? From what you showed I suspected that you copied from the terminal. Is that it? If it's, there's a way to save it direct into a file, using this command:

scrapy crawl special_offers -o <where save the file>/special_offers.json

Hopefully this solves your problem. Please, let me know.

Scrapy output JSON or CSV

1 Answers