Scrapy modify cookie

Question

I'm able to create a cookie with scrapy but unable to modify one existing cookie. In the ecommerce website i'm working, this cookie handles the postal code and the postal is used by each page to modify product attributes. I can modify the postal code using selenium, scrape every page, but the scrape process is too slow. I wanted to use scrapy only, modifying this request/response postcode cookie.

I can create a cookie on my requests using this code

in SETTINGS.PY COOKIES_ENABLED = True

in spider.py yield scrapy.Request(response.urljoin(url), self.parsePage, cookies={'cp': codpost})

I get the list of cookies using: cookies = response.headers.getlist("Set-Cookie")

The relevant result I get on every page is: [..., b'cp=28029; Expires=Sun, 29-Feb-2032 07:58:44 GMT; Path=/', ...]

It doesn't look like to be a pair key/value. How can I modify this cookie? Any suggestions?

Answer 1

I don't think you need to manually modify the cookies, you probably can get through using cookiejar.

You can yield several requests for different postal codes, and include a different cookiejar in each request, that way Scrapy will manage different sessions for each request.

for code in postal_code:
    yield Request(url, meta={'cookiejar': code}, callback=self.callback_func)

This is a minimal example, just to show how to pass the cookiejar in the request. Keep in mind that the values represented by code must be different for each different session you need.