set CrawlSpider settings from Job()

[Matthijz98]

Hi,

I am trying to use a CrawlSpider with Celery. The settings for the CrawlSpider are stored in the database. But what i try i can not get the settings passed to the CrawlSpider class.

My spider

class FindProductSpider(CrawlSpider):
    name = 'FindProductSpider'
    allowed_domains = ['']
    start_urls = ['']
    webshopid = ''
    rule = ''

    rules = [Rule(LinkExtractor(allow=rule), callback='parse_item', follow=True)]

    def parse_item(self, response):
        p = Product(url=response.url, from_webshop_id=self.webshopid)
        p.save()

and my celery task:

@shared_task()
def getproducts():
    webshops = Webshop.objects.all()

    for webshop in webshops:
        job = Job(FindProductSpider,
                  start_urls=[webshop.spider_start_url],
                  allowed_domain=[webshop.spider_allowed_domain],
                  rule=webshop.spider_allow_regex,
                  webshopid=webshop.id
                  )
        processor = Processor(settings=settings)
        data = processor.run([job])

But when i print the settings they keep being empty.
Some help would be super nice