[FIXED] Scrapy settings work using custom_settings but don't work in settings.py

Issue

I have been trying to edit some settings in my Spider but they only seem to work when I override the custom_settings dictionary in my custom Spider.

custom_settings = {
    'DOWNLOAD_DELAY': 1,
    'FEED_URI': 'generalspider.json',
    'FEED_FORMAT': 'json'
}

When I put them in settings.py they don't seem to work. settings.py was supposed to work for all spiders. Am I missing something?

Solution

custom_settings has priority over settings.py. So you'll have to remove the variables in custom_settings for the variables in settings.py to work.

Also please check if the class of your spider is derived from other classes (maybe spiders) and those base classes have their own custom_settings.

Answered By - eLRuLL