[FIXED] HTTPCACHE not working when deploying scrapy project to scrapyd

Issue

When I try to use HTTPCACHE with scrapyd I get the following error:

[scrapy] WARNING: Disabled Httpcache Middlware: unable to find scrapy.cfg file to infer project data dir

Solution

The problem is that http caching defaults to using the relative path httpcache.

This works when you run the spider normally from the command line, but not when running it as a service.

The solution is to set the HTTPCACHE_DIR^[docs] setting to an absolute path.

Answered By - Acorn