[FIXED] Turn several "Response" objects represented as nested dicts into multi dimensional list in python

Issue

EDIT : I cannot seem to figure out how to iterate through this object correctly. I believe I am indexing things incorrectly as the error states that blank text does not match the format for datetime conversion I have.

Below is the beginning of one of the response objects, it is long so I opted not to include the entire thing. There are no irregularities at any point, however, and continues to follow the initiated pattern. Please help me understand how to gain access to, and iterate through only the dates and there proceeding information.

{u'Meta Data': {u'1. Information': u'Daily Prices (open, high, low, close) and Volumes',
                u'2. Symbol': u'MSFT',
                u'3. Last Refreshed': u'2018-07-06',
                u'4. Output Size': u'Compact',
                u'5. Time Zone': u'US/Eastern'},
 u'Time Series (Daily)': {u'2018-02-13': {u'1. open': u'88.9300',
                                          u'2. high': u'90.0000',
                                          u'3. low': u'87.8000',
                                          u'4. close': u'89.8300',
                                          u'5. volume': u'26200053'},
                          u'2018-02-14': {u'1. open': u'88.5100',
                                          u'2. high': u'90.9900',
                                          u'3. low': u'88.4100',
                                          u'4. close': u'90.8100',
                                          u'5. volume': u'34314101'},
                          u'2018-02-15': {u'1. open': u'91.2100',
                                          u'2. high': u'92.7200',
                                          u'3. low': u'90.6200',
                                          u'4. close': u'92.6600',
                                          u'5. volume': u'27407914'},

I recieve this error indicating that I am parsing through the dict incorrectly, I have tried many things and I can't seem to get it. I believe this is the last hump to get this program running. (finally)

ValueError: time data '' does not match format '%Y-%b-%d'

And now my attempted UPDATED code

import requests
import json

# Global Variables
url = "https://www.alphavantage.co/query"
function_type = "TIME_SERIES_DAILY"
api_key = "***********"
current_zero_time = ""
main_list = []
symbols = ["AAPL", "HPQ", "IBM", "AMZN", "MSFT", "GOOGL", "INTC", "CSCO", "ORCL", "OCOM"]


def get_data(symbol):
    global api_key
    global function_type
    request_data = {"function": function_type,
                    "symbol": symbol,
                    "apikey": api_key}
    data_dict = requests.get(url, params=request_data)
    return data_dict


def date_dif(date0, date1):
    from datetime import datetime
    d0_obj = datetime.strptime(date0, '%Y-%b-%d')
    d1_obj = datetime.strptime(date1, '%Y-%b-&d')
    return (d1_obj - d0_obj).days


def iterate_data(symbol):
    data_set = get_data(symbol)
    data_set = json.loads(data_set.text)
    global current_zero_time
    list_size = 0
    this_list = [list_size + 1]
    for p_date in data_set['Time Series (Daily)']:
        if current_zero_time is None:
            current_zero_time = p_date
            this_list[list_size] = 0
        else:
            t = date_dif(current_zero_time, p_date)
            this_list[list_size] = t
        list_size += list_size
        for key in p_date:
            this_list[list_size] = p_date[key]
            list_size += list_size
    return this_list


def make_list(stocks):
    global main_list
    for i in range(len(stocks)):
        this_list = iterate_data(symbols[i])
        main_list = main_list + this_list


make_list(symbols)
print(main_list)

Solution

This is getting a bit too much for comments. Please, keep in mind we should take care to make a useful question/answer pair out of this for benefit of any future readers. Perhaps when reasonable (now?), we close an iteration over the code and corresponding question and start a new one.

So far, we've covered that requests.get() returns Response object. To obtain the content read by the request, we need to access its text attribute. This text if a JSON representation, can be parsed by using json.loads().

Now for the currently presented problem, you've defined a variable current_zero_time to be "", but first call into iterate_data() will evaluate current_zero_time is None in your condition to be False (which it is not). Perhaps you wanted to say if not current_zero_time: ...? Hence you and up calling date_dif() with first parameter being "" (empty string) which strptime fails to parse into time (->ValueError).

Speaking of strptime. That formatting string does not seem to match your (ISO-format) date. It should be "%Y-%m-%d" (replace b: month name with m: month number). Second line also has a typo &d, whereas it should be %d. You'd run into these as 3rd and 4th problem resp.

Side note: global variables are (to keep things simple) "evil" and are better to be avoided whenever possible (which as it turns out is pretty much always) or unless you have a really good reason to use them (which frankly you rarely should have). They break common understanding of scope and can end up being something people did not expect them to be at the time they've accessed them. Case to the point. If you've made it all the way through iterate_data() for a symbol and called it again for the next one. It's initial value for that next pass would the last value it had on the previous go.

The above information should address the problem you're currently seeing and let you run into the next one. Which is this line:

     for key in p_date:
        this_list[list_size] = p_date[key]

key here would be a character (e.g. "2" on the first iteration for instance) from pdate. Which is not a valid index of a p_date (a str). Unfortunately I am not really sure what the resulting list (or output in general) should look like, hence I can only give limited input for the next step. I mean as this:

    for key in p_date:
        this_list[list_size] = key

i.e. iterate over characters from p_date and assign them as values to this_list[list_size]; along with changing the None / "" mismatch and fixing the couple strptime issues. You could get the script to complete execution without failure... but as mentioned above, possibly still not do what it was meant to do.

Anyways, at least couple hopefully helpful hints I can offer. To get a better understanding what state your script was in when an exception (like ValueError in your question) was raised or just to poke around. You can run it in a debugger. Instead of calling your script python ./script.py (or just ./script.py), use python -m pdb ./script.py. You can run your program, step through it, inspect stack frame, values of variable at given time... type help for basic help and help COMNAND to get more info about a COMMAND.

Second hint is (on top of global variables) about style. All the imports are normally handled at the top of a script/module. As a matter of fact, there are style recommendations in PEP-8 and there are helpful tools like pylint to check your code. I mention that, because some mistakes and confusion can almost magically disappear with just a little clean-up. :)

One side note / clarification on your question: It's tagged python-3.x, but string literals in your dict would suggest otherwise. u'' literals are a Python 2 thing.

Answered By - Ondrej K.