[FIXED] Asyncio tasks executed without being called in asyncio.gather()

Issue

I want to put some handler methods to a dictionary and call only some of them (based on the required handlers). However, when the asyncio.gather() executed, all of the tasks are executed. Here is the code snippet:

import asyncio

class DataHandler():
    def __init__(self):
        pass

    async def generate_coroutines(self):
        all_handlers = {
            'handler1': asyncio.create_task(self.handler1()),
            'handler2': asyncio.create_task(self.handler2()),
            'handler3': asyncio.create_task(self.handler3()),
        }
        return all_handlers

    async def main(self, handlers):
        print('Main method started')

        all_handlers = await self.generate_coroutines()
        print('Handler coroutines created')

        # Only add the handlers that has been given as the argument
        required_handlers = []
        for handler in handlers:
            if handler in all_handlers.keys(): required_handlers.append(all_handlers[handler])

        output = list(await asyncio.gather(*required_handlers))
        print(output)


    async def handler1(self): 
        print('handler1 executed')
        return 'handler1_output'

    async def handler2(self):
        print('handler2 executed') 
        return 'handler2_output'

    async def handler3(self):
        print('handler3 executed') 
        return 'handler3_output'


if __name__ == '__main__':

    dh = DataHandler()
    loop = asyncio.get_event_loop()

    loop.run_until_complete( dh.main(['handler2']))

Output:

Main method started
Handler coroutines created
handler1 executed
handler2 executed
handler3 executed
['handler2_output']

Desired Output:

Main method started
Handler coroutines created
handler2 executed
['handler2_output']

If I cancel the unused tasks, then they are not executed. But isn't it possible to create all the possible tasks and only execute some of them and let others go (without a need to cancel the rest)

Solution

asyncio.create_task creates Tasks that schedule the coroutines in an event loop. You can fix this by delaying creating a coroutine and Task:

class DataHandler():
    
    async def main(self, handlers):
        print('Main method started')
        
        # Only make coroutines from specified handlers
        required_coroutines = []
        for handler in handlers:
            if hasattr(self, handler): # directly check instance for handler
                required_coroutines.append(getattr(self, handler)())
        
        print('Handler coroutines created')

        output = list(await asyncio.gather(*required_coroutines))
        
        print(output)

    async def handler1(self): 
        print('handler1 executed')
        return 'handler1_output'

    async def handler2(self):
        print('handler2 executed') 
        return 'handler2_output'

    async def handler3(self):
        print('handler3 executed') 
        return 'handler3_output'

In your original code, the handler2() coroutine didn't get scheduled a 2nd time by gather because coroutines can't be scheduled more than once. You have to call the async function each time and make a fresh coroutine.

Answered By - BatWannaBe