[FIXED] How to avoid "ValueError: Separator is not found, and chunk exceed the limit"

Issue

I have a server that uses asyncio in the following way:

proc = await asyncio.subprocess.create_subprocess_exec(
    command,
    stdout=asyncio.subprocess.PIPE, 
    stderr=asyncio.subprocess.STDOUT)

async for line in proc.stdout:
    line = line.decode('utf-8').strip()
    print(line)

The problem is, if proc.stdout contains a line that is too long, I get in line of the for statement the error:

ValueError: Separator is not found, and chunk exceed the limit

How can I detect such long lines in order to avoid this error in advance?

Solution

As documentation says:

The limit argument sets the buffer limit for StreamReader wrappers for Process.stdout and Process.stderr (if subprocess.PIPE is passed to stdout and stderr arguments).

Seems to be what we need.

Default limit seems to be 64 KiB, you can try something higher:

proc = await asyncio.subprocess.create_subprocess_exec(
    command,
    limit = 1024 * 128,  # 128 KiB
    stdout=asyncio.subprocess.PIPE, 
    stderr=asyncio.subprocess.STDOUT)

Upd:

I just want to truncate/ignore lines that are too long. Is this possible?

You can suppress errors on long lines and continue afterwards. Following code (tested on Windows) shows the idea:

import asyncio


async def main():
    proc = await asyncio.subprocess.create_subprocess_exec(
        *[
            'wget', 
            '--help'
        ],
        limit = 20,
        stdout=asyncio.subprocess.PIPE, 
        stderr=asyncio.subprocess.STDOUT)

    while True:
        try:
            async for line in proc.stdout:
                line = line.decode('utf-8').strip()
                print(line)
        except ValueError:
            continue
        else:
            break


loop = asyncio.ProactorEventLoop()
asyncio.set_event_loop(loop)
loop.run_until_complete(main())

Answered By - Mikhail Gerasimov