using in-memory stream versus tempfile for csv download has been tested
just for reasons of completeness, this test is documented here:
There were thoughts that csv download might speed up, if all data is directly (from memory) sent via the REST API, instead of using a tempfile at this place.
The following code fragment was tested against its actual implementation:
def iterfile():
buf = StringIO('#' + metadata.json(indent=4,ensure_ascii=False).replace('\n', '\n#') + '\n',newline='\n')
buf.seek(0,SEEK_END)
# write header line
buf.write(",".join([column.name for column in models.Data.__mapper__.columns])+'\n')
# now the data
buf.writelines([",".join([str(getattr(curr, column.name)) for column in models.Data.__mapper__.columns]) + '\n' for curr in data])
buf.seek(0)
yield from buf
return StreamingResponse(iterfile(), media_type="text/csv")
The original code fragment is:
with tempfile.NamedTemporaryFile(mode='w', newline='\n', encoding='utf-8', delete=False) as outfile:
outfile.write('#' + metadata.json(indent=4,ensure_ascii=False).replace('\n', '\n#') + '\n')
outcsv = csv.writer(outfile)
# write header line
outcsv.writerow([column.name for column in models.Data.__mapper__.columns])
# now the data
outcsv.writerows([[getattr(curr, column.name) for column in models.Data.__mapper__.columns] for curr in data])
return FileResponse(outfile.name, media_type="text/csv")
It could be shown, that the StringIO approach was 4 times slower than the actual implementation: It was not only slower in writing to the buffer/creating the buffer, but also StreamingResponse was 10 times slower than File Response.
Therefore, no speed up could be achieved related to csv download.