Buffer Overflow in Power Automate

Isaac Stephens, 07 December 2021

Lately I was using Power Automate to retrieve ALL accounts in our Dynamics 365 system, which is something that I have never really done before. Most of the time I will have an OData filter to be a little more specific, but this time I wanted everything. I was expecting this to take a while but I was not expecting it to crash. At this point in time - if you hadn’t read the title - you may expect this to be a timeout issue, but this was not the case. It crashed relatively quickly, in around 90 seconds, with the error "Buffer Overflow" saying:

Cannot write more bytes to the buffer than the configured maximum buffer size: 104857600

This error was new to me, I could understand what it was saying but I also had never experienced this before so was a little perplexed. It makes sense for a retrieve to have a buffer limit but I wouldn’t have assumed it would ever be reached, especially when our Accounts entity only has 2051 records. But it turns out this is something we should be careful of, especially when the solution is rather simple.

When retrieving records through Power Automate you are able to apply a range of parameters:

· Select columns

· Filter rows

· Sort By

· Expand Query

· Fetch Xml Query

· Row count

· Skip token

· Partition ID

Obviously one way to lower the buffer usage would be to lower the number of records retrieved, we could do this with Filter rows, Fetch Xml Query or Row count. However, that means we are not meeting the criteria of retrieving all records so we need to look for another solution!

The solution is to use the select columns option and only retrieve the columns we actually need. This makes a lot of sense when you think about, if you were to make an Advanced Find query would you add all columns or just the ones you wanted to see? You may also know that adding more columns in a view/advanced find increases the time to retrieve the records. So why add an action to retrieve records in CRM with ALL columns, it just doesn’t make sense. The only downside is that you need to type them in manually using their logical names in a comma separated list. This is okay for a customisation-savy user or a developer, but might be a little more confusing for a casual user.