Beam: reduce MaximumRecordSize to 6900000 bytes

[chuwy]

We have an enrich job with growing backlog and very strange workers behavior that constantly dumps following message into Dataflow logs:

Could not commit request due to validation error: INVALID_ARGUMENT: Pubsub publish requests are limited to 10MB, rejecting message over 7MB (size 7548K) to avoid exceeding limit with byte64 request encoding.

We believe that reducing max size from current 10000000 bytes can mitigate the problem. Enriched event of 5Mb is already big enough to alarm.

[fedoz]

Hi!
I get an error "Could not commit request due to validation error: INVALID_ARGUMENT: Pubsub publish requests are limited to 10MB, rejecting message over to avoid exceeding limit with byte64 request encoding" on my enrich step.
How I can fix it?

[chuwy]

Hi @fedoz. It should be fixed on tracking side - either reduce amount of events you put into POST payload or find out what is addding this significant amount of data. Also, I'd recommend you to post question on our forums, you should get a response sooner and with higher chance.

[fedoz]

@chuwy Thank you for answer.
Maybe do you know, how I can find topics which initiate this error?