In the context of query jobs, a query job will fail with this error reason and error message if the job's billing project exceeds the concurrent rate limit for interactive queries. Refer to Google documentation at https://cloud.google.com/bigquery/quotas#query_jobs.
To see a list of query jobs with this error, start with the query at Section 6.2, “Query Jobs”. Add the filters below to common table expression jobs_selected. Modify the query further as needed.
and j.error_result.reason = 'rateLimitExceeded' and j.error_result.message like '%too many concurrent queries%'
To see a list of in-flight query jobs at a given time, start with the query at Section 6.6, “List In-Flight Query Jobs at a Given Time”. Modify the query as needed.
Below are some things to look for.
a particular process or user launched a large number of query jobs in a short interval
a particular process or user launched a large number of query jobs over a long interval and they linger in a way that contributes to the concurrency problem
multiple processes and/or users are contributing to the concurrency problem
Below are some possible solutions.
change how queries are executed
if appropriate, change query jobs to execute in batch mode instead of interactive mode; see https://cloud.google.com/bigquery/docs/running-queries#batch
if appropriate, change query jobs to bill some other project or projects, pre-existing or new
audit users who are permitted to bill queries to the project and revoke permissions for those who should not
schedule queries to avoid using up resources in a given interval
if possible, optimize queries to avoid using up resources in a given interval
have the concurrent rate limit for interactive queries raised
an analysis indicates there are currently enough slots available to the project so that raising the concurrent rate limit for interactive queries will not harm performance
an analysis indicates more slots are required for the project so that raising the concurrent rate limit for interactive queries will not harm performance
move slots between reservations
move projects between reservations
buy more slots for a limited time
buy more slots permanently