[go: up one dir, main page]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cloud Tasks Client throws 4 DEADLINE_EXCEEDED #4573

Open
j-christoffersen opened this issue Aug 22, 2023 · 3 comments
Open

Cloud Tasks Client throws 4 DEADLINE_EXCEEDED #4573

j-christoffersen opened this issue Aug 22, 2023 · 3 comments

Comments

@j-christoffersen
Copy link

We're using cloud tasks to enqueue jobs. This is happening for a few jobs that rely on cloud-tasks, the one i'll focus on writes somewhere between 10-25 tasks it creates by reading a file. 10-20% of the time, this job will fail because the cloud tasks client throws with 4 DEADLINE_EXCEEDED.

I've read through the issue here but it seems that was closed over a year ago now and I'm running into this issue with the latest version of cloud tasks (also ran into it before upgrading using major version 3 of the package).

Environment details

  • google-cloud/tasks
  • Deployed to Gen2 cloud function
  • Node 18
  • Yarn 1.22.19

Steps to reproduce

Below is my code I'm using to add new tasks

import { v2beta3 } from "@google-cloud/tasks"

const { GCP_PROJECT } = process.env

if (!GCP_PROJECT) {
 throw new Error("GCP_PROJECT is required")
}

const cloudServiceAccountEmail = process.env.CLOUD_TASK_SERVICE_ACCT_EMAIL;

class CloudTasksClientWrapper {
 private client: v2beta3.CloudTasksClient

 constructor() {
   this.client = new v2beta3.CloudTasksClient()
 }

 async createHttpTaskWithToken({
   queueName,
   payload,
   url,
   location = "us-central1",
 }:
 {
   queueName: string
   payload: string
   url: string
   location?: string
 }) {
   const parent = this.client.queuePath(GCP_PROJECT!, location, queueName)

   const body = Buffer.from(payload).toString("base64")

   const task = {
     httpRequest: {
       httpMethod: "POST" as const,
       url,
       oidcToken: {
         serviceAccountEmail: cloudServiceAccountEmail,
         audience: new URL(url).origin,
       },
       headers: {
         "Content-Type": "application/json",
       },
       body,
     },
   }

   // Send create task request.
   const [response] = await this.client.createTask(
     { parent, task },
     { timeout: 5000 }
   )
   return response
 }
}

export { CloudTasksClientWrapper }

Then I call createHttpTaskWithToken from various jobs, for example, after reading every 50 lines from a 1000-line csv, I'll call this function with a task to process those 50 lines.

@majkelEXE
Copy link

Have you tried applying dispatchDeadline property to your task?
In my case it solved the error - be wary to stay in range between 15seconds to 30 minutes.

Example:

const task = {
        httpRequest: {
            httpMethod: google.cloud.tasks.v2beta3.HttpMethod.POST,
            url,
            headers: {
                "Content-Type": "application/json",
            },
            oidcToken: {
                serviceAccountEmail: email,
                audience: url,
            },
            body: Buffer.from(JSON.stringify(payload)).toString("base64"),
        },
        dispatchDeadline: { seconds: 60 },
    };

@j-christoffersen
Copy link
Author

@majkelEXE Were you seeing that error in client logs or in logs from the cloud task itself? Mine seems to be an issue adding tasks to the queue, not processing them once their in the queue.

@majkelEXE
Copy link

In my case it was probably caused by adding hundreds of tasks simultaneously

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants