Skip to content

how to restore cortex operator normally when too many jobs are requested #2394

Open

Description

@nellaG

hello.

I'm currently using cortex 0.40.0.

I seldom request thousands of jobs to certain cortex api by mistake.
When I do like that, I can't use cortex cli well (the response time is so long, or just hanging) and I guess that cortex operator is overloaded because of me.
(the status of operator-controller-manager pod is continuously goes to OOMKilled -> CrashLoopBackOff)

To resolve this issue, I attempted these so far but It didn't work well.

  1. delete thousands of AWS sqs queue
  2. delete all of enqueuer job and worker job created by mistake
  3. delete certain cortex api and re-deploy it

After all I just down the cluster and up (+ re-deploy all of api) to make cortex work well.
If this is happened, what should I do to restore cortex without down and up cluster?

I glad to your support. Thank you so much.

Activity

  1. miguelvr commented on Oct 4, 2021

    @miguelvr
    Collaborator

    the operator-controller-manager is responsible for the cleanup of all the resources, so if it starts failing, it requires a lot of intervention.

    The first thing I would try is If the operator-controller-manager is getting OOMKilled, is to increase its memory limits.

    If that doesn't work, there are ways to "fix" that weird state, but still require a lot of manual intervention, or eventually an automated script.

    When you create a BatchAPI job this happens:

    1. A BatchJob kubernetes resource is created
    2. The operator-controller-manager creates / updates / deletes the required resources referring to that BatchJob resource.

    In order to fix that weird state you have to:

    1. Delete the created BatchJob resources from the cluster using kubectl delete with the --force flag
    2. Delete all the created SQS queues manually or with a script
    3. Delete S3 resources that might have been created for that BatchJob resource
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionFurther information is requested

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions