tesk.k8s package¶

Submodules¶

tesk.k8s.constants module¶

Constants for Kubernetes API.

class tesk.k8s.constants.AnnotationConstants(**data)[source]¶

Bases: BaseModel

Constants related to Kubernetes annotations.

ANN_TESTASK_NAME_KEY = Key of the annotation, that stores name of TES task: in both taskmaster’s job and executor’s jobs

ANN_JSON_INPUT_KEY = Key of the annotation, that stores whole input TES: task serialized to JSON

ANN_JSON_INPUT_KEY: str¶

ANN_TESTASK_NAME_KEY: str¶

class Config[source]¶

Bases: object

Configuration for class.

frozen = True¶

class tesk.k8s.constants.FTPConstants(**data)[source]¶

Bases: BaseModel

Constants related to FTP configuration.

FTP_SECRET_USERNAME_ENV = Environment variable name for FTP username

FTP_SECRET_PASSWORD_ENV = Environment variable name for FTP password

class Config[source]¶

Bases: object

Configuration for class.

frozen = True¶

FTP_SECRET_PASSWORD_ENV: str¶

FTP_SECRET_USERNAME_ENV: str¶

class tesk.k8s.constants.JobConstants(**data)[source]¶

Bases: BaseModel

Constants related to job and tasks.

TASKMASTER_INPUT = ENV var that serves as taskmaster script input (JSON: format)

TASKMASTER_INPUT_EXEC_KEY = Key in JSON taskmaster input, which holds list: of executors

VOLUME_NAME = Volume name

JOB_CREATE_ATTEMPTS_NO = Number of attempts of job creation in case of name: collision

JOB_NAME_TASKM_PREFIX = Constant prefix of taskmaster's job name (== TES: task ID)

JOB_NAME_EXEC_PREFIX = Part of executor's job name, that follows: taskmaster’s name

JOB_NAME_TASKM_RAND_PART_LENGTH = No of bytes of random part of task: master’s name (which end up encoded to hex)

JOB_NAME_EXEC_NO_LENGTH = No of digits reserved for executor number in: executor’s job name. Ends up padded with ‘0’ for numbers < 10

JOB_NAME_FILER_SUF = Output filer name suffix

RESOURCE_DISK_DEFAULT¶: float = Default resource disk value

COMPLETED_STATES = TES task states, indicating task is not running and: cannot be cancelled

EXECUTOR_BACKOFF_LIMIT = Set a number of retries of a job execution.

FILER_BACKOFF_LIMIT = Set a number of retries of a filer job execution.

COMPLETED_STATES: Set[str]¶

class Config[source]¶

Bases: object

Configuration for class.

frozen = True¶

EXECUTOR_BACKOFF_LIMIT: str¶

FILER_BACKOFF_LIMIT: str¶

JOB_CREATE_ATTEMPTS_NO: int¶

JOB_NAME_EXEC_NO_LENGTH: int¶

JOB_NAME_EXEC_PREFIX: str¶

JOB_NAME_FILER_SUF: str¶

JOB_NAME_TASKM_PREFIX: str¶

JOB_NAME_TASKM_RAND_PART_LENGTH: int¶

RESOURCE_DISK_DEFAULT: float¶

TASKMASTER_INPUT: str¶

TASKMASTER_INPUT_EXEC_KEY: str¶

VOLUME_NAME: str¶

class tesk.k8s.constants.K8sConstants(**data)[source]¶

Bases: BaseModel

Constants related to Kubernetes.

Attribute:: K8S_BATCH_API_VERSION: Kubernetes Batch API version K8S_BATCH_API_JOB_TYPE: Kubernetes Job object type JOB_RESTART_POLICY: Kubernetes Job restart policy RESOURCE_CPU_KEY: Executor CPU resource label RESOURCE_MEM_KEY: Executor memory resource label RESOURCE_MEM_UNIT: Executor memory resource unit RESOURCE_MEM_ONE_GB: One Gibibyte (Gi) in bytes

class Config[source]¶

Bases: object

Configuration for class.

frozen = True¶

JOB_RESTART_POLICY: str¶

K8S_BATCH_API_JOB_TYPE: str¶

K8S_BATCH_API_VERSION: str¶

class PodPhase(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]¶

Bases: Enum

Pod state.

Attribute:: PENDING = pending

Config = <class 'tesk.k8s.constants.K8sConstants.PodPhase.Config'>[source]¶

PENDING = 'Pending'¶

get_code()[source]¶

Return the pod state.

Return type:: str

RESOURCE_CPU_KEY: str¶

RESOURCE_MEM_KEY: str¶

RESOURCE_MEM_ONE_GB: int¶

RESOURCE_MEM_UNIT: str¶

class tesk.k8s.constants.LabelConstants(**data)[source]¶

Bases: BaseModel

Constants related to Kubernetes labels.

LABEL_TESTASK_ID_KEY = Key of the label, that stores taskmaster's name (==: TES task generated ID) in executor jobs

LABEL_JOBTYPE_KEY = Key of the label, that stores type of a job (taskmaster: or executor)

LABEL_JOBTYPE_VALUE_TASKM = Value of the label with taskmaster's job type

LABEL_JOBTYPE_VALUE_EXEC = Value of the label with executor's job type

LABEL_EXECNO_KEY = Key of the label, that holds executor number for: executor jobs

LABEL_TASKSTATE_KEY = Key of the label, that holds executor's state

LABEL_TASKSTATE_VALUE_CANC = Value of the label, that holds executor's: Cancelled state

LABEL_USERID_KEY = Key of the label, that holds user id

LABEL_GROUPNAME_KEY = Key of the label, that holds user's group name

class Config[source]¶

Bases: object

Configuration for class.

frozen = True¶

LABEL_EXECNO_KEY: str¶

LABEL_GROUPNAME_KEY: str¶

LABEL_JOBTYPE_KEY: str¶

LABEL_JOBTYPE_VALUE_EXEC: str¶

LABEL_JOBTYPE_VALUE_TASKM: str¶

LABEL_TASKSTATE_KEY: str¶

LABEL_TASKSTATE_VALUE_CANC: str¶

LABEL_TESTASK_ID_KEY: str¶

LABEL_USERID_KEY: str¶

class tesk.k8s.constants.PatchConstants(**data)[source]¶

Bases: BaseModel

Constants related to patch operations.

CANCEL_PATH = Patch object passed to job API, when cancelling task

CANCEL_PATCH: str¶

class Config[source]¶

Bases: object

Configuration for class.

frozen = True¶

class tesk.k8s.constants.PathValidationConstants(**data)[source]¶

Bases: BaseModel

Constants related to path validation.

ABSOLUTE_PATH_REGEX = Pattern to validate path

ABSOLUTE_PATH_MESSAGE = Message for absolute path validation (to avoid: message.property)

ABSOLUTE_PATH_MESSAGE: str¶

ABSOLUTE_PATH_REGEXP: str¶

class Config[source]¶

Bases: object

Configuration for class.

frozen = True¶

class tesk.k8s.constants.TeskK8sConstants(**data)[source]¶

Bases: BaseModel

All the constants related to k8s and job creation.

job_constants¶

annotation_constants¶

label_constants¶

path_validation_constants¶

ftp_constants¶

patch_constants¶

k8s_constants¶

class Config[source]¶

Bases: object

Configuration for class.

frozen = True¶

annotation_constants: AnnotationConstants¶

ftp_constants: FTPConstants¶

job_constants: JobConstants¶

k8s_constants: K8sConstants¶

label_constants: LabelConstants¶

patch_constants: PatchConstants¶

path_validation_constants: PathValidationConstants¶

tesk.k8s.wrapper module¶

Wrapper Abstraction of Kubernetes Python client API for TESK.

class tesk.k8s.wrapper.KubernetesClientWrapper[source]¶

Bases: object

Kubernetes client wrapper class.

create_config_map(config_map)[source]¶

Create a config map in the Kubernetes cluster.

Parameters:: config_map (V1ConfigMap) – ConfigMap object to create.
Return type:: V1ConfigMap

create_job(job)[source]¶

Create a job in the Kubernetes cluster.

Return type:: V1Job
Returns:: Job object created in the Kubernetes cluster.

get_single_task_output_filer_job(task_id)[source]¶

Get single task output filer job.

Return type:: Optional[V1Job]

label_job_as_cancelled(task_id)[source]¶

Label a job as cancelled.

Parameters:: task_id (str) – Task identifier.
Return type:: None

label_pod_as_cancelled(pod_name)[source]¶

Label a pod as cancelled.

Parameters:: pod_name (str) – Pod name.
Return type:: None

list_all_filer_jobs()[source]¶

List all output filer jobs in the Kubernetes cluster.

Return type:: V1JobList

list_all_job_pods()[source]¶: List all job pods.

list_all_task_executor_jobs()[source]¶

List all executor jobs in the Kubernetes cluster.

Return type:: V1JobList

list_all_taskmaster_jobs()[source]¶

List all taskmaster jobs in the Kubernetes cluster.

Return type:: V1JobList

list_all_taskmaster_jobs_for_user(page_token, items_per_page)[source]¶

Gets all Taskmaster job objects, a User is allowed to see.

Parameters:

page_token (str) – pageToken supplied by user (from previous result; points to next page of results)
items_per_page (int) – Value submitted by user, limiting number of results.

Return type:

V1JobList

Returns:

Job list of Taskmaster jobs that user is allowed to see.

list_jobs(page_token=None, label_selector=None, limit=None)[source]¶

List jobs in the Kubernetes cluster.

Parameters:

page_token (Optional[str]) – pageToken supplied by user (from previous result; points to next page of results)
label_selector – Label selector to filter jobs.
limit – Maximum number of jobs to return.

list_limits(label_selector=None, limit=None)[source]¶

List limit ranges in the Kubernetes cluster.

Parameters:

label_selector – Label selector to filter limit ranges.
limit – Maximum number of limit ranges to return.

Return type:

V1LimitRangeList

list_single_job_pods(job)[source]¶

List pods associated with a single job.

Parameters:: job (V1Job) – Job object to list pods for.
Return type:: V1PodList

list_single_task_executor_jobs(task_id)[source]¶

List single task executor job.

Return type:: V1JobList

minimum_ram_gb()[source]¶

Get the minimum amount of RAM in the cluster.

Return type:: float
Returns:: Minimum amount of RAM in the cluster in GB.

quantity_to_bytes(quantity)[source]¶

Convert quantity(resource) to bytes.

Return type:: int

read_pod_log(pod_name)[source]¶

Read logs from a pod.

Parameters:: pod_name (str) – Name of the pod to read logs from.
Return type:: Optional[str]

read_taskmaster_job(task_id)[source]¶

Read a taskmaster job from the Kubernetes cluster.

task_id: Task identifier.

Return type:: V1Job
Returns:: Job object read from the Kubernetes cluster
Raises:: Exception – If the task is not found.

Module contents¶

Kubernetes API module for TESK.