tesk.services package

Submodules

tesk.services.constants module

tesk.services.exceptions module

exception tesk.services.exceptions.FileProtocolDisabled

Bases: Exception

exception tesk.services.exceptions.InvalidHostPath

Bases: Exception

class tesk.services.exceptions.ServiceStatusCodes

Bases: object

BAD_REQUEST = 400

CONFLICT = 409

CREATED = 201

FORBIDDEN = 403

GATEWAY_TIMEOUT = 504

INTERNAL_SERVER_ERROR = 500

NOT_FOUND = 404

NOT_IMPLEMENTED = 501

OK = 200

REDIRECT = 300

SERVICE_UNAVAILABLE = 503

UNAUTHORIZED = 401

classmethod get(status_name)

exception tesk.services.exceptions.UnknownProtocol

Bases: Exception

tesk.services.filer module

class tesk.services.filer.FTPTransput(path, url, ftype, ftp_conn=None)

Bases: Transput

delete()

download_dir()

download_file()

upload_dir()

upload_file()

class tesk.services.filer.FileTransput(path, url, ftype)

Bases: Transput

download_dir()

download_file()

transfer(copyFn, src, dst)

upload_dir()

upload_file()

class tesk.services.filer.HTTPTransput(path, url, ftype)

Bases: Transput

download_dir()

download_file()

upload_dir()

upload_file()

tesk.services.filer.copyContent(src, dst, symlinks=False, ignore=None)

https://stackoverflow.com/a/12514470/1553043

tesk.services.filer.copyDir(src, dst)

Limitation of shutil.copytree:

The destination directory, named by dst, must not already exist; it will be created as well as missing parent directories.

tesk.services.filer.copyFile(src, dst)

Limitations of shutil.copy:

It does not interpret * as a glob, but as a character.

tesk.services.filer.file_from_content(filedata)

tesk.services.filer.ftp_check_directory(ftp_connection, path)

Following convention with the rest of the code, return 0 if it is a directory, 1 if it is not or failed to do the check

tesk.services.filer.ftp_download_file(ftp_connection, remote_source_path, local_destination_path)

tesk.services.filer.ftp_login(ftp_connection, netloc, netrc_file)

tesk.services.filer.ftp_make_dirs(ftp_connection, path)

tesk.services.filer.ftp_upload_file(ftp_connection, local_source_path, remote_destination_path)

tesk.services.filer.logConfig(loglevel)

tesk.services.filer.main()

tesk.services.filer.newTransput(scheme, netloc)

tesk.services.filer.process_file(ttype, filedata)

@param ttype: str Can be ‘inputs’ or ‘outputs’

tesk.services.filer.subfolders_in(whole_path)

Returns all subfolders in a path, in order

>>> subfolders_in('/')
['/']

>>> subfolders_in('/this/is/a/path')
['/this', '/this/is', '/this/is/a', '/this/is/a/path']

>>> subfolders_in('this/is/a/path')
['this', 'this/is', 'this/is/a', 'this/is/a/path']

tesk.services.filer_class module

class tesk.services.filer_class.Filer(name, data, filer_name='eu.gcr.io/tes-wes/filer', filer_version='v0.5', pullPolicyAlways=False, json_pvc=None)

Bases: object

add_netrc_mount(netrc_name='netrc')

Sets $HOME to an arbitrary location (to prevent its change as a result of runAsUser), currently hardcoded to /opt/home Mounts the secret netrc into that location: $HOME/.netrc.

add_s3_mount()

Mounts the s3 configuration file. The secret name is hardcoded and set to ‘aws-secret’.

add_volume_mount(pvc)

getContainer(i)

getEnv()

getImagePullPolicy()

getVolumeMounts()

getVolumes()

get_spec(mode, debug=False)

set_backoffLimit(limit)

Set a number of retries of a job execution (default value is 6). Use the environment variable TESK_API_TASKMASTER_ENVIRONMENT_FILER_BACKOFF_LIMIT to explicitly set this value.

Parameters:

limit – The number of retries before considering a Job as failed.

set_ftp(user, pw)

tesk.services.filer_s3 module

class tesk.services.filer_s3.S3Transput(path, url, ftype)

Bases: Transput

check_if_bucket_exists(client)

download_dir()

download_file()

extract_endpoint()

get_bucket_name_and_file_path()

If the S3 url is similar to s3://idr-bucket-1/README.txt format

get_s3_file(file_name, key)

upload_dir()

upload_file()

tesk.services.job module

class tesk.services.job.Job(body, name='task-job', namespace='default')

Bases: object

delete()

get_status(is_all_pods_running)

run_to_completion(poll_interval, check_cancelled, pod_timeout)

tesk.services.path module

tesk.services.path.containerPath(path)

tesk.services.path.fileEnabled()

tesk.services.path.getEnv(varName)

tesk.services.path.getPath(url)

tesk.services.path.getPathEnv(varName)

Gets a path from env var ‘varName’ and normalizes it

e.g. removes trailing slashes. This removes some cases from the rest of the code.

tesk.services.path.isDescendant(base, path)

Is ‘path’ is a descendant of ‘base’?

tesk.services.path.validatePath(path)

tesk.services.pvc module

class tesk.services.pvc.PVC(name='task-pvc', size_gb=1, namespace='default')

Bases: object

create()

delete()

get_subpath()

set_volume_mounts(mounts)

tesk.services.taskmaster module

tesk.services.taskmaster.append_mount(volume_mounts, name, path, pvc)

tesk.services.taskmaster.check_cancelled()

tesk.services.taskmaster.clean_on_interrupt()

tesk.services.taskmaster.dirname(iodata)

tesk.services.taskmaster.exit_cancelled(reason='Unknown reason')

tesk.services.taskmaster.generate_mounts(data, pvc)

tesk.services.taskmaster.init_pvc(data, filer)

tesk.services.taskmaster.main()

tesk.services.taskmaster.newLogger(loglevel)

tesk.services.taskmaster.newParser()

tesk.services.taskmaster.run_executor(executor, namespace, pvc=None)

tesk.services.taskmaster.run_task(data, filer_name, filer_version, have_json_pvc=False)

tesk.services.transput module

class tesk.services.transput.Transput(path, url, ftype)

Bases: object

delete()

download()

download_dir()

download_file()

upload()

upload_dir()

upload_file()

class tesk.services.transput.Type(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)

Bases: Enum

Directory = 'DIRECTORY'

File = 'FILE'

tesk.services.utils module

tesk.services.utils.pprint(data)

Module contents

_summary_:service (tesk_core) package initialization file.