Skip to content

Functions Definition Language (OSCAR)

Example:

functions:
  oscar:
  - oscar-test:
      name: plants
      memory: 2Gi
      cpu: '1.0'
      image: grycap/oscar-theano-plants
      script: plants.sh
      input:
      - storage_provider: minio.default
        path: example-workflow/in
      output:
      - storage_provider: minio.default
        path: example-workflow/med
  - oscar-test:
      name: grayify
      memory: 1Gi
      cpu: '1.0'
      image: grycap/imagemagick
      script: grayify.sh
      expose:
        min_scale: 3 
        max_scale: 7 
        port: 5000  
        cpu_threshold: 70 
      input:
      - storage_provider: minio.default
        path: example-workflow/med
      output:
      - storage_provider: minio.default
        path: example-workflow/res
      - storage_provider: onedata.my_onedata
        path: result-example-workflow
      - storage_provider: webdav.dcache
        path: example-workflow/res

storage_providers:
  onedata:
    my_onedata:
      oneprovider_host: my_provider.com
      token: my_very_secret_token
      space: my_onedata_space
  webdav:
    dcache:
      hostname: my_dcache.com
      login: my_username
      password: my_password

Top level parameters

Field Description
functions
Functions
Mandatory parameter to define a Functions Definition Language file. Note that "functions" instead of "services" has been used in order to keep compatibility with SCAR
storage_providers
StorageProviders
Parameter to define the credentials for the storage providers to be used in the services
clusters
map[string]Cluster
configuration for the OSCAR clusters that can be used as service's replicas, being the key the user-defined identifier for the cluster. Optional

Functions

Field Description
oscar
map[string]Service array
Main object with the definition of the OSCAR services to be deployed. The components of the array are Service maps, where the key of every service is the identifier of the cluster where the service (defined as the value of the entry on the map) will be deployed.

Service

Field Description
name
string
The name of the service
cluster_id
string
Identifier for the current cluster, used to specify the cluster's StorageProvider in job delegations. OSCAR-CLI sets it using the ClusterID from the FDL. Optional. (default: "")
image
string
Docker image for the service
vo
string
Virtual Organization (VO) in which the user creating the service is enrolled. Optional (default: "")
allowed_users
string array
Array EGI UIDs to grant specific users permissions over the service. If empty, the service is viewed as public. (Enabled since OSCAR version v3.0.0).
alpine
boolean
Alpine parameter to set if image is based on Alpine. If true a custom release of faas-supervisor will be used. Optional (default: false)
script
string
Local path to the user script to be executed in the service container
file_stage_in
bool
Parameter to skip the download of the input files by the FaaS Supervisor (default: false)
image_pull_secrets
string array
Array of Kubernetes secrets. Only needed to use private images located on private registries.
memory
string
Memory limit for the service following the kubernetes format. Optional (default: 256Mi)
cpu
string
CPU limit for the service following the kubernetes format. Optional (default: 0.2)
enable_gpu
bool
Parameter to enable the use of GPU for the service. Requires a device plugin deployed on the cluster (More info: Kubernetes device plugins). Optional (default: false)
enable_sgx
bool
Parameter to enable the use of SGX plugin on the cluster containers. (More info: SGX plugin documentation). Optional (default: false)
image_prefetch
bool
Parameter to enable the use of image caching. Optional (default: false)
total_memory
string
Limit for the memory used by all the service's jobs running simultaneously. Apache YuniKorn scheduler is required to work. Same format as Memory, but internally translated to MB (integer). Optional (default: "")
total_cpu
string
Limit for the virtual CPUs used by all the service's jobs running simultaneously. Apache YuniKorn scheduler is required to work. Same format as CPU, but internally translated to millicores (integer). Optional (default: "")
synchronous
SynchronousSettings
Struct to configure specific sync parameters. This settings are only applied on Knative ServerlessBackend. Optional.
expose
ExposeSettings
Struct to expose services. Optional.
replicas
Replica array
List of replicas to delegate jobs. Optional.
rescheduler_threshold
string
Time (in seconds) that a job (with replicas) can be queued before delegating it. Optional.
log_level
string
Log level for the FaaS Supervisor. Available levels: NOTSET, DEBUG, INFO, WARNING, ERROR and CRITICAL. Optional (default: INFO)
input
StorageIOConfig array
Array with the input configuration for the service. Optional
output
StorageIOConfig array
Array with the output configuration for the service. Optional
environment
EnvVarsMap
The user-defined environment variables assigned to the service. Optional
annotations
map[string]string
User-defined Kubernetes annotations to be set in job's definition. Optional
labels
map[string]string
User-defined Kubernetes labels to be set in job's definition. Optional

SynchronousSettings

Field Description
min_scale
integer
Minimum number of active replicas (pods) for the service. Optional. (default: 0)
max_scale
integer
Maximum number of active replicas (pods) for the service. Optional. (default: 0 (Unlimited))

ExposeSettings

Field Description
min_scale
integer
Minimum number of active replicas (pods) for the service. Optional. (default: 1)
max_scale
integer
Maximum number of active replicas (pods) for the service. Optional. (default: 10 (Unlimited))
port
integer
Port inside the container where the API is exposed. (value: 0 , the service wont be exposed.)
cpu_threshold
integer
Percent of use of CPU before creating other pod (default: 80 max:100)

Replica

Field Description
type
string
Type of the replica to re-send events (can be oscar or endpoint)
cluster_id
string
Identifier of the cluster as defined in the "clusters" FDL field. Only used if Type is oscar
service_name
string
Name of the service in the replica cluster. Only used if Type is oscar
url
string
URL of the endpoint to re-send events (HTTP POST). Only used if Type is endpoint
ssl_verify
boolean
Parameter to enable or disable the verification of SSL certificates. Only used if Type is endpoint. Optional. (default: true)
priority
integer
Priority value to define delegation priority. Highest priority is defined as 0. If a delegation fails, OSCAR will try to delegate to another replica with lower priority. Optional. (default: 0)
headers
map[string]string
Headers to send in delegation requests. Optional

StorageIOConfig

Field Description
storage_provider
string
Reference to the storage provider defined in storage_providers. This string is composed by the provider's name (minio, s3, onedata) and identifier (defined by the user), separated by a point (e.g. "minio.myidentifier")
path
string
Path in the storage provider. In MinIO and S3 the first directory of the specified path is translated into the bucket's name (e.g. "bucket/folder/subfolder")
suffix
string array
Array of suffixes for filtering the files to be uploaded. Only used in the output field. Optional
prefix
string array
Array of prefixes for filtering the files to be uploaded. Only used in the output field. Optional

EnvVarsMap

Field Description
Variables
map[string]string
Map to define the environment variables that will be available in the service container

StorageProviders

Field Description
minio
map[string]MinIOProvider
Map to define the credentials for a MinIO storage provider, being the key the user-defined identifier for the provider
s3
map[string]S3Provider
Map to define the credentials for a Amazon S3 storage provider, being the key the user-defined identifier for the provider
onedata
map[string]OnedataProvider
Map to define the credentials for a Onedata storage provider, being the key the user-defined identifier for the provider
webdav
map[string]WebDavProvider
Map to define the credentials for a storage provider accesible via WebDav protocol, being the key the user-defined identifier for the provider

Cluster

Field Description
endpoint
string
Endpoint of the OSCAR cluster API
auth_user
string
Username to connect to the cluster (basic auth)
auth_password
string
Password to connect to the cluster (basic auth)
ssl_verify
boolean
Parameter to enable or disable the verification of SSL certificates

MinIOProvider

Field Description
endpoint
string
MinIO endpoint
verify
bool
Verify MinIO's TLS certificates for HTTPS connections
access_key
string
Access key of the MinIO server
secret_key
string
Secret key of the MinIO server
region
string
Region of the MinIO server

S3Provider

Field Description
access_key
string
Access key of the AWS S3 service
secret_key
string
Secret key of the AWS S3 service
region
string
Region of the AWS S3 service

OnedataProvider

Field Description
oneprovider_host
string
Endpoint of the Oneprovider
token
string
Onedata access token
space
string
Name of the Onedata space

WebDAVProvider

Field Description
hostname
string
Provider hostname
login
string
Provider account username
password
string
Provider account password