Lemmings Machine

backstage

The machine setup is like the backstage of a concert hall. It backs up different shows the workflow, but it is the same room with the same machines

Lemmings requires basic information of the environment (job scheduler) it’s being used in: “Lemmings needs to know how to interact with the job scheduler.” This is specified in the form of a {machine}.yml file which we will simply call machine.yml.

This file can be enforced through the command lines, repeating --machine-file bsolute_path_to_your_machine.ymlis tedious. Moreover this file should be the same for all users on the HPC server.

An environment variable LEMMINGS_MACHINE must be defined and can be done with the following command:

export LEMMINGS_MACHINE='absolute_path_to_your_machine.yml'

Once defined, the machine.yml file can be changed at any moment. The only thing that has to be fixed at least once is the above environment variable. The machine.yml file can be edited, for instance, as follow:

 vim $LEMMINGS_MACHINE

Examples of machine.yml files can be found in lemmings/src/lemmings/chain/machine_template/ which could already be suitable for your environment. If it’s not the case, no worries, you can easily create your own. locate your machine.yml file somewhere outside of lemmings.

Create your Machine configuration {machine}.yml

A machine.yml file, as shown below, contains two main groups

commands:
  submit: sbatch
  cancel: scancel
  get_cpu_time: sacct -j -LEMMING-JOBID- --format=Elapsed -n
  dependency: "--dependency=afterany:"
queues:
  debug:      #--> user defined name
    wall_time: '00:20:00'
    # core_nb is a parameter REQUIRED BY LEMMINGS in order to correctly
    # compute the elapsed CPU time during the chain
    core_nb: 24    # core_nb = nodes*ntaks-per-node !!!
    header: |
            #!/bin/bash
            #SBATCH --partition debug
            #SBATCH --nodes=1
            #SBATCH --ntasks-per-node=24
            #SBATCH --job-name -LEMMING-JOB_NAME-
            #SBATCH --time=-LEMMING-WALL-TIME-
  
            -EXEC-
  debug_pj:   #--> user defined name
    wall_time: '00:02:00'
    header: |
           	#!/bin/bash
            #SBATCH --partition debug
            #SBATCH --nodes=1
            #SBATCH --ntasks-per-node=1
            #SBATCH --job-name -LEMMING-POSTJOB_NAME-
            #SBATCH --time=-LEMMING-WALL-TIME-

            -EXEC_PJ-

commands:
- groups a basic set of commands to interact with the job scheduler
- has to be defined only once per machine you plan to run on
queues:
- Groups information on the queues that the user wishes to use and exist on the cluster:
  - It is basically a list of aliases containing information on partitions to run on and how many resources we want to use.
  - In the above example we have a debug and a debug_pj queue which are both using the “debug” partition but different wall times and resources.
- The commands to request the resources are introduced through the header and dependent on the type of machine environment.
- The wall_time parameter is the wall clock time limit of the machine queue. It could be in HH:MM:SS format or a Float in seconds.
- The core_nb parameter represents the number of cores a given queue will use. This is a parameter REQUIRED BY LEMMINGS in order to CORRECTLY compute the elapsed CPUh cost. Be careful, sometimes in the batch (header part) you use directly core_nb, sometimes you have to use “node number” * “ntasks-per-node”. In any case, the core number will ONLY be taken via the core_nb parameter and NEVER in the BATCH parameters.
- The machine.yml requires at least two queues : job and post job (or pjob). This is simply a consequence of the working strategy of lemmings. A job queue is indicated by the -EXEC- keyword at the end of the header whereas -EXEC_PJ- indicates a pjob queue. These two strings will be replaced by the associated executable information specified in the workflow.yml file as exemplified below.
```
# associated with -EXEC-
exec: |
      source "path/to/virtualenv/bin/activate"
      module load avbp
      mpirun -np $SLURM_NPROCS path/to/exec
# associated with -EXEC_PJ-
exec_pj: |
      source "path/to/virtualenv/bin/activate"
```

The user can define as many queues as wished (with the required information), e.g. a queue called prod_long which has a wall_time of 24h, a prod_short with wall_time of 5h, etc., as longs as the value does not exceed the maximum limit set by the machine on the specific partition. Moreover, multiple queues can be associated with the same partition. The machine.yml should be seen as a list of options, or better, option pairs given the job / pjob link, from which the user can select in the workflow definition.