Getting started

First of all, if you read this, you are a super-user of lemmings. Indeed only super-users will have to configure the machine file or create a new workflow. If you are a user, just find the dude who provided your workflow.

Read the tutorials

Go to the HowTos, and…

  • If you are confident (the type to do crosswords with a ballpoint pen), just read tutorials then proceed with the actual setup here.

  • If you are more cautious, the tutorials will help you to use lemmings without touching at your run, nor actually submitting to an HPC machine. Once you are done, use this page to create your own show…

Set up your machine configuration

Make a machine.yml configuration file adapted to your machine, here is an example. Adapt the commands and the queues:

commands:
  submit: sbatch
  get_cpu_time: sacct -j -LEMMING-JOBID- --format=Elapsed -n
  dependency: "--dependency=afterany:"
queues:
  debug:
    wall_time: 00:25:00
    # core_nb is a parameter REQUIRED BY LEMMINGS in order to correctly
    # compute the elapsed CPU time during the chain
    core_nb: 24   # core_nb = nodes*ntaks-per-node !!!
    header: |
            #!/bin/bash
            #SBATCH --partition debug
            #SBATCH --nodes=1
            #SBATCH --ntasks-per-node=24
            #SBATCH --job-name -LEMMING-JOB_NAME-
            #SBATCH --time=-LEMMING-WALL-TIME-

            -EXEC-
  debug_pj:
    wall_time: 00:02:00
    header: |
            #!/bin/bash
            #SBATCH --partition debug
            #SBATCH --nodes=1
            #SBATCH --ntasks-per-node=24
            #SBATCH --job-name -LEMMING-POSTJOB_NAME-
            #SBATCH --time=-LEMMING-WALL-TIME-

            -EXEC_PJ-

Learn more about this file in the explanations.

Create a simple recursive workflow

The recursive workflow is the easiest way. Try your own workflow for example mywaffle.py. At first,it should look like:

"""
Lemmings workflow
"""
from lemmings.chain.lemmingjob_base import LemmingJobBase

class LemmingJob(LemmingJobBase):
    """
    A lemming job follows always the same pattern.

              +--------------+ True   +-----------+                     +------------+True +-------------+
    Start----->Check on Start+------->|Prepare Run+--->Job submission--->Check on end+----->After End Job+------>Happy
              +-----+--------+    ^   +-----------+                     +------+-----+     +-------------+        End
                    |             |                                            |
                    |False        |                                            |False
                    |             |                   +-------------+          |
                                  |                   |  Prior to   |          |
                  Early           +-------------------+new iteration<----------+
                   End                                +-------------+
    """
    def prepare_run(self):
        """
        Refresh the input file
        """

        with open("./run_params.dat", "w") as fout:
            # Write down the input file, changing with the runs


    def check_on_end(self):
        """
        What to do after post job
        """
        # define a condition controlling the job end
        if condition_reached:
            return False
        else:
            return True

Add a control file mywaffle.yml looking like this:

#---Required Settings--#
exec: |                      #write here what is needed to complete the batch file
    echo "hello world"
    mpirun -n
exec_pj: |                  #write here what is needed to do between jobs
    echo "lorem ipsum"
job_queue: long         # name of the job queue based on your '{machine}.yml'
pjob_queue: short        # name of the post-job queue based on your '{machine}.yml'

Try (and correct)

Try your workflow in the same folder with:

>lemmings run mywaffle --machine-file machine.yml --job-prefix yumyum

If you are lucky, this will work on the first try. If not, use the .log files to find the problem.