Commands

List of useful Slurm commands

• Check Slurm cluster state:

  • sinfo will display general cluster information.
  • sinfo -s will display the summary of cluster information.
  • sinfo -N -l will display the status of each node
  • scontrol show nodes will display detailed information about the state of each node, helpful for debugging purposes
  • scontrol show partitions will display all available partitions
  • scontrol show partition <partition_name> will display detailed partition information regarding the partition

• Check Slurm user information:

  • sacctmgr show user will display general user information
  • sacctmgr show association will display user associations (to quotas, resource limitations, etc)

• Check Slurm job states:

  • watch squeue -u $USER will display information about jobs that are scheduled for execution or are currently running
  • squeue -u $USER -o "%.18i %.20P %.15j %.8u %.8T %.10M %.20R" will also display information about the scheduled jobs with more details
  • watch sacct will display the current state of each job (press Ctrl+C to exit)
  • sacct -N slurm-worker-cpu-1 will show the list of executed jobs in which the given node was involved
  • sacct -u konrad --format=JobID,JobName,Partition,State,Elapsed -S now-1hour will display recent job history per user
  • scontrol show job <job_id> will display a general information regarding the job
  • sstat <job_id> will display a summary information regarding the job
  • scontrol getaddrs $(scontrol show job <job_id> | grep "NodeList=slurm" | cut -d '=' -f 2) | col2 | cut -d ':' -f 1 will display the worker node IP on which the current interactive job is running

• Schedule Slurm jobs:

  • use sbatch test_job.batch to schedule a job
  • to schedule a job against one particular partition, for example a GPU partition, use sbatch test_job.batch -p batch_gpu_g2.2xlarge_32 command
  • use scancel <job_id> to cancel any jobs

• Check resource limitations:

  • use the quota -u $USER -s command to check the storage space available
  • sacctmgr show qos format=name%30,MaxJobsPerUser%30,MaxSubmitJobsPerUser%30,MaxTRESPerJob%30 will display detailed information regarding the partition-level resource limitations

---

Wigner site commands

For general background on the Wigner shared storage and GPU nodes, see the Wigner site section.

• Copying files to /wigner_storage:

  • as with the main storage, always initiate transfers from your home folder, then move files into your personal folder under /wigner_storage

    # upload a file to your home folder with SCP
    scp -i SSH_KEY ~/SrcFile username@slurm.science-cloud.hu:DstFile
    
    # copy the file from your home folder into your Wigner storage folder
    cp ~/DstFile /wigner_storage/username/
    
    # upload a folder directly into your Wigner storage folder with rsync
    rsync -a -e "ssh -i SSH_KEY" SrcFolder username@slurm.science-cloud.hu:/wigner_storage/username/
    

  • Note: write as your own user, not as root — root-owned writes are remapped and will fail against your own directories.

• Check Wigner storage quota:

  • use the quota wigner command to check your storage usage on /wigner_storage
  • usage is capped at a 100 GB soft limit and a 110 GB hard limit per user; usage above the soft limit is only allowed for a grace period before writes are refused

• Schedule a job against the Wigner GPU partition:

  • use sbatch -p batch_gpu_wigner together with --gres=gpu:1 to target the Wigner A100 GPU nodes, for example:

    #!/bin/bash
    #SBATCH --job-name=wigner_job            # Job name
    #SBATCH --output=wigner_job_%j.out       # Output file name
    #SBATCH --error=wigner_job_%j.err        # Error file name
    #SBATCH --partition=batch_gpu_wigner     # Wigner GPU partition
    #SBATCH --gres=gpu:1                     # Request 1 GPU
    #SBATCH --time=1-00:00:00                # Time limit: 1 day (max for this partition)
    
    srun hostname
    srun nvidia-smi
    

  • Note: the batch_gpu_wigner partition is currently running in beta, with access limited to a restricted set of users — it will be opened up to all users soon.

---