Our Slurm configuration aims to ensure efficient and fair usage of /tmp directories, which are private to each job and cleared up after the job completes.
Private Local /tmp Directories
On the local storage of compute nodes, /tmp is private to each job.
Storage is a consumable shared resource, meaning the storage used by one job cannot be used by another job. Therefore, it is critical that Slurm cleans up after each job to ensure all space on the local node is available for the next job. This cleanup is managed by the job_container/tmpfs Slurm plugin.
This plugin creates a Linux namespace for each job and bind mounts /tmp to a location on the local storage. This mount is only visible to the currently running job, and each job, even from the same user, gets its own /tmp directory. Once a job terminates, Slurm removes the directory and all of its content, ensuring that all temporary data is removed.