Today we have a guide to ‘terminal multiplexing’ including suggestions on how to use it on computer clusters such as ShARC and Iceberg.
If the answer to any of these is “yes” then terminal multiplexing may help!
First, we need to delve a little deeper into some of the problems we are trying to solve.
(Skip over this section if you want!)
Every process (bar the
systemd process or
init process with a
process ID of 1) has a parent process. If a process is sent a signal
telling it to cleanly terminate (or ‘hang
up’) then typically its child
processes will be told to do the same.
When you SSH to a remote machine, the SSH service on that machine creates a shell for you within which you can run commands.
To illustrate, here I logged into a server and used the
to view the tree of child-parent relationships between processes. Notice
in the excerpt shown below that the SSH service (
sshd) has spawned a
(bash) shell process for my SSH session, which in turn has spawned my
pstree process: :
[will@acai ~]$ ssh sharc ... [will@sharc-login1 ~]$ pstree -a systemd --switched-root --system --deserialize 21 ... ├─sshd -D │ └─sshd │ └─sshd │ └─bash │ └─pstree -a ...
So if the SSH service decides that your connection has timed out then it
will send a signal to
bash process were to die then any child
processes started by that
bash process would also die.
If the remote servers you work with are primarily High-Performance Computing (HPC) clusters running scheduling software such as Grid Engine then you have a simple, robust way of ensuring that the sucess of your processes doesn’t depend on the reliability of your connection to the clusters: submit your work to the scheduler as batch jobs. There are many other benefits to submitting batch jobs over using interactive sessions when using such clusters but we won’t go into those here.
However, what do you do when there is no HPC-style scheduling software availble?
Neither of these allow you to easily return to interactive sessions though. For that we need terminal multiplexers.
Here we look at demonstrating the above using tmux. I recommend tmux over GNU Screen as the documentation is clearer and it makes fewer references to legacy infrastructure. Plus, it is easier to google for it! However, it may use more memory (true for older versions).
Let’s create and attach to a new tmux session, start a long-running command in it then detach and reattach to the session:
</embed> Used keys:
where <prefix> is Control and b by default. Here <prefix> d means press Control and b then release that key combination before pressing d.
In this case we started tmux on the local machine. tmux is much more useful though when you start it on a remote machine after connecting via ssh.
What else can we do with terminal multiplexers? Well, as the name implies, they can be used to view and control multiple virtual consoles from one session.
A given tmux session can have multiple windows, each of which can contain multiple panes, each of which is a virtual console!
Here’s a demonstration of creating, renaming, switching and deleting tmux windows:
</embed> Used keys:
rename a window
create a new window
switch to next window
switch to previous window
delete current window (actually deletes the current pane in the window but will also delete the window if it contains only one pane)
Now let’s look at creating, switching and deleting panes within a window:
</embed> Used keys:
split the active window vertically
split the active window horizontally
switch to pane in that direction
You can scroll back up through the terminal history of the current pane/window using:
scroll back through terminal history
If you have multiple panes side-by-side then attempt to copy text using the mouse, you’ll copy lines of characters that span all panes, which is almost certainly not going to be what you want. Instead you can
toggle the maximisation of the current pane
then copy the text you want.
Alternively, if you want to copy and paste between tmux panes/windows you can
enter copy mode
move the cursor using the arrow keys to where you want to start copying then
(in copy mode) mark start of section to copy
move the cursor keys to the end of the section you want to copy then
(in copy mode) mark end of section to copy and exit copy mode
You can then move to another pane/window and press
paste copied text
I find this mechanism very useful.
Things not covered in detail here include:
On ShARC and Iceberg can:
Starting tmux on worker nodes is also useful if you want to have multiple windows/panes on a worker node but less useful if you want to disconnect/reconnect from/to a session as if you run qrsh a second time you cannot guarantee that you will be give an interactive job on on the node you started the tmux session from.
However, note that you can have nested tmux sessions (with <prefix><prefix> <key> used to send tmux commands to the ‘inner’ tmux session).
Warning: many clusters have multiple login nodes for redundancy, with only one being the default active login node at any given time. If the active login node requires maintenance then logged-in users may be booted off and long-running processes may be terminated (before the system administrator makes a ‘standby’ login node the currently active one). Under such circumstances your tmux/Screen session may be killed.
Your interactive job (on a cluster worker node) will be terminated by the cluster’s Grid Engine job scheduler after a fixed amount of time (the default is 8 hours) but your tmux/Screen session was started on a login node so is outside the control of the cluster and will keep running indefinitely unless you kill it.
Each tmux/Screen session requires memory on the login node (which is used by all users) so to be a good HPC citizen you should:
Tip: with tmux you can ensure that you either reconnect to an existing session (with a given name) if it already exists or create a new session using: :
tmux new-session -A -s mysession
This should help avoid accidentally creating more than one tmux session.
For queries relating to collaborating with the RSE team on projects: email@example.com
Join our mailing list so as to be notified when we advertise talks and workshops by subscribing to this Google Group.