Remote terminal application that allows roaming, supports intermittent connectivity, and provides intelligent local echo and line editing of user keystrokes.
Mosh is a replacement for SSH. It's more robust and responsive, especially over Wi-Fi, cellular, and long-distance links.
Mosh is free software, available for GNU/Linux, FreeBSD, Solaris, Mac OS X, and Android.
Mosh automatically roams as you move between Internet connections. Use Wi-Fi on the train, Ethernet in a hotel, and LTE on a beach: you'll stay logged in. Most network programs lose their connections after roaming, including SSH and Web apps like Gmail. Mosh is different.
With Mosh, you can put your laptop to sleep and wake it up later, keeping your connection intact. If your Internet connection drops, Mosh will warn you — but the connection resumes when network service comes back.
SSH waits for the server's reply before showing you your own typing. That can make for a lousy user interface. Mosh is different: it gives an instant response to typing, deleting, and line editing. It does this adaptively and works even in full-screen programs like emacs and vim. On a bad connection, outstanding predictions are underlined so you won't be misled.
You don't need to be the superuser to install or run Mosh. The client and server are executables run by an ordinary user and last only for the life of the connection.
Mosh doesn't listen on network ports or authenticate users. The mosh client logs in to the server via SSH, and users present the same credentials (e.g., password, public key) as before. Then Mosh runs the mosh-server remotely and connects to it over UDP.
Mosh is a command-line program, like ssh. You can use it inside xterm, gnome-terminal, urxvt, Terminal.app, iTerm, emacs, screen, or tmux. But mosh was designed from scratch and supports just one character set: UTF-8. It fixes Unicode bugs in other terminals and in SSH.
Unlike SSH, mosh's UDP-based protocol handles packet loss gracefully, and sets the frame rate based on network conditions. Mosh doesn't fill up network buffers, so Control-C always works to halt a runaway process.
$ sudo apt-get install python-software-properties $ sudo add-apt-repository ppa:keithw/mosh $ sudo apt-get update $ sudo apt-get install mosh
Ubuntu also includes older versions of mosh in the universe repository (12.04 and later) and backports repository (10.04–11.10).
Operating system logos are trademarks or registered trademarks and are displayed for identification only. The vendors shown aren't affiliated with and haven't endorsed Mosh.
Extract mosh-1.2.4.tar.gz, then
$ cd mosh-1.2.4 $ ./configure $ make # make install
$ git clone https://github.com/keithw/mosh $ cd mosh $ ./autogen.sh $ ./configure $ make # make install
|Protocol Buffers||protobuf-compiler, libprotobuf-dev|
mosh-client receives an AES session key as an environment
variable. If you are porting Mosh to a new operating system, please make sure that a
running process's environment variables are not readable by other users. We have
confirmed that this is the case on GNU/Linux, OS X, and FreeBSD.
$ mosh chewbacca.norad.mil
Mosh will log the user in via SSH, then start a connection on a UDP port between 60000 and 61000.
$ mosh email@example.com
$ mosh --server=/tmp/mosh-server r2d2
The user can specify an alternate path for the
mosh-server on the remote host. The server binary can even
be installed in the user's home directory.
$ mosh -p 1234 darth
Useful when the server is behind a port-forwarder or NAT.
$ mosh --ssh="ssh -p 2222" figrindan
$ mosh --ssh="~/bin/ssh -i ./identity" fett
$ mosh --predict=never niennunb
-n switch is a synonym. By contrast,
will enable instant local echo even on low-delay
$ mosh pello -- screen -dr
This reattaches to a long-running screen session.
Normally, logout or exit on the remote host will close
the session. Mosh accepts the escape sequence
. (typically typed with Control-Shift-6, then a
period) to end the connection forcibly. To send a
literal Ctrl-^, type
More details can be found in
mosh-server(1) manual pages.
The Mosh research paper describes the design and evaluation of Mosh in more detail than you may want. The paper was presented at the 2012 USENIX Annual Technical Conference, held June 13–15, 2012, in sunny Boston, Mass.
(Why you should trust Mosh with your remote terminal needs: we worry about details so obscure, even USENIX reviewers don't want to hear about them.)
Remote-shell protocols traditionally work by conveying a byte-stream from the server to the client, to be interpreted by the client's terminal. (This includes TELNET, RLOGIN, and SSH.) Mosh works differently and at a different layer. With Mosh, the server and client both maintain a snapshot of the current screen state. The problem becomes one of state-synchronization: getting the client to the most recent server-side screen as efficiently as possible.
This is accomplished using a new protocol called the State Synchronization Protocol, for which Mosh is the first application. SSP runs over UDP, synchronizing the state of any object from one host to another. Datagrams are encrypted and authenticated using AES-128 in OCB mode. While SSP takes care of the networking protocol, it is the implementation of the object being synchronized that defines the ultimate semantics of the protocol.
Roaming with SSP becomes easy: the client sends datagrams to the server with increasing sequence numbers, including a "heartbeat" at least once every three seconds. Every time the server receives an authentic packet from the client with a sequence number higher than any it has previously received, the IP source address of that packet becomes the server's new target for its outgoing packets. By doing roaming “statelessly” in this manner, roaming works in and out of NATs, even ones that may themselves be roaming. Roaming works even when the client is not aware that its Internet-visible IP address has changed. The heartbeats allow Mosh to inform the user when it hasn't heard from the server in a while (unlike SSH, where users may be unaware of a dropped connection until they try to type).
Mosh runs two copies of SSP, one in each direction of the connection. The connection from client to server synchronizes an object that represents the keys typed by the user, and with TCP-like semantics. The connection from server to client synchronizes an object that represent the current screen state, and the goal is always to convey the client to the most recent server-side state, possibly skipping intermediate frames.
Because SSP works at the object layer and can control the rate of synchronization (in other words, the frame rate), it does not need to send every byte it receives from the application. That means Mosh can regulate the frames so as not to fill up network buffers, retaining the responsiveness of the connection and making sure Control-C always works quickly. Protocols that must send every byte can't do this.
One benefit of working at the terminal layer was the opportunity to build a clean UTF-8 terminal emulator from scratch. Mosh fixes several Unicode bugs in existing terminals and in SSH, and was designed as a fresh start to try to be robust and correct even for pathological inputs.
Only Mosh and the OS X Terminal correctly handle a Unicode combining character in the first column.
Only Mosh will never get stuck in hieroglyphs when a nasty program writes to the terminal. (See Markus Kuhn's discussion of the relationship between ISO 2022 and UTF-8.)
Only Mosh and GNOME Terminal have a defensible rendering when Unicode mixes with an ECMA-48/ANSI escape sequence. The OS X Terminal unwisely tries to normalize its input before the vt500 state machine, causing it to misinterpret and become unusable after receiving the following input!* (This also means the OS X Terminal's interpretation of the incoming octet stream varies depending on how the incoming octets are split across TCP segments, because the normalization only looks ahead to available bytes.)
* We earlier wrote that this misbehaving sequence "crashes" the OS X Terminal.app. This was mistaken—instead, Terminal.app interprets the escape sequence as shutting off keyboard input, and because of an unrelated bug in Terminal.app, it is not possible for the user to restore keyboard input by resetting the terminal from the menu.
In the POSIX framework, the kernel needs to know whether the user is typing in an 8-bit character set or in UTF-8, because in canonical mode (i.e. "cooked" mode), the kernel needs to be able to delete a typed multibyte character sequence from an input buffer. On OS X and Linux, this is done with the "IUTF8" termios flag.) (See diagnostic explaining the need for this flag.)
Mosh sets the IUTF8 flag when possible and stubbornly refuses to start up unless the user has a UTF-8-clean environment. SSH does not set the IUTF8 flag, which can lead to garbage in input buffers.
The other major benefit of working at the terminal-emulation layer is that the Mosh client is free to scribble on the local screen without lasting consequence. We use this to implement intelligent local echo. The client runs a predictive model in the background of the server's behavior, hypothesizing that each keystroke will be echoed at the cursor location and that the backspace and left- and right-arrow keys will have their traditional effect. But only when a prediction is confirmed by the server are these effects actually shown to the user. (In addition, by default predictions are only displayed on high-delay connections or during a network “glitch.”) Predictions are done in epochs: when the user does something that might alter the echo behavior — like hit ESC or carriage return or an up- or down-arrow — Mosh goes back into making background predictions until a prediction from the new batch can be confirmed as correct.
Thus, unlike previous attempts at local echo with TELNET and RLOGIN, Mosh's local echo can be used everywhere, even in full-screen programs like emacs and vi.
We evaluated Mosh using traces contributed by six users, covering about 40 hours of real-world usage and including 9,986 total keystrokes. These traces included the timing and contents of all writes from the user to the host and vice versa. The users were asked to contribute "typical, real-world sessions." In practice, the traces include use of popular programs such as the bash shell and zsh shells, the alpine and mutt e-mail clients, the emacs and vim text editors, the irssi and barnowl chat clients, the links text-mode Web browser, and several programs unique to each user.
To evaluate typical usage of a "mobile" terminal, we replayed the traces over an otherwise unloaded Sprint commercial EV-DO (3G) cellular Internet connection in Cambridge, Mass. A client-side process played the user portion of the traces, and a server-side process waited for the expected user input and then replied (in time) with the prerecorded server output. We speeded up long periods with no activity. The average round-trip time on the link was about half a second.
We replayed the traces over two different transports, SSH and Mosh, and recorded the user interface response latency to each simulated user keystroke. The Mosh predictive algorithm was frozen prior to collecting the traces and was not adjusted in response to their contents or results.
Mosh reduced the median keystroke response time from 503 ms to nearly instant (because more than 70% of the keystrokes could be immediately displayed), and reduced the mean keystroke response time from 515 ms to 173 ms. Qualitatively, Mosh makes remote servers "feel" more like the local machine!
Mosh was written by Keith Winstein, along with Anders Kaseorg, Quentin Smith, Richard Tibbetts and Keegan McAllister.
Practical latency on the Internet is on the increase, with the rise of bufferbloat and sophisticated wireless links that optimize for throughput over delay. And roaming is more common than ever, now that laptops and handheld devices have largely displaced desktops. SSH is great, but frustrating to use when you want to change IP addresses or have a long-delay link or a dodgy connection.
Moreover, TELNET had some good things going for it — a local-echo mode and a well-defined network virtual terminal. Even today, SSH doesn't properly support UTF-8 end-to-end on a POSIX system.
We think so. The design principles that Mosh stands for are conservative: warning the user if the state being displayed is out of date, serializing and checkpointing all transactions so that if there are no warnings, the user knows every prior transaction has succeeded, and handling expected events (like roaming from one WiFi network to another) gracefully.
Those don't seem too controversial, but fancy apps like Gmail-in-Chromium or on Android still behave atrociously on dodgy connections or after switching IP addresses. (Have you ever had Gmail leave an e-mail message in "Sending..." for ten hours while merrily retrieving new mail and not indicating any kind of error? Us too.) We think there may be considerable room for improvement in many network user interfaces from the application of these values.
To diagnose the problem, run
locale on the local
ssh remotehost locale. To use Mosh,
both sides of the connection will need to show a UTF-8 locale, like
On many systems, SSH will transfer the locale-related
environment variables, which are then inherited by
mosh-server. If this mechanism fails, Mosh (as of
version 1.2) will pass the variables itself. If neither
mechanism is successful, you can do something like
mosh remotehost --server="LANG=en_US.UTF-8 mosh-server"
en_US.UTF-8 does not exist on the remote server,
you can replace this with a UTF-8 locale that does exist. You
may also need to set LANG locally for the benefit of
mosh-client. It is possible that the local and
remote machines will need different locale names. See also this GitHub
This means that
mosh was able to start
mosh-server successfully on the remote machine, but the client is
not able to communicate with the server. This generally means that
some type of firewall is
blocking the UDP packets between the client and the server. If you
had to forward TCP port 22 on a NAT for SSH, then you will have to
forward UDP ports as well. Mosh will use the first available
UDP port, starting at 60001 and stopping at 60999. If you are only
going to have a small handful of concurrent sessions on a server, then you can
forward a smaller range of ports (e.g., 60000 to 60010).
Tools like netstat, netcat, socat, and tcpdump can be useful for debugging networking and firewall problems.
We're really not UTF-8 zealots. But it's a lot easier to correctly implement one terminal emulator than to try to do the right thing in a variety of difficult edge cases. (This is what GNU screen tries to do, and in our experience it leads to some very tricky-to-debug situations.) So mosh just won't start up until the user has everything configured for a UTF-8-clean pathway. It may be annoying, but it also probably reduces frustration down the road. (Unfortunately an 8-bit vt220 and a UTF-8 vt220 are different and incompatible terminal types; the UTF-8 goes in underneath the vt220 state machine.)
As of Mosh 1.2, you can pass arguments to
ssh like so:
mosh remotehost --ssh="ssh -p 2222"
Or configure a host alias in
~/.ssh/config with a
Port directive. Mosh will respect that too.
Please make sure that mosh is installed on the client, and mosh (or at least mosh-server) is installed on the server you are trying to connect to. If you install mosh-server in your home directory, please see the "Server binary outside path" instructions in the Usage section, above.
In some configurations, SSH canonicalizes the hostname before passing it to the Kerberos GSSAPI plugin. This breaks for Mosh, because the initial forward DNS lookup is done by the Mosh wrapper script. To work around this, invoke Mosh as
mosh remotehost --ssh="ssh -o GSSAPITrustDns=no"
This will often fail on a round-robin DNS setup. In that case it is probably best to pick a specific host from the round-robin pool.
Mosh 1.2 synchronizes only the visible state of the terminal. Mosh 1.3 will have complete scrollback support; see this issue and the others which are linked from there. For now, the workaround is to use screen or tmux on the remote side.
Make sure you are running mosh in a terminal that advertises itself as 256-color capable. (This generally means TERM will be xterm-256color or screen-256color-bce.)
No. Mosh is actively used and has been read over by security-minded crypto nerds who think its design is reasonable, but any novel datagram protocol is going to have to prove itself, and SSP is no exception. We use the reference implementations of AES-128 and OCB, and we welcome your eyes on the code. We think the radical simplicity of the design is an advantage, but of course others have thought that and have been wrong. We don't doubt it will (properly!) take time for the security community to get comfortable with mosh.
Yes, it works great, but please remember to open up UDP ports 60000–61000 on the EC2 firewall.
After you run
mosh user@server, if successful you will be dropped into your login
shell on the remote machine.
If you want
to check that mosh is being used instead of ssh, try typing
to suspend the session (with mosh 1.2.4 or later on the client). Running
fg will then return.
mosh command is a wrapper script that is designed to be the primary way that
you use mosh. In most cases, you can simply just replace "ssh" with "mosh" in your command line.
Behind the scenes, the
mosh wrapper script will SSH to the server, start up
mosh-server, and then close the SSH connection. Then it will start up the
mosh-client executable on the client, passing it the necessary information for
it to connect to the newly spawned
In normal usage,
mosh-server don't need to be run directly.
mosh wrapper script isn't working for you, you can try running
mosh-server programs separately to
form a connection. This can be a useful debugging technique.
1. Log in to the remote host, and run
It will give output like:
$ mosh-server MOSH CONNECT 60004 4NeCCgvZFe2RnPgrcU1PQw mosh-server (mosh 1.1.3) Copyright 2012 Keith Winstein <firstname.lastname@example.org> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. [mosh-server detached, pid = 30261]
2. On the local host, run:
$ MOSH_KEY=key mosh-client remote-IP remote-PORT
where "key" is the 22-byte string printed by mosh-server (in this example, "4NeCCgvZFe2RnPgrcU1PQw"), "remote-PORT" is the port number given by the server (60004 in this case), and "remote-IP" is the IP address of the server. You can look up the server's IP address with "host remotehost".
3. If all goes well, you should have a working Mosh connection. Information about where the process fails can help us debug why Mosh isn't working for you.
This bug is fixed in Mosh 1.2. Thanks to Ed Schouten and Peter Jeremy for tracking this down.
We're very grateful for assistance and support from:
Mosh development and discussion. Sign up or view archives at http://mailman.mit.edu/mailman/listinfo/mosh-devel.
Mosh user discussion and site best practices. Sign up or view archives at http://mailman.mit.edu/mailman/listinfo/mosh-users.
At the recommendation of the security community, confidential security-related matters may be sent to:
pub 4096R/FE254C69 2012-02-05 [expires: 2015-02-04] Key fingerprint = B1A4 7069 121F 6642 BB3D 7F3E 20B7 283A FE25 4C69