9.8 KiB
Best Practice -- Connecting Hawk to the world
Change history:
- Initial version ; Jose Gracia, 15 May 2024
TODOs:
- Add use-cases for e.g. SVN, R, etc.
- Do we recommend setting
https_proxyvariables in.bashrc? - Do we recommend one of the git remote repo protocols, i.e. SSH or HTTP, over the other?
Hawk is surrounded by a strict outgoing firewall. Users can not open arbitrary connections to hosts on the internet. Connections can only be established through a proxy host. This document describes how to set up a proxy, and how to use this proxy in a number of common use cases.
Common use cases include:
- working on remote repositories with git
- installing Python packages from PyPi, etc
- access to package registries for languages such as Golang, Rust, etc.
When following this guide, you will have to execute actions or commands on Hawk, but also on your local workstation or laptop. For clarity, examples below will use prompts hawk> andlocal>, respectively.
Setting up a proxy with ssh
Please note, that there is numerous ways to set up proxies. However, HLRS only supports the method describe in the following.
Ssh allows to create reverse tunnels from Hawk back to your local machine. In addition, it can run a proxy, more specifically a SOCKS5 proxy, on the end of this tunnel. SOCKS5 is a widely supported standard which allows many applications to use such proxies to connect to remote hosts.
Note however, that the tunnel is open on the remote side. Every user on Hawk will be able to use the tunnel and thus have a network connection to your local network. We strongly suggest to keep the tunnel open only when necessary.
Tunnels are associated to ports. In the following, you will have to specify a port which is referred by the environment variable $MY_PROXY_PORT. You may choose the port numbers freely in the suggested range 10000 to 60000. It may happen, that a port is currently used by another user ... just choose a different one.
Ok, let's do it.
The following will open a regular ssh connection to Hawk. In addition, the argument -R $MY_PROXY_PORT will establish a reverse port forwarding tunnel which opens at port $MY_PROXY_PORT on the Hawk side and provides a SOCKS5 proxy.
local> MY_PROXY_PORT=XXXX # choose a port number between 10000 and 60000
local> ssh -R $MY_PROXY_PORT hawk.hww.hlrs.de
The tunnel will remain open as long as the connection to Hawk is open. The tunnel will close as soon as your connection closes. Please note, that we do recommend to not use command line arguments to keep the connection open in the background as you may loose track of open reverse tunnels.
Once you proxy is established, you may proceed to use it for various tasks.
HTTPS/HTTP proxy variables
Requirements:
- set up a proxy with ssh as describe in this section
Most applications which support https or http protocols can use proxies. Most of them will take the proxy from the environment variables $https_proxy and $http_proxy. Sometimes, you will have to specify the proxy through command line arguments.
Example usage:
# set up proxy with ssh and continue in the terminal which just opened
hawk> MY_PROXY_PORT=XXXX # choose same port as above
hawk> curl ifconfig.me # timeout, no connection to internet
hawk> export https_proxy=socks5://localhost:$MY_PROXY_PORT # set proxy environment variables
hawk> export http_proxy=$https_proxy
hawk> curl ifconfig.me
hawk> # last command should show your IP address
hawk> # close terminal to close reverse tunnel when done
Most of the other use cases will build on the usage of these environment variables. Therefore we recommend to set them in you .bashrc. ==TODO: do we recommend this?==
Note that some applications will not honour these environment variables. Often the proxy can be provided as a command line argument as for instance --proxy $https_proxy.
Installing Python packages with pip
Requirements:
- set up a proxy with ssh as describe in this section
- set https proxy variables as describe in this section
Python's package manager pip will honour proxy settings through environment variables or command line arguments. However, it will need the package pysocks to do so. Unfortunately, pysocks is not a part of the core of Python and you may end up without it (for instance if you create a Python environment without including the system site-packages). Note, that you might have to add the command line argument --user to pip commands below.
# set up proxy with ssh and continue in the terminal which just opened
hawk> MY_PROXY_PORT=XXXX # choose same port as above
hawk> python3 -m pip install six # timeout, no connection to internet
hawk> python3 -m pip --proxy socks5://localhost:$MY_PROXY_PORT
# set https/http proxy environment variables as describe above, then
hawk> python3 -m pip install six
If you get error messages related to missing SOCKS5 support in pip, do
# set up proxy with ssh and continue in the terminal which just opened
# set https/http proxy environment variables as describe above, then
hawk> python3 -m pip install /sw/general/x86_64/development/python/share/PySocks-1.7.1-py3-none-any.whl
Using proxy with git
Git essentially supports two protocols for communication with remote repositories:
- HTTP protocol
- SSH protocol
Both protocols can routed through a SOCKS5 proxy.
==TODO: do we recommend one over the other?==
Git HTTP protocol
Requirements:
- set up a proxy with ssh as describe in this section
- set https proxy variables as describe in this section
The HTTP protocol is mostly used to clone public git repositories for read-only access (without the intension to push changes upstream). In these cases, git does not use any kind of authentication. However, git can also use authentication through the HTTP protocol.
In the simplest case, cloning public repositories without authentication just requires setting up the SOCKS5 proxy and defining the https/http proxy environment variables as explained above. Then just clone with
# set up proxy with ssh and continue in the terminal which just opened
# set https/http proxy environment variables as describe above, then
hawk> git clone https://github.com:user/project.git
hawk> git pull
To access remote repositories with authentication, we recommend using access tokens. These access tokens are supported my most repositories. Follow the links to the respective documentation: github, gitlab, gitea, etc.
We recommend to set up dedicated access tokens for access from Hawk. Then work on the repository with
# set up proxy with ssh and continue in the terminal which just opened
# set https/http proxy environment variables as describe above, then
# set up access tokens for repository
hawk> git clone https://github.com:user/project.git
# enter user name
# enter access token as password
hawk> git fetch
hawk> git pull
hawk> git push
Any of these commands may ask you for credentials such as username and password. Enter your access code when prompted for the password. You may consider managing credentials in git.
Git with SSH protocol
Requirements:
- set up a proxy with ssh as describe in this section
The other protocol for accessing remote git repositories is the SSH protocol. Repository URLs for this protocol in general look like
[user@]server:[somepath/]project.git
On github.com, for instance, it is
git@github.com:user/project.git
Repositories like these require to authenticate with an ssh public key. We recommend to create a dedicated key for access to the repository from Hawk, and to register (upload) it with the repository. The following will assume the public key file is name repo_from_hawk.pub.
When using the SSH protocol, git will start ssh to establish a connection from Hawk to the repository server (e.g. github) but fail. Instead, you need to tell ssh to routed the connection through your reverse tunnel, the one you have set up above. The easiest is to add the following to your ssh configuration file (.ssh/config):
Host github.com
# replace XXXX below with port number of your reverse tunnel
ProxyCommand ncat %h %p --proxy localhost:XXXX --proxy-type=socks5
IdentityFile ~/.ssh/repo_from_hawk
PasswordAuthentication no
Replace the XXXX above with the port number of your reverse tunnel, i.e. $MY_PROXY_PORT in the instructions above. Please note, that the proxy command above works on Hawk, but may fail on other systems. In particular, there is various versions of the netcat utility, all of which use different command line arguments.
Now all operations on remote git repositories should work as long as your reverse tunnel is up
git clone git@github.com:user/project.git
git fetch
git pull
git push
Any of these commands my ask you for credentials to unlock the ssh public key.
Again, we recommend to keep the reverse tunnel open as briefly as possible.
SVN
Seems to work after setting https/http proxy variables.
hawk> cat ~/.subversion/servers
[groups]
testhttps = svn.mplayerhq.hu
[testhttps]
http-proxy-host = localhost
http-proxy-port = XXXX
hawk> svn co https://svn.mplayerhq.hu:/ffmep/trunk
hawk> svn co svn://svn.mplayerhq.hu:/ffmep/trunk # times out