stjohn_piano 2021-03-09 no This article contains my log of: - setting up a new server instance running Ubuntu 20. - installing and configuring Nginx and Gunicorn. ### STEP: Set up new server instance. On DigitalOcean, create a new droplet. edgecase3: - Ubuntu 20.04 (LTS) x64 - 1 vCPUs - 1GB / 25GB Disk - ($5/mo) - Authentication: SSH keys -- I have already uploaded the relevant SSH keys to my DigitalOcean account: 1) A key on my WSL Ubuntu instance, and 2) a key for my Putty instance. Select both of them. - IP address: 157.245.39.46 Work machine: - Name: Judgement - Windows 10 - Windows Subsystem for Linux (WSL): Ubuntu 16.04 On WSL Ubuntu, I can use this command to log in: ssh root@157.245.39.46 ### STEP: Set up a domain. I've already bought this domain: edgecase.ink pointed it at DigitalOcean's nameservers: ns1.digitalocean.com ns2.digitalocean.com ns3.digitalocean.com and added it to the domain list on my DigitalOcean account, via Manage / Networking / Add Domain. Go to Projects / \ / Domains, and click edgecase.ink. It already has 3 Nameserver records pointing to the DigitalOcean nameservers. Create an A record for this domain. Use "@" for hostname. Set "Will direct to" to the droplet edgecase3. Set "TTL (Seconds)" to 3600. Click Create Record. 3600 seconds = 1 hour. After the new A record has propagated (about 1 hour), I will be able to use this command to log in: ssh root@edgecase.ink Excerpt from the "Create new record" / A page: Use @ to create the record at the root of the domain or enter a hostname to create it elsewhere. A records are for IPv4 addresses only and tell a request where your domain should direct to. I've also added a CNAME record that redirects all subdomains to the main domain. Details: Type=CNAME, Hostname=*.edgecase.ink, "Is An Alias Of"=edgecase.ink, TTL=43200. ### STEP: Set up Putty. Goals: - Configure Putty for SSH access to a DigitalOcean droplet. - Use a non-root user and set colour preferences. I'm using Putty 0.72. I have already created a key for Putty, and set the droplet to accept it for authentication, but for future reference I'll record the details here as Part 0. Part 0: Create an SSH key for Putty to use to log in to a DigitalOcean server. Source material: http://www.digitalocean.com/docs/droplets/how-to/add-ssh-keys/create-with-putty http://www.digitalocean.com/docs/droplets/how-to/add-ssh-keys/to-existing-droplet A Putty-generated SSH key has to be reformatted in order to be uploaded onto a DigitalOcean account. Using puttygen, generate a new private key and save it as e.g. putty_id_rsa_for_digital_ocean.ppk Do not use the "Save public key" button. The format PuTTYGen uses when it saves the public key is incompatible with the OpenSSH authorized_keys files used for SSH key authentication on Linux servers. The public key is displayed in the text field under "Key / Public key for pasting into OpenSSH authorized_keys file". Select and copy the public key and save it as a file on your local machine. Example name: putty_id_rsa_for_digital_ocean.pub If, in future, you need to re-generate the OpenSSH-formatted public key from the private key, you can open PuTTYGen, choose Load, and open the private key file. The public key will be displayed in the same text field as before. Open the public key file and copy the text. Then, on your DigitalOcean account, go to Account / Security, click "Add SSH Key", and paste in the key text. Part 1: Create a new Putty configuration for connecting to this droplet. SHORTCUT: Copy the configuration from a previous session. - In Session, select a previous session and click Load. Change the session name and the IP address, and click Save. - Change Connection / Data / Auto-login username to "root". - Click Open. - New window: PuTTY Security Alert ("The server's host key is not cached in the registry"). Click Yes. - Putty terminal window opens and logs in. - Skip to Part 2. My default new configuration is shown below. I have not listed all of the available settings. New Putty configuration: - Session: -- Host Name (or IP address) = [the IP address of the new droplet] -- [default] Port = 22 -- [default] Connection type = SSH -- Under "Saved Sessions", type a new name [e.g. the hostname of the new droplet, in this case "edgecase3"] and click Save. - Window: -- [default] Columns = 80 -- Rows = 40 --- [Note: You can always resize the Putty window after it has started.] - Window / Behaviour: -- Tick for "Full screen on Alt-Enter" - Window / Selection: -- Mouse paste action = No action -- {Ctrl,Shift} + Ins = No action -- Ctrl + Shift + {C, V} = System Clipboard - Connection / Data: -- Auto-login username = root - Connection / SSH / Auth: -- Private key file for authentication = Click "Browse" and browse to the ssh private key file on your local machine e.g. work/ssh/putty_id_rsa_for_digital_ocean.ppk. Additionally, my colour preferences: - Window / Colours: -- Note: Colours are RGB (Red, Green, Blue). -- Default Foreground = 187,187,187 -- Default Bold Foreground = 255,255,255 -- Default Background = 0,0,0 -- Default Bold Background = 85,85,85 -- Cursor Text = 0,0,0 -- Cursor Colour = 0,255,0 -- ANSI Black = 0,0,0 -- ANSI Black Bold = 85,85,85 -- ANSI Red = 255,128,64 -- ANSI Red Bold = 255,128,64 -- ANSI Green = 85,255,85 -- ANSI Green Bold = 85,255,85 -- ANSI Yellow = 187,187,0 -- ANSI Yellow Bold = 255,255,85 -- ANSI Blue = 60,160,255 -- ANSI Blue Bold = 60,160,255 -- ANSI Magenta = 187,0,187 -- ANSI Magenta Bold = 255,85,255 -- ANSI Cyan = 0,187,187 -- ANSI Cyan Bold = 85,255,255 -- ANSI White = 187,187,187 -- ANSI White Bold = 255,255,255 To preserve this configuration, go to Session, select the correct session name (e.g. edgecase3), and click Save. To connect to the droplet, go to Session and click Open. A new window will open: - Title: PuTTY Security Alert - Text: The server's host key is not cached in the registry. You have no guarantee that the server is the computer you think it is. The server's ssh-ed25519 key fingerprint is: [deleted] If you trust this host, hit Yes to add the key to PuTTY's cache and carry on connecting. If you want to carry on connecting just once, without adding the key to the cache, hit No. If you do not trust this host, hit Cancel to abandon the connection. Click Yes. Putty terminal window opens, with the prompt "login as". Type "root" and press enter. The next line should be something like: Authenticating with public key "rsa-key-20190730" Part 2: Create a new non-root user account, give it sudo privileges, create a .ssh directory if it doesn't exist, and copy the authorized_keys file from the root user's .ssh directory (this will allow us to log in to the new user account directly with Putty). Command sequence: [as the user 'root'] adduser stjohn [you'll need to supply a password] visudo In the visudo interface, add the following line at the end of the file: stjohn ALL=(ALL) NOPASSWD: ALL login stjohn [as the user 'stjohn'] sudo -l The output should contain the following text, which will confirm that the user 'stjohn' has sudo privileges: User stjohn may run the following commands on edgecase3: (ALL) NOPASSWD: ALL mkdir .ssh sudo cp /root/.ssh/authorized_keys .ssh/ sudo chown stjohn:stjohn .ssh/authorized_keys exit [as the user 'root'] exit [Putty window closes] Open Putty again. In Session, load edgecase3. Change this setting: - Connection / Data: -- Auto-login username = [the new username e.g. "stjohn"] Go to Session, select edgecase3, and click Save. Click Open. A new Putty window should open. It should use the selected username e.g. "stjohn", and log in automatically using the ssh public key. ### STEP: Configure vim. Below is my current .vimrc file. Open ~/.vimrc, run :set formatoptions-=cro to stop automatic commenting, and paste in the text below. Afterwards, configure the root user's vim by copying the file to its home directory: sudo cp ~/.vimrc /root/ """ START NOTES " A line starting with a double quotation mark (") is a comment. " When in vim, to reload the .vimrc file, without restarting or leaving vim, run this command: " :source ~/.vimrc " If you are using putty to connect to a screen session on another machine, the function key effects in this file won't work unless the TERM environment variable on the remote machine is set to 'putty'. " To set this while in Bash: Run the commmand { $ TERM=putty } " To set this while in screen: Run the command { :set term=putty } """ END NOTES """ START VIM CONFIGURATION " Keep the existing indentation level when pressing enter to move to the next line. set autoindent " Show existing tabs with 4 spaces width. set tabstop=4 " When using '\>' or '\<' to indent, use 4 spaces width. set shiftwidth=4 " Show line numbers. set number " Pressing the space bar will now allow the insertion of a single character while in command mode. :nnoremap \ i_\r " Make F2 key into a set paste / nopaste toggle, with visual feedback shown in the status line. nnoremap \ :set invpaste paste?\ set pastetoggle=\ set showmode " Setting the 'paste' toggle turns off vim's automatic addition of indentation, which is good if you are pasting in multi-line text that already contains its own indentation (i.e. various tabs and spaces). " Make F3 key into a set number / nonumber toggle, with visual feedback shown in the status line. " nnoremap line is for command mode, inoremap line is for insert mode. nnoremap \ :set invnumber\ inoremap \ \:set invnumber\ """ END VIM CONFIGURATION ### STEP: Install nginx. Check if nginx is already installed. apt list --installed 2\>&1 | grep nginx Search for available installation options. apt search ^nginx --names-only Install nginx. sudo apt-get install nginx stjohn@edgecase3:/usr$ nginx -v nginx version: nginx/1.18.0 (Ubuntu) Browse to http://edgecase.ink The "Welcome to nginx!" page is displayed. This default page is located at: /var/www/html/index.nginx-debian.html Create a new site config for nginx. server { listen 80 default_server; server_name _; root /srv/edgecase; location /hello_nginx { return 200 'hello world from nginx\\n'; } location / { # Pass everything to gunicorn via a socket. proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_pass http://unix:/run/gunicorn/gunicorn.sock; } } Create a new file /etc/nginx/sites-available/edgecase and paste the above text into it. Next, soft-link this site config into the sites-enabled directory. sudo ln --symbolic sites-available/edgecase sites-enabled/ Example: stjohn@edgecase3:/etc/nginx$ ls -1 sites-enabled/ default stjohn@edgecase3:/etc/nginx$ sudo ln --symbolic /etc/nginx/sites-available/edgecase sites-enabled/ stjohn@edgecase3:/etc/nginx$ sudo rm sites-enabled/default stjohn@edgecase3:/etc/nginx$ ls -1 sites-enabled/ edgecase Check that the new configuration is ok: stjohn@edgecase3:/etc/nginx$ sudo nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful Tell nginx service to reload its configuration: sudo nginx -s reload Test that we can talk to nginx on edgecase3 from an external machine: stjohn@judgement:~$ curl edgecase.ink/hello_nginx hello world from nginx We haven't set up the gunicorn server yet, so this is the result we get for any other request: stjohn@judgement:~$ curl edgecase.ink \ \\502 Bad Gateway\\ \ \\502 Bad Gateway\\ \\nginx/1.18.0 (Ubuntu)\ \ \ The access log and error log file paths are set in the Nginx configuration file: /etc/nginx/nginx.conf. By default, they are /var/log/nginx/access.log /var/log/nginx/error.log ### STEP: Install gunicorn. Check if gunicorn is already installed. apt list --installed 2\>&1 | grep gunicorn Search for available installation options. apt search ^gunicorn --names-only Install nginx. sudo apt-get install gunicorn stjohn@edgecase3:/etc/nginx$ apt list --installed 2\>&1 | grep gunicorn gunicorn/focal,now 20.0.4-3 all [installed] python3-gunicorn/focal,now 20.0.4-3 all [installed,automatic] Unlike nginx, gunicorn is installed only as a package, not as a working systemd service. Find the location paths for the installed files for the gunicorn package: dpkg --listfiles gunicorn The binary is located at: /usr/bin/gunicorn stjohn@edgecase3:/usr$ gunicorn --version gunicorn (version 20.0.4) stjohn@edgecase3:/usr$ python3 --version Python 3.8.5 stjohn@edgecase3:/usr$ python --version Command 'python' not found, did you mean: command 'python3' from deb python3 command 'python' from deb python-is-python3 stjohn@edgecase3:/usr$ sudo apt install python-is-python3 stjohn@edgecase3:/usr$ python --version Python 3.8.5 From the output of ps -ef | grep nginx we can see that nginx is running with a master process (owned by root) and a worker process (owned by www-data). sudo find / -name "nginx.service" The location of nginx systemd service file is: /usr/lib/systemd/system/nginx.service Create a new user for gunicorn. Let's call it 'alice'. [you'll need to supply a password] "The Unicorn looked dreamily at Alice, and said "Talk, child." Alice could not help her lips curling up into a smile as she began: "Do you know, I always thought Unicorns were fabulous monsters, too? I never saw one alive before!" "Well, now that we have seen each other," said the Unicorn, "If you'll believe in me, I'll believe in you. Is that a bargain?" If gunicorn needs some additional python packages, these can be installed in the 'alice' user's home directory. sudo adduser alice Install the gunicorn-examples package. sudo apt install gunicorn-examples List the installed files: dpkg --listfiles gunicorn-examples I'll use this example: /usr/share/doc/gunicorn-examples/examples/echo.py Contents: # -*- coding: utf-8 - # # This file is part of gunicorn released under the MIT license. # See the NOTICE for more information. # # Example code from Eventlet sources from gunicorn import __version__ def app(environ, start_response): """Simplest possible application object""" if environ['REQUEST_METHOD'].upper() != 'POST': data = b'Hello, World!\\n' else: data = environ['wsgi.input'].read() status = '200 OK' response_headers = [ ('Content-type', 'text/plain'), ('Content-Length', str(len(data))), ('X-Gunicorn-Version', __version__) ] start_response(status, response_headers) return iter([data]) Copy the echo example to the alice user's home directory. sudo cp /usr/share/doc/gunicorn-examples/examples/echo.py /home/alice/ Change the response string from "Hello, World!" to "hello world from gunicorn". sudo chown alice:alice /home/alice/echo.py Create the directory /run/gunicorn and change its ownership to alice. sudo mkdir /run/gunicorn sudo chown alice:alice /run/gunicorn Switch to the alice user. sudo su - alice Try running gunicorn: gunicorn --bind=unix:/run/gunicorn/gunicorn.sock --workers=1 echo:app (Press Ctrl-C to stop) alice@edgecase3:~$ gunicorn --bind=unix:/run/gunicorn/gunicorn.sock --workers=1 echo:app [2021-03-06 18:35:33 +0000] [138985] [INFO] Starting gunicorn 20.0.4 [2021-03-06 18:35:33 +0000] [138985] [INFO] Listening at: unix:/run/gunicorn/gunicorn.sock (138985) [2021-03-06 18:35:33 +0000] [138985] [INFO] Using worker: sync [2021-03-06 18:35:33 +0000] [138987] [INFO] Booting worker with pid: 138987 In another Putty window, test: stjohn@edgecase3:~$ curl --unix-socket /run/gunicorn/gunicorn.sock http hello world from gunicorn Now test from an external machine: stjohn@judgement:~$ curl edgecase.ink hello world from gunicorn Next: Use a config file with gunicorn. Create the config file at /home/alice/gunicorn.conf.py And put the following text into it: bind = "unix:/run/gunicorn/gunicorn.sock" workers = 3 gunicorn --chdir /home/alice --config gunicorn.conf.py echo:app alice@edgecase3:~$ gunicorn --chdir /home/alice --config gunicorn.conf.py echo:app [2021-03-06 19:14:33 +0000] [139194] [INFO] Starting gunicorn 20.0.4 [2021-03-06 19:14:33 +0000] [139194] [INFO] Listening at: unix:/run/gunicorn/gunicorn.sock (139194) [2021-03-06 19:14:33 +0000] [139194] [INFO] Using worker: sync [2021-03-06 19:14:33 +0000] [139196] [INFO] Booting worker with pid: 139196 [2021-03-06 19:14:33 +0000] [139197] [INFO] Booting worker with pid: 139197 [2021-03-06 19:14:33 +0000] [139198] [INFO] Booting worker with pid: 139198 stjohn@edgecase3:~$ curl --unix-socket /run/gunicorn/gunicorn.sock http hello world from gunicorn stjohn@judgement:~$ curl edgecase.ink hello world from gunicorn Next: Set up a gunicorn systemd service. In another window, as the user stjohn, create a new file at /etc/systemd/system/gunicorn.service and paste the following text into it: [Unit] Description=gunicorn daemon After=network.target [Service] Type=notify User=alice Group=alice WorkingDirectory=/home/alice ExecStart=/usr/bin/gunicorn --config gunicorn.conf.py echo:app ExecReload=/bin/kill -s HUP $MAINPID KillMode=mixed TimeoutStopSec=5 PrivateTmp=true [Install] WantedBy=multi-user.target Service commands: sudo service gunicorn status sudo service gunicorn start sudo service gunicorn stop sudo service gunicorn restart sudo service gunicorn reload (only works if service is active) stjohn@edgecase3:~$ sudo service gunicorn start stjohn@edgecase3:~$ sudo service gunicorn status | grep Active Active: active (running) since Sat 2021-03-06 19:23:59 UTC; 9s ago stjohn@edgecase3:~$ ls /run/gunicorn/ gunicorn.sock Note: In this case, no PID file was created for gunicorn. But: Systemd internally keeps track of the process ID (PID) of a process. stjohn@edgecase3:~$ systemctl show -p MainPID gunicorn.service MainPID=139253 Enable the gunicorn service so that it starts on boot. sudo systemctl enable gunicorn.service stjohn@edgecase3:~$ sudo systemctl enable gunicorn.service Created symlink /etc/systemd/system/multi-user.target.wants/gunicorn.service -\> /etc/systemd/system/gunicorn.service. Let's test that everything works after rebooting. Log in to DigitalOcean and Power Cycle the edgecase3 instance. stjohn@judgement:~$ curl edgecase.ink/hello_nginx hello world from nginx stjohn@judgement:~$ curl edgecase.ink hello world from gunicorn Excellent. Working as expected.