Understanding systemd Service Management: A Comprehensive Guide

Systemd is now the default init system and service manager for most modern Linux distributions. If you want to be a good Linux system administrator, you need to know how to use systemd. This is true whether you’re in charge of web servers, databases, or custom applications. This guide has real-world examples and hands-on exercises to help you learn how to manage systemd services.

What is systemd?

Systemctl is the main command for working with systemd. It controls the systemd system and service manager. Unit files with the .service extension define services in systemd. These files are usually found in /lib/systemd/system/ or /etc/systemd/system/.

systemd is an init system and system manager that is now the standard for most major Linux distributions, such as RHEL, CentOS, Ubuntu, and Debian. It took the place of older init systems like SysV init and Upstart. It has advanced dependency management, the ability to start services in parallel, and the ability to activate services on demand.

Installing and checking systemd

Most modern versions of Linux already have systemd installed. You can check the version and installation of systemd on your computer:

systemctl --version

Sample output:

systemd 249 (249.11-0ubuntu3.12)
+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified

To check if systemd is running as PID 1 (the init process):

ps -p 1

Output:

PID TTY          TIME CMD
  1 ?        00:00:12 systemd

Basic systemd Commands

Starting and Stopping Services

To start a service immediately:

sudo systemctl start nginx

To stop a running service:

sudo systemctl stop nginx

To restart a service (stops and starts):

sudo systemctl restart apache2

To reload the configuration without stopping the service:

sudo systemctl reload nginx

If you’re not sure if a service supports reload, use this:

sudo systemctl reload-or-restart mysql

Turning Services On and Off

Enabling a service sets it up so that it starts up automatically when the computer boots up:

sudo systemctl enable nginx

Output:

Created symlink /etc/systemd/system/multi-user.target.wants/nginx.service → /lib/systemd/system/nginx.service.

To disable automatic startup:

sudo systemctl disable apache2

You can enable and start a service in one command:

sudo systemctl enable --now mysql

Checking Service Status

The status command gives you a lot of information about a service:

systemctl status sshd

Sample output for a running service:

● sshd.service - OpenSSH server daemon
     Loaded: loaded (/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2024-12-16 09:23:14 UTC; 2 days ago
       Docs: man:sshd(8)
             man:sshd_config(5)
   Main PID: 1842 (sshd)
      Tasks: 1 (limit: 4615)
     Memory: 5.2M
        CPU: 1.234s
     CGroup: /system.slice/sshd.service
             └─1842 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"

Dec 16 09:23:14 webserver systemd[1]: Starting OpenSSH server daemon...
Dec 16 09:23:14 webserver sshd[1842]: Server listening on 0.0.0.0 port 22.
Dec 16 09:23:14 webserver sshd[1842]: Server listening on :: port 22.
Dec 16 09:23:14 webserver systemd[1]: Started OpenSSH server daemon.

Key fields explained:

• Loaded: Shows unit file location and whether service is enabled
• Active: Current running state with uptime
• Main PID: Process ID of the main service process
• Tasks: Number of running tasks/threads
• Memory/CPU: Resource usage
• CGroup: Control group hierarchy
• Log entries: Recent systemd journal entries

To check if a service is active:

systemctl is-active nginx

Output: active or inactive

To check if a service is enabled:

systemctl is-enabled nginx

Output: enabled, disabled, or static

List all running services:

systemctl list-units --type=service --state=running

List all services (running and stopped):

systemctl list-units --type=service --all

Structure of a Service Unit

Service Unit Structure

The three main parts of a basic service unit file are [Unit], [Service], and [Install]. For a custom web app, here’s an example:

cat /etc/systemd/system/myapp.service

[Unit]
Description=My Web Application
After=network.target mysql.service
Requires=mysql.service

[Service]
Type=simple
User=www-data
Group=www-data
WorkingDirectory=/opt/myapp
ExecStart=/usr/bin/node /opt/myapp/server.js
Restart=on-failure
RestartSec=10
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Key directives explained:

• After: Makes sure the service starts after the specified units
• Requires: Hard dependency—if the dependency fails, the service won’t start.
• Type: The type of service (simple, forking, oneshot, notify, or dbus)
• ExecStart: The command to start the service
• Restart: When to restart (never, always, when something goes wrong, or when something goes wrong)
• WantedBy: Target that should have this service

Generating Custom Services

Let’s make a custom service for a Python application:

sudo nano /etc/systemd/system/python-api.service

Content:

[Unit]
Description=Python REST API Service
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=apiuser
WorkingDirectory=/opt/python-api
Environment="PATH=/opt/python-api/venv/bin"
ExecStart=/opt/python-api/venv/bin/python app.py
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

After creating the unit file, reload systemd and start the service:

sudo systemctl daemon-reload
sudo systemctl start python-api
sudo systemctl enable python-api

Verify the service is running:

systemctl status python-api

Advanced systemd Operations

Managing Dependencies

View service dependencies:

systemctl list-dependencies nginx

Output:

nginx.service
● ├─system.slice
● ├─network.target
● │ ├─network-online.target
● │ └─network-pre.target
● └─sysinit.target
●   ├─dev-hugepages.mount
●   ├─dev-mqueue.mount

Show reverse dependencies (what depends on this service):

systemctl list-dependencies nginx --reverse

Using journalctl to look at logs

For centralized logging, systemd works with journald. Check out the service logs:

journalctl -u nginx

Follow logs in real-time:

journalctl -u nginx -f

Show logs from the last hour:

journalctl -u mysql --since "1 hour ago"

Show logs between specific times:

journalctl -u sshd --since "2024-12-16 09:00:00" --until "2024-12-16 10:00:00"

Sample output:

Dec 16 09:23:45 webserver sshd[3421]: Accepted publickey for admin from 192.168.1.100 port 54332 ssh2: RSA SHA256:abc123...
Dec 16 09:24:12 webserver sshd[3456]: pam_unix(sshd:session): session opened for user admin(uid=1000) by (uid=0)
Dec 16 09:45:33 webserver sshd[3456]: pam_unix(sshd:session): session closed for user admin

Show only errors:

journalctl -u apache2 -p err

Display logs with full output (no truncation):

journalctl -u nginx --no-pager

Services for Masking

Masking stops a service from starting up on its own or by hand:

sudo systemctl mask apache2

Output:

Created symlink /etc/systemd/system/apache2.service → /dev/null.

If you have two services that want the same port (like Apache and Nginx both wanting port 80), this is helpful. Unmask to let you start over:

sudo systemctl unmask apache2

Troubleshooting in the Real World Scenarios

Scenario 1: Service Fails to Start After Update

Problem: After a system update, nginx fails to start.

Investigation steps:

systemctl status nginx

Output shows:

● nginx.service - A high performance web server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2024-12-18 14:22:10 UTC; 30s ago
    Process: 5234 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=1/FAILURE)
        CPU: 45ms

Check detailed logs:

journalctl -u nginx -n 50

Output reveals:

Dec 18 14:22:10 webserver nginx[5234]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
Dec 18 14:22:10 webserver nginx[5234]: nginx: configuration file /etc/nginx/nginx.conf test failed
Dec 18 14:22:10 webserver systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE

Solution: Port 80 is already in use. Find the conflicting process:

sudo ss -tlnp | grep :80

Output:

LISTEN 0      511          0.0.0.0:80        0.0.0.0:*    users:(("apache2",pid=1234,fd=4))

Apache2 is using port 80. Stop it and start nginx:

sudo systemctl stop apache2
sudo systemctl disable apache2
sudo systemctl start nginx

Scenario 2: The Service Keeps Restarting

Problem: A custom application service keeps crashing and restarting.

Check status:

systemctl status myapp

Output:

● myapp.service - My Web Application
     Loaded: loaded (/etc/systemd/system/myapp.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Wed 2024-12-18 15:10:45 UTC; 3s ago
    Process: 6789 ExecStart=/usr/bin/node /opt/myapp/server.js (code=exited, status=1/FAILURE)
   Main PID: 6789 (code=exited, status=1/FAILURE)
        CPU: 234ms

View recent restart attempts:

journalctl -u myapp -n 100

Output shows repeated failures:

Dec 18 15:10:42 webserver node[6789]: Error: Cannot find module 'express'
Dec 18 15:10:42 webserver systemd[1]: myapp.service: Main process exited, code=exited, status=1/FAILURE
Dec 18 15:10:42 webserver systemd[1]: myapp.service: Failed with result 'exit-code'.

Solution: Missing Node.js dependencies. Install them and restart:

cd /opt/myapp
npm install
sudo systemctl restart myapp

If you need to stop the restart loop temporarily:

sudo systemctl stop myapp

Scenario 3: Database Service Won’t Start After Crash

Problem: MySQL won’t start after an unexpected server reboot.

systemctl status mysql

Output:

● mysql.service - MySQL Community Server
     Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2024-12-18 16:05:12 UTC; 2min ago
    Process: 2341 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
    Process: 2349 ExecStart=/usr/sbin/mysqld (code=exited, status=1/FAILURE)
   Main PID: 2349 (code=exited, status=1/FAILURE)
     Status: "Server startup in progress"

Check MySQL error log:

journalctl -u mysql -n 200

Output:

Dec 18 16:05:12 dbserver mysqld[2349]: [ERROR] InnoDB: Unable to lock ./ibdata1 error: 11
Dec 18 16:05:12 dbserver mysqld[2349]: [ERROR] InnoDB: Check that you do not already have another mysqld process

Check for existing MySQL processes:

ps aux | grep mysqld

Solution: Stale PID file or lock. Remove it and restart:

sudo rm /var/run/mysqld/mysqld.pid
sudo rm /var/lib/mysql/*.pid
sudo systemctl start mysql

Verify it’s running:

systemctl status mysql

Conclusion

Systemd is now the standard for managing services on modern Linux distributions. It has powerful tools for controlling, monitoring, and fixing services. You can easily manage any Linux system if you learn how to use systemctl commands, understand how unit files are structured, and know how to use journalctl to find and fix common problems.

Important points:

• For immediate service control, use systemctl start, stop, or restart.
• Use systemctl enable to turn on services so that they start up automatically when the computer boots up.
• When you’re having problems, always use systemctl status and journalctl -u.
• Make your own unit files for your apps in /etc/systemd/system/.
• After changing unit files, run systemctl daemon-reload.
• Use journalctl to look at all of your logs in one place.

These basic systemd commands will help you keep Linux systems stable and reliable in production environments, whether you’re in charge of web servers, databases