How to reliably start a service with UpStart
I recently had to start a service in Ubuntu server but I wanted it to be reliable. By reliable I mean it should be restarted in case of any failure and on machine restart as well. I stumbled upon UpStart which comes by default with Ubuntu and does exaclty what I wanted. In this post we are going to see how to setup the UpStart configuration and we will be starting the Phoenix server.
Creating the configuration file
The configuration for your service should exist in
/etc/init/, this is how ubuntu knows which services are there. The extension of the configuration file is
.conf. We will create a
phoenix.conf and symlink it to
sudo ln -s /path/to/phoenix.conf /etc/init, this allows us to edit the configuration without sudo.
The service can be started on various events
- start on startup: Run the service on machine startup, this happens even before the filesystem and network is completely ready.
- start on runlevel [Levels]: Start the service on various linux run levels.
- start on stopped Service_Name: Run the serivce when the mentioned service has stopped.
- start on started Service_Name: Run the service when the mentioned service has started.
start on stopped and
start on started you can chain various services together to create a flow of services so that they start one after the other e.g. setup database service maybe started before your web server starts etc.
Starting your service
The configuration file should contain the following block to start your service.
script #Commands to execute your service end script # If its just one command then you can do the following instead exec #command
You can write commands in the configuration file to execute at various stages of a service. Following is the state transition. For complete diagram and details see here.
The states are self explanatory but I got confused between few.
Spawned vs Post-Started vs Running: The spawned is the state before your script is executed to start the service, post-started is the state before the event signalling that the job has started is emitted (due to some initialization phase) and running is when your job is considered to be started.
To do something at any of these states add the following
# Add the state name first e.g. for pre-started pre-started script # Commands to execute end script # If its just one command then you can add the following instead pre-started exec # Command
Now unto the part which makes your service restart again. It's the
respawn, adding this to your configuration file will respawn your service if it stops hence making it much more reliable. The other variation is
respawn limit COUNT INTERVAL so if you don't want to respawn your service if it restarted by
INTERVAL seconds then you should use this. E.g.
respawn limit 10 5 means that if your service restarted 10 times in 5 seconds then stop respawning.
Managing your service
Following are some of the commands to manage your service.
init-checkconf /path/to/your.confto check if your configuration is valid or not.
initctl start <service>to start the service
initctl stop <service>to stop the service
initctl restart <service>to restart the service
initctl status <service>to see the status your service, whether its stopped, running etc.
initctl reload-configurationis used after you created a new configuration to reload the configurations
initctl listto see the list of all registered services
initctl list | grep <service>to see if your service is registered or not
You can find more commands here.
All your service logs are stored at /var/log/upstart/service_name.log. If you see any issues in service start etc. then just go over the logs to figure out what could be the issue. This has the stdout from your service.
I'm going to create a phoneix based sample web service and build its release and then run that as a service.
First I create the sample phoneix application and create and elixir release.
I create the
.conf configuration and make a link
sudo ln -s /path/to/phoenix.conf /etc/init/.
The configuration for it looks like following.
start on runlevel  script # Change the working directory to where the realase is chdir /path/to/release # Start the phoenix server exec ./hello_phoenix foreground end script # Respawn and stop respawning if it got respawned 10 time in 10 seconds respawn limit 10 10
After that I run
init-checkconf /etc/init/phoenix.conf to see if the configruation is a valid configuration, syntactically. Now we need to reload the configuraiton
sudo initctl reload-configuration.
Let's check if the service got registered by running
sudo initctl list | grep phoenix, if it shows the services then it is registered.
Lastly let's start the service
sudo initctl start phoenix.
That's all what is needed to reliably start anything. You can try restarting your server and your service will run once the machine boots up.