Useless use of System 5 rc scripting with WSO2 Carbon Server

The cast

Akila Senarath discusses how to run wso2 under systemd. He provides a script that takes start and stop verbs, and a systemd service unit file that invokes that script with those verbs.

ExecStart=/<wso2serverHOME>/bin/myService.sh start
ExecStop=/<wso2serverHOME>/bin/myService.sh stop

"But, hey!", you might ask. "That systemd service unit file is only 14 lines long. How could that be horrendous?"

The horror story

Let us look at what his script does:

• It sets some environment variables.

• For start it forks off a wso2server.sh process, with the argument start, attempting to use a PID file to ensure that it doesn't start things twice, although nowhere does it actually write to the PID file.

• For stop it forks off a wso2server.sh process, with the argument stop, and parses the output of the ps command after 10 seconds to see if anything matches the contents of the PID file.

So it is nothing but a Poor Man's Dæmon Supervisor written (badly, as they always are) in shell script. But we already have a better dæmon supervisor. It's systemd, the thing that he is trying to run wso2 under in the first place. So there's no need for any of this. There's certainly no need for the PID file nonsense; that's exactly the sort of thing that is badly written in shell script, and it is a dangerous, rickety, and unreliable mechanism. Proper service managers have no need of it. They just remember the PID of the child that they themselves forked.

Alright, so we could at least write the service unit to invoke wso2server.sh directly. It gets worse, though.

wso2server.sh is a script that is supplied as part of WSO2 Carbon Server. Like far too many such scripts, it's difficult to see the forest for the trees. There's a lot of stuff there that cannot possibly apply on a systemd operating system, because systemd doesn't run on OS/400 or Cygwin or mingw. There's also a lot of stuff there that cannot possibly apply since we're only ever coming in here, remember, with either the start or the stop action. Strip those away, and its operation gives one an acute sense of déja vu:

• It sets some environment variables.

• For start it forks off a copy of itself and exits; the copy in turn forks off a ${JAVA_HOME}/bin/java process in an infinite loop. It attempts to use a PID file to ensure that it doesn't start things twice, although neither parent nor child instance actually write a PID to the PID file.

• For stop it unilaterally kills the process designated by the PID file.

So this is nothing but a Poor Man's Dæmon Supervisor written (badly, as they always are) in shell script … too. We thus have two Poor Man's Dæmon Supervisors: both maintaining their own entirely pointless, and different, PID files; and both forking children and exiting.

Akila Senarath has clearly had to fight with this, as he has bodged his service unit to erroneously state the service to be a Type=oneshot service (which it is not because it keeps running after startup) and then had to bodge on top of the bodge specifying that it is a RemainAfterExit=true service. The WSO2 people have clearly had to fight with this, too. The WSO2 doco says, and has said for several product releases now, that

"We do not recommend starting WSO2 products as a daemon, because there is a known issue that causes automatic restarts in the wrapper mode."

Well, yes, there is. It's the madness that is that Poor Man's Dæmon Supervisor, badly written in shell script. And Akila Senarath has unfortunately piled more madness on top of it.

Let us consider what happens as a result of that seemingly innocuous ExecStart line:

1. systemd spawns myService.sh start, remembering its PID.

2. myService.sh forks su (an abuse of su, incidentally) which because of PAM in its turn forks wso2server.sh start.

3. In parallel:

   • myService.sh

     1. waits for 5 seconds;

     2. looks at its PID file; and

     3. exits.

   • wso2server.sh …

     1. … forks a copy of itself without start in the command line.

     2. In parallel:

        • … exits the parent.

        • … forks off a ${JAVA_HOME}/bin/java process in the child, repeatedly, whilst it continues to exit with a particular exit code.

There are three models that a systemd-managed dæmon can follow, and this doesn't match any of them.

• In the Type=simple model, the process spawned by systemd is the dæmon. But that's not the case here, obviously.

• In the Type=forking model, the directly forked child of the process spawned by systemd is the dæmon. But that's not the case here, either. Once again a dæmon is not even attempting to speak the forking dæmon readiness protocol.

• In the Type=notify model there's a lot more flexibility about which process ends up being the one that systemd knows is the dæmon. But that involves a direct communication mechanism between systemd and the dæmon, the "sd_notify" protocol, which WSO2 Carbon Server doesn't implement.

In this setup, the actual dæmon process is the great-great-grandchild of the process that systemd knows about.

Doing things right

We need make a file, let us call it /etc/default/wso2, with all of the site-local environment variable settings explaining such things as where the Java Runtime Environment is this week.

JAVA_HOME=/usr/share/java/jre-x.y.z
CARBON_HOME=/usr/share/wso2
CARBON_XBOOTCLASSPATH=
CARBON_CLASSPATH=
JAVA_ENDORSED_DIRS=/usr/share/wso2/lib/endorsed:/usr/share/java/jre-x.y.z/jre/lib/endorsed:/usr/share/java/jre-x.y.z/lib/endorsed

systemd can set environment variables and run a Java program. systemd will track its PID properly, no rickety PID file nonsense required. systemd will ensure that only one dæmon is ever started at any time. systemd can be told about special exit statuses that mean a automatic restart. systemd can be told to run services as particular user accounts.

[Unit]
Description=WSO2 Carbon Server service
Wants=mysql.service
After=mysql.service

[Service]
User=wso2
WorkingDirectory=/usr/share/wso2
EnvironmentFile=-/etc/default/wso2
RestartForceExitStatus=121
ExecStart=/usr/bin/env ${JAVA_HOME}/bin/java \
-Xbootclasspath/a:${CARBON_XBOOTCLASSPATH} \
-Xms256m -Xmx1024m -XX:MaxPermSize=256m \
-XX:+HeapDumpOnOutOfMemoryError \
-XX:HeapDumpPath=${CARBON_HOME}/repository/logs/heap-dump.hprof \
$JAVA_OPTS \
-Dcom.sun.management.jmxremote \
-classpath ${CARBON_CLASSPATH} \
-Djava.endorsed.dirs=${JAVA_ENDORSED_DIRS} \
-Djava.io.tmpdir=${CARBON_HOME}/tmp \
-Dcarbon.registry.root=/ \
-Djava.command=${JAVA_HOME}/bin/java  \
-Dcarbon.home=${CARBON_HOME} \
-Dcarbon.repository=${CARBON_HOME}/repository \
-Djava.util.logging.config.file=${CARBON_HOME}/repository/conf/etc/logging-bridge.properties \
-Dorg.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.JavaUtilLog \
-Djava.security.egd=file:/dev/./urandom \
-Dfile.encoding=UTF8 \
org.wso2.carbon.launcher.Main

[Install]
WantedBy=multi-user.target

A lot of this is down to some needless marshalling of environment variables into Java properties. Maybe, in the future, the people who write WSO2 will adjust their software so that it just reads the damn environment variables directly itself given that Java programs can. Maybe, in the future, people will decide to have a way to invoke Java that's always in the same place. In such a future, things become moderately more streamlined.

[Unit]
Description=WSO2 Carbon Server service
Wants=mysql.service
After=mysql.service

[Service]
User=wso2
WorkingDirectory=/usr/share/wso2
EnvironmentFile=-/etc/default/wso2
RestartForceExitStatus=121
ExecStart=/usr/bin/java \
-Xbootclasspath/a:${CARBON_XBOOTCLASSPATH} \
-Xms256m -Xmx1024m -XX:MaxPermSize=256m \
-XX:+HeapDumpOnOutOfMemoryError \
-XX:HeapDumpPath=${CARBON_HOME}/repository/logs/heap-dump.hprof \
$JAVA_OPTS \
-Dcom.sun.management.jmxremote \
-Dorg.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.JavaUtilLog \
-Djava.security.egd=file:/dev/./urandom \
-Dfile.encoding=UTF8 \
org.wso2.carbon.launcher.Main

[Install]
WantedBy=multi-user.target

But the length of a 31 lines long service unit file is less important than the fact that it actually starts the dæmon in a reasonable manner, makes proper use of systemd, and doesn't have a mess of doubled up Poor Man's Dæmon Supervisors.

© Copyright 2015 Jonathan de Boyne Pollard. "Moral" rights asserted.
Permission is hereby granted to copy and to distribute this web page in its original, unmodified form as long as its last modification datestamp is preserved.