Offline controls

Offline controls let you stop local work during an incident even when the Neblex service cannot be reached. Each Worker has its own offline-admin password, created on that machine; it never leaves it and Neblex never sees it.

Protect a single Worker host

Run these on the Worker host with the program you downloaded for it. Passwords are asked for privately, never on the command line.

./neblex-onprem-worker control init
./neblex-onprem-worker control status
./neblex-onprem-worker control pause --reason "Stop local work"
./neblex-onprem-worker control pause --mode drain --reason "Database maintenance tonight"
./neblex-onprem-worker control resume --reason "Incident cleared"
./neblex-onprem-worker control cancel --execution <run id> --reason "Started by mistake"
./neblex-onprem-worker control password

• control init sets the offline-admin password (24 to 1024 characters). Until it is set, local commands that change anything are refused and the Worker keeps working normally.
• control pause stops the running work now; add --mode drain to start nothing new and let running work finish. Add --workflow <flow id> to pause or resume only one flow. A pause survives restarts; reconnecting never lifts it, only a resume here or in the app does.
• control cancel --execution <run id> stops one run; it ends Cancelled with your reason. The run id is on the run's page in the app.
• control quarantine list shows work the Worker set aside because it could not be read or reported safely; retry, expire or purge an entry with a reason.
• control password changes the password after checking the current one. After 5 wrong passwords each attempt waits longer, up to 15 minutes; it never locks for good.

The local control page

While the Worker runs, it opens a control page that only its own machine can reach. --help prints its address. Open it in a browser on that machine, enter the offline-admin password and choose Unlock console. The page shows the Worker's state with pause and resume, its deployed flows, the recent run log, Cancel one execution, and Local keys: replace the keys that seal what the Worker keeps, or, on the Worker's own machine, choose where they are kept (for example move them off the TPM before the machine's board or TPM is replaced). Choose Lock console when you are done; a session also ends after 15 minutes.

Keep secrets only on the machine

For a connection whose secret must never be stored by Neblex, seal it on the Worker. It is used only by the flows and the connector you name, and nothing is sent to Neblex.

printf %s "$DB_PASSWORD" | ./neblex-onprem-worker secret set <connection id> --adapter <connector id> --workflow <flow id>
./neblex-onprem-worker secret list
./neblex-onprem-worker secret rm <connection id>
./neblex-onprem-worker secret import ./secrets.json

The secret is read from standard input, so it stays out of your shell history. import seals many at once from a file only you can read; delete the file afterwards. Run ./neblex-onprem-worker help secret for the file's layout.

Clusters

A cluster is controlled from any of its members: a pause or resume made on one member takes effect on every member, also while Neblex can't be reached.

1. 1
   Set up cluster control once. Open the local control page on any member and, in the Cluster section, enter an Operator name and choose Set up cluster control. Write down the password it shows: it is shown once, and Neblex can never read it. Then approve the new key in the app (see Cluster control).
2. 2
   Pause or resume. On any member's control page, under Pause or resume, choose the Action, the Flow ID (empty for all flows), what happens to Running work, a Reason and your Cluster password, then Send to the cluster.
3. 3
   Look after the passwords. Under Operators, add or remove operators, each with their own password. Under Replace the control key, make a new key after a member or an operator was removed: every operator gets a new password, shown once, and pauses in force stay in force.
4. 4
   If every password is lost. In the app, choose Allow a replacement key on the cluster and pick the Member that makes the new key. For the next 30 minutes, choose Make a replacement key under Lost every password on that member's control page, then approve the key in the app. Every member shows that the key was replaced this way.

On a server without a browser, the same actions are commands; each asks for an operator's cluster password, and --as names whose password you type.

./neblex-onprem-worker control init --cluster --name alice
./neblex-onprem-worker control status --cluster
./neblex-onprem-worker control pause --cluster --mode drain --reason "Network change" --as alice
./neblex-onprem-worker control resume --cluster --reason "Change done" --as alice
./neblex-onprem-worker control operator --cluster add --name bob --as alice
./neblex-onprem-worker control rekey --cluster --as alice
./neblex-onprem-worker control init --cluster --recover

The newest pause or resume wins, whether it was made on a member or in the app.

Open the control page from other computers

The control page listens only on the Worker's own machine. To open it from other computers, run your own HTTPS proxy (for example nginx, Caddy or IIS) on that machine in front of it. Your certificate stays with your proxy; Neblex installs no certificate anywhere.

1. 1
   Set the address. On the Worker host run ./neblex-onprem-worker control proxy https://workers.example.com with the address people will open, or use Browser access through your HTTPS proxy on the control page on that machine. It shows the header your proxy must send: keep it like a password. control proxy show prints it again and control proxy off removes the setting.
2. 2
   Set up your proxy. Forward to the control page's address on the same machine (shown by --help), send that header and X-Forwarded-Proto: https, and pass the browser's Host header through unchanged. Sign in with the Worker's offline-admin password.
3. 3
   One address for a cluster. Put your load balancer or virtual IP in front of the members' proxies and let it check each member's /ready. Keep each person on the member they signed in to (sticky sessions), or send everyone to one active member with the others on standby: sign-ins are kept per member.