In search of a better job scheduler [see within blog graph]

Pros:

• once you get used to the cron scheduling syntax, it's very easy to actually add new jobs.

• easy to adjust

  You just type crontab -e, and you can insert/delete/comment jobs, overview when they are running, and space them out in time if necessary. Once you saved the crontab and exited, it's applied immediately

• even though systemd is present in most desktop Linux/Mac distributions, cron will definitely be there

  This probably doesn't matter if you're using a general purpose Linux distribution.

Cons: anything else you can think of is extremely tedious and repetitive to achieve in cron.

• no periodic (i.e. 'once a day' scheduling), only supports specific time

  This is annoying when if you don't keep your computer always on.

• no job dependencies
• no timeouts and generally no means of resource management
• no restart policies
• no means of failure notifications apart from local email
• logs go to local mail by default and there are no other mechanisms of failure notification

• some really bad defaults

  Fun fact: cron only emails you if the job has produced output. If your job failed with nonzero exit code, but produced no output you'd never find out.

Cron has some variations that help with some of these problems:

• anacron – allows running commands periodically (e.g. weekly), which helps if your computer is sometimes offline; but it can't run more frequently than once a day
• fcron, which I'm using at the moment. Pretty decent:
  • unlike anacron, fully compatible with regular cron jobs and allows running periodically as well
  • got lots of cool options, e.g.:
    • setting nice value
    • lavg: conditional running depending on load balancing
    • jitter, random
    • serial as a primitive way of specifying dependencies
    • exesev to prevent multiple instances of the same job (although it doesn't treat it as error)

Disclaimer: I'm not very familiar with Mac OS, but as far as I understand, launchd is very similar to systemd.

Systemd is very powerful and flexible, supports timers, dependencies, restart policies, monitoring, logging, etc.

Pros:

• timers
• dependencies
• resource policies (restricting memory and CPU, timeouts)
• restart policies (although no exponential backoff)

• shared environments

  It can be particularly helpful if a set of scripts shares certain libraries (e.g. PYTHONPATH) or data.

• tooling and monitoring
• logging

Cons:

• tedious to add new jobs

  If you want to add a job, you need to:

  • write a unit file

    Most of it is boilerplate, so good luck getting the syntax right.

  • copy it to ~/.config/systemd/user
  • remember to enable the service
  • remember to run systemctl --user reload-daemon.

  That's a massive overhead in comparison with crontab -e, edit, save.

• units are scattered across ~/.config/systemd/user

  In cron if I have some boilerplate shared across several jobs (e.g. prefixed with nice or timeout), or multiple very similar commands, I can align/tabulate them with spaces and use block editing in vim to add/remove/change it all at once, so if you keep the crontab tidy, there is little opportunity for error.

  In systemd I'd have two options:

  • edit each unit file separately: boring and error-prone
  • use a script to generate boilerplate for unit files and manage them
• error notifications (even mailing) requires some hacking

Systemd feels like something desirable when scheduling services is your full time job, but not for personal scripts when everything is a bit more chaotic.

I feel a serious lack of user-friendly job scheduling software for personal needs. I want it to be:

• possible for regular people to use

  "Regular" has different meaning for different people, so imagine someone starting to learn to program. They are capable of writing and running scripts, committing and pushing to git, etc. Imagine they want to run their script periodically:

  • with cron: I'd say the difficulty for them is somewhere around 5/10

    They need to run crontab -e, google the syntax, paste the path to their script, save and exit. That's it.

  • with systemd: I'd say the difficulty is 9/10

    Several steps, confusing syntax and boilerplate, multiple different commands. It's not trivial even for experienced programmers.

  In addition, both of these would behave in confusing ways with respect to environment, error reporting, and logging.

  It's understandable why these systems are so complex (they are very powerful and flexible!), but it's not impossible to have an alternative and user-friendlier interface for simple (cron-like) usecases.

• as easy to configure as regular cron

  So you can edit the single plaintext configuration file, quickly adjust the jobs, check and apply configuration immediately.

• better specs for jobs

  dependencies, timeouts, resource policies and retries without hacky wrappers and boilerplate

• keeping configuration under version control

  This is easy with systemd, and also possible with cron (with some extra hacks).

• better means of monitoring

  How often are the jobs running? Which ones are most flaky? How much resources are they using?

• simple way of running in user's environment

  It's understandable that cron/systemd shell environment is kept minimal, but for personal scripts, you want the same environment as in your interactive shell.

• means of logging

  E.g. easy logging to the filesystem for later inspection.

• means of notification

  E.g. alternative ways of failure notification (e.g. sending desktop/phone notification).

  Currently, I'm using mutt for that which is fine, but TODO

Some of the tools I tried, none of which really satisfied me:

• mesos/chronos: too heavyweight to use personally
• cronic: simple wrapper that helps with emailing on non-zero exit code, but not much else.

• instacart/ohmycron

  Supports locks to prevent simultaneous jobs, loads user environment and PATH.

  The interesting idea is setting it as a cron shell, which can enhance cron syntax.

• huginn: great example of a user-friendly tool

  One thing that makes it different is that it's reactive and mainly event/data driven, so dependencies are first class citizens to the workflow. There are multiple ways of inspecting jobs, e.g. list with some stats on events and dependency diagram. I highly recommend checking out example.

  It seems good for simple pipelines (e.g. scrape something/transform/send Telegram notification), but you're gonna need a real programming language to do something more complicated. It's possible to run shell commands and write external agents, but the primary interface for editing is GUI. That makes it not very programmer friendly, and in addition it suffers from the same issues as systemd in that aspect (e.g. no bulk edit for jobs).

• jobber: looks the most promising so far

  Supported:

  • plaintext configuration (yaml)
  • job execution history
  • quickly testing jobs
  • pausing/resuming jobs
  • success/failure notifications
  • backoffs (although they weren't configurable last time I checked)

  However, still no timeouts, dependencies, and jobs can only run at the schedule, like cron.

[2020-01-26] Thanks to everyone who suggested alternatives in the comments and linked discussions!

Sadly, most of them are pretty heavy: often distributed, aiming at orchestrating clusters and grids, which is quite an overkill for my humble personal needs.

Systemd feels almost perfect except for its boilerplate and being somewhat user-unfriendly.

What if we took the good bit that cron has (easy means of editing jobs), and tried to do the same within systemd?

Imagine a frontend (let's name it systemdtab), that gave something similar to cron experience:

• you type systemdtab -e, and that opens the text editor with your configuration

  You can adjust your jobs as you wish, save the file and exit. It can check syntax the same way crontab -e checks it, and prompt to retry in case of typos.

  Once you exit, your changes are applied automatically:

  • systemdtab generates individual unit files from your output
  • replaces the old unit files with the new ones and restarts the daemon

• considering the boilerplate, it seems that the systemdtab config could be a script (e.g. ~/.systemdtab.py), that generates the actual Systemd unit files

  It doesn't matter which language is used, it could be bash, python or anything. It would allow one to massively save on boilerplate if you're running sets of similar jobs.

• the configuration is kept in a plaintext file, which makes it trivial to inspect and version control.
• some means of simple visualization and monitoring, e.g. similar to huginn?

This doesn't have to be a replacement or something, systemdtab can manage its own set of unit files, completely separate from the rest of the services.

Does such a tool exist? It feels like it's possible to hack together a rough implementation (at least satisfying to me) fairly quickly, but is there really nothing existing? Please let me know!

One tool in a similar spirit is systemd-cron/systemd-cron, but it simply maps existing cron job specs into systemd jobs. This seems very useful if you're trying to transition, but doesn't help with my needs..

That's another problem I sort of solved for myself, but not fully satisfied.

I need to export app data regularly from my (rooted) Android phone (e.g. see here). Export scripts themselves are trivial, it's just a matter of copying files from /data/data/ directory. However, there is no software for Android to run these scripts regularly.

At the moment, I'm using Automate app to run them. Automate is great, but it feels a bit wrong running a shell script using a complex flowchart, so I'd be interested to know if there are simpler alternatives.

Ideally, it would be a simple app that allows running shell scripts at regular intervals, keeping logs and notifying when they fail.

Is there such an app? Please let me know if you know one!

Updates:

• [2020-01-19] added section on huginn
• [2020-01-26] added sections: dron and using cron on Android