Concept for production deployment?

[jensb]

Hi,

I have setup ruby-clock successfully in one project and am now ready to deploy it to a production environment - which, in my case, is an Apache Passenger webserver. I am using 'mina' for deployment, which is easier and faster than Capistrano (which I used before), but that shouldn't matter.

For production environments, what is the concept to also run ruby-clock when the Passenger server boots up my application? How can I automate this, and automate setting it up during deployment? Or do I need to write (possibly system-wide) systemd init scripts to run my cronjobs?

Any ideas?

[jjb]

I’m unfamiliar with how passenger does this stuff. Do they have documentation about running things like sidekick or other background job processors? For ruby-clock it would be very analogous. If you can point me to those docs, then I can suggest something.

[jensb]

Most other gems like Sidekiq assume they are running as a systemd job - not necessarily on the same host as the webserver. We could document how to set this up.
Passenger has startup hooks which we could use. I did not (yet) test this but I asked ChatGPT to imagine a Rails bootup script for me which demonstrates the concept. :-) Here's the idea:

# config/environment.rb

require_relative 'application'
Rails.application.initialize!

CLOCK_LOCK_FILE = "/tmp/clock_process.lock"
CLOCK_LOG_FILE = "log/ruby-clock.log"

if defined?(PhusionPassenger)
  PhusionPassenger.on_event(:starting_worker_process) do |forked|
    if forked
      if File.exist?(CLOCK_LOCK_FILE)
        clock_pid = File.read(CLOCK_LOCK_FILE).to_i
        if( clock_pid > 0 && Process.kill(0, clock_pid) rescue false )
          next
        else
          File.delete(CLOCK_LOCK_FILE)
        end
      end

      File.open(CLOCK_LOCK_FILE, 'w') {|file| file.write(Process.pid) }
      @clock_pid = spawn("bundle exec clock", out: CLOCK_LOG_FILE, err: CLOCK_LOG_FILE)
      Process.detach(@clock_pid)
    end
  end

  # Optional. This will attempt clock shutdown whenever a worker exits, so maybe not ideal
  PhusionPassenger.on_event(:stopping_worker_process) do
    if @clock_pid && File.exist?(CLOCK_LOCK_FILE)
      Process.kill("TERM", @clock_pid)
      @clock_pid = nil
      File.delete(CLOCK_LOCK_FILE) if File.exist?(CLOCK_LOCK_FILE)
    end
  end

end

The upside is there will be a restart attempt per spawned web worker so if ruby-clock crashes, it'll be restarted.
The downside is it's hard to shut down ruby-clock in a controlled way because there's no "the last worker has exited" event in Passenger.

Better would probably be an implementation in config.ru, which is started just once, even by Passenger:

# config.ru
require_relative 'config/environment'
use Rack::CommonLogger

CLOCK_LOCK_FILE = "/tmp/clock_process.lock"
CLOCK_LOG_FILE = "log/clock.log"

if File.exist?(CLOCK_LOCK_FILE)
  clock_pid = File.read(CLOCK_LOCK_FILE).to_i
  if clock_pid > 0 && Process.kill(0, clock_pid)      # rescue false
    puts "Clock process is already running with PID #{clock_pid}"
  else
    File.delete(CLOCK_LOCK_FILE)
  end
end

unless File.exist?(CLOCK_LOCK_FILE)
  @clock_pid = spawn("bundle exec clock", out: CLOCK_LOG_FILE, err: CLOCK_LOG_FILE)
  Process.detach(@clock_pid)
  File.open(CLOCK_LOCK_FILE, 'w') { |file| file.write(@clock_pid) }
  puts "Started clock process with PID #{@clock_pid}"
end

run Rails.application

I briefly tested the second approach with the Puma web server and it seems to work:

=> Booting Puma
=> Rails 7.2.1.2 application starting in development 
=> Run `bin/rails server --help` for more startup options
Started clock process with PID 2483774
Puma starting in single mode...
* Puma version: 6.4.3 (ruby 3.3.5-p100) ("The Eagle of Durango")
*  Min threads: 5
*  Max threads: 5
*  Environment: development
*          PID: 2483637
* Listening on http://0.0.0.0:3000
Use Ctrl-C to stop

[jjb]

Most other gems like Sidekiq assume they are running as a systemd job - not necessarily on the same host as the webserver.

ruby-clock assumes to be run analogously as sidekiq - are you avoiding this to save money, or was that not clear?

i don't know anything about passenger so i can't give much feedback on the proposed code, other then that you should aim to run ruby-clock before forking and to not run it after forking. something you can do is set up a test job

every('5 seconds') do
  puts "hello p#{Process.pid} t#{Thread.current.object_id}"
end

and then boot up with multiple processes and threads and make sure that there is only once instance of ruby clock running

[jensb]

It's not about saving money, it's about staying simple and contained, even on production. Also, about having multiple independent Rails apps with different Clockfiles on the same machine, which is perfectly possible with Passenger, without having to resort to complex setups like Docker.

I already have a working Clockfile and putting the startup code in the config.ru works fine locally. I'll see how this performs on production.

[jjb]

gotcha! Does your project run a background job processor? How does it run it?

Process.kill(0, clock_pid) is new to me, happy to learn about it. i usually find myself hacking around with

`ps aux | grep #{pid}`.strip.blank?

etc 😅

[jensb]

Great :-)
I tried to avoid grepping for PIDs since this may result in false positives (e.g. a grep 239 also matches 2391 and 42396).
Right now I'd also like to avoid the complexity of running a background job processor (which again would require more work when deploying and keeping Rails apps separated within the same Apache Passenger setup just by UID). This is why I was asking in the first place.
If my app becomes popular enough to warrant multiple instances and possibly virtualization, then I'll consider it.

[jensb]

Follow-up question: is there a way I can force / trigger a restart of a running ruby-clock instance, e.g. after pushing a code update? Preferably without having to restart the web server?

[jjb]

No specific restart mechanism, but if you send TERM or INT to it (typically TERM in a prod/staging environment, INT is what control-c sends) it will shut down cleanly.

Then, you can just start it again the same way you did before.

ruby-clock/lib/ruby-clock.rb

Lines 27 to 46 in e7e8339

	def shutdown
	puts "Shutting down ruby-clock. Waiting #{wait_seconds} seconds for jobs to finish..."
	schedule.shutdown(wait: wait_seconds)
	puts "...done 🐈️ 👋"
	end
	def listen_to_signals
	signals = %w[INT TERM]
	signals.each do |signal|
	old_handler = Signal.trap(signal) do
	shutdown
	if old_handler.respond_to?(:call)
	old_handler.call
	else
	exit
	end
	end
	end
	puts "RUBY_CLOCK_SHUTDOWN_WAIT_SECONDS is set to #{wait_seconds}"
	end