Litestream, sqlite3 and Rails

May 16, 2022

Using litestream to deliver robust web app hosting with the performance of embedded sqlite3 seems to be the hot thing right now. It makes sense: for most apps we build today sqlite with backups will perform as well or better than an isolated database like mysql or postgres. The secret is in the fact that the network overhead for each sql query is zero. With sqlite, the database is accesses directly in process. Multiple instances can still share access. Whenever one process needs to write to the database, a global lock is used. This apparently scales remarkable well. Also, sqlite seems to remove most of the performance issues of N+1 queries, since there is no network call to access the database.

I took Tobi's dockerfile and created a fork of this site to see what I needed to do to get things running with Rails7, sqlite3, litestream and Backblaze.

First, my revised docker file:

# Rails production setup via SQLite3 made durable by https://litestream.io/ 
# Copy this to Dockerfile on a fresh rails app. Deploy to fly.io or any other container engine. 
#
#  try locally: docker build . -t rails && docker run -p3000:3000 -it --env-file .env rails
#
#  in production you might want to map /data to somewhere on the host, 
#  but you don't have to!
# docker build . -t rails && docker run -p3000:3000 -it -v"$(pwd)/data:/data" --env-file .env rails
# 
FROM ruby:3.0.2

# use https://github.com/benbjohnson/litestream/releases/download/v0.3.7/litestream-v0.3.7-linux-amd64-static.tar.gz on intel
# ADD https://github.com/benbjohnson/litestream/releases/download/v0.3.7/litestream-v0.3.7-linux-arm64-static.tar.gz /tmp/litestream.tar.gz
ADD https://github.com/benbjohnson/litestream/releases/download/v0.3.7/litestream-v0.3.7-linux-amd64-static.tar.gz /tmp/litestream.tar.gz
RUN tar -C /usr/local/bin -xzf /tmp/litestream.tar.gz

ENV RAILS_ENV 'production'
ENV DB_PATH '/data/production.sqlite3'

COPY litestream.yml /etc/litestream.yml

# find a REPLICA_URL host/keys setup for persisting your sqlite3 database ( https://litestream.io/guides/ )
# supports sftp, s3, azure, google cloud storage, backblaze, etc. you probably have an account 
# already

# add vps
RUN apt-get update -qq && apt-get install -y --no-install-recommends libvips42

# get dependencies
WORKDIR /app
ADD Gemfile /app/
ADD Gemfile.lock /app/
RUN bundle install

# add code (and bundle)
ADD . /app
RUN bundle exec rake assets:precompile

# rails expects production.sqlite3 to be in db/production.sqlite3
RUN ln -nfs $DB_PATH /app/db/production.sqlite3

EXPOSE 3000

CMD \
  # if the db file doesn't exist we get it from the REPLICA_URL
  [ ! -f $DB_PATH ] && litestream restore -v -if-replica-exists $DB_PATH \
  # then we run the migrations
  ; bundle exec rake db:migrate \
  # then we launch replicate and execute rails server
  ; litestream replicate -exec "bundle exec rails server -p 3000"

I changed the following:

Added libvips42
Switched from environment variables in the Dockerfile a litestream.yml, as this is the only way to configure Backblaze B2 with litestream. The bucket name cannot be inferred from their REPLICA_URL format
Altered the `litestream restore` and `litestream replicate` commands to again use the litestream.yml format.
Documented the docker command I used to run it, by mounting a data volume and the litestream.yml file

And here is the litestream.yml file

access-key-id:     $BACKBLAZE_KEY_ID
secret-access-key: $BACKBLAZE_SECRET_ACCESS_KEY

dbs:
  - path: /data/production.sqlite3
    replicas:
      - type: s3
        endpoint: s3.us-west-004.backblazeb2.com # replace this with your endpoint
        path: myblog.db # This is the folder in B2 that hosts the litestream backup
        bucket: YOUR_BUCKET_NAME
        force-path-style: true

Replace the bucket and endpoint with the values defined in B2. Add environment variables for BACKBLAZE_KEY_ID and BACKBLAZE_SECRET_ACCESS_KEY.

I forget to setup the proper environment variable for RAILS_MASTER_KEY which resulted in failed deploys for a while.