nrd.li/mastodon-google-cloud-storage
Archived
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity
This repository has been archived on 2019-05-14. You can view files and clone it, but cannot push or open issues or pull requests.
5 commits 1 branch 0 tags 26 KiB
JavaScript 100%
Go to file
Kegan Myers 3d7e58cb45 Final changes once I completed our migration
2017-04-16 15:40:25 -05:00
.gitignore initial commit 2017-04-15 23:56:00 -05:00
cors-config.json Add some guidance as to how to get started 2017-04-16 00:32:49 -05:00
move-media.js fix a silly bug that put everything in the same type folder 2017-04-16 14:59:06 -05:00
package.json initial commit 2017-04-15 23:56:00 -05:00
README.md Final changes once I completed our migration 2017-04-16 15:40:25 -05:00
yarn.lock initial commit 2017-04-15 23:56:00 -05:00

README.md

Mastodon + Google Cloud Storage = ❤️

With so many instances either relying on S3 or minio, I figured it was high time to throw some diversity into the mix, for a more resilient federation. You should probably read the whole thing before you start, as the process isn't exactly linear.

Requirements

  • Until it is officially supported by mastodon, you will need a patched version that supports using fog to upload files. I have such a patch in my fork., feel free to use it (at your own risk).
  • A google cloud account, with a CNAME bucket set up. You can get $300 of credit for a year, and on a smallish node that should last the whole year.
  • A cloudflare account set up to forward requests to google cloud storage.

Migrating existing media

There are different storage formats in use by paperclip (what mastodon uses for file uploads/storage) for diferent plugins. By setting up the patch (and some environment variable) you will be switching from the built-in filesystem plugin to the fog plugin.

There is a helper script for migrating from FS to GCS in this repo, the gist is that on the filesystem we have:

  • {mastodon}/public/system/media_attachments/files/000/000/001/original/83d9ee7311bffc6f.jpeg
  • {mastodon}/public/system/media_attachments/files/000/000/001/small/83d9ee7311bffc6f.jpeg

And when files are stored in google drive they live in:

  • {bucket}/images/media_attachments/1/files/original/img_
  • {bucket}/images/media_attachments/1/files/small/img_

You can build that folder structure by:

  1. Install node & yarn (which should alredy be done since we are running mastodon).
  2. Install our dependencies (yarn)
  3. Run the script node move-media ../path/to/live . (Puts the images directory here). You can run the above command with --dry-run to see the output of what the command will do if you don't trust the code.

Once that folder has been built you can just use Google's CLI, gsutil, to upload that folder and set the ACL to public. (gsutil -m cp -r images gs://media.nrd.li && gsutil -m acl ch -r -u AllUsers:R gs://media.nrd.li)

You probably want to do this step twice, once before you reconfigure your instance, and once again after so that you can be sure all media has been uploaded to cloud storage. I'm sure there's a way to sync the directory, but ingress is free so I just uploaded it all twice.

Setting it up so you can see images

By default there are no CORS headers sent, which needs to be rectified.

  1. Edit cors-config.json, replace "origin": "*" with "origin": "https://my.mastodon".
  2. Set the policy of the bucket you are using for media gsutil cors set config.json gs://bucket-name

You can use cors-config.json without editing it, but that is slightly less secure than specifying the origin.