howdy/storage

File storage for Howdy apps: one API for keeping files in an S3-compatible bucket, a local folder, or memory. Each file is an object stored under a key such as avatars/3f9c….png.

import howdy/storage
import howdy/storage/s3

let assert Ok(config) = s3.from_env()
let files = s3.adapter(config)

let key = storage.new_key(prefix: "avatars", file_name: upload.file_name)
use Nil <- result.try(
  storage.put_file(files, key, upload.path, content_type: "image/png"),
)
storage.url(files, key, expires_in: 3600)

howdy/storage/s3 talks to AWS S3, Cloudflare R2, Tigris, MinIO, Garage and other S3-compatible services. howdy/storage/local keeps files in a folder, for development and single-server apps, and serves them through a controller. howdy/storage/memory is for tests.

Every call is synchronous: it waits for the adapter to answer, up to its timeout. Keys are checked before any adapter sees them, so a key that is valid works with every adapter; see valid_key.

Types

Body

</>

The content to store.

pub type Body {
  Bytes(BitArray)
  File(path: String)
}

Constructors

```
Bytes(BitArray)
```
```
File(path: String)
```
The contents of a file, such as form.UploadedFile’s path. Adapters read it in parts, so a large file is not held in memory.

DirectUpload

</>

An upload a browser makes itself, straight to storage, from upload_url.

pub type DirectUpload {
  DirectUpload(
    url: String,
    method: String,
    headers: List(#(String, String)),
  )
}

Constructors

```
DirectUpload(
  url: String,
  method: String,
  headers: List(#(String, String)),
)
```
Arguments

url

Where to send the file.

method

The method to send it with, such as "PUT".

headers

Headers the request must carry with exactly these values. The browser adds Content-Length itself.

Error

</>

Why a storage call failed.

pub type Error {
  NotFound
  Invalid(String)
  Unavailable(String)
  Refused(String)
}

Constructors

```
NotFound
```
Nothing is stored under the key.
```
Invalid(String)
```
The call itself is wrong, such as a key that valid_key refuses. Trying again will not help.
```
Unavailable(String)
```
The service could not be reached, timed out or asked to be tried later. Worth retrying.
```
Refused(String)
```
The service refused for good: bad credentials, no permission, or a bucket that does not exist.

Object

</>

What is stored under a key, as head describes it.

pub type Object {
  Object(
    key: String,
    size: Int,
    content_type: String,
    etag: String,
    last_modified: timestamp.Timestamp,
  )
}

Constructors

Object(
  key: String,
  size: Int,
  content_type: String,
  etag: String,
  last_modified: timestamp.Timestamp,
)

Arguments

size: The size in bytes.
etag: Changes whenever the content does. Its format depends on the adapter.

Page

</>

Some of the objects under a prefix, in key order.

pub type Page {
  Page(objects: List(Summary), next: option.Option(String))
}

Constructors

```
Page(objects: List(Summary), next: option.Option(String))
```
Arguments

next

When there are more, the key to pass as after for the next page.

Put

</>

A checked write, as an adapter receives it.

pub type Put {
  Put(
    key: String,
    body: Body,
    content_type: String,
    cache_control: option.Option(String),
    download_as: option.Option(String),
  )
}

Constructors

Put(
  key: String,
  body: Body,
  content_type: String,
  cache_control: option.Option(String),
  download_as: option.Option(String),
)

Arguments

cache_control: A Cache-Control header to serve the object with.
download_as: A file name to download the object as, rather than show it.

Storage

opaque </>

Somewhere objects are kept: howdy/storage/s3, howdy/storage/local, howdy/storage/memory, or one of your own.

pub opaque type Storage

Summary

</>

One object in a Page from list. Listing does not tell the content type; head does.

pub type Summary {
  Summary(
    key: String,
    size: Int,
    etag: String,
    last_modified: timestamp.Timestamp,
  )
}

Constructors

Summary(
  key: String,
  size: Int,
  etag: String,
  last_modified: timestamp.Timestamp,
)

UploadRequest

</>

An upload URL, as an adapter is asked for one.

pub type UploadRequest {
  UploadRequest(
    key: String,
    content_type: String,
    size: Int,
    expires_in: Int,
  )
}

Constructors

```
UploadRequest(
  key: String,
  content_type: String,
  size: Int,
  expires_in: Int,
)
```
Arguments

size

The exact size in bytes the upload must have.

expires_in

Seconds from now until the URL stops working.

UrlRequest

</>

A signed URL, as an adapter is asked for one.

pub type UrlRequest {
  UrlRequest(
    key: String,
    expires_in: Int,
    download_as: option.Option(String),
  )
}

Constructors

```
UrlRequest(
  key: String,
  expires_in: Int,
  download_as: option.Option(String),
)
```
Arguments

expires_in

Seconds from now until the URL stops working.

download_as

A file name to download the object as, rather than show it.

Write

opaque </>

Something to store, built with write and stored with put_with.

pub opaque type Write

Values

adapter

</>

pub fn adapter(
  named name: String,
  put put: fn(Put) -> Result(Nil, Error),
  get get: fn(String) -> Result(BitArray, Error),
  head head: fn(String) -> Result(Object, Error),
  delete delete: fn(String) -> Result(Nil, Error),
  list list: fn(String, option.Option(String), Int) -> Result(
    Page,
    Error,
  ),
  url url: fn(UrlRequest) -> Result(String, Error),
) -> Storage

Storage for a service this package does not cover. Keys and URL requests are checked before these functions see them. delete of a missing key succeeds; list takes a prefix, the key to start after and the most objects to return, at least 1 and at most 1000.

attachment

</>

pub fn attachment(file_name: String) -> String

A Content-Disposition value that downloads as file_name: a plain ASCII name for old clients, and the exact name encoded as RFC 6266 allows.

cache_control

</>

pub fn cache_control(
  write: Write,
  cache_control: String,
) -> Write

A Cache-Control header to serve the object with, such as "public, max-age=31536000, immutable" for a key that never changes.

content_type

</>

pub fn content_type(write: Write, content_type: String) -> Write

The content type to serve the object with.

delete

</>

pub fn delete(
  storage: Storage,
  key: String,
) -> Result(Nil, Error)

Remove the object under key. Removing one that is not there succeeds.

delete_unused

</>

pub fn delete_unused(
  storage: Storage,
  prefix prefix: String,
  before before: timestamp.Timestamp,
  in_use in_use: fn(String) -> Bool,
) -> Result(Int, Error)

Delete the objects under prefix last changed before before that in_use says nothing refers to, and say how many went. For uploads that were never confirmed: a browser that uploads directly and never says it is done leaves an object no record points to. Run it now and then, such as from a scheduled job. in_use is asked about each old object; answer True when unsure, such as when the database cannot say, so the object is kept until next time.

let day_ago = timestamp.add(timestamp.system_time(), duration.hours(-24))
storage.delete_unused(files, prefix: "uploads/", before: day_ago,
  in_use: fn(key) { documents.refers_to(db, key) |> result.unwrap(True) },
)

direct_upload_to_json

</>

pub fn direct_upload_to_json(upload: DirectUpload) -> json.Json

A direct upload as JSON, for the endpoint a browser asks for one: {"url": "...", "method": "PUT", "headers": {"content-type": "..."}}.

download_as

</>

pub fn download_as(write: Write, file_name: String) -> Write

Download the object as a file with this name when it is opened, rather than show it in the browser.

download_url

</>

pub fn download_url(
  storage: Storage,
  key: String,
  expires_in expires_in: Int,
  file_name file_name: String,
) -> Result(String, Error)

Like url, but the browser downloads the object as file_name rather than showing it.

error_to_string

</>

pub fn error_to_string(error: Error) -> String

exists

</>

pub fn exists(
  storage: Storage,
  key: String,
) -> Result(Bool, Error)

Whether anything is stored under key.

get

</>

pub fn get(
  storage: Storage,
  key: String,
) -> Result(BitArray, Error)

The whole content of the object under key, in memory. To send a file to a browser, redirect it to a url instead.

head

</>

pub fn head(
  storage: Storage,
  key: String,
) -> Result(Object, Error)

What is stored under key, without its content.

list

</>

pub fn list(
  storage: Storage,
  prefix prefix: String,
  after after: option.Option(String),
  limit limit: Int,
) -> Result(Page, Error)

Up to limit objects whose keys start with prefix, in key order, starting after the key after. limit is kept between 1 and 1000. A prefix is matched as text, so "avatars/" finds avatars/a.png but "avatars" also finds avatars-old/b.png.

max_expiry

</>

pub const max_expiry: Int

The longest a signed URL may last: seven days, the most S3 allows.

max_upload_size

</>

pub const max_upload_size: Int

The largest object one upload may make: 5 GiB, the most S3 takes in a single request.

name

</>

pub fn name(storage: Storage) -> String

new_key

</>

pub fn new_key(
  prefix prefix: String,
  file_name file_name: String,
) -> String

A new, unguessable key under prefix that keeps the extension of file_name, such as avatars/3f9c0b…e1.png. Never build a key from a name a user chose: it may clash with another object or reveal more than intended.

percent_encode

</>

pub fn percent_encode(text: String) -> String

Percent-encode everything but RFC 3986’s unreserved characters, as S3 signing and RFC 5987 both want.

public_url

</>

pub fn public_url(
  storage: Storage,
  key: String,
) -> Result(String, Error)

A permanent URL for the object, for storage set up with with_public_url, such as a public bucket behind a CDN.

put

</>

pub fn put(
  storage: Storage,
  key: String,
  bytes: BitArray,
  content_type type_: String,
) -> Result(Nil, Error)

Store bytes under key, replacing anything there.

put_file

</>

pub fn put_file(
  storage: Storage,
  key: String,
  path: String,
  content_type type_: String,
) -> Result(Nil, Error)

Store the file at path under key, replacing anything there. The file is read in parts, so it need not fit in memory.

put_with

</>

pub fn put_with(
  storage: Storage,
  write: Write,
) -> Result(Nil, Error)

Store what write describes, replacing anything under its key.

retryable

</>

pub fn retryable(error: Error) -> Bool

Whether trying again later might work.

to_service_error

</>

pub fn to_service_error(error: Error) -> service.Error

A service error for a failed call. NotFound is a 404; failures of the service become Internal, which is logged and not shown.

upload_url

</>

pub fn upload_url(
  storage: Storage,
  key: String,
  content_type content_type: String,
  size size: Int,
  expires_in expires_in: Int,
) -> Result(DirectUpload, Error)

A URL a browser can upload a file to itself, without it passing through the app. The upload must be exactly size bytes of content_type and start within expires_in seconds; storage refuses anything else.

The browser sends its file’s size and type first. Check both against your own limits before asking for the URL, choose the key yourself, and check the object with head when the browser says it is done.

url

</>

pub fn url(
  storage: Storage,
  key: String,
  expires_in expires_in: Int,
) -> Result(String, Error)

A URL anyone holding it can read the object at, until expires_in seconds from now. It is signed, so it cannot be changed to name another object.

valid_key

</>

pub fn valid_key(key: String) -> Bool

Whether key can name an object in every adapter: 1 to 1024 bytes of UTF-8, made of /-separated segments that are neither empty, . nor .., nor longer than 255 bytes, with no control characters or backslashes.

with_direct_uploads

</>

pub fn with_direct_uploads(
  storage: Storage,
  upload: fn(UploadRequest) -> Result(DirectUpload, Error),
) -> Storage

Let browsers upload straight to the storage with upload_url. The request is checked before upload sees it.

with_public_url

</>

pub fn with_public_url(
  storage: Storage,
  url: fn(String) -> String,
) -> Storage

Give the storage permanent, unsigned URLs, such as through a CDN in front of a public bucket. public_url fails without one.

write

</>

pub fn write(key: String, body: Body) -> Write

Something to store under key, as application/octet-stream until content_type says otherwise.