How to create a Google Cloud Storage Signed URL using Elixir or Erlang?

Question

I'm currently trying to get my elixir web server to generate signed urls for Google Cloud Storage so that I can generate file urls that expire. Unfortunately when I try to use the generated urls I get the following error:

<Code>SignatureDoesNotMatch</Code>
<Message>
The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.
</Message>

I am able to generate signed urls that work via the gsutil tool, although it is quite slow, and also via the python example given here:

Google Cloud Storage Signed URLs Example

My current implementation in Elixir is based on the above Python example and looks like this:

@default_expiration 1000
  def construct_string(http_verb, content_md5, content_type, expiration_timestamp, canonicalized_extension_headers, canonicalized_resource) do
    "#{http_verb}\n
    #{content_md5}\n
    #{content_type}\n
    #{expiration_timestamp}\n
    #{canonicalized_extension_headers}
    #{canonicalized_resource}"
  end

  def load_secret_pem do
    load_local_key_file("/path/to/key")
  end

  def load_local_key_file(path) do
    {ok, pem_bin} = File.read(path)
    [rsa_entry] = :public_key.pem_decode(pem_bin)
    key = :public_key.pem_entry_decode(rsa_entry)
  end

  def base64Sign(plaintext) do
    key = load_secret_pem()
    signature_bytes = :public_key.sign(plaintext, :sha256, key )
    Base.url_encode64(signature_bytes)
    |> String.replace("-", "%2B")
    |> String.replace("_", "%2F")
    |> URI.encode_www_form
  end

  def make_url(verb, path, content_md5 \\ "", content_type \\ "") do
    client_id = GCloud.Credentials.client_email() |> URI.encode_www_form
    expiration =  :os.system_time(:seconds) + @default_expiration
    base_url = GCloud.Storage.base_uri() <> path
    signature_string = construct_string(verb, content_md5, content_type, expiration, "", path )
    url_encoded_signature = base64Sign(signature_string)
    IO.puts "#{base_url}?GoogleAccessId=#{client_id}&Expires=#{expiration}&Signature=#{url_encoded_signature}"
  end

How are signed urls correctly signed using Elixir or Erlang?

Answer 1

Your string construction in construct_string may be doing things you do not realize. Remember that Python syntax is not the same, and has other opinions on spaces.

defmodule Test do
  def foo(a,b) do
    "#{a}\n
    #{b}"
  end
end
IO.inspect Test.foo(1,2)
# output:
"1\n\n    2"

If you use a heredoc with """ instead, the leading spaces go away, but your newlines are still duplicated. This approach is probably a bad idea, though, because if you save the file from a windows machine, you may have \r\n as the line ending in the editor, and getting rid of those is an unnecessary annoyance anyway.

Instead, I think you should change your approach here to be something like this:

def construct_string(http_verb, content_md5, content_type, expiration_timestamp, canonicalized_extension_headers, canonicalized_resource) do
  headers = Enum.join([http_verb, content_md5, content_type, expiration_timestamp], "\n")
  "#{headers}\n#{canonicalized_extension_headers}#{canonicalized_resource}"
end

I'm not sure if there are any other errors, but this stands out to me immediately.

Answer 2

I managed to get this working, I did this by opening a python and elixir REPL side by side, executed each step in both with a test string and compared the output for discrepancies, there where no discrepancies after hashing or signing the test string, but there was after base64 encoding, so I changed:

def base64Sign(plaintext) do
    key = load_secret_pem()
    signature_bytes = :public_key.sign(plaintext, :sha256, key )
    Base.url_encode64(signature_bytes)
    |> String.replace("-", "%2B")
    |> String.replace("_", "%2F")
    |> URI.encode_www_form
end

to

def base64Sign(plaintext) do
    key = GCloud.Credentials.load_secret_pem()
    signature_bytes = :public_key.sign(plaintext, :sha256, key )
    Base.encode64(signature_bytes)
    |> URI.encode_www_form
end

This combined with asonge's string construction advice solved the issue.

Answer 3

using gcs_signer from hex.pm:

Application.get_env(:goth, :json)
|> Poison.decode!
|> GcsSigner.Client.from_keyfile()
|> GcsSigner.sign_url(bucket, object)

In Google Cloud Platform environments, such as Cloud Functions and App Engine, you usually don't provide a keyFilename or credentials during instantiation,
then you can use signBlob api see this example