Broken pipe when connecting to Azure Datamarket using http-client-tls

Question

This is now a fixed bug in hs-tls.

Before I can hit the bing translation api, I have to obtain an access token. This basically amounts to the following snippet which works fine from curl

curl -Lp --data-urlencode client_id=$APPID \
         --data-urlencode client_secret=$SECRET \
         --data-urlencode scope='http://api.microsofttranslator.com' \
         --data-urlencode grant_type="client_credentials" \
         https://datamarket.accesscontrol.windows.net/v2/OAuth2-13

and responds immediately with some JSON containing the token. Bear in mind that even bad/malformed requests elicit some response containing an error message although also typically a 400 Bad Request HTTP response code.

When I try to replicate this in Haskell it fails right away (reformatted):

*** Exception: FailedConnectionException2 "datamarket.accesscontrol.windows.net"
    443 True <socket: 11>: hPutBuf: resource vanished (Broken pipe)

Further, when I try a simpler (https) request to a different host everything works as expected.

This question is solved when someone can tell me what is going on here.

If you get it working with another library that's useful too, but not really what this question is about.

A self contained example is shown below.

{-# LANGUAGE OverloadedStrings #-}

module Test where

import Control.Monad
import Data.Maybe
import Network.HTTP.Client as N
import Network.HTTP.Client.TLS as N
import qualified Data.ByteString.Lazy.Char8 as B
import qualified Data.ByteString.Char8 as BS
import Debug.Trace


badURL :: String
badURL = "https://datamarket.accesscontrol.windows.net/v2/OAuth2-13"

goodURL :: String
goodURL = "https://httpbin.org/post"

requestData :: [(BS.ByteString, BS.ByteString)]
requestData = [
  ("client_id", "APPID"),
  ("client_secret", "SECRETSECRET"),
  ("scope", "http://api.microsofttranslator.com"),
  ("grant_type", "client_credentials")]

mkReq :: String -> Request
mkReq url = urlEncodedBody requestData . fromJust $ parseUrl url

go :: String -> IO ()
go url = do
  let addHeaders h r = r { requestHeaders = h ++ requestHeaders r }
      headers = [("User-Agent", "curl/7.39.0")
                ,("Accept", "*/*")
                ]
      req = addHeaders headers (mkReq url)
  N.withManager N.tlsManagerSettings $ \man -> do
    resp <- httpLbs req man
    guard (trace "wefwef" True)
    B.putStrLn $ responseBody resp
    return ()

{-
 - *Main> go goodURL 
 - wefwef
 - {
 -   "args": {}, 
 -   "data": "", 
 -   "files": {}, 
 -   "form": {
 -     "client_id": "APPID", 
 -     "client_secret": "SECRETSECRET", 
 -     "grant_type": "client_credentials", 
 -     "scope": "http://api.microsofttranslator.com"
 -   }, 
 -   "headers": {
 -     "Accept": "*/*", 
 -     "Accept-Encoding": "gzip", 
 -     "Connect-Time": "1", 
 -     "Connection": "close", 
 -     "Content-Length": "119", 
 -     "Content-Type": "application/x-www-form-urlencoded", 
 -     "Host": "httpbin.org", 
 -     "Total-Route-Time": "0", 
 -     "User-Agent": "curl/7.39.0", 
 -     "Via": "1.1 vegur", 
 -     "X-Request-Id": "3ed2de84-a1a3-4560-b7e3-f2dabfe45727"
 -   }, 
 -   "json": null, 
 -   "origin": "108.85.134.2", 
 -   "url": "https://httpbin.org/post"
 - }
 - *Main> go badURL 
 - *** Exception: FailedConnectionException2 "datamarket.accesscontrol.windows.net" 443 True <socket: 11>: hPutBuf: resource vanished (Broken pipe)
 -}

EDIT:

I've tried tls-simpleclient and tls-retrievecertificate as well but also got broken pipe errors.

$ tls-simpleclient -d -v datamarket.accesscontrol.windows.net                                          
sending query:
GET / HTTP/1.0



debug: >> Handshake [ClientHello TLS12 (ClientRandom "\235=wnV\156z\143M\168+n\165`\193\217\132G\144\204\187\178\SOHG\156\EM\195\168\251l\232+") (Session Nothing) [107,103,57,51,56,50,47,53,4,5,10] [0] [(0,"\NUL'\NUL\NUL$datamarket.accesscontrol.windows.net"),(65281,"\NUL")] Nothing]
debug: >> Alert [(AlertLevel_Fatal,InternalError)]
tls-simpleclient: send: resource vanished (Broken pipe)

$ .cabal-sandbox/bin/tls-retrievecertificate datamarket.accesscontrol.windows.net 443                                     
connecting to datamarket.accesscontrol.windows.net on port 443 ...
tls-retrievecertificate: send: resource vanished (Broken pipe)

Answer 1

It seems that datamarket.accesscontrol.windows.net doesn't like Haskell's TLS implementation.

You can add logging of the TLS packets by doing this:

In the connection package, make the following changes to Network/Connection.hs

Add import Network.TLS ((Logging(..))

Modify tlsEstablish as follows:

  tlsEstablish :: Handle -> TLS.ClientParams -> IO TLS.Context
  tlsEstablish handle tlsParams = do
      rng <- RNG.makeSystem
      ctx <- TLS.contextNew handle tlsParams rng
 +    let logging = def { loggingPacketSent = (\s -> putStrLn $ "<-- sent packet " ++ s)
 +                      , loggingPacketRecv = (\s -> putStrLn $ "--> recv packet " ++ s)
 +                      }
 +    TLS.contextHookSetLogging ctx logging
      TLS.handshake ctx
      return ctx

When you execute go goodURL you see the following traffic:

<-- sent packet Handshake [ClientHello TLS12 ...
--> recv packet Handshake [ServerHello TLS12 ...
...

But when you execute go badURL you just see:

<-- sent packet Handshake [ClientHello TLS12
<-- sent packet Alert [(AlertLevel_Fatal,InternalError)]
*** Exception: FailedConnectionException2 "datamarket.accesscontrol.windows.net" 443 True <socket: 13>: hPutBuf: resource vanished (Broken pipe)

The interpretation is:

1. client sends Handshake packet
2. remote side closes connections
3. Broken pipe exception raised on client
4. client sends Alert packet to notify remote side of an internal error (i.e. the broken pipe exception)
5. client raises FailedConnectionException2

Update: You can accomplish the same thing by running:

tls-simpleclient -d -v datamarket.accesscontrol.windows.net

where tls-simpleclient can be found in https://github.com/vincenthz/hs-tls