Wed 11 March 2015
By Tristan Terpelle howtos

nginx User-Agent spoofing to circumvent LinkedIn's blacklisting

I have a few virtual hosts that redirect to my LinkedIn profile thanks to a rewrite rule in nginx:

server {
  server_name  example.com;
  listen       80;

  location / {
    rewrite    ^  http://be.linkedin.com/in/tristanterpelle/;
  }
}

This worked beautifully when I tested it, but today I noticed it was broken. LinkedIn returned a 999 error. After looking around a bit, it turns out that LinkedIn actively blocks HTTP requests from clients with certain User-Agent strings.

 $ curl -I --url https://be.linkedin.com/in/tristanterpelle 
HTTP/1.1 999 Request denied

$  curl -A "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36" -I --url https://be.linkedin.com/in/tristanterpelle
HTTP/1.1 200 OK

OK, so curl's User-Agent is blacklisted, and apparently so is nginx' (wget also fails). Luckily, nginx can spoof its User-Agent thanks to the HttpHeadersMoreModule.

server {
  server_name  example.com;
  listen       80;

  location / {
    rewrite    ^  http://be.linkedin.com/in/tristanterpelle/;
    more_set_input_headers 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36';
  }
}

On Debian, you need to install the nginx-extras package, or you will get a [emerg] unknown directive "more_set_input_headers" error.

Tagged , ,