A Cloud Foundry deployment can be configured to log extra HTTP headers on every request to enrich the log messages produced by the gorouters. This is highly useful to add trace headers to all requests going through the Cloud Foundry deployment. As of writing this, there is a configuration option for this feature available in the OpsManager API, but not in the OpsManager UI itself. So to configure the extra headers to log with Cloud Foundry, we’ll need to hop on a shell.

Gathering Required Information

To enable the logging of extra headers you will need three pieces of information: the address of the OpsManager, the BOSH GUID if the elastic runtime, and an access token for the OpsManager.

OpsManager Address

<OPSMGR>: get either the FQDN or the IP address of your OpsManager instance, depending on what you’ve used to install OpsManager. You will need this to target requests to the OpsManager API.

Elastic Runtime GUID

<CF_GUID>: The GUID of the elastic runtime is required to target the correct BOSH deployment with our requests. To determine the elastic runtime GUID, you may either use the BOSH CLI or issue another curl request to the OpsManager API as follows:

# either with BOSH CLI
bosh deployments
# or with curl
curl https://<OPSMGR>/api/v0/staged/products

In both cases, look for the cf deployment and get its GUID.

OpsManager Access Token

<ACCESS_TOKEN>: To authorize the request you are about to send you’ll need an access token for the OpsManager API. Retrieving the access token is as simple as:

$ gem install cf-uaac
$ uaac target https://<OPSMGR>/uaa
$ uaac token owner get
Client name: opsman
Client secret:
User name: YOUR_USERNAME_HERE
Password: YOUR_PASSWORD_HERE
$ uaac context

Skim through the output of the uaac context to find the access token.

Note that there is already a pre-installed uaac client on your OpsManager VM, so you may just SSH there and use that instead of fighting Ruby on your machine.

Enable Extra Headers with the OpsManager API

To log extra headers the following curl request is sufficient. Simply replace the placeholders with the values gathered above and you are good to go:

curl "https://<OPSMGR>/api/v0/staged/products/<CF_GUID>/properties" \
    -X PUT \
    -H "Authorization: Bearer <ACCESS_TOKEN>" \
    -H "Content-Type: application/json" \
    -d '{
        "properties": {
            ".router.extra_headers_to_log": {"value": "<HEADER_ONE,HEADER_TWO>" }
        }
    }'

Note that if you want to log multiple headers the .router.extra_headers_to_log property simply expects a comma-separated list of header names to log.

After issuing the requets, go to your OpsManager UI and hit the Apply Changes button. Or if you will, issue another curl request:

curl "https://<OPSMGR>/api/v0/installations" \ 
    -X POST \ 
    -H "Authorization: Bearer <ACCESS_TOKEN>" \ 
    -H "Content-Type: application/json"

Check Extra Headers

To check whether the extra headers work correctly issue an HTTP request to an app running on Cloud Foundry. Then, with the CF CLI check the logs with cf logs <APP> --recent and see whether the extra headers are being logged. If so, you should see something like:

$ cf logs <APP> --recent
2017-05-31T08:34:36.12+0200 [RTR/2] OUT winter-music.mycf.io - [2017-05-31T06:34:35.679+0000] 
  "GET /albums HTTP/1.1" 200 0 4297 "http://winter-music.mycf.io/" [...] 
  vcap_request_id:"6d36e138-7fd4-25e0-6425-511350e31145" response_time:0.450099021 
  app_id:"f1765de2-dbe7-4a12-198a-36b6248ea093" app_index:"0" 
  header_one:"my_extra_header" header_two:"another extra header"

Now that those extra headers are being logged, make sure to forward the log messages of your foundation to a log message tool capable of leveraging that extra piece of information.