JonLuca's Blog

12 Oct 2019

Reversing private APIs, Safeway, and not-so-extreme couponing

Safeway is an American supermarket chain that has historically had a pretty comprehensive coupon program. They somewhat recently tried to modernize their offerings (as they were founded in 1915, and are pretty much the definition of “old guard” companies), by allowing online coupons. This brought what used to be a painstaking process of clipping coupons from magazines to a slightly less painful process of having to click “Add” on all the coupons every week after logging onto their site.

I’m not particularly invested in couponing, but figured with so many coupons offered and how often I need to get groceries, it might be interesting to explore automating adding all the coupons. It probabl won’t be worthwhile in the long run, but at the very least it should provide for an interesting afternoon.

Background

You’re able to automatically add coupons to your Safeway account (“just for U”) online at https://www.safeway.com/justforu/coupons-deals.html.

Safeway offers

Safeway offers

Navigating to the coupon page lets you see 30 coupons. At the bottom there is a “Load more” button, which will load an additional 30 coupons.

Safeway add

Safeway Load More

I assumed that it was requesting the new coupons from some API, with a page parameter, but the network tab was suspiciously empty every time I clicked “Load More”.

Getting all the coupons

It turns out that the coupon page makes a request to https://www.safeway.com/abs/pub/web/j4u/api/ecomgallery?offerPgm=PD-CC&storeId=3031&transformOfferbyUpc=y when it’s your first time on the site, and stores them in localstorage on subsequent requests. This loads all the coupons (over 340 for me) on first page load.

Unfortunately this is not a completely open URL, so you’ll need to have a valid “API” key (that is included in the first authenticated request to the page) to be able to access it.

Safeway Coupons

Safeway coupons stored in the coupon grid component state

Each of these coupons have specific information related to it:

{
  "offerId": "1769990482",
  "priceType": "88",
  "image": "https://www.safeway.com/CMS/j4u/offers/images/5_dollars_off.gif",
  "offerPrice": "$5 OFF",
  "description": "*Standard exclusions apply.",
  "status": "U",
  "startDate": "1570644000000",
  "endDate": "10/15/2019",
  "offerTs": "1570554250722",
  "usageType": "O",
  "offerPgm": "PD",
  "purchaseInd": "B",
  "extlOfferId": "2588K4119_760013",
  "offerProvider": "NextGenAutoLoadStoreCpns",
  "imageId": "5_dollars_off.gif",
  "hierarchies": {
    "categories": ["Special Offers"],
    "events": []
  },
  "name": "$5 Off Your Purchase of $50 or More",
  "offerID": "1769990482",
  "disclaimer": "Save $5 when you spend $50 or more in qualifying purchases in a single transaction using your just for U® account ($50 minimum purchase calculated after deduction of promotional and loyalty card savings, coupons and all other discounts, offers and savings). Offer is valid 10/09/2019 - 10/15/2019 only at participating Safeway stores located in Northern California and Western Nevada. Qualifying purchases exclude: alcoholic beverages, tobacco products, fluid dairy, fuel, prescription items and co-payments, fishing/hunting licenses and tags, postage stamps, money orders/transfers, bus/commuter passes, amusement park/ski/event/lottery tickets, phone cards, gift cards/certificates and applicable taxes, bottle/container deposits and bag fees.",
  "priceTitle": "Special Offer",
  "limits": "O",
  "categoryType": "Special Offers",
  "purchaseRank": "2",
  "arrivalRank": "16",
  "expiryRank": "16",
  "deleted": false,
  "category": "Special Offers",
  "events": [],
  "offerType": "PD",
  "isGroupSplit": true,
  "updBtnText": "Added",
  "updBtnMode": true,
  "usageTypeTxt": "One time use",
  "couponID": "1769990482",
  "brand": "$5 Off Your Purchase of $50 or More",
  "newOfferPrice": "Special Offer $5 OFF",
  "refineBrand": "$5 Off Your Purchase of $50 or More",
  "refineDescription": "*Standard exclusions apply."
}

There would be two paths forward to make this automatable - either go through the sign in procedure in a headless browser like puppeteer, and then just dump localStorage.getItem('abCoupons'), like so:

Safeway local storage

Safeway coupons retrieved from local storage

The other way would be to reverse their authentication route and make the request directly using raw requests. Once we get the API token, then we can make requests directly to the above endpoint. This would be a much more reproducible (and faster) way of accomplishing the goal, so we’ll stick to this way.

First we’ll explore how to add coupons to our account, and then connect it all together to make it fully automated.

Adding a coupon to your account

Fortunately reversing the add coupon functionality is fairly trivial. Clicking “Add” fires off a POST request to https://www.safeway.com/abs/pub/web/j4u/api/offers/clip with storeId as a URL parameter.

Safeway add coupon

Adding a coupon to your account on safeway

The body of the request is, interestingly enough, the same ID with two clipTypes. I’m not entirely sure what C and L stand for, but it seems they’re the same for any coupon type or category.

{
  "items": [
    {
      "clipType": "C",
      "itemId": "169451067",
      "itemType": "PD"
    },
    {
      "clipType": "L",
      "itemId": "169451067",
      "itemType": "PD"
    }
  ]
}

From here it’s easy to convert the request into code. My preferred way is to right click -> Copy as cURL, then use a tool like curlconverter to turn it into python:

import requests

cookies = {
    'abc': 'abc',
    # Cookies omitted due to session
}

headers = {
    # headers omitted due to session
}

params = (
    ('storeId', 'my_store_id'),
)

data = '{"items":[{"clipType":"C","itemId":"836190395","itemType":"PD"},{"clipType":"L","itemId":"836190395","itemType":"PD"}]}'

response = requests.post('https://www.safeway.com/abs/pub/web/j4u/api/offers/clip', headers=headers, params=params, cookies=cookies, data=data)

After playing around with it, I figured out that you only needed two headers, and none of the cookies, to successfully add a coupon to your account - the SWY_SSO_TOKEN header, and 'Content-Type': 'application/json' to designate the content type of the request.

SWY_SSO_TOKEN is actually a jwt with the following format:

{
  "ver": 1,
  "jti": "<redacted>",
  "iss": "https://albertsons.okta.com/oauth2/<redacted>",
  "aud": "Albertsons",
  "iat": 1570908253,
  "exp": 1570910953,
  "cid": "<redacted>",
  "uid": "<redacted>",
  "scp": ["used_credentials", "offline_access", "openid", "email", "profile"],
  "zip": "<my zip code>",
  "sub": "<my email address>",
  "hid": "<redacted>",
  "dnm": "<my name>",
  "aln": "<redacted>",
  "gid": "<redacted>",
  "ecc": "",
  "fnm": "<my name>",
  "lnm": "<my name>",
  "uuid": "",
  "jpr": "",
  "ban": "safeway",
  "str": "<my store id>",
  "phn": "<my phone number>",
  "bid": ""
}

We can’t construct our own JWTs as we don’t have their signing secret, but it’s interesting nonetheless.

Cleaning up the code a little and it looks like this:

import requests



def add_coupon_by_id(id, store_id, type):
	headers = {
    'SWY_SSO_TOKEN': '<SWY_SSO_TOKEN> jwt',
    'Content-Type': 'application/json',
	}

	params = (
	    ('storeId', store_id),
	)
	t = Template('{"items":[{"clipType":"C","itemId":"${id}","itemType":"${type}"},{"clipType":"L","itemId":"${id}","itemType":"${type}"}]}')
	data = t.substitute(id=str(id), type=str(type))

	response = requests.post('https://www.safeway.com/abs/pub/web/j4u/api/offers/clip', headers=headers, params=params, data=data)
	print(response.status_code)

add_coupon_by_id(836190395, 0)

This looks like an atomic action, and calling clip multiple times doesn’t perform any state changes.

Adding all coupons

To start I just dumped the JSON object containing all the coupon offers into the test python file and iterated over it like so:

for id in coupons['offers'].keys():
	add_coupon_by_id(id, 0)

This worked really well. It turns out there are also two ways offers are stored - by regular coupon ID, like above, and then by UPC, so you need to make sure to do both.

These are stored as UPC keys with an array of offers.

Safeway upc

Safeways second offer storage, by UPC

If you know the UPC of an item you’re following, it would be pretty easy to add functionality that shows you all offers for that item.

After running my script (which took a little over a minute), I refreshed the page and saw that all the offers had been added.

Safeway all added

Safeways coupon screen after running the script

Reversing the Auth API

The last step of the process is to find a way to generate SWY_SSO_TOKEN programmatically, so it can all be done in one function (and run on a cron job).

Safeway auth flow

Authentication flow on safeway.com

Upon filling in your credentials and hitting enter, the auth flow seems to function as follows:

  1. The browser fires an OPTIONS request to https://albertsons.okta.com/api/v1/authn. Since we’re doing this programmatically, we don’t need to worry about this (as it’s for cross origin request safety, a browser safety feature). The best part of this is that they’re actually using Okta, as their auth, more info on this later.

  2. The browser fires a POST request with a body of {username: "<email>", password: "<password>"}.

  3. The browser fires a GET request to https://albertsons.okta.com/oauth2/ with the state from the previous request.

  4. The browser fires a GET request to the static redirect URL, which is https://shop.safeway.com/bin/safeway/sso/authorize?code=<code>&state=<state>

  5. That route replies back with Set-Cookie that sets our SWY_SSO_TOKEN.

The interesting thing about (2) is there is no unique token or requirements on the route - all you need to do is set the Content-Type and the request will go through:

headers = {
    'Content-Type': 'application/json',
}

data = '{"username":"<email>","password":"<password>"}'

response = requests.post('https://albertsons.okta.com/api/v1/authn', headers=headers, data=data)
print(response.text)

This replies back with:

{
  "expiresAt": "2019-10-12T20:56:08.000Z",
  "status": "SUCCESS",
  "sessionToken": "<session token>",
  "_embedded": {
    "user": {
      "id": "<id>",
      "passwordChanged": "2019-09-18T01:51:47.000Z",
      "profile": {
        "login": "<email>",
        "firstName": "<firstname>",
        "lastName": "<lastname>",
        "locale": "en",
        "timeZone": "America/Los_Angeles"
      }
    }
  },
  "_links": {
    "cancel": {
      "href": "https://albertsons.okta.com/api/v1/authn/cancel",
      "hints": {
        "allow": ["POST"]
      }
    }
  }
}

The great part of all this is that there’s no guessing involved - this is actually the Okta auth flow, which is heavily documented on their own site. The last part of this is to just take this session and pass it along with the static application ID to the last okta, along with the static redirect URL.

Since this is just using okta’s sign in flow, we can play around with the parameters and authorization type - instead of a code response, we could accept a bearer token instead.

Having a bearer token lets us hit their nimbus API directly. I extracted the API route by man-in-the-middle-ing myself on iOS and used BurpSuite to find the route, but I’ll skip that section for brevity. Just know that all the above sections remain true, the only difference is I’ll just be hitting their mobile API first instead (which is hosted on nimbus.safeway.com).

All it takes to authenticate is the following code:

import requests

headers = {
    'Host': 'albertsons.okta.com',
    'Accept': 'application/json',
    'Authorization': 'Basic <basic auth>',
    'Accept-Encoding': 'gzip, deflate',
    'Accept-Language': 'en-us',
    'Content-Type': 'application/x-www-form-urlencoded',
    'charset': 'utf-8',
}

data = 'username=<username>&password=<password>&grant_type=password&scope=openid profile offline_access'

response = requests.post('https://albertsons.okta.com/oauth2/ausp6soxrIyPrm8rS2p6/v1/token', headers=headers, data=data, verify=False)

Note that ausp6soxrIyPrm8rS2p6 is the ID okta assigned to Safeway.

The response has a access_token that can be used to authenticate as myself against their raw API.

The final request to their Nimbus API will look like:

import requests

cookies = {
    'swyConsumerDirectoryPro': '<same bearer token>',
}

headers = {
    'Host': 'nimbus.safeway.com',
    'Authorization': 'Bearer <same bearer token>',
}

params = (
    ('storeId', '<store id>'),
)

response = requests.get('https://nimbus.safeway.com/emmd/service/gallery/offer/mfg>', headers=headers, params=params, cookies=cookies, verify=False)

The response format is slightly different for mobile, but still has all the requisite information (specifically, we need our store ID, the offerPgm, and the couponID).

{
  "ack": "0",
  "errors": null,
  "resultCount": 326,
  "manufacturerCoupons": [
    {
      "priceType": null,
      "priceSubType": null,
      "image": "https://www.safeway.com/CMS/j4u/offers/images/440/979440_95beddb0-deab-4943-84f4-30c4a7b1fafc.gif",
      "offerPrice": "SAVE $1.00",
      "description": "on ONE (1) Stonyfield® Organic Kids Multipacks, Tubes or Pouches (4pk or larger)",
      "status": "C",
      "startDate": "\\/Date(1569866400000)\\/",
      "endDate": "\\/Date(1572544800000)\\/",
      "offerTs": "1569341944086",
      "usageType": "O",
      "offerPgm": "MF",
      "offerSubPgm": "09",
      "purchaseInd": "L",
      "commonPromoCodes": null,
      "categoryRank": "null",
      "clipTs": null,
      "events": [],
      "redemptions": [],
      "couponID": "1116974738",
      "brand": "Stonyfield® Organic",
      "generalDisclaimer": "Not subject to doubling. Void if sold, reproduced, altered, or expired and wherever taxed, regulated, restricted or prohibited by law. Limit one coupon per purchase of specified product(s). Consumer pays applicable sales tax. Coupon may not be combined with any other offer. Coupons not properly redeemed will be voided. Retailer: For each coupon accepted as an authorized agent we will pay you the face value of the coupon plus 8 cents handling. Invoices proving purchase of sufficient stock to cover all coupons redeemed must be shown upon request. Cash value 1/20 cent. Redeem by mail to: Stonyfield Farm, Inc., CMS Dept. # 52159, One Fawcett Drive, Del Rio, TX 78840. Void in LA & NV.",
      "category": "Dairy, Eggs & Cheese",
      "vndrBannerCd": "2bbf4b36-a75e-11e6-80f5-76304dec7eb7#C",
      "purchaseRank": "1",
      "arrivalRank": "1",
      "expiryRank": "1"
    }
  ]
}

This returns all the coupons for a given store, which allows me to use the code from above to completely automate the process. I use the response from this code block to populate all the calls to add_coupon_by_id, and now it’s a closed loop system!

Automating it

I saved all this in a file called safeway_coupons, that I set up in a cron job. I threw it up on one of my personal production boxes and have it run every night at midnight with 0 0 * * * /usr/bin/python3 ~/safeway_coupons. Tada! Every day all my coupons are synced and added automatically, and I don’t need to think about it when I go to the store. Whatever coupons I may have will be applied automatically when I check out, and I don’t need any of the mental overhead of couponing (especially since I’m not someone that “needs” to get the best deal - if what I’m buying has a coupon then great, but I’m not going to change my purchases based on coupons)

Bonus: Speeding things up

It took nearly a minute to make 300 network requests - only about 5 requests per second, which was way too slow. I was running the code in a linear fashion - we only make the request to add the next coupon once the first has completed.

I reused a thread pool from one of my previous articles (/r/churning stats), as below:


class Worker(Thread):
  """ Thread executing tasks from a given tasks queue """

  def __init__(self, tasks):
    Thread.__init__(self)
    self.tasks = tasks
    self.daemon = True
    self.start()

  def run(self):
    while True:
      func, args, kargs = self.tasks.get()
      try:
        func(*args, **kargs)
      except Exception as e:
        # An exception happened in this thread
        print(e)
      finally:
        # Mark this task as done, whether an exception happened or not
        self.tasks.task_done()


class ThreadPool:
  """ Pool of threads consuming tasks from a queue """

  def __init__(self, num_threads):
    self.tasks = Queue(num_threads)
    for _ in range(num_threads):
      Worker(self.tasks)

  def add_task(self, func, *args, **kargs):
    """ Add a task to the queue """
    self.tasks.put((func, args, kargs))

  def map(self, func, args_list):
    """ Add a list of tasks to the queue """
    for args in args_list:
      self.add_task(func, args)

  def wait_completion(self):
    """ Wait for completion of all the tasks in the queue """
    self.tasks.join()

And then wrote a quick wrapper to the above function (since Python mapping doesn’t support multiple arguments without using partial 🙄).

def add_coupon_by_id_wrapper(params):
	add_coupon_by_id(*params)


pool = ThreadPool(5)

values = []

for id in coupons['offers'].keys():
	values.append([id, 0, coupons['offers'][id]['offerPgm']])


pool.map(add_coupon_by_id_wrapper, values)
pool.wait_completion()

Now it makes roughly 25 RPS, which completes in about 10 seconds. Much better!

Comments on reddit and hackernews.

JonLuca

JonLuca at 13:16

To get notified when I publish a new essay, please subscribe here.