Every bank creates offers for its customers on their websites and apps. Customers find it challenging to discover these offers and redeem before they expire, especially when holding multiple credit cards and figuring out how to redeem them at retail stores and restaurants.
91social had to come up with a solution that would consolidate all offers from all banks into a single place, classify by bank, card type, offer domain (travel, dining etc), and keep them in sync with the bank’s website. Banks do not publish these offers by API or feeds, so they have to be extracted from websites, interpreted and standardized so they can be presented in the same way to the customer irrespective of the bank.
A lowcode crawler was built to configure websites using a simple ruleset and extensible with code at runtime, so new websites could be configured fast without extensive code changes. The crawler was built to mimic a real user’s behavior, to handle static and dynamic websites where offers are visible after a few user interactions. It was built as a cluster of virtual browsers handling multiple pages and a pipeline to handle offers as they were identified, data extracted and normalized to a standard format.