
Credit card information is stored in a third party gateway service to meet PCI-DSS compliance requirements. Besides cost at scale, it was limiting the user experience to present the card information in innovative ways.
The brief was to build an inhouse secure store that:
Could integrate easily with existing apps and services
Could scale to support millions of customers with API operations at p99 less than 100ms
Was flexible to secure data beyond cards
Supported PCI-DSS compliant and non-compliant clients
The solution was built using lightweight secure libraries and keys were managed in AWS KMS.
Single-use tokens were provided to retrieve PANs for short duration
The store provided masked/truncated data for PANs to display in apps
The multi-tenant store isolated PANs by any combination of client or group
Expired PANs were deleted on a scheduled maintenance cycle
An audit trail was created of every API call with latency and customer ID
Communication between servers was via GRPC and Protobuf was the in-memory data format
The data was secured using the following design principles:
Card data was isolated from the token and key handling system. Separate databases were deployed
Keys were rotated by lifetime and number of cards encrypted, so any exposure of a single key limited the number of cards affected
Keys and unencrypted data were never stored on a disk
Card data and keys when needed were held in memory in encrypted form for short durations
Sensitive info was never written to logs
A design view of the system

The solution delivered card retrieval at a p99 of 50ms and card additions at p99 of 100ms, delivered from 2 cores and 2GB RAM in a cluster mode of load balancer and distributed cache, under a sustained load of 100 RPS for 10 minutes.