At Evooq, we have been working on a life-cycle management platform (LCM) for structured products. Investors can have thousands of structured products in their portfolio (both present and past), and through LCM, we strive to offer them a transparent view of all their products’ current performances.
The price of structured products, like many other investment instruments, is susceptible to market change. Thus, it is vital for investors to monitor their products’ prices and get notified when the price goes above or below a particular value as well as when the daily fluctuation is greater than a threshold. This would allow investors to react quickly during a volatile market.
Requirements
The general flow is straightforward:
- We consume near real-time structured product prices through our market data provider.
- When our market provider pushes us a structured product price, we check our database if any user has set an alert on this product and whether its new price satisfies the user’s conditions.
- We generate notifications for affected users via the user’s chosen means of communication, i.e., currently, our internal webapp notification and email.
While the general flow sounds straightforward, it poses several considerable technical challenges:
- The prices of structured products, which are pushed to us from the market data provider, have high throughput (every 5s for a single product). While we can deduplicate incoming events or cache processed events to reduce the frequency, it may not be beneficial under volatile market conditions when product price continuously fluctuates.
- Many users can monitor the same products. Therefore any price change to this particular product can potentially lead to many notifications generated by our system. Thus, it’s desirable that our system can withstand an unpredictable load and still generate notifications at a comfortable rate.
- While we strive for near real-time notification delivery for users, we also need to send users exactly one notification per channel. We should not miss any event or send users duplicated notifications. This requirement should be true, regardless if our system is being deployed or crashes.
- Extensibility: In the future, we can add different kinds of alerts and also more communication channels for the notification (e.g., mobile push notifications).
Implementation
With those requirements, it’s only natural for us to utilize a queue system. The price event pushed to us will go through the queue before being distributed to process.
This can help us make sure that we can still process the price event at a fixed rate and not crash our system under a high volume of incoming events. Our requirement for a queue system is simple that it supports delivery retry and dead letter queue. Since our application is also hosted on Google Cloud Platform, we decided to use Pubsub.
We also split the notification generation flow into two parts:
- Notification service has the responsibility to consume prices and identify affected users. When a matched user is found, the service dispatches a task to the second queue (on the right) to deliver notifications. Note that this task is per user per channel, so if the user has set alerts on both webapp and email, the notification service will push two events to the queue.
- Notification delivery service picks up the tasks from the queue and generates notifications for that user. The isolation of delivery logic will simplify the integration of new delivery channels.
By splitting the flow, we gain additional benefits:
- We have better control of the processing time of each event. When a hit alert happens for a single product, it can lead to notifications for a large number of users and channels, so it might take a long time to finish. In our approach, since each notification event is per user per channel, its processing time does not depend on how many users subscribed to that particular product.
- If a delivery event fails during processing, Pubsub will automatically re-queue it for processing again. By splitting the delivery for each user, we will only need to re-deliver to a single user.
- We can enforce idempotency better at each step. Before delivering notification, we ensure that the user does not have the notification record in the database for the requested price alert. If the user already has, the event is skipped.
- We can appropriately scale the services since the price event queue has a much higher throughput than the delivery event queue.
Future improvement
We have deployed the above system for a few months, and this architecture has been our foundation to continuously add more types of notifications and delivery channels for our users.
In the future, we want to scale our notification delivery service depending on the number of events in the queue. This helps to ensure that we have a low waiting time for the event, and our users can receive notifications as soon as possible.
