Algolia
Tech Stack
# Admin portal
(1) Rails with bootstrap UI
(2) Admin: AWS hosting with RDS (MySQL) and EC2 in 2 different availability zones
(3) ELB for load balancing.
(4) Use CloudFont to store all assets in order to provide a good experience anywhere in the world
====
# API server
(1) Store some real-time data using Redis. They are:
* the last logs of users
* the last error logs
* api counter
* Full log processing is done by another cluster (elasticsearch).
(3) If data is sent to our API, we replicate the data on 3 different hosts which are the API servers. So, indexing is done on 3 different hosts.
(4) They use a consensus algorithm to be sure to be synchronized. (ie. RAFT, kinda Paxos algo).
(5) You’ve got to ensure not only that you have replicated the data, but that it is in the same exact order, so these three hosts must always be synchronized and in the same state.
(6) We also use redundancy for performance b/c we share teh queries across 3 different hosts.
(7) Search engine is written in C++ that is directly embedded in Nginx.
(8) High end machines
* Dedicated servers
* 256G RAM
* Attached 1TB of SSD in RAID-0
* 16 physical cores >= 3.5GHz
(9) Blazingly fast bandwidth
* 4.5 GBps of dedicated bandwidth per cluster
(10) Each index is a binary file in our own format. We put the information in a specific order so that it is very fast to perform queries on it. This is the algorithm that we have developed for mobile.
(11) Nginx C++ module will directly open the index file in memory-mapped mode in order to share memory between the different Nginx processes and will apply teh query on the memory-mapped data structure.
(12) API Load balancing (select random servers to failover)
(13) Chaos Monkey approach from Netflix for Testing.
(14) Code Climate to test code quality
(15) Coveralls for code coverage
(16) For production, we have an automatic deployment script that applies a big set of unit tests and non-regression tests before deployment. We also have an automatic script for rollback operation if a problem is found with the new version.
(17) Use Server Density to monitor our servers. For the frontend (Admin), we use cloudwatch.
(18) Geo routing: Route53 => Another one
(19) Use intercom for user onboarding in the dashboard
(20) Use GA and KISSmetrics to track growth metrics and identify the steps we can improve in our funnel.
(21) Use HipChat internally or for customer to feedback.
(22) Use Twilio to send SMS when a probe detects an issue
(23) Use Stripe for payments.
(24) Google SparseHash Library - very low level and high performance hashtable that are using in all the low level C++ code.
(25) Depstack - tool to find and vote for popular libraries.
Search and Relevancy
- ElasticSearch: Regular TF-IDF ranks higher documents that contain many times the query terms. And it is a statictical formula that will generate a score for each document. It is a bit hard for you to figure out why needless to say tune it.
- Algolia: It believes that the statistics make sense if you have a webpage like wikipedia where you have a huge amount of text. For ones don't have such large amount of text, it is better we put more focus on where the term appears than its occurrences in the text.
Example
If you search for an iPhone on an e-commerce website, what results you are preferring to see?
- iPhone 6s ranks higher than iPhone 3G
- Ranks the ones with higher sales better
- Misspell
In Algolia, you can customize the search result via:
- Specify the importance of attributes like title > desc.
- Define the popularity of the objects. For example, the # of likes or # of sales.