10K's

Scale From Zero To Millions of Users

A step by step example

Single server setup

1. User access web throught domain names. The Domain Name System(DNS) is a paid service provided by 3rd party and not hosted by our servers.

2. Internel Protocal(IP) address is returned to browser or app.

3. Once the IP is obtained, Hypertext Tranfer Protocal(HTTP) requests are sent directly to web server.

4. The web server return HTML pages or JSON response for rendering.

5. The trafic

   1. Web application. It uses a combination of server-side language(Java, Python, etc.) to handle business logic, storage, etc., and client side languages(HTML and Javascript) for presentaion.

   2. Mobile Application. HTTP is the communication protocal. Javascript Object Notation (JSON) is commonly used for API response format to trasfer data due to the simplicity. A example JSON response:

      GET /users/12 – Retrieve user object for id = 12

      {
         "id":12,
         "firstName":"John",
         "lastName":"Smith",
         "address":{
            "streetAddress":"21 2nd Street",
            "city":"New York",
            "state":"NY",
            "postalCode":10021
         },
         "phoneNumbers":[
            "212 555-1234",
            "646 555-4567"
         ]
      }
      

   Database

   With growth of the user base, one server is not enough, we need multiple servers: one for web and the other for the database. Seperating traficts allows them to be scaled dependently.

   

Which database to use?

1. Traditional relational databases
   • also called relational database management system(RDBMS) or SQL database.
   • Example: MySQL, Oracle database, PostgreSQL..
   • represent and store data in rows and columns, You use join across tables to 'connect' data in different tables.
2. Non-relational databases
   • Also called NoSQL databases.
   • CouchDB, Neo4J, Cassandra, Hbase, Amazon DynamoDB.
   • Grouped in for categories: Key-value, graph, column, document
   • Notmally do not support JOIN
   • Might be right choice when:
     • requires super-low latency
     • unstructured data or non-relational data
     • data serialization and deserialzation needed (JSON, YAML, XML,...)
     • store massive amount of data..

Vertical Scaling vs horizental scaling

1. Vertical scalling: scale up. means adding more power(cpu, RAM etc.) to servers.
   • Advantage: simplicity
   • disadvantge:1) limit: you cannot add unlimited CPU and memory 2) have no failover and redundency
2. Horizontal scaling: scale out. adding more servers to pools.
3. If the server is offline; or too many user access server which cause a slowdonw or fail to connection, a load balancer is the best technique.

Load Balancer

1. A load balancer evenly distrbuts incoming traffic among web servers.

   

2. Users connect to the public IP of load balancer rather than directly to the servers.

3. Load balancer communicate with web servers via private IPs.

4. Solved no failover issue and imporve availability.

The single point of database doesn't support failover and redundency.

Data Replication

1. Master database: accepts write operations.

2. Slave database: accepts read operations.

3. Most applications requires a high ratio of read write so the slave is more than master in a system. (According DDIA this is only one kind of replication (Master-Slave))

   

4. Advantage

   • Better performace: it allows more queries run in parallel;
   • Reliability: If one database replocation is destroyed, others still have you data.
   • High availability: if a database is offline, others are still available to support data quering.

5. How data replication improve availability?

   • if only one slave available and down, the traffic will go to master temporarily and once the issue was resolve, a new slave will repalce the old one.

   • If multiple slave available and some of them down, the read request will redirect to other healthy ones. and a new database server will replace the old one.

   • If the master goes down. A slave will be prompted to master and a new slave database server will replace the old one.

     one problem is maybe the new master do not have updated data and needs to be update. This will be more complicated when doing this.

it is time to improve the load/response time. This can be done by adding a cache layer and shifting static content (JavaScript/CSS/image/video files) to the content delivery network (CDN).

Cache

1. Cache: temporary storage are that stores expensive resopnses or frequently accessed data in memory so that subsequent requests are served more quickly.

Cache tier

1. Benefits: better performance, reduce database workloads, scale the cache independently.

   

2. this Caching strategy： read-through

3. interacting with cache servers is simple because most cache servers provide APIs for common programming laguagues.

4. A typical Memchached APIs:

   SECONDS = 1
   cache.set('myKey, 'hi there', 3600 * SECONDS)
   cache.get('myKey')
   

Considerations for using cache

1. When to use: More read, less write; not for persistance vital data.
2. Expiration policy: expiration date not too soon thus cause reload data too frequently; and not to long to prevent data to be not fresh.
3. Consistency:
4. Mitigation failures
   • a cache server represents a potential point of failure.
   • multiple cache servers accross different data centers are recommended
   • Or overprovison the required memory to provide buffer for memory increasing.
5. Eviction policy: LRU/FIFO/LFU

Content delivery network(CDN)

1. geographically distributed servers to deliver static contents(iamges, videos, CSS, javascript files.
2. The further the CDN is from a client, the longer the request time it is normally.
3. How does it work:
   1. User A visist a website, a CDN server close to it will deliver the static content.
   2. If the CDN server does not have the content, it will request from the orginal server, and stores in the CDN server for further use, with a expiration time(TTL-time_To_Live);
   3. if the server has the content, the contents would be returned directly unless they are expired.

4. Considerations of using CDN
   • Cost: if you cache infrequenly used things, that may be worthless.
   • Cache expiry: not to long, not too short.
   • fallback: how to handle CDN down.
   • Invalidate the CDN objects:
     • via API provided by vendors
     • via a versioning server to version things and you fetch the latest version each time.

5. better perfomance for retriving static contents; less pressure on database by caching data.

Stateless web tier

It's time to scale up.

Stateful data: user session data, etc.

We need to move these data out of single server and store them into persistence storage. So each server can access them. This is called stateless web tier.

Stateful architecture

1. stateful server vs stageless server

   

2. The issue is that every request from the same client must route to the same server. This can be done in load balance but added overheads.

3. Adding or removing(or server fail) is more difficut.

Stateless architecture

1. Auto scaling(add or remove server based on trafic load) is easy by moving out state data from server.

Data centers

1. we could use geoDNS (provided by DNS service) to route to closest data center.
2. geoDNS, also know as geo-routed, allows domain names to be resolved to IP address based on the location of a user.

3. If one data center ourage, the trafict will be routed to a healthy one.

4. Techinical challenges for achiving multi-data center
   • traffic redirection
   • data synchronozation
   • Test in diffrent locations to make sure they are good
   • Auto delopyment tool is vital for consistent data in all data centers.

To further scale, we need to decouple components to allow them scale independently. Messaging queue is a key stretegy to solve this.

Message queue

1. a message queue is a component that supppots asynchronose communication bewteen components of a systme and can be either stored in memory or on disk to achive durability.

2. The architectur

   

3. Decoupling makes it a prefered arechtecture for scaling system.

4. The producer and consumer can scale or shirink based on the workload.

Logging, Metrics, automation

1. Logging

2. Metrics

   • host level: CPU, Memory...
   • Aggragated level metrics: perfromance of database tier..
   • business metrics: daily active users...

3. Automation tools

   • continuous integration
   • autmating build, test and deploy process..
   

As the data grows, you should also scale the data tier.

Database scaling

1. vertical scaling: more powerful server

   • Not unlimited

   • single point failuer

   • high cost

2. horizontal scaling: Sharding seperated large databases into smaller, more easily managed parts -- shards.

3. Sharding key(partition key) consists one or more columns that determine how data is distributed.

4. Cretiria for choosing a sharding key: evenly distributed to servers.

5. Challenges

   • Rehashing
     • Why: one server takes too much due to uneven distribution; or one server is full
     • Consistent hashing
   • Celebrity problem
     • hotspot key problem.
     • To address this issue: allocate partition for hotspot( dedicated)
   • Join and de-normalization
     • It's hard for tables to join in different partitions.

Millions of users and beyond

1. scaling system is an iterative process.
2. summary
   • Keep web tier stateless
   • Build redundancy at every tier
   • Cache data as much as you can
   • Support multiple data centers
   • Host static assets in CDN
   • Scale your data tier by sharding
   • Split tiers into individual services
   • Monitor your system and use automation tools