rhetenor/awesome-awesomeness
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Updating conversion, creating readmes

Browse Source
This commit is contained in:
Jonas Zeunert
2024-04-19 23:37:46 +02:00
parent 3619ac710a
commit 08e75b0f0a
635 changed files with 30878 additions and 37344 deletions
Download Patch File Download Diff File Expand all files Collapse all files

57
terminal/scalability
View File

@@ -1,32 +1,29 @@
!Logo (/logo.png) (http://awesome-scalability.com/)
An updated and organized reading list for illustrating the patterns of scalable, reliable, and performant large-scale systems. Concepts are explained in the articles of prominent engineers and credible 
references. Case studies are taken from battle-tested systems that serve millions to billions of users.
An updated and organized reading list for illustrating the patterns of scalable, reliable, and performant large-scale systems. Concepts are explained in the articles of prominent engineers and credible references. Case studies are taken
from battle-tested systems that serve millions to billions of users.
If your system goes slow
▐ Understand your problems: scalability problem (fast for a single user but slow under heavy load) or performance problem (slow for a single user) by reviewing some design principles (#principle) and checking 
▐ how scalability (#scalability) and performance (#performance) problems are solved at tech companies. The section of intelligence (#intelligence) are created for those who work with data and machine learning at
▐ big (data) and deep (learning) scale.
▐ Understand your problems: scalability problem (fast for a single user but slow under heavy load) or performance problem (slow for a single user) by reviewing some design principles (#principle) and checking how scalability 
▐ (#scalability) and performance (#performance) problems are solved at tech companies. The section of intelligence (#intelligence) are created for those who work with data and machine learning at big (data) and deep (learning) scale.
If your system goes down
▐ "Even if you lose all one day, you can build all over again if you retain your calm!" - Thuan Pham, former CTO of Uber. So, keep calm and mind the availability (#availability) and stability (#stability) 
▐ matters! 
▐ "Even if you lose all one day, you can build all over again if you retain your calm!" - Thuan Pham, former CTO of Uber. So, keep calm and mind the availability (#availability) and stability (#stability) matters! 
If you are having a system design interview
▐ Look at some interview notes (#interview) and real-world architectures with completed diagrams (#architecture) to get a comprehensive view before designing your system on whiteboard. You can check some talks 
▐ (#talk) of engineers from tech giants to know how they build, scale, and optimize their systems. Good luck!
▐ Look at some interview notes (#interview) and real-world architectures with completed diagrams (#architecture) to get a comprehensive view before designing your system on whiteboard. You can check some talks (#talk) of engineers from 
▐ tech giants to know how they build, scale, and optimize their systems. Good luck!
If you are building your dream team
▐ The goal of scaling team is not growing team size but increasing team output and value. You can find out how tech companies reach that goal in various aspects: hiring, management, organization, culture, and 
▐ communication in the organization (#organization) section.
▐ The goal of scaling team is not growing team size but increasing team output and value. You can find out how tech companies reach that goal in various aspects: hiring, management, organization, culture, and communication in the 
▐ organization (#organization) section.
Community power
▐ Contributions are greatly welcome! You may want to take a look at the contribution guidelines (CONTRIBUTING.md). If you see a link here that is no longer maintained or is not a good fit, please submit a pull 
▐ request!
▐ Contributions are greatly welcome! You may want to take a look at the contribution guidelines (CONTRIBUTING.md). If you see a link here that is no longer maintained or is not a good fit, please submit a pull request!
▐ Many long hours of hard work have gone into this project. If you find it helpful, please share on Facebook, on Twitter (https://ctt.ec/V8B2p), on Weibo (http://t.cn/RnjFLCB), or on your chat groups! Knowledge 
▐ is power, knowledge shared is power multiplied. Thank you!
▐ Many long hours of hard work have gone into this project. If you find it helpful, please share on Facebook, on Twitter (https://ctt.ec/V8B2p), on Weibo (http://t.cn/RnjFLCB), or on your chat groups! Knowledge is power, knowledge 
▐ shared is power multiplied. Thank you!
Content
- Principle (#principle)
@@ -248,8 +245,7 @@
* **WebAuthn Support for Secure Sign In at Dropbox** (https://blogs.dropbox.com/tech/2018/05/introducing-webauthn-support-for-secure-dropbox-sign-in/) 
* **Security Development Lifecycle at Slack** (https://slack.engineering/moving-fast-and-securing-things-540e6c5ae58a) 
* **Unprivileged Container Builds at Kinvolk** (https://kinvolk.io/blog/2018/04/towards-unprivileged-container-builds/) 
* **Diffy: Differencing Engine for Digital Forensics in the Cloud at Netflix** (https://medium.com/netflix-techblog/netflix-sirt-releases-diffy-a-differencing-engine-for-digital-forensics-in-the-cloud-37b71abd26 
98) 
* **Diffy: Differencing Engine for Digital Forensics in the Cloud at Netflix** (https://medium.com/netflix-techblog/netflix-sirt-releases-diffy-a-differencing-engine-for-digital-forensics-in-the-cloud-37b71abd2698)
* **Detecting Credential Compromise in AWS at Netflix** (https://medium.com/netflix-techblog/netflix-cloud-security-detecting-credential-compromise-in-aws-9493d6fd373a) 
* **Scalable User Privacy at Spotify** (https://labs.spotify.com/2018/09/18/scalable-user-privacy/) 
* **AVA: Audit Web Applications at Indeed** (https://engineering.indeedblog.com/blog/2018/09/application-scanning/) 
@@ -269,8 +265,7 @@
* **Qmessage: Distributed, Asynchronous Task Queue at Quora** (https://engineering.quora.com/Qmessage-Handling-Billions-of-Tasks-Per-Day) 
* **Cherami: Message Queue System for Transporting Async Tasks at Uber** (https://eng.uber.com/cherami/) 
* **Dynein: Distributed Delayed Job Queueing System at Airbnb** (https://medium.com/airbnb-engineering/dynein-building-a-distributed-delayed-job-queueing-system-93ab10f05f99) 
* **Timestone: Queueing System for Non-Parallelizable Workloads at Netflix** (https://netflixtechblog.com/timestone-netflixs-high-throughput-low-latency-priority-queueing-system-with-built-in-support-1abf249ba95 
f) 
* **Timestone: Queueing System for Non-Parallelizable Workloads at Netflix** (https://netflixtechblog.com/timestone-netflixs-high-throughput-low-latency-priority-queueing-system-with-built-in-support-1abf249ba95f)
* **Messaging Service at Riot Games** (https://engineering.riotgames.com/news/riot-messaging-service) 
* **Debugging Production with Event Logging at Zillow** (https://www.zillow.com/engineering/debugging-production-event-logging/) 
* **Cross-platform In-app Messaging Orchestration Service at Netflix** (https://medium.com/netflix-techblog/building-a-cross-platform-in-app-messaging-orchestration-service-86ba614f92d8) 
@@ -401,8 +396,7 @@
* **Handling Growth with Postgres at Instagram** (https://engineering.instagram.com/handling-growth-with-postgres-5-tips-from-instagram-d5d7e7ffdfcb) 
* **Scaling the Analytics Database (Postgres) at TransferWise** (http://tech.transferwise.com/scaling-our-analytics-database/) 
* **Updating a 50 Terabyte PostgreSQL Database at Adyen** (https://medium.com/adyen/updating-a-50-terabyte-postgresql-database-f64384b799e7) 
* **Scaling Database Access for 100s of Billions of Queries per Day at PayPal** (https://medium.com/paypal-engineering/scaling-database-access-for-100s-of-billions-of-queries-per-day-paypal-introducing-hera-e192 
adacda54) 
* **Scaling Database Access for 100s of Billions of Queries per Day at PayPal** (https://medium.com/paypal-engineering/scaling-database-access-for-100s-of-billions-of-queries-per-day-paypal-introducing-hera-e192adacda54)
* **Minimizing Read-Write MySQL Downtime at Yelp** (https://engineeringblog.yelp.com/2020/11/minimizing-read-write-mysql-downtime.html) 
* **Migrating MySQL from 5.6 to 8.0 at Facebook** (https://engineering.fb.com/2021/07/22/data-infrastructure/mysql/) 
* **Migration from HBase to MyRocks at Quora** (https://quoraengineering.quora.com/Migration-from-HBase-to-MyRocks-at-Quora) 
@@ -455,8 +449,7 @@
 * **Benchmarking Cassandra Scalability on AWS at Netflix** (https://medium.com/netflix-techblog/benchmarking-cassandra-scalability-on-aws-over-a-million-writes-per-second-39f45f066c9e) 
 * **Service Decomposition at Scale with Cassandra at Intuit QuickBooks** (https://quickbooks-engineering.intuit.com/service-decomposition-at-scale-70405ac2f637) 
 * **Cassandra for Keeping Counts In Sync at SoundCloud** (https://developers.soundcloud.com/blog/keeping-counts-in-sync) 
 * **Cassandra Driver Configuration for Improved Performance and Load Balancing at Glassdoor** (https://medium.com/glassdoor-engineering/cassandra-driver-configuration-for-improved-performance-and-load-balancing- 
1b0106ce12bb) 
 * **Cassandra Driver Configuration for Improved Performance and Load Balancing at Glassdoor** (https://medium.com/glassdoor-engineering/cassandra-driver-configuration-for-improved-performance-and-load-balancing-1b0106ce12bb)
 * **cstar: Cassandra Orchestration Tool at Spotify** (https://labs.spotify.com/2018/09/04/introducing-cstar-the-spotify-cassandra-orchestration-tool-now-open-source/) 
 * **HBase** (https://hbase.apache.org/) 
 * **HBase at Salesforce** (https://engineering.salesforce.com/investing-in-big-data-apache-hbase-b9d98661a66b) 
@@ -511,8 +504,7 @@
* **Dynamic Configuration at GoDaddy** (https://sg.godaddy.com/engineering/2019/03/06/dynamic-configuration-for-nodejs/) 
⟡ Scaling Continuous Integration and Continuous Delivery (https://www.synopsys.com/blogs/software-security/agile-cicd-devops-glossary/)
* **Continuous Integration Stack at Facebook** (https://code.fb.com/web/rapid-release-at-massive-scale/) 
* **Continuous Integration with Distributed Repositories and Dependencies at Netflix** (https://medium.com/netflix-techblog/towards-true-continuous-integration-distributed-repositories-and-dependencies-2a2e3108c 
051) 
* **Continuous Integration with Distributed Repositories and Dependencies at Netflix** (https://medium.com/netflix-techblog/towards-true-continuous-integration-distributed-repositories-and-dependencies-2a2e3108c051)
* **Continuous Integration and Deployment with Bazel at Dropbox** (https://blogs.dropbox.com/tech/2019/12/continuous-integration-and-deployment-with-bazel/) 
* **Continuous Deployments at BuzzFeed** (https://tech.buzzfeed.com/continuous-deployments-at-buzzfeed-d171f76c1ac4) 
* **Screwdriver: Continuous Delivery Build System for Dynamic Infrastructure at Yahoo** (https://yahooeng.tumblr.com/post/155765242061/open-sourcing-screwdriver-yahoos-continuous) 
@@ -712,8 +704,7 @@
* **Analytics Pipeline at Grammarly** (https://tech.grammarly.com/blog/building-a-versatile-analytics-pipeline-on-top-of-apache-spark) 
* **Analytics Pipeline at Teads** (https://medium.com/teads-engineering/give-meaning-to-100-billion-analytics-events-a-day-d6ba09aa8f44) 
* **ML Data Pipelines for Real-Time Fraud Prevention at PayPal** (https://www.infoq.com/presentations/paypal-ml-fraud-prevention-2018) 
* **Big Data Analytics and ML Techniques at LinkedIn** (https://cdn.oreillystatic.com/en/assets/1/event/269/Big%20data%20analytics%20and%20machine%20learning%20techniques%20to%20drive%20and%20grow%20business%20P 
resentation%201.pdf) 
* **Big Data Analytics and ML Techniques at LinkedIn** (https://cdn.oreillystatic.com/en/assets/1/event/269/Big%20data%20analytics%20and%20machine%20learning%20techniques%20to%20drive%20and%20grow%20business%20Presentation%201.pdf)
* **Self-Serve Reporting Platform on Hadoop at LinkedIn** (https://cdn.oreillystatic.com/en/assets/1/event/137/Building%20a%20self-serve%20real-time%20reporting%20platform%20at%20LinkedIn%20Presentation%201.pdf) 
* **Privacy-Preserving Analytics and Reporting at LinkedIn** (https://engineering.linkedin.com/blog/2019/04/privacy-preserving-analytics-and-reporting-at-linkedin) 
* **Analytics Platform for Tracking Item Availability at Walmart** (https://medium.com/walmartlabs/how-we-build-a-robust-analytics-platform-using-spark-kafka-and-cassandra-lambda-architecture-70c2d1bc8981) 
@@ -741,8 +732,7 @@
* **Log Analysis Platform at LINE** (https://www.slideshare.net/wyukawa/strata2017-sg) 
* **Data Visualisation Platform at Myntra** (https://medium.com/myntra-engineering/universal-dashboarding-platform-udp-data-visualisation-platform-at-myntra-5f2522fcf72d) 
* **Building and Scaling Data Lineage at Netflix** (https://medium.com/netflix-techblog/building-and-scaling-data-lineage-at-netflix-to-improve-data-infrastructure-reliability-and-1a52526a7977) 
* **Building a scalable data management system for computer vision tasks at Pinterest** (https://medium.com/@Pinterest_Engineering/building-a-scalable-data-management-system-for-computer-vision-tasks-a6dee8f1c58 
0) 
* **Building a scalable data management system for computer vision tasks at Pinterest** (https://medium.com/@Pinterest_Engineering/building-a-scalable-data-management-system-for-computer-vision-tasks-a6dee8f1c580) 
* **Structured Data at Etsy** (https://codeascraft.com/2019/07/31/an-introduction-to-structured-data-at-etsy/) 
* **Scaling a Mature Data Pipeline - Managing Overhead at Airbnb** (https://medium.com/airbnb-engineering/scaling-a-mature-data-pipeline-managing-overhead-f34835cbc866) 
* **Spark Partitioning Strategies at Airbnb** (https://medium.com/airbnb-engineering/on-spark-hive-and-small-files-an-in-depth-look-at-spark-partitioning-strategies-a9a364f908) 
@@ -825,8 +815,7 @@
* **Personalized Search at Etsy** (https://codeascraft.com/2020/10/29/bringing-personalized-search-to-etsy/) 
* **ML Feature Serving Infrastructure at Lyft** (https://eng.lyft.com/ml-feature-serving-infrastructure-at-lyft-d30bf2d3c32a) 
* **Context-Specific Bidding System at Etsy** (https://codeascraft.com/2021/03/23/how-we-built-a-context-specific-bidding-system-for-etsy-ads/) 
* **Moderating Promotional Spam and Inappropriate Content in Photos at Scale at Yelp** (https://engineeringblog.yelp.com/2021/05/moderating-promotional-spam-and-inappropriate-content-in-photos-at-scale-at-yelp.h 
tml) 
* **Moderating Promotional Spam and Inappropriate Content in Photos at Scale at Yelp** (https://engineeringblog.yelp.com/2021/05/moderating-promotional-spam-and-inappropriate-content-in-photos-at-scale-at-yelp.html)
* **Optimizing Payments with Machine Learning at Dropbox** (https://dropbox.tech/machine-learning/optimizing-payments-with-machine-learning) 
* **Scaling Media Machine Learning at Netflix** (https://netflixtechblog.com/scaling-media-machine-learning-at-netflix-f19b400243) 
* **Similarity Engine at eBay** (https://tech.ebayinc.com/engineering/ebays-blazingly-fast-billion-scale-vector-similarity-engine/) 
@@ -842,8 +831,7 @@
⟡ Architecture of API Gateway at Uber (https://eng.uber.com/architecture-api-gateway/)
⟡ Architecture of API Gateway at Tinder (https://medium.com/tinder/how-we-built-the-tinder-api-gateway-831c6ca5ceca)
⟡ Basic Architecture of Slack (https://slack.engineering/how-slack-built-shared-channels-8d42c895b19f)
⟡ Lightweight Distributed Architecture to Handle Thousands of Library Releases at eBay
 (https://tech.ebayinc.com/engineering/a-lightweight-distributed-architecture-to-handle-thousands-of-library-releases-at-ebay/)
⟡ Lightweight Distributed Architecture to Handle Thousands of Library Releases at eBay (https://tech.ebayinc.com/engineering/a-lightweight-distributed-architecture-to-handle-thousands-of-library-releases-at-ebay/)
⟡ Back-end at LinkedIn (https://engineering.linkedin.com/architecture/brief-history-scaling-linkedin)
⟡ Back-end at Flickr (https://yahooeng.tumblr.com/post/157200523046/introducing-tripod-flickrs-backend-refactored)
⟡ Infrastructure (3 parts) at Zendesk (https://medium.com/zendesk-engineering/the-history-of-infrastructure-at-zendesk-part-3-foundation-team-forming-and-evolving-9859e40f5390)
@@ -967,8 +955,7 @@
⟡ Scaling Infrastructure at Instagram - Lisa Guo, Instagram Engineering (https://www.youtube.com/watch?v=hnpzNAPiC0E)
⟡ Scaling Infrastructure at Twitter - Yao Yue, Staff Software Engineer at Twitter (https://www.youtube.com/watch?v=6OvrFkLSoZ0)
⟡ Scaling Infrastructure at Etsy - Bethany Macri, Engineering Manager at Etsy (https://www.youtube.com/watch?v=LfqyhM1LeIU)
⟡ Scaling Real-time Infrastructure at Alibaba for Global Shopping Holiday - Xiaowei Jiang, Senior Director at Alibaba
 (https://atscaleconference.com/videos/scaling-alibabas-real-time-infrastructure-for-global-shopping-holiday/)
⟡ Scaling Real-time Infrastructure at Alibaba for Global Shopping Holiday - Xiaowei Jiang, Senior Director at Alibaba (https://atscaleconference.com/videos/scaling-alibabas-real-time-infrastructure-for-global-shopping-holiday/)
⟡ Scaling Data Infrastructure at Spotify - Matti (Lepistö) Pehrs, Spotify (https://www.youtube.com/watch?v=cdsfRXr9pJU)
⟡ Scaling Pinterest - Marty Weiner, Pinterest’s founding engineer (https://www.youtube.com/watch?v=jQNCuD_hxdQ&list=RDhnpzNAPiC0E&index=11)
⟡ Scaling Slack - Bing Wei, Software Engineer (Infrastructure) at Slack (https://www.infoq.com/presentations/slack-scalability)