We prepared a dataset from the GH Archive that contains all the events in all GitHub repositories since 2011 in structured format. The dataset was uploaded into ClickHouse, where it contains 11.1 billion records. We redistribute it for research purposes and it can be downloaded at this direct link. This dataset can help answer almost any question about GitHub that you can imagine.
This article shows the usage of the dataset and offers insights into the GitHub ecosystem. It emphasizes how easy it is to play with data using modern tools.
The dataset is exactly as complete as the GH Archive feed. Since the middle of 2025 that feed reports almost only PushEvent events, so everything else (stars, forks, issues, pull requests, comments) is heavily undercounted for 2025–2026. Keep it in mind while reading the reports for the most recent years.
The WatchEvent is the event when someone gives a star to a repo.
:) SELECT count() FROM github_events WHERE event_type = 'WatchEvent'
┌───count()─┐
│ 554401487 │
└───────────┘
1 rows in set. Elapsed: 0.007 sec. Processed 202.19 thousand rows, 202.20 KB (31.05 million rows/s., 31.05 MB/s.)
There is no event when someone removes the star. This means that star removal is invisible, but it should be rare in any case, so we can calculate the approximate number of stars from these events.
:) SELECT action, count() FROM github_events WHERE event_type = 'WatchEvent' GROUP BY action
┌─action──┬───count()─┐
│ started │ 554401487 │
└─────────┴───────────┘
1 rows in set. Elapsed: 0.095 sec. Processed 554.51 million rows, 1.11 GB (5.83 billion rows/s., 11.67 GB/s.)
There are over 550 million stars on GitHub!
Let's validate the star count by comparing it for my favorite repository:
:) SELECT count() FROM github_events WHERE event_type = 'WatchEvent' AND repo_name IN ('ClickHouse/ClickHouse', 'yandex/ClickHouse') GROUP BY action
┌─count()─┐
│ 46077 │
└─────────┘
1 rows in set. Elapsed: 0.011 sec. Processed 249.76 thousand rows, 2.37 MB (22.92 million rows/s., 217.12 MB/s.)
The number looks consistent with the real star count.
(The repository was moved from yandex/ClickHouse to ClickHouse/ClickHouse on October 2019, so we have to sum up the numbers for two repos.)
This is the first report that comes to mind.
:) SET output_format_pretty_row_numbers = 1
:) SELECT repo_name, count() AS stars FROM github_events WHERE event_type = 'WatchEvent' GROUP BY repo_name ORDER BY stars DESC LIMIT 50
┌─repo_name──────────────────────────────────┬──stars─┐
1. │ sindresorhus/awesome │ 450317 │
2. │ 996icu/996.ICU │ 378812 │
3. │ kamranahmedse/developer-roadmap │ 363470 │
4. │ public-apis/public-apis │ 338225 │
5. │ donnemartin/system-design-primer │ 336938 │
6. │ jwasham/coding-interview-university │ 326155 │
7. │ EbookFoundation/free-programming-books │ 312481 │
8. │ codecrafters-io/build-your-own-x │ 297720 │
9. │ vinta/awesome-python │ 292818 │
10. │ facebook/react │ 290876 │
11. │ freeCodeCamp/freeCodeCamp │ 273819 │
12. │ vuejs/vue │ 251379 │
13. │ torvalds/linux │ 243571 │
14. │ TheAlgorithms/Python │ 231991 │
15. │ tensorflow/tensorflow │ 229323 │
16. │ awesome-selfhosted/awesome-selfhosted │ 226992 │
17. │ FreeCodeCamp/FreeCodeCamp │ 225340 │
18. │ trekhleb/javascript-algorithms │ 207717 │
19. │ getify/You-Dont-Know-JS │ 206031 │
20. │ flutter/flutter │ 202194 │
21. │ trimstray/the-book-of-secret-knowledge │ 199529 │
22. │ jackfrued/Python-100-Days │ 197647 │
23. │ practical-tutorials/project-based-learning │ 196636 │
24. │ github/gitignore │ 185238 │
25. │ ossu/computer-science │ 181404 │
26. │ ant-design/ant-design │ 177645 │
27. │ avelino/awesome-go │ 171577 │
28. │ CyC2018/CS-Notes │ 170749 │
29. │ jlevy/the-art-of-command-line │ 170294 │
30. │ airbnb/javascript │ 169898 │
31. │ Snailclimb/JavaGuide │ 168315 │
32. │ twbs/bootstrap │ 167745 │
33. │ AUTOMATIC1111/stable-diffusion-webui │ 158350 │
34. │ golang/go │ 153423 │
35. │ massgravel/Microsoft-Activation-Scripts │ 152573 │
36. │ facebook/react-native │ 146964 │
37. │ huggingface/transformers │ 144801 │
38. │ danistefanovic/build-your-own-x │ 143392 │
39. │ labuladong/fucking-algorithm │ 141944 │
40. │ n8n-io/n8n │ 138806 │
41. │ 521xueweihan/HelloGitHub │ 137952 │
42. │ yangshun/tech-interview-handbook │ 137527 │
43. │ microsoft/PowerToys │ 136633 │
44. │ Genymobile/scrcpy │ 134421 │
45. │ f/awesome-chatgpt-prompts │ 133666 │
46. │ yt-dlp/yt-dlp │ 129410 │
47. │ aplus-framework/app │ 127305 │
48. │ justjavac/free-programming-books-zh_CN │ 127009 │
49. │ microsoft/vscode │ 124831 │
50. │ nodejs/node │ 123699 │
└────────────────────────────────────────────┴────────┘
50 rows in set. Elapsed: 2.871 sec. Processed 554.51 million rows, 3.39 GB (193.12 million rows/s., 1.18 GB/s.)
The top repository ever is the "Awesome" list — a list of lists of awesome things. But wait... it's not the top repo. We still see two copies of "FreeCodeCamp": freeCodeCamp/freeCodeCamp and FreeCodeCamp/FreeCodeCamp. If we sum up the stars it will be 273819 + 225340 = 499159 — more than the "Awesome" list. So, the top repository on GitHub is an educational resource.
Next we see: "996.ICU" — not software, but more like a project to improve awareness about work schedules in different Chinese companies; then a roadmap for developers, a list of public APIs, a system design primer, an interview preparation course, free programming books, and "build your own x". Only in 10th place there is the first piece of software: React. Then another JavaScript framework, and in 13th place there is Linux. Let's stop at this point.
Bottomline: GitHub is full of JavaScript frameworks and education content. But don't jump to conclusions yet — we've only scratched the surface!
By the way, what is the distribution of repositories by stars?
SELECT
exp10(floor(log10(c))) AS stars,
uniq(k)
FROM
(
SELECT
repo_name AS k,
count() AS c
FROM github_events
WHERE event_type = 'WatchEvent'
GROUP BY k
)
GROUP BY stars
ORDER BY stars ASC
┌──stars─┬──uniq(k)─┐
│ 1 │ 43428394 │
│ 10 │ 2815597 │
│ 100 │ 482968 │
│ 1000 │ 59518 │
│ 10000 │ 4773 │
│ 100000 │ 85 │
└────────┴──────────┘
6 rows in set. Elapsed: 3.605 sec. Processed 554.51 million rows, 3.39 GB (153.80 million rows/s., 940.27 MB/s.)
There are only 47 million repositories that have at least one star. There are only 3.4 million repositories with 10+ stars. Just 547 000 repositories have over 100 stars, 64 000 have over 1k stars, 4900 have over 10k stars and there are only 85 repositories with more than 100k stars.
:) SELECT uniq(repo_name) FROM github_events
┌─uniq(repo_name)─┐
│ 559616139 │
└─────────────────┘
1 rows in set. Elapsed: 8.618 sec. Processed 11.08 billion rows, 55.87 GB (1.29 billion rows/s., 6.48 GB/s.)
If we will take all stars and distribute uniformly across all the repositories, every repository will get 0.99 stars.
2015 | 2016 | 2017 | 2018 | 2019 | 2020 | 2021 | 2022 | 2023 | 2024 | 2025 | 2026
:) SELECT repo_name, count() AS stars FROM github_events WHERE event_type = 'WatchEvent' AND toYear(created_at) = '2026' GROUP BY repo_name ORDER BY stars DESC LIMIT 50 ┌─repo_name─────────────────────────────────────┬─stars─┐ 1. │ openclaw/openclaw │ 73357 │ 2. │ obra/superpowers │ 40256 │ 3. │ affaan-m/everything-claude-code │ 39887 │ 4. │ anomalyco/opencode │ 32891 │ 5. │ NousResearch/hermes-agent │ 27516 │ 6. │ anthropics/skills │ 27038 │ 7. │ nextlevelbuilder/ui-ux-pro-max-skill │ 19469 │ 8. │ clawdbot/clawdbot │ 19371 │ 9. │ msitarzewski/agency-agents │ 19166 │ 10. │ forrestchang/andrej-karpathy-skills │ 18319 │ 11. │ garrytan/gstack │ 16946 │ 12. │ anthropics/claude-code │ 16416 │ 13. │ karpathy/autoresearch │ 15686 │ 14. │ VoltAgent/awesome-design-md │ 14937 │ 15. │ thedotmack/claude-mem │ 14683 │ 16. │ ultraworkers/claw-code │ 14288 │ 17. │ farion1231/cc-switch │ 14094 │ 18. │ mattpocock/skills │ 13950 │ 19. │ moltbot/moltbot │ 13576 │ 20. │ codecrafters-io/build-your-own-x │ 13227 │ 21. │ koala73/worldmonitor │ 12602 │ 22. │ ComposioHQ/awesome-claude-skills │ 12600 │ 23. │ paperclipai/paperclip │ 12289 │ 24. │ 666ghj/MiroFish │ 12022 │ 25. │ sindresorhus/awesome │ 11804 │ 26. │ firecrawl/firecrawl │ 11695 │ 27. │ code-yeongyu/oh-my-opencode │ 11509 │ 28. │ VoltAgent/awesome-openclaw-skills │ 11289 │ 29. │ HKUDS/nanobot │ 11258 │ 30. │ public-apis/public-apis │ 11218 │ 31. │ instructkr/claw-code │ 10942 │ 32. │ KeygraphHQ/shannon │ 10702 │ 33. │ shanraisshan/claude-code-best-practice │ 10237 │ 34. │ shareAI-lab/learn-claude-code │ 10233 │ 35. │ JuliusBrussee/caveman │ 10133 │ 36. │ badlogic/pi-mono │ 10103 │ 37. │ github/spec-kit │ 10010 │ 38. │ daytonaio/daytona │ 9870 │ 39. │ x1xhlol/system-prompts-and-models-of-ai-tools │ 9704 │ 40. │ gsd-build/get-shit-done │ 9668 │ 41. │ microsoft/markitdown │ 9494 │ 42. │ DigitalPlatDev/FreeDomain │ 9472 │ 43. │ ZhuLinsen/daily_stock_analysis │ 9239 │ 44. │ sickn33/antigravity-awesome-skills │ 9122 │ 45. │ D4Vinci/Scrapling │ 9051 │ 46. │ safishamsi/graphify │ 8882 │ 47. │ bytedance/deer-flow │ 8865 │ 48. │ TauricResearch/TradingAgents │ 8838 │ 49. │ vercel-labs/agent-browser │ 8651 │ 50. │ vercel-labs/agent-skills │ 8527 │ └───────────────────────────────────────────────┴───────┘ 50 rows in set. Elapsed: 0.222 sec. Processed 508.65 million rows, 2.21 GB (2.29 billion rows/s., 9.96 GB/s.)
Almost the whole list is about coding agents: agent harnesses, collections of "skills" and prompt libraries. Keep in mind that 2026 is both unfinished and undercounted — see the note in the abstract — so these are lower bounds.
:) SELECT repo_name, count() AS stars FROM github_events WHERE event_type = 'WatchEvent' AND toYear(created_at) = '2025' GROUP BY repo_name ORDER BY stars DESC LIMIT 50 ┌─repo_name─────────────────────────────────────┬─stars─┐ 1. │ codecrafters-io/build-your-own-x │ 99603 │ 2. │ deepseek-ai/DeepSeek-R1 │ 86882 │ 3. │ deepseek-ai/DeepSeek-V3 │ 85274 │ 4. │ unionlabs/union │ 79643 │ 5. │ n8n-io/n8n │ 77991 │ 6. │ x1xhlol/system-prompts-and-models-of-ai-tools │ 71357 │ 7. │ DigitalPlatDev/FreeDomain │ 66670 │ 8. │ langflow-ai/langflow │ 65669 │ 9. │ sindresorhus/awesome │ 62531 │ 10. │ punkpeye/awesome-mcp-servers │ 59862 │ 11. │ browser-use/browser-use │ 56916 │ 12. │ langgenius/dify │ 54592 │ 13. │ modelcontextprotocol/servers │ 53343 │ 14. │ open-webui/open-webui │ 52306 │ 15. │ google-gemini/gemini-cli │ 51218 │ 16. │ Shubhamsaboo/awesome-llm-apps │ 49691 │ 17. │ public-apis/public-apis │ 46361 │ 18. │ ollama/ollama │ 45364 │ 19. │ TapXWorld/ChinaTextbook │ 42743 │ 20. │ mannaandpoem/OpenManus │ 42647 │ 21. │ microsoft/markitdown │ 41406 │ 22. │ inkonchain/node │ 40839 │ 23. │ inkonchain/docs │ 40608 │ 24. │ inkonchain/ink-kit │ 40410 │ 25. │ awesome-selfhosted/awesome-selfhosted │ 40163 │ 26. │ openai/codex │ 38870 │ 27. │ massgravel/Microsoft-Activation-Scripts │ 38542 │ 28. │ yeongpin/cursor-free-vip │ 35819 │ 29. │ infiniflow/ragflow │ 35225 │ 30. │ vinta/awesome-python │ 35105 │ 31. │ donnemartin/system-design-primer │ 34803 │ 32. │ practical-tutorials/project-based-learning │ 34361 │ 33. │ trimstray/the-book-of-secret-knowledge │ 34358 │ 34. │ kamranahmedse/developer-roadmap │ 33890 │ 35. │ microsoft/ai-agents-for-beginners │ 33889 │ 36. │ virattt/ai-hedge-fund │ 32147 │ 37. │ yt-dlp/yt-dlp │ 32003 │ 38. │ cline/cline │ 31523 │ 39. │ clash-verge-rev/clash-verge-rev │ 31301 │ 40. │ unclecode/crawl4ai │ 30821 │ 41. │ deepseek-ai/awesome-deepseek-integration │ 30809 │ 42. │ astral-sh/uv │ 30720 │ 43. │ pathwaycom/pathway │ 30300 │ 44. │ rasbt/LLMs-from-scratch │ 30167 │ 45. │ hacksider/Deep-Live-Cam │ 30095 │ 46. │ 521xueweihan/HelloGitHub │ 29882 │ 47. │ anthropics/claude-code │ 28978 │ 48. │ microsoft/generative-ai-for-beginners │ 28789 │ 49. │ linera-io/linera-protocol │ 28037 │ 50. │ CherryHQ/cherry-studio │ 27848 │ └───────────────────────────────────────────────┴───────┘ 50 rows in set. Elapsed: 0.565 sec. Processed 512.83 million rows, 2.83 GB (908.23 million rows/s., 5.02 GB/s.)
The year of open-weight models: DeepSeek R1 and V3 appeared within days of each other. Then workflow automation (n8n, Langflow) and the first awesome-list for MCP servers. x1xhlol/system-prompts-and-models-of-ai-tools is a collection of leaked system prompts.
:) SELECT repo_name, count() AS stars FROM github_events WHERE event_type = 'WatchEvent' AND toYear(created_at) = '2024' GROUP BY repo_name ORDER BY stars DESC LIMIT 50 ┌─repo_name──────────────────────────────────┬──stars─┐ 1. │ Rabbitminers/Extended-Bogeys │ 100547 │ 2. │ codecrafters-io/build-your-own-x │ 88066 │ 3. │ ollama/ollama │ 70695 │ 4. │ practical-tutorials/project-based-learning │ 67701 │ 5. │ sindresorhus/awesome │ 64977 │ 6. │ massgravel/Microsoft-Activation-Scripts │ 53332 │ 7. │ public-apis/public-apis │ 52853 │ 8. │ zed-industries/zed │ 52516 │ 9. │ awesome-selfhosted/awesome-selfhosted │ 49646 │ 10. │ krahets/hello-algo │ 48599 │ 11. │ donnemartin/system-design-primer │ 48457 │ 12. │ xai-org/grok-1 │ 47610 │ 13. │ open-webui/open-webui │ 46847 │ 14. │ kamranahmedse/developer-roadmap │ 45780 │ 15. │ microsoft/generative-ai-for-beginners │ 44701 │ 16. │ EbookFoundation/free-programming-books │ 43071 │ 17. │ langgenius/dify │ 42914 │ 18. │ vinta/awesome-python │ 41951 │ 19. │ hacksider/Deep-Live-Cam │ 40957 │ 20. │ DigitalPlatDev/FreeDomain │ 40746 │ 21. │ comfyanonymous/ComfyUI │ 39955 │ 22. │ clash-verge-rev/clash-verge-rev │ 39720 │ 23. │ jwasham/coding-interview-university │ 38879 │ 24. │ trimstray/the-book-of-secret-knowledge │ 38855 │ 25. │ shadcn-ui/ui │ 38223 │ 26. │ lobehub/lobe-chat │ 37362 │ 27. │ RVC-Boss/GPT-SoVITS │ 36850 │ 28. │ shardeum/shardeum │ 35874 │ 29. │ mlabonne/llm-course │ 35373 │ 30. │ rasbt/LLMs-from-scratch │ 35097 │ 31. │ Stirling-Tools/Stirling-PDF │ 34431 │ 32. │ maybe-finance/maybe │ 34384 │ 33. │ abi/screenshot-to-code │ 33684 │ 34. │ freeCodeCamp/freeCodeCamp │ 32480 │ 35. │ astral-sh/uv │ 32443 │ 36. │ 2noise/ChatTTS │ 32158 │ 37. │ yt-dlp/yt-dlp │ 32044 │ 38. │ lizongying/my-tv │ 31704 │ 39. │ AUTOMATIC1111/stable-diffusion-webui │ 31269 │ 40. │ immich-app/immich │ 31097 │ 41. │ FuelLabs/fuel-core │ 30533 │ 42. │ karpathy/LLM101n │ 30484 │ 43. │ dockur/windows │ 29732 │ 44. │ OpenDevin/OpenDevin │ 29295 │ 45. │ myshell-ai/OpenVoice │ 29294 │ 46. │ localsend/localsend │ 28812 │ 47. │ microsoft/markitdown │ 28034 │ 48. │ hiyouga/LLaMA-Factory │ 27623 │ 49. │ GrowingGit/GitHub-Chinese-Top-Charts │ 27295 │ 50. │ excalidraw/excalidraw │ 27082 │ └────────────────────────────────────────────┴────────┘ 50 rows in set. Elapsed: 0.608 sec. Processed 495.34 million rows, 3.16 GB (814.99 million rows/s., 5.19 GB/s.)
ollama and zed are the actual software here. The repository on top is a Minecraft add-on that collected 27 891 stars in a single day, and Microsoft-Activation-Scripts is... something else entirely.
:) SELECT repo_name, count() AS stars FROM github_events WHERE event_type = 'WatchEvent' AND toYear(created_at) = '2023' GROUP BY repo_name ORDER BY stars DESC LIMIT 50 ┌─repo_name──────────────────────────────────┬─stars─┐ 1. │ AUTOMATIC1111/stable-diffusion-webui │ 87369 │ 2. │ base-org/chains │ 85316 │ 3. │ Significant-Gravitas/Auto-GPT │ 84899 │ 4. │ f/awesome-chatgpt-prompts │ 81664 │ 5. │ base-org/node │ 74042 │ 6. │ practical-tutorials/project-based-learning │ 61102 │ 7. │ twitter/the-algorithm │ 60262 │ 8. │ sindresorhus/awesome │ 56970 │ 9. │ Torantulino/Auto-GPT │ 56840 │ 10. │ codecrafters-io/build-your-own-x │ 55452 │ 11. │ nomic-ai/gpt4all │ 55190 │ 12. │ public-apis/public-apis │ 53599 │ 13. │ Yidadaa/ChatGPT-Next-Web │ 52076 │ 14. │ krahets/hello-algo │ 51961 │ 15. │ hwchase17/langchain │ 51289 │ 16. │ xtekky/gpt4free │ 50720 │ 17. │ EbookFoundation/free-programming-books │ 50566 │ 18. │ awesome-selfhosted/awesome-selfhosted │ 50153 │ 19. │ ByteByteGoHq/system-design-101 │ 48198 │ 20. │ openai/openai-cookbook │ 46942 │ 21. │ ggerganov/llama.cpp │ 46297 │ 22. │ facebookresearch/llama │ 45826 │ 23. │ PlexPt/awesome-chatgpt-prompts-zh │ 45526 │ 24. │ AntonOsika/gpt-engineer │ 45369 │ 25. │ vinta/awesome-python │ 44294 │ 26. │ FuelLabs/sway │ 43909 │ 27. │ imartinez/privateGPT │ 42947 │ 28. │ lencx/ChatGPT │ 41916 │ 29. │ kamranahmedse/developer-roadmap │ 41271 │ 30. │ huggingface/transformers │ 40771 │ 31. │ facebookresearch/segment-anything │ 39553 │ 32. │ dair-ai/Prompt-Engineering-Guide │ 36081 │ 33. │ jwasham/coding-interview-university │ 35580 │ 34. │ THUDM/ChatGLM-6B │ 35486 │ 35. │ KillianLucas/open-interpreter │ 34738 │ 36. │ massgravel/Microsoft-Activation-Scripts │ 34682 │ 37. │ donnemartin/system-design-primer │ 34207 │ 38. │ opentffoundation/manifesto │ 33225 │ 39. │ XingangPan/DragGAN │ 33026 │ 40. │ LAION-AI/Open-Assistant │ 33001 │ 41. │ trimstray/the-book-of-secret-knowledge │ 32496 │ 42. │ openai/whisper │ 32322 │ 43. │ geekan/MetaGPT │ 31737 │ 44. │ abi/screenshot-to-code │ 31166 │ 45. │ oven-sh/bun │ 30311 │ 46. │ FuelLabs/fuel-core │ 29980 │ 47. │ microsoft/visual-chatgpt │ 29967 │ 48. │ lm-sys/FastChat │ 29224 │ 49. │ oobabooga/text-generation-webui │ 28707 │ 50. │ suno-ai/bark │ 28354 │ └────────────────────────────────────────────┴───────┘ 50 rows in set. Elapsed: 0.603 sec. Processed 493.45 million rows, 3.07 GB (818.01 million rows/s., 5.10 GB/s.)
The year generative AI arrived on GitHub: Stable Diffusion WebUI, two copies of Auto-GPT and a collection of ChatGPT prompts. base-org/chains and base-org/node come from a crypto airdrop campaign, and twitter/the-algorithm is the day Twitter published its recommendation code.
:) SELECT repo_name, count() AS stars FROM github_events WHERE event_type = 'WatchEvent' AND toYear(created_at) = '2022' GROUP BY repo_name ORDER BY stars DESC LIMIT 50 ┌─repo_name──────────────────────────────────┬─stars─┐ 1. │ aplus-framework/app │ 92632 │ 2. │ sindresorhus/awesome │ 55839 │ 3. │ aplus-framework/database │ 53822 │ 4. │ public-apis/public-apis │ 52515 │ 5. │ donnemartin/system-design-primer │ 51603 │ 6. │ Anduin2017/HowToCook │ 47303 │ 7. │ kamranahmedse/developer-roadmap │ 44122 │ 8. │ EbookFoundation/free-programming-books │ 44083 │ 9. │ jwasham/coding-interview-university │ 43695 │ 10. │ vinta/awesome-python │ 43230 │ 11. │ awesome-selfhosted/awesome-selfhosted │ 41641 │ 12. │ codecrafters-io/build-your-own-x │ 41372 │ 13. │ CompVis/stable-diffusion │ 37259 │ 14. │ trimstray/the-book-of-secret-knowledge │ 30879 │ 15. │ tauri-apps/tauri │ 30714 │ 16. │ aplus-framework/framework │ 30177 │ 17. │ aplus-framework/one │ 30127 │ 18. │ aplus-framework/aplus │ 30124 │ 19. │ aplus-framework/image │ 30108 │ 20. │ aplus-framework/cli │ 30082 │ 21. │ aplus-framework/pagination │ 30056 │ 22. │ aplus-framework/email │ 30044 │ 23. │ aplus-framework/routing │ 30029 │ 24. │ aplus-framework/http │ 30001 │ 25. │ aplus-framework/session │ 29988 │ 26. │ aplus-framework/mvc │ 29830 │ 27. │ aplus-framework/validation │ 29820 │ 28. │ aplus-framework/http-client │ 29791 │ 29. │ aplus-framework/language │ 29791 │ 30. │ trekhleb/javascript-algorithms │ 29281 │ 31. │ carbon-language/carbon-lang │ 28621 │ 32. │ ossu/computer-science │ 27241 │ 33. │ TheAlgorithms/Python │ 27211 │ 34. │ httpie/httpie │ 25862 │ 35. │ AUTOMATIC1111/stable-diffusion-webui │ 25830 │ 36. │ The-Run-Philosophy-Organization/run │ 25373 │ 37. │ UsergeTeam/Userge │ 25309 │ 38. │ facebook/react │ 25290 │ 39. │ freeCodeCamp/freeCodeCamp │ 25285 │ 40. │ UsergeTeam/Userge-Assistant │ 24996 │ 41. │ practical-tutorials/project-based-learning │ 24921 │ 42. │ microsoft/Web-Dev-For-Beginners │ 24261 │ 43. │ torvalds/linux │ 23833 │ 44. │ jlevy/the-art-of-command-line │ 23617 │ 45. │ Developer-Y/cs-video-courses │ 22827 │ 46. │ avelino/awesome-go │ 22571 │ 47. │ yt-dlp/yt-dlp │ 21901 │ 48. │ PKUFlyingPig/cs-self-learning │ 21718 │ 49. │ vercel/next.js │ 21640 │ 50. │ yangshun/tech-interview-handbook │ 21402 │ └────────────────────────────────────────────┴───────┘ 50 rows in set. Elapsed: 0.585 sec. Processed 493.12 million rows, 3.62 GB (843.14 million rows/s., 6.18 GB/s.)
Two repositories of the same PHP framework on top, which is suspicious at the very least. Anduin2017/HowToCook is a Chinese cooking guide — arguably the most useful repository in this list.
:) SELECT repo_name, count() AS stars FROM github_events WHERE event_type = 'WatchEvent' AND toYear(created_at) = '2021' GROUP BY repo_name ORDER BY stars DESC LIMIT 50 ┌─repo_name──────────────────────────────┬─stars─┐ 1. │ ant-design/ant-design │ 79848 │ 2. │ public-apis/public-apis │ 69555 │ 3. │ jwasham/coding-interview-university │ 56058 │ 4. │ EbookFoundation/free-programming-books │ 50575 │ 5. │ kamranahmedse/developer-roadmap │ 45149 │ 6. │ donnemartin/system-design-primer │ 41829 │ 7. │ trekhleb/javascript-algorithms │ 41684 │ 8. │ danistefanovic/build-your-own-x │ 36829 │ 9. │ sindresorhus/awesome │ 35556 │ 10. │ ossu/computer-science │ 33680 │ 11. │ TheAlgorithms/Python │ 31985 │ 12. │ microsoft/Web-Dev-For-Beginners │ 30904 │ 13. │ flutter/flutter │ 30472 │ 14. │ microsoft/ML-For-Beginners │ 29522 │ 15. │ CyC2018/CS-Notes │ 29079 │ 16. │ 30-seconds/30-seconds-of-code │ 26296 │ 17. │ labuladong/fucking-algorithm │ 24335 │ 18. │ freeCodeCamp/freeCodeCamp │ 24300 │ 19. │ Snailclimb/JavaGuide │ 23249 │ 20. │ facebook/react │ 23235 │ 21. │ torvalds/linux │ 22975 │ 22. │ vercel/next.js │ 22556 │ 23. │ tiangolo/fastapi │ 22178 │ 24. │ vitejs/vite │ 21859 │ 25. │ trimstray/the-book-of-secret-knowledge │ 21655 │ 26. │ awesome-selfhosted/awesome-selfhosted │ 21413 │ 27. │ supabase/supabase │ 21403 │ 28. │ google/zx │ 21379 │ 29. │ conwnet/github1s │ 21130 │ 30. │ vinta/awesome-python │ 21021 │ 31. │ iptv-org/iptv │ 20902 │ 32. │ coder2gwy/coder2gwy │ 20595 │ 33. │ microsoft/PowerToys │ 20447 │ 34. │ microsoft/vscode │ 20317 │ 35. │ ytdl-org/youtube-dl │ 19954 │ 36. │ BottleSome/AutoApiSecret │ 19526 │ 37. │ anuraghazra/github-readme-stats │ 19487 │ 38. │ youngyangyang04/leetcode-master │ 19235 │ 39. │ jlevy/the-art-of-command-line │ 19216 │ 40. │ tauri-apps/tauri │ 18984 │ 41. │ tailwindlabs/tailwindcss │ 18982 │ 42. │ sveltejs/svelte │ 18810 │ 43. │ huggingface/transformers │ 18617 │ 44. │ ryanmcdermott/clean-code-javascript │ 18584 │ 45. │ ohmyzsh/ohmyzsh │ 18550 │ 46. │ ibraheemdev/modern-unix │ 18521 │ 47. │ vuejs/vue │ 18387 │ 48. │ jackfrued/Python-100-Days │ 18057 │ 49. │ getify/You-Dont-Know-JS │ 17851 │ 50. │ Genymobile/scrcpy │ 17692 │ └────────────────────────────────────────┴───────┘ 50 rows in set. Elapsed: 0.520 sec. Processed 498.86 million rows, 3.36 GB (958.66 million rows/s., 6.46 GB/s.)
Education and awesome lists, as before. ant-design is on top only because of a single day — see the report about the most stars over one day.
:) SELECT repo_name, count() AS stars FROM github_events WHERE event_type = 'WatchEvent' AND toYear(created_at) = '2020' GROUP BY repo_name ORDER BY stars DESC LIMIT 50 ┌─repo_name────────────────────────────────────┬─stars─┐ 1. │ labuladong/fucking-algorithm │ 81959 │ 2. │ jwasham/coding-interview-university │ 63213 │ 3. │ kamranahmedse/developer-roadmap │ 54303 │ 4. │ public-apis/public-apis │ 40661 │ 5. │ donnemartin/system-design-primer │ 40402 │ 6. │ EbookFoundation/free-programming-books │ 40059 │ 7. │ TheAlgorithms/Python │ 36526 │ 8. │ danistefanovic/build-your-own-x │ 36524 │ 9. │ CyC2018/CS-Notes │ 35381 │ 10. │ microsoft/PowerToys │ 35331 │ 11. │ trekhleb/javascript-algorithms │ 35022 │ 12. │ flutter/flutter │ 33115 │ 13. │ Snailclimb/JavaGuide │ 32795 │ 14. │ denoland/deno │ 32786 │ 15. │ sindresorhus/awesome │ 30582 │ 16. │ ytdl-org/youtube-dl │ 29005 │ 17. │ vuejs/vue │ 28954 │ 18. │ florinpop17/app-ideas │ 28897 │ 19. │ jackfrued/Python-100-Days │ 28007 │ 20. │ CSSEGISandData/COVID-19 │ 26978 │ 21. │ facebook/react │ 25807 │ 22. │ bradtraversy/design-resources-for-developers │ 23641 │ 23. │ microsoft/vscode │ 23347 │ 24. │ ohmyzsh/ohmyzsh │ 23269 │ 25. │ geekxh/hello-algorithm │ 23253 │ 26. │ cli/cli │ 22195 │ 27. │ torvalds/linux │ 22113 │ 28. │ goldbergyoni/nodebestpractices │ 22061 │ 29. │ ossu/computer-science │ 21259 │ 30. │ github/gitignore │ 21202 │ 31. │ getify/You-Dont-Know-JS │ 21105 │ 32. │ microsoft/playwright │ 20496 │ 33. │ huggingface/transformers │ 20351 │ 34. │ Genymobile/scrcpy │ 19778 │ 35. │ willmcgugan/rich │ 19603 │ 36. │ tiangolo/fastapi │ 19532 │ 37. │ gothinkster/realworld │ 19300 │ 38. │ macrozheng/mall │ 19283 │ 39. │ PanJiaChen/vue-element-admin │ 18310 │ 40. │ AobingJava/JavaFamily │ 18126 │ 41. │ jlevy/the-art-of-command-line │ 17661 │ 42. │ microsoft/terminal │ 17552 │ 43. │ GitSquared/edex-ui │ 17532 │ 44. │ tensorflow/tensorflow │ 17323 │ 45. │ evanw/esbuild │ 17213 │ 46. │ freeCodeCamp/freeCodeCamp │ 16904 │ 47. │ vinta/awesome-python │ 16851 │ 48. │ anuraghazra/github-readme-stats │ 16785 │ 49. │ angular/angular │ 16641 │ 50. │ kon9chunkit/GitHub-Chinese-Top-Charts │ 16468 │ └──────────────────────────────────────────────┴───────┘ 50 rows in set. Elapsed: 0.503 sec. Processed 503.38 million rows, 3.38 GB (1.00 billion rows/s., 6.72 GB/s.)
The top repo is a Chinese educational resource to prepare for an algorithms interview. Actually, almost all of the top 10 are related to education; the only exception is microsoft/PowerToys. Yes, I expected that 2020 would be about COVID-19, but happily enough it's not — CSSEGISandData/COVID-19 only reaches 20th place, and Deno is 14th.
:) SELECT repo_name, count() AS stars FROM github_events WHERE event_type = 'WatchEvent' AND toYear(created_at) = '2019' GROUP BY repo_name ORDER BY stars DESC LIMIT 50 ┌─repo_name──────────────────────────────────┬──stars─┐ 1. │ 996icu/996.ICU │ 332955 │ 2. │ jackfrued/Python-100-Days │ 76794 │ 3. │ Snailclimb/JavaGuide │ 53399 │ 4. │ TheAlgorithms/Python │ 48840 │ 5. │ CyC2018/CS-Notes │ 46596 │ 6. │ MisterBooo/LeetCodeAnimation │ 41517 │ 7. │ vuejs/vue │ 39004 │ 8. │ flutter/flutter │ 38657 │ 9. │ doocs/advanced-java │ 35315 │ 10. │ microsoft/Terminal │ 34320 │ 11. │ M4cs/BabySploit │ 31761 │ 12. │ jlevy/the-art-of-command-line │ 30837 │ 13. │ kamranahmedse/developer-roadmap │ 30826 │ 14. │ facebook/react │ 28974 │ 15. │ donnemartin/system-design-primer │ 27028 │ 16. │ tensorflow/tensorflow │ 27006 │ 17. │ testerSunshine/12306 │ 26380 │ 18. │ macrozheng/mall │ 26360 │ 19. │ sindresorhus/awesome │ 26274 │ 20. │ azl397985856/leetcode │ 25755 │ 21. │ jwasham/coding-interview-university │ 25549 │ 22. │ PanJiaChen/vue-element-admin │ 25069 │ 23. │ 0voice/interview_internal_reference │ 24545 │ 24. │ lib-pku/libpku │ 23002 │ 25. │ getify/You-Dont-Know-JS │ 22637 │ 26. │ microsoft/terminal │ 21864 │ 27. │ EbookFoundation/free-programming-books │ 21859 │ 28. │ scutan90/DeepLearning-500-questions │ 21774 │ 29. │ deepfakes/faceswap │ 21228 │ 30. │ sveltejs/svelte │ 20964 │ 31. │ torvalds/linux │ 20428 │ 32. │ vinta/awesome-python │ 20187 │ 33. │ justjavac/free-programming-books-zh_CN │ 20128 │ 34. │ 30-seconds/30-seconds-of-code │ 19774 │ 35. │ formulahendry/955.WLB │ 19764 │ 36. │ NationalSecurityAgency/ghidra │ 19645 │ 37. │ robbyrussell/oh-my-zsh │ 19598 │ 38. │ imhuay/Algorithm_Interview_Notes-Chinese │ 19469 │ 39. │ trekhleb/javascript-algorithms │ 19453 │ 40. │ LisaDziuba/Awesome-Design-Tools │ 19135 │ 41. │ freeCodeCamp/freeCodeCamp │ 18922 │ 42. │ golang/go │ 18801 │ 43. │ alibaba/flutter-go │ 18370 │ 44. │ ziishaned/learn-regex │ 18344 │ 45. │ github/gitignore │ 18210 │ 46. │ danistefanovic/build-your-own-x │ 18152 │ 47. │ ossu/computer-science │ 17787 │ 48. │ Advanced-Frontend/Daily-Interview-Question │ 17708 │ 49. │ dcloudio/uni-app │ 17693 │ 50. │ codercom/code-server │ 17686 │ └────────────────────────────────────────────┴────────┘ 50 rows in set. Elapsed: 0.439 sec. Processed 505.74 million rows, 3.38 GB (1.15 billion rows/s., 7.70 GB/s.)
996.ICU by a huge margin, and then education, education and more education. MS Terminal is worth noting, and a pentesting framework sits right after it.
:) SELECT repo_name, count() AS stars FROM github_events WHERE event_type = 'WatchEvent' AND toYear(created_at) = '2018' GROUP BY repo_name ORDER BY stars DESC LIMIT 50 ┌─repo_name──────────────────────────────┬─stars─┐ 1. │ vuejs/vue │ 51345 │ 2. │ trekhleb/javascript-algorithms │ 39232 │ 3. │ facebook/react │ 38603 │ 4. │ flutter/flutter │ 38319 │ 5. │ danistefanovic/build-your-own-x │ 37811 │ 6. │ tensorflow/tensorflow │ 36960 │ 7. │ kamranahmedse/developer-roadmap │ 35266 │ 8. │ x64dbg/x64dbg │ 32364 │ 9. │ donnemartin/system-design-primer │ 31359 │ 10. │ CyC2018/Interview-Notebook │ 29933 │ 11. │ kelseyhightower/nocode │ 26311 │ 12. │ Microsoft/vscode │ 26210 │ 13. │ xingshaocheng/architect-awesome │ 25749 │ 14. │ sindresorhus/awesome │ 24969 │ 15. │ ry/deno │ 22784 │ 16. │ getify/You-Dont-Know-JS │ 22006 │ 17. │ tensorflow/models │ 21499 │ 18. │ GoogleChrome/puppeteer │ 21130 │ 19. │ leonardomso/33-js-concepts │ 20996 │ 20. │ facebook/create-react-app │ 20297 │ 21. │ axios/axios │ 20082 │ 22. │ houshanren/hangzhou_house_knowledge │ 19226 │ 23. │ ant-design/ant-design │ 19143 │ 24. │ facebookresearch/Detectron │ 18606 │ 25. │ jwasham/coding-interview-university │ 18247 │ 26. │ robbyrussell/oh-my-zsh │ 18247 │ 27. │ EbookFoundation/free-programming-books │ 18121 │ 28. │ github/gitignore │ 17868 │ 29. │ airbnb/javascript │ 17611 │ 30. │ justjavac/free-programming-books-zh_CN │ 17268 │ 31. │ PanJiaChen/vue-element-admin │ 17252 │ 32. │ yangshun/front-end-interview-handbook │ 17135 │ 33. │ vinta/awesome-python │ 16794 │ 34. │ golang/go │ 16419 │ 35. │ facebook/react-native │ 16016 │ 36. │ torvalds/linux │ 15933 │ 37. │ TheAlgorithms/Python │ 15900 │ 38. │ tabler/tabler │ 15845 │ 39. │ kubernetes/kubernetes │ 15793 │ 40. │ Avik-Jain/100-Days-Of-ML-Code │ 15773 │ 41. │ iluwatar/java-design-patterns │ 15365 │ 42. │ parcel-bundler/parcel │ 15253 │ 43. │ Meituan-Dianping/mpvue │ 15009 │ 44. │ storybooks/storybook │ 14825 │ 45. │ dawnlabs/carbon │ 14799 │ 46. │ toddmotto/public-apis │ 14780 │ 47. │ electron/electron │ 14741 │ 48. │ shadowsocks/shadowsocks-windows │ 14614 │ 49. │ kdn251/interviews │ 14580 │ 50. │ scutan90/DeepLearning-500-questions │ 14495 │ └────────────────────────────────────────┴───────┘ 50 rows in set. Elapsed: 0.413 sec. Processed 503.94 million rows, 3.36 GB (1.22 billion rows/s., 8.14 GB/s.)
JavaScript frameworks and education.
:) SELECT repo_name, count() AS stars FROM github_events WHERE event_type = 'WatchEvent' AND toYear(created_at) = '2017' GROUP BY repo_name ORDER BY stars DESC LIMIT 50 ┌─repo_name──────────────────────────────────────┬─stars─┐ 1. │ freeCodeCamp/freeCodeCamp │ 90938 │ 2. │ tensorflow/tensorflow │ 49263 │ 3. │ vuejs/vue │ 48090 │ 4. │ facebook/react │ 34488 │ 5. │ mr-mig/every-programmer-should-know │ 30986 │ 6. │ kamranahmedse/developer-roadmap │ 30489 │ 7. │ sindresorhus/awesome │ 29066 │ 8. │ getify/You-Dont-Know-JS │ 28898 │ 9. │ thedaviddias/Front-End-Checklist │ 24735 │ 10. │ facebookincubator/create-react-app │ 24463 │ 11. │ Microsoft/vscode │ 23498 │ 12. │ donnemartin/system-design-primer │ 22580 │ 13. │ GoogleChrome/puppeteer │ 22401 │ 14. │ airbnb/javascript │ 22115 │ 15. │ sdmg15/Best-websites-a-programmer-should-visit │ 21756 │ 16. │ jwasham/coding-interview-university │ 21383 │ 17. │ twbs/bootstrap │ 21128 │ 18. │ toddmotto/public-apis │ 20878 │ 19. │ kamranahmedse/design-patterns-for-humans │ 19419 │ 20. │ robbyrussell/oh-my-zsh │ 19323 │ 21. │ airbnb/lottie-android │ 18695 │ 22. │ facebook/react-native │ 18667 │ 23. │ vuejs/awesome-vue │ 18497 │ 24. │ tipsy/github-profile-summary │ 17840 │ 25. │ FreeCodeCampChina/freecodecamp.cn │ 17782 │ 26. │ electron/electron │ 17470 │ 27. │ kdn251/interviews │ 17469 │ 28. │ ryanmcdermott/clean-code-javascript │ 17431 │ 29. │ mzabriskie/axios │ 17412 │ 30. │ vinta/awesome-python │ 17234 │ 31. │ torvalds/linux │ 17134 │ 32. │ tensorflow/models │ 17057 │ 33. │ nodejs/node │ 17051 │ 34. │ github/gitignore │ 16864 │ 35. │ python/cpython │ 16622 │ 36. │ yangshun/tech-interview-handbook │ 16344 │ 37. │ ElemeFE/element │ 16316 │ 38. │ bailicangdu/vue2-elm │ 16228 │ 39. │ d3/d3 │ 16163 │ 40. │ Hack-with-Github/Awesome-Hacking │ 16146 │ 41. │ golang/go │ 16080 │ 42. │ k88hudson/git-flight-rules │ 15943 │ 43. │ webpack/webpack │ 15867 │ 44. │ EbookFoundation/free-programming-books │ 15315 │ 45. │ mbeaudru/modern-js-cheatsheet │ 15307 │ 46. │ angular/angular │ 14370 │ 47. │ prettier/prettier │ 14336 │ 48. │ vhf/free-programming-books │ 14302 │ 49. │ JetBrains/kotlin │ 14203 │ 50. │ parcel-bundler/parcel │ 14188 │ └────────────────────────────────────────────────┴───────┘ 50 rows in set. Elapsed: 0.406 sec. Processed 501.66 million rows, 3.34 GB (1.23 billion rows/s., 8.22 GB/s.)
:) SELECT repo_name, count() AS stars FROM github_events WHERE event_type = 'WatchEvent' AND toYear(created_at) = '2016' GROUP BY repo_name ORDER BY stars DESC LIMIT 50 ┌─repo_name───────────────────────────────────────┬──stars─┐ 1. │ FreeCodeCamp/FreeCodeCamp │ 182073 │ 2. │ jwasham/google-interview-university │ 31516 │ 3. │ vhf/free-programming-books │ 28860 │ 4. │ vuejs/vue │ 28786 │ 5. │ tensorflow/tensorflow │ 28276 │ 6. │ facebook/react │ 26020 │ 7. │ getify/You-Dont-Know-JS │ 24967 │ 8. │ sindresorhus/awesome │ 24820 │ 9. │ chrislgarry/Apollo-11 │ 23478 │ 10. │ yarnpkg/yarn │ 21512 │ 11. │ facebook/react-native │ 19755 │ 12. │ twbs/bootstrap │ 19374 │ 13. │ airbnb/javascript │ 19010 │ 14. │ joshbuchea/HEAD │ 18436 │ 15. │ facebookincubator/create-react-app │ 18174 │ 16. │ firehol/netdata │ 17901 │ 17. │ robbyrussell/oh-my-zsh │ 17360 │ 18. │ FallibleInc/security-guide-for-developers │ 15894 │ 19. │ open-guides/og-aws │ 15611 │ 20. │ github/gitignore │ 15197 │ 21. │ electron/electron │ 15130 │ 22. │ googlesamples/android-architecture │ 14635 │ 23. │ torvalds/linux │ 14403 │ 24. │ getlantern/lantern │ 14176 │ 25. │ reactjs/redux │ 13985 │ 26. │ nodejs/node │ 13597 │ 27. │ Microsoft/vscode │ 13270 │ 28. │ angular/angular │ 13190 │ 29. │ apple/swift │ 13051 │ 30. │ ParsePlatform/parse-server │ 12830 │ 31. │ braydie/HowToBeAProgrammer │ 12562 │ 32. │ docker/docker │ 12476 │ 33. │ atom/atom │ 12461 │ 34. │ webpack/webpack │ 12184 │ 35. │ ZuzooVn/machine-learning-for-software-engineers │ 11913 │ 36. │ jgthms/bulma │ 11685 │ 37. │ vinta/awesome-python │ 11622 │ 38. │ angular/angular.js │ 11440 │ 39. │ golang/go │ 11332 │ 40. │ toddmotto/public-apis │ 11263 │ 41. │ ReactiveX/RxJava │ 11242 │ 42. │ daneden/animate.css │ 11099 │ 43. │ naptha/tesseract.js │ 10954 │ 44. │ tensorflow/models │ 10871 │ 45. │ josephmisiti/awesome-machine-learning │ 10545 │ 46. │ FortAwesome/Font-Awesome │ 10492 │ 47. │ juliangarnier/anime │ 10461 │ 48. │ AHAAAAAAA/PokemonGo-Map │ 10446 │ 49. │ Microsoft/TypeScript │ 10339 │ 50. │ caesar0301/awesome-public-datasets │ 10330 │ └─────────────────────────────────────────────────┴────────┘ 50 rows in set. Elapsed: 0.404 sec. Processed 498.41 million rows, 3.31 GB (1.23 billion rows/s., 8.18 GB/s.)
:) SELECT repo_name, count() AS stars FROM github_events WHERE event_type = 'WatchEvent' AND toYear(created_at) = '2015' GROUP BY repo_name ORDER BY stars DESC LIMIT 50 ┌─repo_name────────────────────────────────────┬─stars─┐ 1. │ FreeCodeCamp/FreeCodeCamp │ 38467 │ 2. │ facebook/react-native │ 25868 │ 3. │ apple/swift │ 25740 │ 4. │ sindresorhus/awesome │ 24408 │ 5. │ facebook/react │ 22950 │ 6. │ jlevy/the-art-of-command-line │ 22101 │ 7. │ NARKOZ/hacker-scripts │ 20444 │ 8. │ twbs/bootstrap │ 19504 │ 9. │ google/material-design-lite │ 17897 │ 10. │ airbnb/javascript │ 17574 │ 11. │ nvbn/thefuck │ 17164 │ 12. │ vhf/free-programming-books │ 16920 │ 13. │ getify/You-Dont-Know-JS │ 16402 │ 14. │ atom/electron │ 16150 │ 15. │ tensorflow/tensorflow │ 15988 │ 16. │ FreeCodeCamp/freecodecamp │ 15317 │ 17. │ nylas/N1 │ 14924 │ 18. │ angular/angular.js │ 14810 │ 19. │ atom/atom │ 13716 │ 20. │ mbostock/d3 │ 13495 │ 21. │ Dogfalo/materialize │ 13143 │ 22. │ ripienaar/free-for-dev │ 12843 │ 23. │ robbyrussell/oh-my-zsh │ 12276 │ 24. │ torvalds/linux │ 12263 │ 25. │ github/gitignore │ 11356 │ 26. │ bevacqua/dragula │ 11059 │ 27. │ meteor/meteor │ 10960 │ 28. │ docker/docker │ 10830 │ 29. │ FortAwesome/Font-Awesome │ 10791 │ 30. │ daneden/animate.css │ 10722 │ 31. │ zenorocha/clipboard.js │ 10468 │ 32. │ 0xAX/linux-insides │ 10454 │ 33. │ typicode/json-server │ 10242 │ 34. │ wasabeef/awesome-android-ui │ 10181 │ 35. │ babel/babel │ 10142 │ 36. │ h5bp/Front-end-Developer-Interview-Questions │ 10009 │ 37. │ prakhar1989/awesome-courses │ 9944 │ 38. │ lukehoban/es6features │ 9887 │ 39. │ driftyco/ionic │ 9770 │ 40. │ gorhill/uBlock │ 9758 │ 41. │ minimaxir/big-list-of-naughty-strings │ 9547 │ 42. │ Semantic-Org/Semantic-UI │ 9540 │ 43. │ alex/what-happens-when │ 9477 │ 44. │ Microsoft/vscode │ 9473 │ 45. │ vinta/awesome-python │ 9303 │ 46. │ golang/go │ 9243 │ 47. │ phanan/htaccess │ 9240 │ 48. │ nwjs/nw.js │ 9196 │ 49. │ johnpapa/angular-styleguide │ 9127 │ 50. │ google/material-design-icons │ 9093 │ └──────────────────────────────────────────────┴───────┘ 50 rows in set. Elapsed: 0.400 sec. Processed 495.32 million rows, 3.26 GB (1.24 billion rows/s., 8.14 GB/s.)
Bonus: all these results in a single report:
SELECT
year,
lower(repo_name) AS repo,
count()
FROM github_events
WHERE (event_type = 'WatchEvent') AND (year >= 2015)
GROUP BY
repo,
toYear(created_at) AS year
ORDER BY
year ASC,
count() DESC
LIMIT 10 BY year
┌─year─┬─repo──────────────────────────────────────────┬─count()─┐
│ 2015 │ freecodecamp/freecodecamp │ 53784 │
│ 2015 │ facebook/react-native │ 25868 │
│ 2015 │ apple/swift │ 25740 │
│ 2015 │ sindresorhus/awesome │ 24408 │
│ 2015 │ facebook/react │ 22950 │
│ 2015 │ jlevy/the-art-of-command-line │ 22101 │
│ 2015 │ narkoz/hacker-scripts │ 20444 │
│ 2015 │ twbs/bootstrap │ 19504 │
│ 2015 │ google/material-design-lite │ 17897 │
│ 2015 │ airbnb/javascript │ 17574 │
│ 2016 │ freecodecamp/freecodecamp │ 182073 │
│ 2016 │ jwasham/google-interview-university │ 31516 │
│ 2016 │ vhf/free-programming-books │ 28860 │
│ 2016 │ vuejs/vue │ 28786 │
│ 2016 │ tensorflow/tensorflow │ 28276 │
│ 2016 │ facebook/react │ 26020 │
│ 2016 │ getify/you-dont-know-js │ 24967 │
│ 2016 │ sindresorhus/awesome │ 24820 │
│ 2016 │ chrislgarry/apollo-11 │ 23478 │
│ 2016 │ yarnpkg/yarn │ 21512 │
│ 2017 │ freecodecamp/freecodecamp │ 96305 │
│ 2017 │ tensorflow/tensorflow │ 49263 │
│ 2017 │ vuejs/vue │ 48090 │
│ 2017 │ facebook/react │ 34488 │
│ 2017 │ mr-mig/every-programmer-should-know │ 30986 │
│ 2017 │ kamranahmedse/developer-roadmap │ 30489 │
│ 2017 │ sindresorhus/awesome │ 29066 │
│ 2017 │ getify/you-dont-know-js │ 28898 │
│ 2017 │ thedaviddias/front-end-checklist │ 24735 │
│ 2017 │ facebookincubator/create-react-app │ 24463 │
│ 2018 │ vuejs/vue │ 51345 │
│ 2018 │ trekhleb/javascript-algorithms │ 39232 │
│ 2018 │ facebook/react │ 38603 │
│ 2018 │ flutter/flutter │ 38319 │
│ 2018 │ danistefanovic/build-your-own-x │ 37811 │
│ 2018 │ tensorflow/tensorflow │ 36960 │
│ 2018 │ kamranahmedse/developer-roadmap │ 35266 │
│ 2018 │ x64dbg/x64dbg │ 32364 │
│ 2018 │ donnemartin/system-design-primer │ 31359 │
│ 2018 │ cyc2018/interview-notebook │ 29933 │
│ 2019 │ 996icu/996.icu │ 332955 │
│ 2019 │ jackfrued/python-100-days │ 76794 │
│ 2019 │ microsoft/terminal │ 56791 │
│ 2019 │ snailclimb/javaguide │ 53399 │
│ 2019 │ thealgorithms/python │ 48840 │
│ 2019 │ cyc2018/cs-notes │ 46596 │
│ 2019 │ misterbooo/leetcodeanimation │ 41517 │
│ 2019 │ vuejs/vue │ 39004 │
│ 2019 │ flutter/flutter │ 38657 │
│ 2019 │ doocs/advanced-java │ 35315 │
│ 2020 │ labuladong/fucking-algorithm │ 81959 │
│ 2020 │ jwasham/coding-interview-university │ 63213 │
│ 2020 │ kamranahmedse/developer-roadmap │ 54303 │
│ 2020 │ public-apis/public-apis │ 40661 │
│ 2020 │ donnemartin/system-design-primer │ 40402 │
│ 2020 │ ebookfoundation/free-programming-books │ 40059 │
│ 2020 │ thealgorithms/python │ 36526 │
│ 2020 │ danistefanovic/build-your-own-x │ 36524 │
│ 2020 │ cyc2018/cs-notes │ 35381 │
│ 2020 │ microsoft/powertoys │ 35331 │
│ 2021 │ ant-design/ant-design │ 79848 │
│ 2021 │ public-apis/public-apis │ 69555 │
│ 2021 │ jwasham/coding-interview-university │ 56058 │
│ 2021 │ ebookfoundation/free-programming-books │ 50575 │
│ 2021 │ kamranahmedse/developer-roadmap │ 45149 │
│ 2021 │ donnemartin/system-design-primer │ 41829 │
│ 2021 │ trekhleb/javascript-algorithms │ 41684 │
│ 2021 │ danistefanovic/build-your-own-x │ 36829 │
│ 2021 │ sindresorhus/awesome │ 35556 │
│ 2021 │ ossu/computer-science │ 33680 │
│ 2022 │ aplus-framework/app │ 92632 │
│ 2022 │ sindresorhus/awesome │ 55839 │
│ 2022 │ aplus-framework/database │ 53822 │
│ 2022 │ public-apis/public-apis │ 52515 │
│ 2022 │ donnemartin/system-design-primer │ 51603 │
│ 2022 │ anduin2017/howtocook │ 47303 │
│ 2022 │ kamranahmedse/developer-roadmap │ 44122 │
│ 2022 │ ebookfoundation/free-programming-books │ 44083 │
│ 2022 │ jwasham/coding-interview-university │ 43695 │
│ 2022 │ vinta/awesome-python │ 43230 │
│ 2023 │ automatic1111/stable-diffusion-webui │ 87369 │
│ 2023 │ base-org/chains │ 85316 │
│ 2023 │ significant-gravitas/auto-gpt │ 84899 │
│ 2023 │ f/awesome-chatgpt-prompts │ 81664 │
│ 2023 │ base-org/node │ 74042 │
│ 2023 │ practical-tutorials/project-based-learning │ 61102 │
│ 2023 │ twitter/the-algorithm │ 60262 │
│ 2023 │ sindresorhus/awesome │ 56970 │
│ 2023 │ torantulino/auto-gpt │ 56840 │
│ 2023 │ codecrafters-io/build-your-own-x │ 55452 │
│ 2024 │ rabbitminers/extended-bogeys │ 100547 │
│ 2024 │ codecrafters-io/build-your-own-x │ 88066 │
│ 2024 │ ollama/ollama │ 70695 │
│ 2024 │ practical-tutorials/project-based-learning │ 67701 │
│ 2024 │ sindresorhus/awesome │ 64977 │
│ 2024 │ massgravel/microsoft-activation-scripts │ 53332 │
│ 2024 │ public-apis/public-apis │ 52853 │
│ 2024 │ zed-industries/zed │ 52516 │
│ 2024 │ awesome-selfhosted/awesome-selfhosted │ 49646 │
│ 2024 │ krahets/hello-algo │ 48599 │
│ 2025 │ codecrafters-io/build-your-own-x │ 99603 │
│ 2025 │ deepseek-ai/deepseek-r1 │ 86882 │
│ 2025 │ deepseek-ai/deepseek-v3 │ 85274 │
│ 2025 │ unionlabs/union │ 79643 │
│ 2025 │ n8n-io/n8n │ 77991 │
│ 2025 │ x1xhlol/system-prompts-and-models-of-ai-tools │ 71357 │
│ 2025 │ digitalplatdev/freedomain │ 66670 │
│ 2025 │ langflow-ai/langflow │ 65669 │
│ 2025 │ sindresorhus/awesome │ 62531 │
│ 2025 │ punkpeye/awesome-mcp-servers │ 59862 │
│ 2026 │ openclaw/openclaw │ 73357 │
│ 2026 │ obra/superpowers │ 40256 │
│ 2026 │ affaan-m/everything-claude-code │ 39887 │
│ 2026 │ anomalyco/opencode │ 32891 │
│ 2026 │ nousresearch/hermes-agent │ 27516 │
│ 2026 │ anthropics/skills │ 27038 │
│ 2026 │ nextlevelbuilder/ui-ux-pro-max-skill │ 19469 │
│ 2026 │ clawdbot/clawdbot │ 19371 │
│ 2026 │ msitarzewski/agency-agents │ 19166 │
│ 2026 │ forrestchang/andrej-karpathy-skills │ 18319 │
└──────┴───────────────────────────────────────────────┴─────────┘
120 rows in set. Elapsed: 4.862 sec. Processed 553.94 million rows, 5.61 GB (113.93 million rows/s., 1.15 GB/s.)
And a multi-line graph:
SELECT
repo AS name,
groupArrayInsertAt(toUInt32(c), toUInt64(dateDiff('month', toDate('2015-01-01'), month))) AS data
FROM
(
SELECT
lower(repo_name) AS repo,
toStartOfMonth(created_at) AS month,
count() AS c
FROM github_events
WHERE (event_type = 'WatchEvent') AND (toYear(created_at) >= 2015) AND (repo IN
(
SELECT lower(repo_name) AS repo
FROM github_events
WHERE (event_type = 'WatchEvent') AND (toYear(created_at) >= 2015)
GROUP BY repo
ORDER BY count() DESC
LIMIT 10
))
GROUP BY
repo,
month
)
GROUP BY repo
ORDER BY repo ASC
10 rows in set. Elapsed: 4.544 sec. Processed 1.11 billion rows, 8.46 GB (243.80 million rows/s., 1.86 GB/s.)
:) SELECT toYear(created_at) AS year, count() AS stars, bar(stars, 0, 80000000, 10) AS bar FROM github_events WHERE event_type = 'WatchEvent' GROUP BY year ORDER BY year
┌─year─┬────stars─┬─bar────────┐
│ 2011 │ 1831523 │ ▏ │
│ 2012 │ 4047169 │ ▌ │
│ 2013 │ 7427951 │ ▉ │
│ 2014 │ 11943791 │ █▍ │
│ 2015 │ 18980008 │ ██▎ │
│ 2016 │ 26152326 │ ███▎ │
│ 2017 │ 32616858 │ ████ │
│ 2018 │ 37042824 │ ████▋ │
│ 2019 │ 46024600 │ █████▊ │
│ 2020 │ 49382862 │ ██████▏ │
│ 2021 │ 50034134 │ ██████▎ │
│ 2022 │ 55932066 │ ██████▉ │
│ 2023 │ 73422352 │ █████████▏ │
│ 2024 │ 74747313 │ █████████▎ │
│ 2025 │ 54409114 │ ██████▊ │
│ 2026 │ 10406627 │ █▎ │
└──────┴──────────┴────────────┘
16 rows in set. Elapsed: 0.184 sec. Processed 554.52 million rows, 2.77 GB (3.01 billion rows/s., 15.06 GB/s.)
Growth was steady until 2024. The decline in 2025 and the collapse in 2026 are not a change in the behavior of GitHub users — they are a change in the data source: since the middle of 2025 the GH Archive feed reports almost only PushEvent, so most star events never reach the dataset.
:) SELECT actor_login, count() AS stars FROM github_events WHERE event_type = 'WatchEvent' GROUP BY actor_login ORDER BY stars DESC LIMIT 50
┌─actor_login─────┬───stars─┐
1. │ Good4lien │ 1140636 │
2. │ insolitum │ 465461 │
3. │ italy │ 264778 │
4. │ 4148 │ 232492 │
5. │ elhadjx │ 226339 │
6. │ daweedkob │ 215747 │
7. │ salifm │ 202516 │
8. │ gaoypChina │ 157813 │
9. │ CodePromoter │ 149686 │
10. │ aplus-developer │ 141409 │
11. │ shahradelahi │ 133952 │
12. │ dinosoid │ 132747 │
13. │ edom71 │ 126018 │
14. │ alineai18 │ 117868 │
15. │ seanpm2001 │ 109723 │
16. │ gkze │ 109689 │
17. │ fly51fly │ 105881 │
18. │ tetrabyyte │ 100408 │
19. │ 5l1v3r1 │ 97717 │
20. │ quarkusbot │ 97329 │
21. │ 411112 │ 78485 │
22. │ JeffCarpenter │ 78276 │
23. │ x0rzkov │ 74165 │
24. │ esneko │ 73698 │
25. │ hangga │ 73269 │
26. │ BottleSome │ 69621 │
27. │ nikitavoloboev │ 64124 │
28. │ romanofficial │ 61465 │
29. │ MilleniumSpark │ 60507 │
30. │ romanbugrin │ 60036 │
31. │ gauravssnl │ 60032 │
32. │ dkapt │ 59875 │
33. │ ik5 │ 56519 │
34. │ tkersey │ 55537 │
35. │ Sandalots │ 55058 │
36. │ Maxxum69 │ 54642 │
37. │ trippup │ 51870 │
38. │ nschlemm │ 51479 │
39. │ thanhtoan1196 │ 51368 │
40. │ baslr │ 50365 │
41. │ denji │ 49257 │
42. │ mitoksim │ 49229 │
43. │ missAnneThorpe │ 47212 │
44. │ abhijit1990 │ 46720 │
45. │ diffblogbot │ 45692 │
46. │ StarTheWorld │ 45691 │
47. │ mvandermeulen │ 44800 │
48. │ mcanthony │ 44324 │
49. │ tuanductran │ 44297 │
50. │ athosss23 │ 44161 │
└─────────────────┴─────────┘
50 rows in set. Elapsed: 8.159 sec. Processed 554.51 million rows, 9.12 GB (67.96 million rows/s., 1.12 GB/s.)
The top account gave 1 140 636 stars. That is not a person — at one star per minute it would take more than two years without sleep. Star farms have become an industry, and 4148, the leader in the first edition of this article, has been pushed down to 4th place.
In comparison, I only gave 497 stars:
:) SELECT actor_login, count() AS stars FROM github_events WHERE event_type = 'WatchEvent' AND actor_login = 'alexey-milovidov' GROUP BY actor_login ORDER BY stars DESC LIMIT 50
┌─actor_login──────┬─stars─┐
│ alexey-milovidov │ 497 │
└──────────────────┴───────┘
1 rows in set. Elapsed: 1.381 sec. Processed 554.51 million rows, 8.45 GB (401.47 million rows/s., 6.12 GB/s.)
And here are my favorites sorted by the total number of stars:
SELECT
repo_name,
count() AS stars
FROM github_events
WHERE (event_type = 'WatchEvent') AND (repo_name IN
(
SELECT repo_name
FROM github_events
WHERE (event_type = 'WatchEvent') AND (actor_login = 'alexey-milovidov')
))
GROUP BY repo_name
ORDER BY stars DESC
LIMIT 50
┌─repo_name─────────────────────────────┬─stars─┐
1. │ microsoft/playwright │ 80473 │
2. │ ocornut/imgui │ 74424 │
3. │ minimaxir/big-list-of-naughty-strings │ 49711 │
4. │ koalaman/shellcheck │ 40526 │
5. │ makeplane/plane │ 40352 │
6. │ llvm/llvm-project │ 37114 │
7. │ httpie/httpie │ 35463 │
8. │ SerenityOS/serenity │ 34735 │
9. │ nothings/stb │ 32872 │
10. │ kovidgoyal/kitty │ 32126 │
11. │ acmesh-official/acme.sh │ 29722 │
12. │ danny-avila/LibreChat │ 29186 │
13. │ gabime/spdlog │ 29080 │
14. │ sharkdp/hyperfine │ 26278 │
15. │ qdrant/qdrant │ 26238 │
16. │ mxgmn/WaveFunctionCollapse │ 25549 │
17. │ Swordfish90/cool-retro-term │ 24739 │
18. │ SigNoz/signoz │ 23920 │
19. │ plausible/analytics │ 23030 │
20. │ copy/v86 │ 21508 │
21. │ tabler/tabler-icons │ 20669 │
22. │ jart/cosmopolitan │ 20195 │
23. │ dolthub/dolt │ 18494 │
24. │ githubnext/monaspace │ 17212 │
25. │ rsms/inter │ 16545 │
26. │ rqlite/rqlite │ 16447 │
27. │ asciinema/asciinema │ 16278 │
28. │ HubSpot/youmightnotneedjquery │ 16189 │
29. │ chrislusf/seaweedfs │ 15113 │
30. │ spotify/annoy │ 14399 │
31. │ projectdiscovery/katana │ 14204 │
32. │ mikefarah/yq │ 13740 │
33. │ HackerNews/API │ 13515 │
34. │ bytebase/bytebase │ 12932 │
35. │ cube-js/cube.js │ 12176 │
36. │ GTFOBins/GTFOBins.github.io │ 12149 │
37. │ darlinghq/darling │ 12124 │
38. │ google/oss-fuzz │ 12013 │
39. │ bentoml/OpenLLM │ 11128 │
40. │ jemalloc/jemalloc │ 10947 │
41. │ Canner/WrenAI │ 10890 │
42. │ hashicorp/packer │ 10702 │
43. │ google/magika │ 9921 │
44. │ giscus/giscus │ 9867 │
45. │ olivernn/lunr.js │ 9756 │
46. │ leeoniya/uPlot │ 9745 │
47. │ lemire/simdjson │ 9523 │
48. │ catboost/catboost │ 9415 │
49. │ antirez/kilo │ 9365 │
50. │ ange-yaghi/engine-sim │ 9307 │
└───────────────────────────────────────┴───────┘
50 rows in set. Elapsed: 0.124 sec. Processed 23.81 million rows, 221.78 MB (191.26 million rows/s., 1.78 GB/s.)
A quick question I'm eager to ask: what are the top repositories sorted by the number of stars from people who starred the ClickHouse repository?
SELECT
repo_name,
count() AS stars
FROM github_events
WHERE (event_type = 'WatchEvent') AND (actor_login IN
(
SELECT actor_login
FROM github_events
WHERE (event_type = 'WatchEvent') AND (repo_name IN ('ClickHouse/ClickHouse', 'yandex/ClickHouse'))
)) AND (repo_name NOT IN ('ClickHouse/ClickHouse', 'yandex/ClickHouse'))
GROUP BY repo_name
ORDER BY stars DESC
LIMIT 50
┌─repo_name──────────────────────────────┬─stars─┐
1. │ donnemartin/system-design-primer │ 14064 │
2. │ ant-design/ant-design │ 12908 │
3. │ sindresorhus/awesome │ 12245 │
4. │ golang/go │ 12078 │
5. │ tensorflow/tensorflow │ 11710 │
6. │ kubernetes/kubernetes │ 11430 │
7. │ avelino/awesome-go │ 10667 │
8. │ prometheus/prometheus │ 10416 │
9. │ facebook/react │ 10258 │
10. │ grafana/grafana │ 10224 │
11. │ minio/minio │ 10172 │
12. │ n8n-io/n8n │ 10171 │
13. │ tauri-apps/tauri │ 10156 │
14. │ kamranahmedse/developer-roadmap │ 10098 │
15. │ torvalds/linux │ 9936 │
16. │ gin-gonic/gin │ 9900 │
17. │ pingcap/tidb │ 9669 │
18. │ flutter/flutter │ 9557 │
19. │ rust-lang/rust │ 9406 │
20. │ supabase/supabase │ 9156 │
21. │ excalidraw/excalidraw │ 9126 │
22. │ vuejs/vue │ 8668 │
23. │ jwasham/coding-interview-university │ 8627 │
24. │ vinta/awesome-python │ 8533 │
25. │ elastic/elasticsearch │ 8493 │
26. │ huggingface/transformers │ 8318 │
27. │ jlevy/the-art-of-command-line │ 8276 │
28. │ fatedier/frp │ 8235 │
29. │ TheAlgorithms/Python │ 8125 │
30. │ public-apis/public-apis │ 8093 │
31. │ rustdesk/rustdesk │ 8074 │
32. │ microsoft/playwright │ 8068 │
33. │ metabase/metabase │ 8039 │
34. │ facebook/rocksdb │ 7975 │
35. │ ansible/ansible │ 7803 │
36. │ cockroachdb/cockroach │ 7714 │
37. │ istio/istio │ 7570 │
38. │ EbookFoundation/free-programming-books │ 7459 │
39. │ syncthing/syncthing │ 7425 │
40. │ sveltejs/svelte │ 7353 │
41. │ facebook/react-native │ 7294 │
42. │ pytorch/pytorch │ 7290 │
43. │ tiangolo/fastapi │ 7261 │
44. │ 996icu/996.ICU │ 7223 │
45. │ langgenius/dify │ 7184 │
46. │ milvus-io/milvus │ 7148 │
47. │ electron/electron │ 7095 │
48. │ denoland/deno │ 7054 │
49. │ grpc/grpc │ 7030 │
50. │ danistefanovic/build-your-own-x │ 7030 │
└────────────────────────────────────────┴───────┘
50 rows in set. Elapsed: 2.141 sec. Processed 554.66 million rows, 11.60 GB (259.13 million rows/s., 5.42 GB/s.)
The head of the list is now taken by the same educational content that tops the global list, but right behind it there is Go, Kubernetes, Prometheus and Grafana. The overall list is still clearly biased towards data-intensive applications: database engines, distributed filesystems, data orchestration, observability.
Let's calculate the affinity index: the ratio of stars to repositories that were given by users that gave a star to ClickHouse | Linux | LLVM. Don't worry, the query is pretty simple.
SELECT repo_name, uniq(actor_login) AS total_stars, uniqIf(actor_login, actor_login IN ( SELECT actor_login FROM github_events WHERE (event_type = 'WatchEvent') AND (repo_name IN ('ClickHouse/ClickHouse', 'yandex/ClickHouse')) )) AS clickhouse_stars, round(clickhouse_stars / total_stars, 2) AS ratio FROM github_events WHERE (event_type = 'WatchEvent') AND (repo_name NOT IN ('ClickHouse/ClickHouse', 'yandex/ClickHouse')) GROUP BY repo_name HAVING total_stars >= 100 ORDER BY ratio DESC LIMIT 50 ┌─repo_name─────────────────────────────┬─total_stars─┬─clickhouse_stars─┬─ratio─┐ │ yandex/clickhouse-jdbc │ 340 │ 246 │ 0.72 │ │ yandex/clickhouse-presentations │ 244 │ 173 │ 0.71 │ │ f1yegor/clickhouse_exporter │ 206 │ 140 │ 0.68 │ │ roistat/go-clickhouse │ 202 │ 133 │ 0.66 │ │ Vertamedia/clickhouse-grafana │ 563 │ 365 │ 0.65 │ │ yandex/graphouse │ 214 │ 140 │ 0.65 │ │ ClickHouse/clickhouse_exporter │ 214 │ 139 │ 0.65 │ │ cloudflare/sqlalchemy-clickhouse │ 342 │ 222 │ 0.65 │ │ Altinity/clickhouse-rpm-install │ 154 │ 99 │ 0.64 │ │ smi2/clickhouse-frontend │ 200 │ 129 │ 0.64 │ │ Percona-Lab/clickhousedb_fdw │ 215 │ 135 │ 0.63 │ │ ClickHouse/clickhouse-docs │ 165 │ 104 │ 0.63 │ │ datafusedev/fuse-query │ 291 │ 183 │ 0.63 │ │ grafana/clickhouse-datasource │ 171 │ 107 │ 0.63 │ │ Vertamedia/chproxy │ 743 │ 471 │ 0.63 │ │ Slach/clickhouse-flamegraph │ 200 │ 124 │ 0.62 │ │ kshvakov/clickhouse │ 637 │ 395 │ 0.62 │ │ ClickHouse/click-ui │ 101 │ 63 │ 0.62 │ │ smi2/tabix.ui │ 300 │ 185 │ 0.62 │ │ suharev7/clickhouse-rs │ 360 │ 218 │ 0.61 │ │ artpaul/clickhouse-cpp │ 105 │ 64 │ 0.61 │ │ adjust/clickhouse_fdw │ 191 │ 116 │ 0.61 │ │ Altinity/clicktail │ 289 │ 172 │ 0.6 │ │ lomik/graphite-clickhouse │ 178 │ 107 │ 0.6 │ │ ClickHouse/JSONBench │ 151 │ 90 │ 0.6 │ │ housepower/clickhouse_sinker │ 531 │ 319 │ 0.6 │ │ Percona-Lab/PromHouse │ 274 │ 165 │ 0.6 │ │ HouseOps/HouseOps │ 334 │ 197 │ 0.59 │ │ ClickHouse/clickhouse-cpp │ 350 │ 205 │ 0.59 │ │ hatarist/clickhouse-cli │ 261 │ 153 │ 0.59 │ │ zhihu/zetta │ 142 │ 83 │ 0.58 │ │ lomik/carbon-clickhouse │ 159 │ 93 │ 0.58 │ │ ClickHouse/clickhouse-odbc │ 199 │ 116 │ 0.58 │ │ Altinity/clickhouse-mysql-data-reader │ 390 │ 225 │ 0.58 │ │ mailru/go-clickhouse │ 448 │ 256 │ 0.57 │ │ icelake-io/icelake │ 159 │ 90 │ 0.57 │ │ ClickHouse/clickhouse-jdbc │ 800 │ 457 │ 0.57 │ │ pingcap/titan │ 102 │ 57 │ 0.56 │ │ ClickHouse/clickhouse-jdbc-bridge │ 162 │ 90 │ 0.56 │ │ ClickHouse/clickhouse-presentations │ 820 │ 462 │ 0.56 │ │ analysys/olap │ 130 │ 73 │ 0.56 │ │ housepower/ClickHouse-Native-JDBC │ 534 │ 301 │ 0.56 │ │ ClickHouse-China/ClickhouseMeetup │ 165 │ 93 │ 0.56 │ │ vectorengine/vectorsql │ 324 │ 182 │ 0.56 │ │ nikepan/clickhouse-bulk │ 498 │ 277 │ 0.56 │ │ korchasa/awesome-clickhouse │ 164 │ 91 │ 0.55 │ │ kszucs/pandahouse │ 242 │ 132 │ 0.55 │ │ crobox/clickhouse-scala-client │ 123 │ 68 │ 0.55 │ │ mindis/prom2click │ 165 │ 90 │ 0.55 │ │ loyd/clickhouse.rs │ 243 │ 134 │ 0.55 │ └───────────────────────────────────────┴─────────────┴──────────────────┴───────┘ 50 rows in set. Elapsed: 7.872 sec. Processed 554.66 million rows, 11.83 GB (70.46 million rows/s., 1.50 GB/s.)
This is the perfect exploration tool for related repositories!
I also added a threshold on the number of stars and I'm counting stars from unique authors instead of just star events, because otherwise I face some data anomalies.
SELECT repo_name, uniq(actor_login) AS total_stars, uniqIf(actor_login, actor_login IN ( SELECT actor_login FROM github_events WHERE (event_type = 'WatchEvent') AND (repo_name IN ('torvalds/linux')) )) AS clickhouse_stars, round(clickhouse_stars / total_stars, 2) AS ratio FROM github_events WHERE (event_type = 'WatchEvent') AND (repo_name NOT IN ('torvalds/linux')) GROUP BY repo_name HAVING total_stars >= 100 ORDER BY ratio DESC LIMIT 50 ┌─repo_name─────────────────────────┬─total_stars─┬─clickhouse_stars─┬─ratio─┐ │ MeherRushi/wolverine │ 253 │ 247 │ 0.98 │ │ torvalds/libdc-for-dirk │ 375 │ 290 │ 0.77 │ │ torvalds/libgit2 │ 266 │ 192 │ 0.72 │ │ torvalds/test-tlb │ 966 │ 682 │ 0.71 │ │ torvalds/subsurface-for-dirk │ 463 │ 325 │ 0.7 │ │ torvalds/subsurface │ 1218 │ 820 │ 0.67 │ │ torvalds/pesconvert │ 557 │ 367 │ 0.66 │ │ mirror/make │ 254 │ 168 │ 0.66 │ │ freedesktop/xorg-xserver │ 164 │ 105 │ 0.64 │ │ GNOME/gdm │ 121 │ 76 │ 0.63 │ │ archlinux/archlinux-repro │ 143 │ 88 │ 0.62 │ │ xfce-mirror/xfwm4 │ 125 │ 77 │ 0.62 │ │ coreos/grub │ 460 │ 285 │ 0.62 │ │ torvalds/uemacs │ 1865 │ 1153 │ 0.62 │ │ Subsurface-divelog/subsurface │ 718 │ 438 │ 0.61 │ │ DROPCitizenShip/linux │ 103 │ 62 │ 0.6 │ │ svenkatr/linux │ 110 │ 64 │ 0.58 │ │ subsurface/subsurface │ 1391 │ 801 │ 0.58 │ │ archlinux/linux │ 1435 │ 829 │ 0.58 │ │ gregkh/kernel-history │ 120 │ 70 │ 0.58 │ │ joshumax/hurd │ 212 │ 124 │ 0.58 │ │ archlinux/mkinitcpio │ 227 │ 131 │ 0.58 │ │ GNOME/gnome-software │ 101 │ 59 │ 0.58 │ │ archlinux/devtools │ 115 │ 67 │ 0.58 │ │ letolabs/nasm │ 113 │ 65 │ 0.58 │ │ lvmteam/lvm2 │ 147 │ 84 │ 0.57 │ │ wayland-project/wayland-protocols │ 155 │ 88 │ 0.57 │ │ openSUSE/kernel │ 248 │ 142 │ 0.57 │ │ rhboot/efivar │ 229 │ 130 │ 0.57 │ │ mirrors/gcc │ 435 │ 242 │ 0.56 │ │ rust-cli/man │ 108 │ 60 │ 0.56 │ │ Alexpux/Cygwin │ 101 │ 57 │ 0.56 │ │ apple/darwin-libplatform │ 142 │ 80 │ 0.56 │ │ GNOME/gnome-control-center │ 114 │ 64 │ 0.56 │ │ GNOME/gnome-desktop │ 160 │ 90 │ 0.56 │ │ rhboot/grub2 │ 288 │ 161 │ 0.56 │ │ haskell/HTTP │ 229 │ 129 │ 0.56 │ │ google/kmsan │ 449 │ 251 │ 0.56 │ │ gregkh/linux │ 650 │ 361 │ 0.56 │ │ aquasecurity/btfhub-archive │ 116 │ 64 │ 0.55 │ │ huawei-openlab/oct │ 109 │ 60 │ 0.55 │ │ archlinux/archweb │ 393 │ 218 │ 0.55 │ │ anakryiko/bpf-ringbuf-examples │ 134 │ 74 │ 0.55 │ │ mirror/vbox │ 287 │ 157 │ 0.55 │ │ apple/darwin-libpthread │ 176 │ 97 │ 0.55 │ │ gregkh/kernel-development │ 719 │ 392 │ 0.55 │ │ rlane/ubpf │ 107 │ 59 │ 0.55 │ │ gtk-rs/glib │ 108 │ 59 │ 0.55 │ │ GNOME/gnome-terminal │ 371 │ 205 │ 0.55 │ │ xfce-mirror/thunar │ 291 │ 159 │ 0.55 │ └───────────────────────────────────┴─────────────┴──────────────────┴───────┘ 50 rows in set. Elapsed: 8.098 sec. Processed 554.66 million rows, 11.82 GB (68.49 million rows/s., 1.46 GB/s.)
SELECT repo_name, uniq(actor_login) AS total_stars, uniqIf(actor_login, actor_login IN ( SELECT actor_login FROM github_events WHERE (event_type = 'WatchEvent') AND (repo_name IN ('llvm/llvm-project')) )) AS clickhouse_stars, round(clickhouse_stars / total_stars, 2) AS ratio FROM github_events WHERE (event_type = 'WatchEvent') AND (repo_name NOT IN ('llvm/llvm-project')) GROUP BY repo_name HAVING total_stars >= 100 ORDER BY ratio DESC LIMIT 50 ┌─repo_name──────────────────────────────────────────────────────────────────────────────┬─total_stars─┬─clickhouse_stars─┬─ratio─┐ │ llvm/clangir │ 526 │ 337 │ 0.64 │ │ llvm/llvm-test-suite │ 212 │ 130 │ 0.61 │ │ facebookincubator/clangir │ 137 │ 80 │ 0.58 │ │ rust-lang/llvm-project │ 149 │ 86 │ 0.58 │ │ Meinersbur/ppcg │ 128 │ 71 │ 0.55 │ │ Jason2013/gslcl │ 127 │ 67 │ 0.53 │ │ llvm/Polygeist │ 473 │ 248 │ 0.52 │ │ llvmenv/llvmenv │ 113 │ 58 │ 0.51 │ │ tensorflow/mlir-hlo │ 420 │ 215 │ 0.51 │ │ PacktPublishing/Learn-LLVM-17 │ 210 │ 107 │ 0.51 │ │ mdadams/clang_libraries_companion │ 118 │ 59 │ 0.5 │ │ kumasento/polymer │ 113 │ 56 │ 0.5 │ │ ksco/rvld │ 103 │ 51 │ 0.5 │ │ rust-lang/ena │ 101 │ 49 │ 0.49 │ │ open64-compiler/open64 │ 100 │ 49 │ 0.49 │ │ spcl/pymlir │ 266 │ 131 │ 0.49 │ │ llvm/mlir-npcomp │ 152 │ 74 │ 0.49 │ │ PacktPublishing/LLVM-Techniques-Tips-and-Best-Practices-Clang-and-Middle-End-Libraries │ 188 │ 93 │ 0.49 │ │ google/llvm-propeller │ 497 │ 237 │ 0.48 │ │ PacktPublishing/Learn-LLVM-12 │ 492 │ 234 │ 0.48 │ │ GPUOpen-Drivers/llpc │ 190 │ 92 │ 0.48 │ │ apache/tvm-rfcs │ 111 │ 53 │ 0.48 │ │ rust-fuzz/libfuzzer │ 183 │ 87 │ 0.48 │ │ Lewuathe/mlir-hello │ 134 │ 64 │ 0.48 │ │ rust-lang/rustc-demangle │ 163 │ 76 │ 0.47 │ │ trailofbits/binrec-tob │ 133 │ 63 │ 0.47 │ │ abenkhadra/llvm-pass-tutorial │ 245 │ 114 │ 0.47 │ │ AliveToolkit/alive2 │ 959 │ 451 │ 0.47 │ │ banach-space/clang-tutor │ 761 │ 358 │ 0.47 │ │ rust-lang/compiler-builtins │ 340 │ 159 │ 0.47 │ │ KhronosGroup/SPIRV-LLVM-Translator │ 591 │ 275 │ 0.47 │ │ rust-lang/libz-sys │ 121 │ 56 │ 0.46 │ │ iml130/mlir-emitc │ 127 │ 58 │ 0.46 │ │ cuviper/autocfg │ 103 │ 47 │ 0.46 │ │ KyleMayes/install-llvm-action │ 137 │ 63 │ 0.46 │ │ tari/llvm-sys.rs │ 216 │ 98 │ 0.45 │ │ ROCm/llvm-project │ 104 │ 47 │ 0.45 │ │ tlc-pack/relax │ 199 │ 89 │ 0.45 │ │ PacktPublishing/LLVM-Code-Generation │ 124 │ 54 │ 0.44 │ │ PaddlePaddle/CINN │ 135 │ 60 │ 0.44 │ │ rust-lang/stdarch │ 363 │ 160 │ 0.44 │ │ ROCm-Developer-Tools/aomp │ 157 │ 69 │ 0.44 │ │ itanium-cxx-abi/cxx-abi │ 555 │ 244 │ 0.44 │ │ rust-lang/cargo-bisect-rustc │ 176 │ 75 │ 0.43 │ │ intel/opencl-clang │ 152 │ 66 │ 0.43 │ │ getsentry/pdb │ 115 │ 50 │ 0.43 │ │ mirror/make │ 254 │ 108 │ 0.43 │ │ rust-lang/pkg-config-rs │ 141 │ 61 │ 0.43 │ │ P2Tree/LLVM_for_cpu0 │ 250 │ 108 │ 0.43 │ │ oneapi-src/oneDPL │ 261 │ 113 │ 0.43 │ └────────────────────────────────────────────────────────────────────────────────────────┴─────────────┴──────────────────┴───────┘ 50 rows in set. Elapsed: 7.079 sec. Processed 554.66 million rows, 11.83 GB (78.35 million rows/s., 1.67 GB/s.)
Surprisingly this simple heuristic works really well!
Let me find a friend by the intersection on starred repositories.
WITH repo_name IN
(
SELECT repo_name
FROM github_events
WHERE (event_type = 'WatchEvent') AND (actor_login IN ('alexey-milovidov'))
) AS is_my_repo
SELECT
actor_login,
sum(is_my_repo) AS stars_my,
sum(NOT is_my_repo) AS stars_other,
round(stars_my / (497 + stars_other), 3) AS ratio
FROM github_events
WHERE event_type = 'WatchEvent'
GROUP BY actor_login
ORDER BY ratio DESC
LIMIT 50
┌─actor_login──────┬─stars_my─┬─stars_other─┬─ratio─┐
1. │ alexey-milovidov │ 497 │ 0 │ 1 │
2. │ andrejs-ps │ 50 │ 11 │ 0.098 │
3. │ xiaoyuhou1 │ 35 │ 20 │ 0.068 │
4. │ pranay01 │ 25 │ 54 │ 0.045 │
5. │ ucasfl │ 32 │ 267 │ 0.042 │
6. │ UnamedRus │ 29 │ 219 │ 0.041 │
7. │ filimonov │ 27 │ 175 │ 0.04 │
8. │ gabime │ 27 │ 183 │ 0.04 │
9. │ nikitamikhaylov │ 20 │ 90 │ 0.034 │
10. │ ANTOSzbk │ 17 │ 10 │ 0.034 │
11. │ devcrafter │ 25 │ 265 │ 0.033 │
12. │ ramazanpolat │ 28 │ 379 │ 0.032 │
13. │ azuyoh │ 22 │ 200 │ 0.032 │
14. │ exitNA │ 32 │ 499 │ 0.032 │
15. │ dqdang │ 18 │ 79 │ 0.031 │
16. │ jions7ihj │ 18 │ 103 │ 0.03 │
17. │ JustFriendlyAlex │ 20 │ 163 │ 0.03 │
18. │ amosbird │ 41 │ 879 │ 0.03 │
19. │ bpf120 │ 15 │ 6 │ 0.03 │
20. │ glebmezh │ 16 │ 61 │ 0.029 │
21. │ sihaixianyu │ 30 │ 534 │ 0.029 │
22. │ hagen1778 │ 27 │ 439 │ 0.029 │
23. │ pavelfeldman │ 15 │ 20 │ 0.029 │
24. │ chhetripradeep │ 30 │ 520 │ 0.029 │
25. │ jeremiahar │ 34 │ 712 │ 0.028 │
26. │ canhld94 │ 16 │ 88 │ 0.027 │
27. │ leicsss │ 14 │ 14 │ 0.027 │
28. │ Venage5603 │ 24 │ 444 │ 0.026 │
29. │ muardle │ 16 │ 112 │ 0.026 │
30. │ jackpgao │ 20 │ 280 │ 0.026 │
31. │ LingangJiang │ 15 │ 84 │ 0.026 │
32. │ jibsen │ 23 │ 393 │ 0.026 │
33. │ kevintop │ 63 │ 1931 │ 0.026 │
34. │ Machtergreifung │ 30 │ 720 │ 0.025 │
35. │ kshvakov │ 44 │ 1290 │ 0.025 │
36. │ qk8 │ 39 │ 1091 │ 0.025 │
37. │ VadimLevin │ 15 │ 110 │ 0.025 │
38. │ gkuznets │ 31 │ 779 │ 0.024 │
39. │ markpapadakis │ 23 │ 447 │ 0.024 │
40. │ the-die │ 20 │ 341 │ 0.024 │
41. │ orginux │ 22 │ 437 │ 0.024 │
42. │ mkabilov │ 17 │ 197 │ 0.024 │
43. │ artpaul │ 15 │ 134 │ 0.024 │
44. │ embg │ 19 │ 282 │ 0.024 │
45. │ CurtizJ │ 13 │ 34 │ 0.024 │
46. │ isublimity │ 14 │ 96 │ 0.024 │
47. │ Snsjshjsn │ 12 │ 0 │ 0.024 │
48. │ BigRedEye │ 35 │ 1022 │ 0.023 │
49. │ vstakhov │ 16 │ 194 │ 0.023 │
50. │ PasinduDew │ 14 │ 104 │ 0.023 │
└──────────────────┴──────────┴─────────────┴───────┘
50 rows in set. Elapsed: 11.209 sec. Processed 557.97 million rows, 11.91 GB (49.78 million rows/s., 1.06 GB/s.)
Here are my friends... and I don't know what to do with this list.
Authors that contributed to ClickHouse also contributed to what repositories?
SELECT
repo_name,
count() AS prs,
uniq(actor_login) AS authors
FROM github_events
WHERE (event_type = 'PullRequestEvent') AND (action = 'opened') AND (actor_login IN
(
SELECT actor_login
FROM github_events
WHERE (event_type = 'PullRequestEvent') AND (action = 'opened') AND (repo_name IN ('yandex/ClickHouse', 'ClickHouse/ClickHouse'))
)) AND (repo_name NOT ILIKE '%ClickHouse%')
GROUP BY repo_name
ORDER BY authors DESC
LIMIT 50
┌─repo_name───────────────────────────────────────┬───prs─┬─authors─┐
│ Homebrew/homebrew-core │ 181 │ 47 │
│ StarRocks/starrocks │ 2090 │ 47 │
│ apache/spark │ 553 │ 45 │
│ grafana/grafana │ 1126 │ 44 │
│ pingcap/tidb │ 1438 │ 38 │
│ apache/incubator-doris │ 737 │ 35 │
│ apache/doris │ 7986 │ 34 │
│ apache/flink │ 459 │ 34 │
│ kubernetes/kubernetes │ 112 │ 33 │
│ apache/arrow │ 1529 │ 33 │
│ apache/airflow │ 1320 │ 31 │
│ ydb-platform/ydb │ 2299 │ 29 │
│ facebook/rocksdb │ 93 │ 28 │
│ llvm/llvm-project │ 308 │ 28 │
│ duckdb/duckdb │ 111 │ 27 │
│ NixOS/nixpkgs │ 12026 │ 27 │
│ rust-lang/rust │ 154 │ 27 │
│ kubernetes/website │ 40 │ 26 │
│ open-telemetry/opentelemetry-collector-contrib │ 13728 │ 24 │
│ golang/go │ 50 │ 24 │
│ ByConity/ByConity │ 822 │ 24 │
│ avelino/awesome-go │ 57 │ 24 │
│ python/cpython │ 160 │ 24 │
│ apache/arrow-rs │ 355 │ 23 │
│ getsentry/sentry │ 1592 │ 22 │
│ ansible/ansible │ 98 │ 22 │
│ elastic/elasticsearch │ 292 │ 21 │
│ rms-support-letter/rms-support-letter.github.io │ 23 │ 21 │
│ decentralized-hse/practice │ 36 │ 21 │
│ helm/charts │ 582 │ 20 │
│ pocoproject/poco │ 44 │ 20 │
│ trinodb/trino │ 396 │ 19 │
│ apache/arrow-datafusion │ 596 │ 19 │
│ cockroachdb/cockroach │ 156 │ 19 │
│ prometheus/prometheus │ 1299 │ 19 │
│ facebookincubator/velox │ 178 │ 19 │
│ pytorch/pytorch │ 106 │ 18 │
│ apache/hadoop │ 168 │ 18 │
│ saltstack/salt │ 514 │ 18 │
│ grafana/loki │ 455 │ 18 │
│ Homebrew/homebrew-cask │ 194 │ 17 │
│ tikv/tikv │ 142 │ 17 │
│ getredash/redash │ 198 │ 17 │
│ ContentSquare/chproxy │ 116 │ 17 │
│ langchain-ai/langchain │ 148 │ 17 │
│ metabase/metabase │ 858 │ 17 │
│ vectordotdev/vector │ 3316 │ 16 │
│ oap-project/gluten │ 917 │ 16 │
│ grpc/grpc │ 304 │ 16 │
│ datafuselabs/databend │ 2423 │ 16 │
└─────────────────────────────────────────────────┴───────┴─────────┘
50 rows in set. Elapsed: 1.405 sec. Processed 814.98 million rows, 6.07 GB (580.08 million rows/s., 4.32 GB/s.)
Authors that filed an issue in ClickHouse also filed issues in what repositories?
SELECT
repo_name,
count() AS prs,
uniq(actor_login) AS authors
FROM github_events
WHERE (event_type = 'IssuesEvent') AND (action = 'opened') AND (actor_login IN
(
SELECT actor_login
FROM github_events
WHERE (event_type = 'IssuesEvent') AND (action = 'opened') AND (repo_name IN ('yandex/ClickHouse', 'ClickHouse/ClickHouse'))
)) AND (repo_name NOT ILIKE '%ClickHouse%')
GROUP BY repo_name
ORDER BY authors DESC
LIMIT 50
┌─repo_name───────────────────────┬──prs─┬─authors─┐
│ grafana/grafana │ 530 │ 191 │
│ golang/go │ 493 │ 173 │
│ StarRocks/starrocks │ 689 │ 115 │
│ kubernetes/kubernetes │ 273 │ 104 │
│ ansible/ansible │ 244 │ 101 │
│ microsoft/vscode │ 166 │ 100 │
│ elastic/elasticsearch │ 259 │ 100 │
│ duckdb/duckdb │ 487 │ 90 │
│ dbeaver/dbeaver │ 171 │ 89 │
│ pingcap/tidb │ 1202 │ 83 │
│ vectordotdev/vector │ 193 │ 77 │
│ apache/superset │ 146 │ 77 │
│ minio/minio │ 120 │ 75 │
│ apache/arrow │ 177 │ 73 │
│ apache/doris │ 334 │ 71 │
│ rust-lang/rust │ 361 │ 71 │
│ metabase/metabase │ 189 │ 69 │
│ prometheus/prometheus │ 131 │ 65 │
│ apache/airflow │ 126 │ 63 │
│ VictoriaMetrics/VictoriaMetrics │ 578 │ 59 │
│ pola-rs/polars │ 333 │ 58 │
│ apache/incubator-superset │ 168 │ 56 │
│ llvm/llvm-project │ 191 │ 55 │
│ Microsoft/vscode │ 132 │ 54 │
│ nodejs/node │ 112 │ 54 │
│ trinodb/trino │ 165 │ 53 │
│ elastic/kibana │ 406 │ 52 │
│ telegramdesktop/tdesktop │ 152 │ 52 │
│ bitnami/charts │ 83 │ 52 │
│ docker/docker │ 121 │ 52 │
│ tensorflow/tensorflow │ 92 │ 52 │
│ influxdata/influxdb │ 489 │ 51 │
│ airbytehq/airbyte │ 124 │ 51 │
│ prestodb/presto │ 140 │ 50 │
│ apache/dolphinscheduler │ 363 │ 49 │
│ NixOS/nixpkgs │ 296 │ 49 │
│ apache/incubator-doris │ 290 │ 48 │
│ pandas-dev/pandas │ 114 │ 48 │
│ systemd/systemd │ 94 │ 46 │
│ cockroachdb/cockroach │ 689 │ 46 │
│ timescale/timescaledb │ 121 │ 45 │
│ alibaba/nacos │ 67 │ 45 │
│ grpc/grpc │ 70 │ 45 │
│ grafana/loki │ 99 │ 44 │
│ alibaba/druid │ 86 │ 44 │
│ fluent/fluent-bit │ 92 │ 44 │
│ kubernetes/ingress-nginx │ 64 │ 44 │
│ travis-ci/travis-ci │ 82 │ 42 │
│ rancher/rancher │ 119 │ 42 │
│ getsentry/sentry │ 266 │ 42 │
└─────────────────────────────────┴──────┴─────────┘
50 rows in set. Elapsed: 0.483 sec. Processed 284.13 million rows, 1.95 GB (588.23 million rows/s., 4.04 GB/s.)
SELECT
repo_name,
toDate(created_at) AS day,
count() AS stars
FROM github_events
WHERE event_type = 'WatchEvent'
GROUP BY
repo_name,
day
ORDER BY count() DESC
LIMIT 1 BY repo_name
LIMIT 50
┌─repo_name───────────────────────────┬────────day─┬─stars─┐
1. │ 996icu/996.ICU │ 2019-03-28 │ 70976 │
2. │ ant-design/ant-design │ 2021-05-16 │ 54716 │
3. │ Rabbitminers/Extended-Bogeys │ 2024-09-29 │ 27891 │
4. │ x64dbg/x64dbg │ 2018-01-06 │ 26459 │
5. │ UsergeTeam/Userge │ 2022-01-02 │ 22625 │
6. │ UsergeTeam/Userge-Assistant │ 2022-01-02 │ 22594 │
7. │ twitter/the-algorithm │ 2023-04-01 │ 20596 │
8. │ xai-org/grok-1 │ 2024-03-18 │ 20150 │
9. │ M4cs/BabySploit │ 2019-09-08 │ 20116 │
10. │ microsoft/Terminal │ 2019-05-07 │ 14402 │
11. │ Rohfosho/CosmosBrowserBackend │ 2014-10-13 │ 13345 │
12. │ Torantulino/Auto-GPT │ 2023-04-14 │ 12617 │
13. │ deepseek-ai/DeepSeek-V3 │ 2025-01-28 │ 12281 │
14. │ arddaxd/Arddaxd │ 2022-06-19 │ 11208 │
15. │ deepseek-ai/DeepSeek-R1 │ 2025-01-28 │ 10603 │
16. │ apple/swift │ 2015-12-04 │ 10567 │
17. │ opentffoundation/manifesto │ 2023-08-29 │ 10369 │
18. │ google-gemini/gemini-cli │ 2025-06-26 │ 10098 │
19. │ desktop/desktop │ 2020-11-16 │ 10044 │
20. │ mannaandpoem/OpenManus │ 2025-03-07 │ 9986 │
21. │ flightlessmango/MangoHud │ 2021-07-16 │ 9790 │
22. │ Genesis-Embodied-AI/Genesis │ 2024-12-19 │ 9252 │
23. │ facebookresearch/segment-anything │ 2023-04-06 │ 9070 │
24. │ base-org/chains │ 2023-03-13 │ 8974 │
25. │ Significant-Gravitas/Auto-GPT │ 2023-04-17 │ 8525 │
26. │ instructkr/claw-code │ 2026-04-01 │ 8404 │
27. │ openbilibili/go-common │ 2019-04-22 │ 8173 │
28. │ TheBreakery/Bloomware │ 2022-02-07 │ 7667 │
29. │ WeNeedHome/SummaryOfLoanSuspension │ 2022-07-14 │ 7650 │
30. │ unionlabs/union │ 2025-01-17 │ 7573 │
31. │ base-org/node │ 2023-03-13 │ 7432 │
32. │ clawdbot/clawdbot │ 2026-01-26 │ 7403 │
33. │ inkonchain/node │ 2025-01-07 │ 7346 │
34. │ inkonchain/ink-kit │ 2025-01-07 │ 7338 │
35. │ inkonchain/docs │ 2025-01-07 │ 7328 │
36. │ openai/codex │ 2025-04-17 │ 7092 │
37. │ deepseek-ai/FlashMLA │ 2025-02-24 │ 7016 │
38. │ httpie/httpie │ 2022-04-15 │ 6740 │
39. │ mrpond/BlockTheSpot │ 2021-07-20 │ 6606 │
40. │ amattson21/gitjob │ 2016-10-08 │ 6511 │
41. │ mr-mig/every-programmer-should-know │ 2017-09-05 │ 6431 │
42. │ feet/audient │ 2022-10-30 │ 6366 │
43. │ skylot/jadx │ 2018-01-06 │ 6319 │
44. │ nomic-ai/gpt4all │ 2023-03-29 │ 6275 │
45. │ phase/sure │ 2015-06-16 │ 6039 │
46. │ Konloch/bytecode-viewer │ 2018-01-06 │ 6000 │
47. │ jlevy/the-art-of-command-line │ 2015-06-16 │ 5895 │
48. │ yarnpkg/yarn │ 2016-10-12 │ 5820 │
49. │ FuelLabs/sway │ 2023-07-04 │ 5813 │
50. │ microsoft/typescript-go │ 2025-03-12 │ 5720 │
└─────────────────────────────────────┴────────────┴───────┘
50 rows in set. Elapsed: 14.682 sec. Processed 554.52 million rows, 5.11 GB (37.77 million rows/s., 347.78 MB/s.)
This report reads like a timeline of the last few years: the day Twitter published its recommendation algorithm, the day Grok-1 was released, two separate days of Auto-GPT, and DeepSeek V3 and R1 arriving within the same day. There is still "openbilibili" — something like Chinese YouTube, closed due to a DMCA takedown.
I accidentially made a similar query for repositories the gained the most stars over just one second:
:) SELECT repo_name, created_at, count() AS stars FROM github_events WHERE event_type = 'WatchEvent' GROUP BY repo_name, created_at ORDER BY count() DESC LIMIT 50
┌─repo_name───────────────────────────────────┬──────────created_at─┬─stars─┐
1. │ expressjs/express │ 2018-04-29 20:30:59 │ 370 │
2. │ expressjs/express │ 2018-04-29 20:31:00 │ 195 │
3. │ m1xxxn/Ap3x-Lunacy-T00l │ 2023-05-15 19:44:13 │ 159 │
4. │ usmanghani-10/DBD-CHT-2024 │ 2024-07-18 23:01:31 │ 150 │
5. │ munenepaul/C4D-2024 │ 2024-07-18 23:02:37 │ 145 │
6. │ ATLAS9581/Atlas-Crypt │ 2023-05-16 11:32:09 │ 138 │
7. │ siddhardhap90/banana-clicker │ 2024-07-18 22:58:59 │ 137 │
8. │ Italo889/Escape-From-Tarkov-hack-ESP-aimbot │ 2024-07-18 22:28:37 │ 136 │
9. │ danieldaeschle/swapy │ 2018-04-29 16:10:44 │ 132 │
10. │ AtlasWaaare/CSGO- │ 2023-06-04 21:42:41 │ 130 │
11. │ i0nnxna/Ap3x-Lunacy-T00l │ 2023-05-12 17:43:48 │ 129 │
12. │ i0nnxna/F0RTNITE-LUNACY-PROJECT │ 2023-05-12 17:44:50 │ 128 │
13. │ AmaruSegovia/Catizen-Crypto-Bot │ 2024-07-19 17:57:48 │ 128 │
14. │ Retnorr/FlpBlxxx-Predictor │ 2023-05-10 16:32:17 │ 127 │
15. │ MaikelGamerWT/MediaHuman-YouTube-Downloader │ 2024-07-22 23:31:47 │ 125 │
16. │ ATLAS9581/Atlas-Spoof-for-Valorant │ 2023-05-16 11:32:22 │ 123 │
17. │ kaif001/VMWare-Workstation-Pro-CRACKED │ 2024-03-10 20:38:14 │ 121 │
18. │ LOLIKPOP/TS4 │ 2024-07-20 18:55:39 │ 121 │
19. │ JonathaCesar/GTA-5-Mod-Menu │ 2024-03-10 20:33:54 │ 120 │
20. │ Stelarrix/Overwatch-2-AltiumWare-PROJECT │ 2023-05-18 14:15:38 │ 120 │
21. │ PolarisSzn/GENERATIVE-AI-PH0T0SHOP │ 2024-07-20 17:45:19 │ 119 │
22. │ m1xxxn/F0RTNITE-LUNACY-PROJECT │ 2023-05-15 19:44:20 │ 119 │
23. │ ArthurRSaldanha/Wallet-Crypto-Stea1er │ 2024-07-19 17:47:02 │ 119 │
24. │ flutter/flutter │ 2018-04-29 15:13:58 │ 118 │
25. │ Allexin0/FlpBlx-Predictor │ 2023-05-05 17:05:35 │ 118 │
26. │ kaif001/Sketchup-Pro-Max │ 2024-03-10 20:38:50 │ 117 │
27. │ munenepaul/CL-DRAW-2024 │ 2024-07-18 23:02:41 │ 116 │
28. │ vhzzz/Cryptowallet-Bruteforce-Tool │ 2024-07-18 16:54:11 │ 116 │
29. │ michaelronin220/AT │ 2024-07-24 11:44:10 │ 115 │
30. │ ricardonts/DriverBooster │ 2024-07-19 19:35:48 │ 115 │
31. │ LOLIKPOP/CUPH │ 2024-07-20 18:54:30 │ 115 │
32. │ SoPCasTl3/LUNACY-SP00FER │ 2023-05-14 16:01:16 │ 115 │
33. │ Euriic/CyberGhostVPN-Cracked │ 2023-05-14 22:34:34 │ 115 │
34. │ i0nnxna/Lunacy-CSGO- │ 2023-05-12 17:45:52 │ 114 │
35. │ GraceTulofs/Crossout-Ultimate-Hack-2024 │ 2024-07-21 13:05:31 │ 114 │
36. │ Yuriann0/LagMonstar-Apex │ 2023-05-11 10:33:47 │ 113 │
37. │ Richardio0/FlpBlx1-Predictor │ 2023-05-14 22:20:40 │ 113 │
38. │ JonathaCesar/AVAST-PREMIUM-SECURITY-CRACKED │ 2024-03-10 20:35:38 │ 113 │
39. │ SaidiAgibu/RUST-DISARRAY-FREE │ 2024-07-20 18:54:38 │ 113 │
40. │ kendreambelieertU/RBLX-HA1K │ 2024-07-21 16:37:39 │ 113 │
41. │ 7Sence/Clover-Launcher │ 2024-12-25 17:31:55 │ 113 │
42. │ lolivmolly/BAB │ 2024-07-20 14:33:07 │ 112 │
43. │ PolarisSzn/voicemod-free │ 2024-07-20 18:05:23 │ 112 │
44. │ ATLAAAAAS/OmWare │ 2023-05-16 17:44:54 │ 112 │
45. │ Gamax256/Adobe-Illustrator-2024-Crack- │ 2024-07-22 23:26:17 │ 112 │
46. │ i0nnxna/LUNACY-FIV3M │ 2023-05-12 17:46:23 │ 111 │
47. │ GagWinsent/SigmaFirSkin-Apex │ 2023-05-05 09:44:38 │ 111 │
48. │ usmanghani-10/GTA-5-CHT-2024 │ 2024-07-18 22:55:49 │ 110 │
49. │ ATLAS9581/ATLASware-Undetected │ 2023-05-16 11:32:32 │ 110 │
50. │ Jesus24-Dev/DaVinci-Resolve-Studio-19 │ 2024-07-18 17:32:56 │ 110 │
└─────────────────────────────────────────────┴─────────────────────┴───────┘
50 rows in set. Elapsed: 37.951 sec. Processed 524.88 million rows, 5.06 GB (13.83 million rows/s., 133.36 MB/s.)
Another JavaScript framework got 370 stars within a single second on a peaceful April evening? It's complete madness. Most of the rest of the list is cheats, cracks and "tools" — the kind of repository that comes with a bought audience.
WITH toYear(created_at) AS year
SELECT
repo_name,
sum(year = 2024) AS stars2024,
sum(year = 2023) AS stars2023,
round(stars2024 / stars2023, 3) AS yoy,
min(created_at) AS first_seen
FROM github_events
WHERE event_type = 'WatchEvent'
GROUP BY repo_name
HAVING (min(created_at) <= '2023-01-01 00:00:00') AND (stars2023 >= 1000)
ORDER BY yoy DESC
LIMIT 50
┌─repo_name───────────────────────────────────┬─stars2024─┬─stars2023─┬────yoy─┬──────────first_seen─┐
1. │ charlax/professional-programming │ 21728 │ 1978 │ 10.985 │ 2016-05-07 03:25:06 │
2. │ ValdikSS/GoodbyeDPI │ 16824 │ 1739 │ 9.675 │ 2017-05-16 19:34:37 │
3. │ gleam-lang/gleam │ 12661 │ 1325 │ 9.555 │ 2019-10-28 02:23:59 │
4. │ 4ian/GDevelop │ 8692 │ 1182 │ 7.354 │ 2018-09-10 19:49:49 │
5. │ xiaolai/everyone-can-use-english │ 12319 │ 1859 │ 6.627 │ 2019-03-15 16:48:50 │
6. │ nocobase/nocobase │ 7890 │ 1293 │ 6.102 │ 2021-03-29 06:12:15 │
7. │ Avaiga/taipy │ 14719 │ 2417 │ 6.09 │ 2022-04-07 14:06:36 │
8. │ THUDM/CogVideo │ 6528 │ 1110 │ 5.881 │ 2022-05-29 14:28:27 │
9. │ miss-mumu/developer2gwy │ 6838 │ 1201 │ 5.694 │ 2021-09-08 02:41:33 │
10. │ atherosai/ui │ 9963 │ 1765 │ 5.645 │ 2022-12-15 13:28:50 │
11. │ coollabsio/coolify │ 25478 │ 4691 │ 5.431 │ 2021-01-26 21:41:12 │
12. │ shidahuilang/shuyuan │ 5311 │ 1001 │ 5.306 │ 2022-04-17 13:14:14 │
13. │ maboloshi/github-chinese │ 5680 │ 1092 │ 5.201 │ 2021-11-18 12:48:55 │
14. │ 0xJacky/nginx-ui │ 4983 │ 1029 │ 4.843 │ 2021-05-19 00:57:56 │
15. │ kestra-io/kestra │ 9723 │ 2075 │ 4.686 │ 2019-12-27 12:51:04 │
16. │ alexta69/metube │ 4722 │ 1024 │ 4.611 │ 2020-08-06 01:35:09 │
17. │ Chalarangelo/30-seconds-of-code │ 6569 │ 1469 │ 4.472 │ 2017-12-11 16:51:38 │
18. │ dream-num/univer │ 6096 │ 1392 │ 4.379 │ 2022-09-29 12:17:25 │
19. │ KRTirtho/spotube │ 26045 │ 6070 │ 4.291 │ 2021-03-18 07:16:37 │
20. │ alyssaxuu/screenity │ 4135 │ 1056 │ 3.916 │ 2020-11-19 10:51:09 │
21. │ krishnadey30/LeetCode-Questions-CompanyWise │ 8632 │ 2211 │ 3.904 │ 2020-08-11 16:57:43 │
22. │ quickemu-project/quickemu │ 4318 │ 1113 │ 3.88 │ 2021-11-19 11:03:53 │
23. │ YueChan/Live │ 4600 │ 1221 │ 3.767 │ 2022-11-18 07:34:55 │
24. │ plankanban/planka │ 4557 │ 1213 │ 3.757 │ 2020-04-26 06:50:04 │
25. │ goauthentik/authentik │ 8720 │ 2404 │ 3.627 │ 2021-04-21 21:10:50 │
26. │ MetaCubeX/ClashMetaForAndroid │ 13382 │ 3723 │ 3.594 │ 2022-06-07 06:50:38 │
27. │ doocs/source-code-hunter │ 8865 │ 2483 │ 3.57 │ 2019-10-28 02:00:51 │
28. │ searxng/searxng │ 8329 │ 2374 │ 3.508 │ 2021-04-22 16:43:27 │
29. │ srush/GPU-Puzzles │ 5636 │ 1633 │ 3.451 │ 2022-07-11 21:01:33 │
30. │ hxu296/leetcode-company-wise-problems-2022 │ 5187 │ 1546 │ 3.355 │ 2022-05-10 16:07:23 │
31. │ johannesjo/super-productivity │ 4905 │ 1492 │ 3.288 │ 2017-01-16 10:09:21 │
32. │ HariSekhon/DevOps-Bash-tools │ 3836 │ 1178 │ 3.256 │ 2019-08-11 09:22:50 │
33. │ mfts/papermark │ 4279 │ 1339 │ 3.196 │ 2022-10-03 22:18:08 │
34. │ mehdihadeli/awesome-software-architecture │ 5488 │ 1765 │ 3.109 │ 2021-02-24 13:54:31 │
35. │ nginx/nginx │ 6206 │ 2050 │ 3.027 │ 2012-02-11 05:52:50 │
36. │ pystardust/ani-cli │ 3330 │ 1116 │ 2.984 │ 2021-06-09 12:44:26 │
37. │ blacklanternsecurity/bbot │ 4075 │ 1372 │ 2.97 │ 2022-08-19 17:06:18 │
38. │ dokku/dokku │ 4438 │ 1502 │ 2.955 │ 2015-12-04 20:05:41 │
39. │ 3b1b/videos │ 3370 │ 1151 │ 2.928 │ 2020-12-31 21:09:03 │
40. │ lich0821/WeChatFerry │ 3219 │ 1119 │ 2.877 │ 2022-09-25 14:30:06 │
41. │ Floorp-Projects/Floorp │ 4506 │ 1591 │ 2.832 │ 2022-05-12 08:43:55 │
42. │ soxoj/maigret │ 4312 │ 1526 │ 2.826 │ 2020-08-30 08:06:59 │
43. │ Permify/permify │ 2796 │ 1023 │ 2.733 │ 2022-07-14 14:44:01 │
44. │ free-educa/books │ 5246 │ 1946 │ 2.696 │ 2021-01-19 17:17:42 │
45. │ ChrisTitusTech/winutil │ 16736 │ 6323 │ 2.647 │ 2022-05-10 15:27:38 │
46. │ gojue/ecapture │ 6083 │ 2310 │ 2.633 │ 2022-11-29 03:52:00 │
47. │ martinvonz/jj │ 5389 │ 2052 │ 2.626 │ 2020-12-18 05:10:27 │
48. │ catdad/canvas-confetti │ 4208 │ 1620 │ 2.598 │ 2018-04-25 01:33:19 │
49. │ PawanOsman/ChatGPT │ 4271 │ 1652 │ 2.585 │ 2022-12-06 18:46:10 │
50. │ quickwit-oss/quickwit │ 3877 │ 1504 │ 2.578 │ 2022-01-27 08:06:46 │
└─────────────────────────────────────────────┴───────────┴───────────┴────────┴─────────────────────┘
50 rows in set. Elapsed: 3.988 sec. Processed 554.52 million rows, 5.61 GB (139.06 million rows/s., 1.41 GB/s.)
I compare 2024 with 2023 instead of the two most recent years on purpose: star events in 2025 and 2026 are heavily undercounted (see the note in the abstract), so any ratio computed over them would measure the gap in the data rather than the growth of a repository. GoodbyeDPI and Gleam grew almost tenfold in 2024.
WITH toYear(created_at) AS year
SELECT
repo_name,
sum(year = 2024) AS stars2024,
sum(year = 2023) AS stars2023,
round(stars2024 / stars2023, 3) AS yoy,
min(created_at) AS first_seen
FROM github_events
WHERE event_type = 'WatchEvent'
GROUP BY repo_name
HAVING (min(created_at) <= '2023-01-01 00:00:00') AND (max(created_at) >= '2024-06-01 00:00:00') AND (stars2023 >= 1000)
ORDER BY yoy ASC
LIMIT 50
┌─repo_name─────────────────────────────────────────┬─stars2024─┬─stars2023─┬───yoy─┬──────────first_seen─┐
1. │ aplus-framework/email │ 3 │ 8984 │ 0 │ 2021-08-05 16:39:18 │
2. │ aplus-framework/http-client │ 2 │ 8959 │ 0 │ 2021-08-05 16:40:05 │
3. │ aplus-framework/validation │ 3 │ 8894 │ 0 │ 2021-08-05 16:38:07 │
4. │ aplus-framework/aplus │ 4 │ 8877 │ 0 │ 2021-09-06 01:53:01 │
5. │ aplus-framework/one │ 1 │ 8905 │ 0 │ 2021-09-06 01:53:04 │
6. │ benphelps/homepage │ 0 │ 5078 │ 0 │ 2022-08-25 04:21:08 │
7. │ ellie/atuin │ 0 │ 5865 │ 0 │ 2021-02-14 00:44:39 │
8. │ aplus-framework/database │ 2 │ 16062 │ 0 │ 2021-08-05 16:38:01 │
9. │ aplus-framework/pagination │ 2 │ 9053 │ 0 │ 2021-08-05 16:38:35 │
10. │ aplus-framework/language │ 2 │ 8987 │ 0 │ 2021-08-05 16:38:41 │
11. │ aplus-framework/http │ 2 │ 9019 │ 0 │ 2021-07-13 12:33:20 │
12. │ aplus-framework/routing │ 4 │ 9047 │ 0 │ 2021-08-05 16:38:31 │
13. │ aplus-framework/image │ 2 │ 8811 │ 0 │ 2021-08-05 16:38:04 │
14. │ aplus-framework/mvc │ 4 │ 9105 │ 0 │ 2021-08-05 16:37:52 │
15. │ aplus-framework/framework │ 5 │ 8963 │ 0.001 │ 2021-09-01 00:07:05 │
16. │ aplus-framework/app │ 15 │ 27148 │ 0.001 │ 2021-08-05 16:37:57 │
17. │ aplus-framework/cli │ 5 │ 8829 │ 0.001 │ 2021-08-05 16:37:42 │
18. │ aplus-framework/session │ 5 │ 9057 │ 0.001 │ 2021-08-05 16:38:10 │
19. │ xiaoming2028/FreePAC │ 3 │ 1850 │ 0.002 │ 2020-01-12 18:03:37 │
20. │ inovua/reactdatagrid │ 44 │ 3363 │ 0.013 │ 2020-10-09 13:12:46 │
21. │ Joystream/joystream │ 24 │ 1537 │ 0.016 │ 2019-04-02 08:30:58 │
22. │ YeeZTech/YeeZ-Privacy-Computing │ 28 │ 1335 │ 0.021 │ 2021-03-23 02:45:01 │
23. │ innnky/so-vits-svc │ 84 │ 3175 │ 0.026 │ 2022-09-19 01:40:11 │
24. │ harryzhangOG/Deep-RL-Notes │ 29 │ 1012 │ 0.029 │ 2020-01-14 07:39:19 │
25. │ casey/ord │ 52 │ 1716 │ 0.03 │ 2022-01-21 19:38:13 │
26. │ hiifeng/V2ray-for-Doprax │ 327 │ 10197 │ 0.032 │ 2022-12-29 03:56:23 │
27. │ gluon-framework/gluon │ 95 │ 2882 │ 0.033 │ 2022-12-13 22:41:13 │
28. │ hwchase17/notion-qa │ 71 │ 1938 │ 0.037 │ 2022-11-14 18:30:26 │
29. │ transitive-bullshit/chatgpt-api │ 428 │ 10972 │ 0.039 │ 2022-12-03 00:23:29 │
30. │ cherish-chat/xxim-server │ 131 │ 3218 │ 0.041 │ 2022-11-17 15:46:26 │
31. │ danielgross/whatsapp-gpt │ 84 │ 1994 │ 0.042 │ 2022-12-02 17:56:13 │
32. │ google/cdc-file-transfer │ 115 │ 2677 │ 0.043 │ 2022-11-03 10:06:24 │
33. │ dominant-strategies/go-quai │ 98 │ 2156 │ 0.045 │ 2022-10-10 21:38:38 │
34. │ gencay/vscode-chatgpt │ 149 │ 3320 │ 0.045 │ 2022-12-06 22:56:34 │
35. │ rawandahmad698/PyChatGPT │ 93 │ 2027 │ 0.046 │ 2022-12-05 09:45:32 │
36. │ iway1/react-ts-form │ 91 │ 1943 │ 0.047 │ 2022-12-27 03:41:32 │
37. │ petykowski/London-Underground-Dot-Matrix-Typeface │ 66 │ 1412 │ 0.047 │ 2020-01-04 16:47:48 │
38. │ Fictiverse/Redream │ 78 │ 1659 │ 0.047 │ 2022-12-13 10:50:08 │
39. │ nolanaatama/sd-1click-colab │ 52 │ 1033 │ 0.05 │ 2022-12-24 22:09:04 │
40. │ Atri-Labs/atrilabs-engine │ 120 │ 2333 │ 0.051 │ 2022-07-23 15:11:27 │
41. │ twitter/the-algorithm │ 3334 │ 60262 │ 0.055 │ 2022-04-25 20:54:10 │
42. │ liady/ChatGPT-pdf │ 56 │ 1008 │ 0.056 │ 2022-12-04 19:55:30 │
43. │ skyline-emu/skyline │ 140 │ 2512 │ 0.056 │ 2019-09-05 01:28:45 │
44. │ massalabs/massa │ 254 │ 4495 │ 0.057 │ 2021-11-19 03:11:49 │
45. │ prophesier/diff-svc │ 166 │ 2847 │ 0.058 │ 2022-10-24 03:22:58 │
46. │ microsoft/BioGPT │ 230 │ 3958 │ 0.058 │ 2022-08-15 09:25:50 │
47. │ liu673cn/box │ 526 │ 9045 │ 0.058 │ 2022-07-10 09:30:32 │
48. │ THUDM/GLM-130B │ 379 │ 6374 │ 0.059 │ 2022-08-04 11:23:25 │
49. │ EmpireMediaScience/A1111-Web-UI-Installer │ 98 │ 1625 │ 0.06 │ 2022-10-29 22:24:18 │
50. │ eryajf/chatgpt-dingtalk │ 69 │ 1148 │ 0.06 │ 2022-12-09 01:36:01 │
└───────────────────────────────────────────────────┴───────────┴───────────┴───────┴─────────────────────┘
50 rows in set. Elapsed: 5.790 sec. Processed 554.52 million rows, 5.61 GB (95.78 million rows/s., 969.33 MB/s.)
Almost the whole list belongs to a single organization that collected thousands of stars in 2023 and then exactly nothing — a strong hint that those stars were never organic in the first place.
SELECT
repo_name,
max(stars) AS daily_stars,
sum(stars) AS total_stars,
total_stars / daily_stars AS rate
FROM
(
SELECT
repo_name,
toDate(created_at) AS day,
count() AS stars
FROM github_events
WHERE event_type = 'WatchEvent'
GROUP BY
repo_name,
day
)
GROUP BY repo_name
ORDER BY rate DESC
LIMIT 50
┌─repo_name─────────────────────────┬─daily_stars─┬─total_stars─┬────rate─┐
1. │ BottleSome/AutoApiSecret │ 58 │ 69610 │ 1200.17 │
2. │ tpope/vim-fugitive │ 20 │ 23302 │ 1165.1 │
3. │ fxsjy/jieba │ 34 │ 38016 │ 1118.12 │
4. │ coreybutler/nvm-windows │ 44 │ 45922 │ 1043.68 │
5. │ senchalabs/connect │ 11 │ 10888 │ 989.82 │
6. │ powerline/fonts │ 28 │ 27642 │ 987.21 │
7. │ plataformatec/devise │ 22 │ 21197 │ 963.5 │
8. │ janl/mustache.js │ 19 │ 18104 │ 952.84 │
9. │ zsh-users/zsh-syntax-highlighting │ 25 │ 23220 │ 928.8 │
10. │ g-truc/glm │ 12 │ 11138 │ 928.17 │
11. │ pxb1988/dex2jar │ 15 │ 13685 │ 912.33 │
12. │ JamesNK/Newtonsoft.Json │ 14 │ 12689 │ 906.36 │
13. │ tornadoweb/tornado │ 21 │ 18967 │ 903.19 │
14. │ FasterXML/jackson │ 12 │ 10691 │ 890.92 │
15. │ square/retrofit │ 56 │ 49605 │ 885.8 │
16. │ spring-projects/spring-framework │ 76 │ 66830 │ 879.34 │
17. │ ansible/ansible-examples │ 15 │ 13130 │ 875.33 │
18. │ nvie/gitflow │ 34 │ 29639 │ 871.74 │
19. │ expressjs/multer │ 15 │ 12949 │ 863.27 │
20. │ badges/shields │ 31 │ 26745 │ 862.74 │
21. │ tweepy/tweepy │ 14 │ 12004 │ 857.43 │
22. │ jenkinsci/jenkins │ 33 │ 27762 │ 841.27 │
23. │ winstonjs/winston │ 27 │ 22285 │ 825.37 │
24. │ mishoo/UglifyJS │ 15 │ 12362 │ 824.13 │
25. │ VundleVim/Vundle.vim │ 25 │ 20370 │ 814.8 │
26. │ rspec/rspec-rails │ 7 │ 5681 │ 811.57 │
27. │ wycats/handlebars.js │ 21 │ 16860 │ 802.86 │
28. │ expressjs/session │ 9 │ 7205 │ 800.56 │
29. │ square/okhttp │ 67 │ 53046 │ 791.73 │
30. │ mybatis/mybatis-3 │ 30 │ 23656 │ 788.53 │
31. │ scrooloose/nerdtree │ 18 │ 14068 │ 781.56 │
32. │ chocolatey/choco │ 16 │ 12492 │ 780.75 │
33. │ miguelgrinberg/flasky │ 13 │ 10135 │ 779.62 │
34. │ OpenVPN/openvpn │ 18 │ 14005 │ 778.06 │
35. │ isaacs/rimraf │ 8 │ 6216 │ 777 │
36. │ motdotla/dotenv │ 27 │ 20928 │ 775.11 │
37. │ oblador/react-native-vector-icons │ 25 │ 19328 │ 773.12 │
38. │ Maximus5/ConEmu │ 13 │ 10036 │ 772 │
39. │ moodle/moodle │ 9 │ 6919 │ 768.78 │
40. │ playframework/playframework │ 16 │ 12282 │ 767.62 │
41. │ amix/vimrc │ 47 │ 35976 │ 765.45 │
42. │ milesial/Pytorch-UNet │ 15 │ 11416 │ 761.07 │
43. │ shimat/opencvsharp │ 8 │ 6088 │ 761 │
44. │ capistrano/capistrano │ 19 │ 14453 │ 760.68 │
45. │ tymondesigns/jwt-auth │ 17 │ 12928 │ 760.47 │
46. │ creationix/nvm │ 49 │ 36658 │ 748.12 │
47. │ NLog/NLog │ 10 │ 7422 │ 742.2 │
48. │ Supervisor/supervisor │ 13 │ 9631 │ 740.85 │
49. │ andymccurdy/redis-py │ 14 │ 10356 │ 739.71 │
50. │ vanhauser-thc/thc-hydra │ 16 │ 11833 │ 739.56 │
└───────────────────────────────────┴─────────────┴─────────────┴─────────┘
50 rows in set. Elapsed: 34.278 sec. Processed 554.52 million rows, 5.11 GB (16.18 million rows/s., 148.96 MB/s.)
:) SELECT toDayOfWeek(created_at) AS day, count() AS stars, bar(stars, 0, 100000000, 10) AS bar FROM github_events WHERE event_type = 'WatchEvent' GROUP BY day ORDER BY day
┌─day─┬────stars─┬─bar───────┐
│ 1 │ 84651961 │ ████████▍ │
│ 2 │ 87848452 │ ████████▊ │
│ 3 │ 86719799 │ ████████▋ │
│ 4 │ 84926999 │ ████████▍ │
│ 5 │ 81446351 │ ████████▏ │
│ 6 │ 63927932 │ ██████▍ │
│ 7 │ 64880024 │ ██████▍ │
└─────┴──────────┴───────────┘
7 rows in set. Elapsed: 0.179 sec. Processed 554.52 million rows, 2.77 GB (3.10 billion rows/s., 15.52 GB/s.)
It is Tuesday. Definitely not the weekend. Maybe Wednesday or Thursday, but not Monday or Friday.
:) SELECT uniq(actor_login) FROM github_events
┌─uniq(actor_login)─┐
│ 102149912 │
└───────────────────┘
1 rows in set. Elapsed: 5.616 sec. Processed 11.08 billion rows, 37.16 GB (1.97 billion rows/s., 6.62 GB/s.)
102 million. Actually, these are users that are not only registered but also participated at least in... something.
Total number of users that gave at least one star:
:) SELECT uniq(actor_login) FROM github_events WHERE event_type = 'WatchEvent'
┌─uniq(actor_login)─┐
│ 28971830 │
└───────────────────┘
1 rows in set. Elapsed: 1.545 sec. Processed 554.51 million rows, 9.12 GB (358.96 million rows/s., 5.90 GB/s.)
Just 29 million. I've heard that some people don't give stars. They just do their job instead.
Total number of users with at least one push:
:) SELECT uniq(actor_login) FROM github_events WHERE event_type = 'PushEvent'
┌─uniq(actor_login)─┐
│ 60171743 │
└───────────────────┘
1 rows in set. Elapsed: 3.179 sec. Processed 6.39 billion rows, 19.75 GB (2.01 billion rows/s., 6.21 GB/s.)
There are actually more people who pushed code than those who gave stars.
Total number of users with at least one created PR:
:) SELECT uniq(actor_login) FROM github_events WHERE event_type = 'PullRequestEvent' AND action = 'opened'
┌─uniq(actor_login)─┐
│ 16169371 │
└───────────────────┘
1 rows in set. Elapsed: 0.614 sec. Processed 814.54 million rows, 3.65 GB (1.33 billion rows/s., 5.94 GB/s.)
What if we count top starred repositories but only from those who made at least one PR in at least one repo?
SELECT
repo_name,
count()
FROM github_events
WHERE (event_type = 'WatchEvent') AND (actor_login IN
(
SELECT actor_login
FROM github_events
WHERE (event_type = 'PullRequestEvent') AND (action = 'opened')
))
GROUP BY repo_name
ORDER BY count() DESC
LIMIT 50
┌─repo_name──────────────────────────────────┬─count()─┐
1. │ donnemartin/system-design-primer │ 229199 │
2. │ kamranahmedse/developer-roadmap │ 225940 │
3. │ sindresorhus/awesome │ 218358 │
4. │ public-apis/public-apis │ 202412 │
5. │ facebook/react │ 197694 │
6. │ jwasham/coding-interview-university │ 196505 │
7. │ EbookFoundation/free-programming-books │ 169732 │
8. │ codecrafters-io/build-your-own-x │ 161107 │
9. │ torvalds/linux │ 157775 │
10. │ getify/You-Dont-Know-JS │ 151594 │
11. │ vuejs/vue │ 148885 │
12. │ 996icu/996.ICU │ 141966 │
13. │ trekhleb/javascript-algorithms │ 139100 │
14. │ tensorflow/tensorflow │ 135318 │
15. │ github/gitignore │ 128528 │
16. │ airbnb/javascript │ 122250 │
17. │ flutter/flutter │ 121196 │
18. │ freeCodeCamp/freeCodeCamp │ 120969 │
19. │ vinta/awesome-python │ 119659 │
20. │ TheAlgorithms/Python │ 117462 │
21. │ ant-design/ant-design │ 117339 │
22. │ jlevy/the-art-of-command-line │ 112904 │
23. │ ossu/computer-science │ 109002 │
24. │ golang/go │ 108172 │
25. │ danistefanovic/build-your-own-x │ 106494 │
26. │ awesome-selfhosted/awesome-selfhosted │ 104210 │
27. │ FreeCodeCamp/FreeCodeCamp │ 98506 │
28. │ twbs/bootstrap │ 98290 │
29. │ yangshun/tech-interview-handbook │ 98157 │
30. │ facebook/react-native │ 97430 │
31. │ practical-tutorials/project-based-learning │ 92823 │
32. │ avelino/awesome-go │ 91143 │
33. │ rust-lang/rust │ 88579 │
34. │ nodejs/node │ 86213 │
35. │ trimstray/the-book-of-secret-knowledge │ 84113 │
36. │ n8n-io/n8n │ 83395 │
37. │ kubernetes/kubernetes │ 82647 │
38. │ huggingface/transformers │ 81976 │
39. │ resume/resume.github.com │ 81461 │
40. │ neovim/neovim │ 80659 │
41. │ angular/angular │ 79901 │
42. │ robbyrussell/oh-my-zsh │ 79886 │
43. │ mrdoob/three.js │ 79240 │
44. │ ripienaar/free-for-dev │ 78859 │
45. │ AUTOMATIC1111/stable-diffusion-webui │ 77393 │
46. │ excalidraw/excalidraw │ 77353 │
47. │ nvbn/thefuck │ 77256 │
48. │ tauri-apps/tauri │ 76822 │
49. │ microsoft/vscode │ 76739 │
50. │ Genymobile/scrcpy │ 76284 │
└────────────────────────────────────────────┴─────────┘
50 rows in set. Elapsed: 36.965 sec. Processed 1.37 billion rows, 15.60 GB (37.03 million rows/s., 421.92 MB/s.)
The list is similar to the overall top list.
What if we take authors who have made at least 10 PRs?
SELECT
repo_name,
count()
FROM github_events
WHERE (event_type = 'WatchEvent') AND (actor_login IN
(
SELECT actor_login
FROM github_events
WHERE (event_type = 'PullRequestEvent') AND (action = 'opened')
GROUP BY actor_login
HAVING count() >= 10
))
GROUP BY repo_name
ORDER BY count() DESC
LIMIT 50
┌─repo_name──────────────────────────────┬─count()─┐
1. │ donnemartin/system-design-primer │ 111045 │
2. │ kamranahmedse/developer-roadmap │ 103627 │
3. │ facebook/react │ 102415 │
4. │ sindresorhus/awesome │ 100731 │
5. │ public-apis/public-apis │ 92415 │
6. │ jwasham/coding-interview-university │ 86323 │
7. │ getify/You-Dont-Know-JS │ 80245 │
8. │ torvalds/linux │ 77918 │
9. │ EbookFoundation/free-programming-books │ 70207 │
10. │ trekhleb/javascript-algorithms │ 70082 │
11. │ vuejs/vue │ 68557 │
12. │ airbnb/javascript │ 65565 │
13. │ codecrafters-io/build-your-own-x │ 64951 │
14. │ github/gitignore │ 62832 │
15. │ tensorflow/tensorflow │ 60586 │
16. │ golang/go │ 59687 │
17. │ ant-design/ant-design │ 58782 │
18. │ danistefanovic/build-your-own-x │ 57523 │
19. │ flutter/flutter │ 56825 │
20. │ resume/resume.github.com │ 56685 │
21. │ jlevy/the-art-of-command-line │ 56504 │
22. │ rust-lang/rust │ 54032 │
23. │ ossu/computer-science │ 50648 │
24. │ facebook/react-native │ 49792 │
25. │ yangshun/tech-interview-handbook │ 49406 │
26. │ vinta/awesome-python │ 48183 │
27. │ awesome-selfhosted/awesome-selfhosted │ 48016 │
28. │ neovim/neovim │ 47390 │
29. │ nodejs/node │ 46966 │
30. │ sveltejs/svelte │ 46896 │
31. │ avelino/awesome-go │ 46638 │
32. │ TheAlgorithms/Python │ 46415 │
33. │ tauri-apps/tauri │ 45515 │
34. │ nvbn/thefuck │ 45476 │
35. │ twbs/bootstrap │ 45245 │
36. │ freeCodeCamp/freeCodeCamp │ 45153 │
37. │ kubernetes/kubernetes │ 44774 │
38. │ nektos/act │ 44153 │
39. │ robbyrussell/oh-my-zsh │ 44123 │
40. │ denoland/deno │ 44078 │
41. │ excalidraw/excalidraw │ 43787 │
42. │ ripienaar/free-for-dev │ 42552 │
43. │ n8n-io/n8n │ 41258 │
44. │ ryanmcdermott/clean-code-javascript │ 41069 │
45. │ papers-we-love/papers-we-love │ 40906 │
46. │ mrdoob/three.js │ 40800 │
47. │ supabase/supabase │ 40700 │
48. │ anuraghazra/github-readme-stats │ 39312 │
49. │ vercel/next.js │ 39105 │
50. │ 996icu/996.ICU │ 39008 │
└────────────────────────────────────────┴─────────┘
50 rows in set. Elapsed: 9.333 sec. Processed 1.37 billion rows, 15.51 GB (146.68 million rows/s., 1.66 GB/s.)
If we count only software, the list looks like this: React, Linux, Vue, Tensorflow, Golang, Flutter.
:) SELECT repo_name, count(), uniq(actor_login) FROM github_events WHERE event_type = 'PullRequestEvent' AND action = 'opened' GROUP BY repo_name ORDER BY count() DESC LIMIT 50
┌─repo_name───────────────────────────────────────────────────┬─count()─┬─uniq(actor_login)─┐
1. │ WolseyBankWitness/rediffusion │ 807658 │ 8 │
2. │ google-test/signclav2-probe-repo │ 603127 │ 3 │
3. │ mhutchinson/mhutchinson-distributor │ 515855 │ 7 │
4. │ google-test/signcla-probe-repo │ 444681 │ 5 │
5. │ leandrohstein/transporteservico-urbs-data │ 435322 │ 1 │
6. │ NixOS/nixpkgs │ 376689 │ 12363 │
7. │ actions-canary/ForkPRCanary │ 357945 │ 9 │
8. │ prawn-test-staging-rw/mergequeue_service_test_commit_status │ 351930 │ 5 │
9. │ microsoft/winget-pkgs │ 266167 │ 5669 │
10. │ Homebrew/homebrew-core │ 225493 │ 9512 │
11. │ Homebrew/homebrew-cask │ 171013 │ 6903 │
12. │ Nix-Drone/mergequeue-github │ 170274 │ 2 │
13. │ MetaMask/eth-phishing-detect │ 170139 │ 666 │
14. │ everypolitician/everypolitician-data │ 158123 │ 19 │
15. │ brianchandotcom/liferay-portal │ 156749 │ 602 │
16. │ GuruCICDCanary-Beta/CICDCanary │ 143669 │ 1 │
17. │ GuruCICDCanary-Prod/CICDCanary │ 142862 │ 1 │
18. │ GuruCICDCanary-Prod-Release/CICDCanary │ 142527 │ 1 │
19. │ elastic/kibana │ 142304 │ 1592 │
20. │ JuliaRegistries/General │ 131488 │ 536 │
21. │ openshift-helm-charts/sandbox │ 126785 │ 13 │
22. │ odoo/odoo │ 121604 │ 4265 │
23. │ trunk-io/mergequeue │ 115108 │ 10 │
24. │ deepsourcestatus/test-repository │ 114665 │ 3 │
25. │ pytorch/pytorch │ 103882 │ 6433 │
26. │ prawn-test-production-rw/mg-service-test │ 97738 │ 6 │
27. │ aelmanaa/cl-docs │ 97450 │ 1 │
28. │ prawn-test-staging-rw/mg-service-test │ 97364 │ 6 │
29. │ firstcontributions/first-contributions │ 90933 │ 82459 │
30. │ openshift/openshift-docs │ 87567 │ 1249 │
31. │ elastic/elasticsearch │ 84000 │ 3060 │
32. │ prawn-test-staging-rw/mergequeue_service_test │ 82094 │ 3 │
33. │ test-organization-kkjeer/bot-validation │ 79773 │ 5 │
34. │ bitnami/containers │ 78981 │ 430 │
35. │ test-organization-kkjeer/app-test │ 78949 │ 8 │
36. │ SmartThingsCommunity/SmartThingsPublic │ 77857 │ 791 │
37. │ getsentry/sentry │ 77286 │ 1100 │
38. │ rust-lang/rust │ 75837 │ 5763 │
39. │ kubernetes/kubernetes │ 74768 │ 6750 │
40. │ cockroachdb/cockroach │ 72712 │ 947 │
41. │ trunk-io/mergequeue-demo │ 71196 │ 3 │
42. │ llvm/llvm-project │ 68416 │ 4428 │
43. │ hmrc/build-and-deploy-canary-service │ 68161 │ 15 │
44. │ Automattic/wp-calypso │ 66399 │ 978 │
45. │ home-assistant/core │ 65615 │ 5194 │
46. │ mergifyio-testing/functional-testing-repo-Greesb │ 65253 │ 5 │
47. │ code-dot-org/code-dot-org │ 65032 │ 209 │
48. │ scalr-automation/terraform-scalr-flat-mirror4 │ 64953 │ 29 │
49. │ openshift/release │ 64312 │ 1924 │
50. │ zephyrproject-rtos/zephyr │ 63708 │ 3992 │
└─────────────────────────────────────────────────────────────┴─────────┴───────────────────┘
50 rows in set. Elapsed: 4.207 sec. Processed 814.41 million rows, 6.51 GB (193.58 million rows/s., 1.55 GB/s.)
Here we can see some very specific data. The top repositories collected hundreds of thousands of pull requests from a handful of accounts — probe and canary repositories for CI machinery, and the authors are obviously bots.
Conclusion: if there are many pull requests from a small number of users, it means that this is just a part of some automated process.
Number six is the package repository for Nix OS. It's quite understandable — if you want your package to be in the OS, just make a pull request. And there are a lot of packages. The same for the Windows Package Manager at №9 and Homebrew at №10.
And there is a canary repository at №7 with 357 945 pull requests from 9 accounts. Looks like a script got out of control.
Repositories with the maximum amount of pull request contributors:
:) SELECT repo_name, count(), uniq(actor_login) AS u FROM github_events WHERE event_type = 'PullRequestEvent' AND action = 'opened' GROUP BY repo_name ORDER BY u DESC LIMIT 50
┌─repo_name─────────────────────────────────────────────────────────────────┬─count()─┬─────u─┐
1. │ firstcontributions/first-contributions │ 90933 │ 82459 │
2. │ jlord/patchwork │ 52082 │ 45028 │
3. │ digitalinnovationone/dio-lab-open-source │ 42215 │ 34632 │
4. │ google/it-cert-automation-practice │ 34701 │ 33285 │
5. │ octocat/Spoon-Knife │ 32679 │ 29558 │
6. │ JetBrains/swot │ 31799 │ 27157 │
7. │ ibm-developer-skills-network/jbbmo-Introduction-to-Git-and-GitHub │ 42543 │ 26079 │
8. │ DefinitelyTyped/DefinitelyTyped │ 53899 │ 19984 │
9. │ zero-to-mastery/start-here-guidelines │ 21457 │ 18746 │
10. │ is-a-dev/register │ 25683 │ 14441 │
11. │ freddier/hyperblog │ 14836 │ 14393 │
12. │ trustwallet/assets │ 28983 │ 12592 │
13. │ NixOS/nixpkgs │ 376689 │ 12363 │
14. │ MicrosoftDocs/azure-docs │ 38584 │ 12224 │
15. │ ibm-developer-skills-network/gkpbt-css-circle │ 16121 │ 11671 │
16. │ archway-network/testnets │ 11542 │ 10734 │
17. │ elidianaandrade/dio-lab-open-source │ 11377 │ 10695 │
18. │ udacity/course-collaboration-travel-plans │ 10476 │ 10109 │
19. │ COGS108/MyFirstPullRequest │ 10645 │ 9995 │
20. │ solana-labs/token-list │ 26526 │ 9893 │
21. │ deadlyvipers/dojo_rules │ 15927 │ 9755 │
22. │ Homebrew/homebrew-core │ 225493 │ 9512 │
23. │ freeCodeCamp/freeCodeCamp │ 33991 │ 9040 │
24. │ udacity/create-your-own-adventure │ 10493 │ 8897 │
25. │ AliceWonderland/hacktoberfest │ 10802 │ 8781 │
26. │ ansible/ansible │ 50243 │ 8256 │
27. │ laravel/framework │ 32274 │ 7995 │
28. │ education/GitHubGraduation-2021 │ 10070 │ 7933 │
29. │ TheOdinProject/curriculum │ 26588 │ 7880 │
30. │ netology-code/git-2-homeworks-pr │ 7863 │ 7565 │
31. │ githubschool/open-enrollment-classes-introduction-to-github │ 9299 │ 7534 │
32. │ learn-co-students/js-from-dom-to-node-bootcamp-prep-000 │ 7472 │ 7420 │
33. │ rails/rails │ 35304 │ 7336 │
34. │ learn-co-students/javascript-intro-to-functions-lab-bootcamp-prep-000 │ 7187 │ 7157 │
35. │ learn-co-students/js-functions-lab-bootcamp-prep-000 │ 7010 │ 6976 │
36. │ Homebrew/homebrew-cask │ 171013 │ 6903 │
37. │ Meta-Front-End-Developer-PC/m3l2_forking_lab │ 7048 │ 6900 │
38. │ learn-co-students/js-node-practice-lab-bootcamp-prep-000 │ 6788 │ 6763 │
39. │ kubernetes/kubernetes │ 74768 │ 6750 │
40. │ kubernetes/website │ 32901 │ 6663 │
41. │ learn-co-students/js-if-else-files-lab-bootcamp-prep-000 │ 6555 │ 6541 │
42. │ learn-co-students/javascript-arithmetic-lab-bootcamp-prep-000 │ 6522 │ 6493 │
43. │ pytorch/pytorch │ 103882 │ 6433 │
44. │ learn-co-students/js-what-is-a-test-lab-bootcamp-prep-000 │ 6369 │ 6355 │
45. │ education/GitHubGraduation-2022 │ 7748 │ 6299 │
46. │ learn-co-students/first-ide-lab-bootcamp-prep-000 │ 6293 │ 6245 │
47. │ ibm-developer-skills-network/Centralized-repository-shipping_calculations │ 8473 │ 6221 │
48. │ mate-academy/layout_hello-world │ 8645 │ 6149 │
49. │ github/docs │ 21521 │ 6076 │
50. │ learn-co-students/js-what-is-a-test-bootcamp-prep-000 │ 6085 │ 6073 │
└───────────────────────────────────────────────────────────────────────────┴─────────┴───────┘
50 rows in set. Elapsed: 6.144 sec. Processed 814.41 million rows, 6.51 GB (132.55 million rows/s., 1.06 GB/s.)
"firstcontributions" is the obvious case: a repository that teaches you how to make a pull request... by allowing you to make a pull request to this repository. "patchwork" is similar. Most of the repositories in this list are similar.
:) SELECT repo_name, count() AS c, uniq(actor_login) AS u FROM github_events WHERE event_type = 'IssuesEvent' AND action = 'opened' GROUP BY repo_name ORDER BY c DESC LIMIT 50
┌─repo_name─────────────────────────────┬───────c─┬─────u─┐
1. │ pddemo/demo │ 1598913 │ 15 │
2. │ h4sh5/pypi-auto-scanner │ 396480 │ 4 │
3. │ ghscr/ghscription │ 383477 │ 17452 │
4. │ koorellasuresh/UKRegionTest │ 379340 │ 4 │
5. │ leo424y/heysiri.ml │ 292255 │ 267 │
6. │ kysports-cn/kytiyu │ 209933 │ 1 │
7. │ kysports-cn/kytiyutiyu │ 201078 │ 1 │
8. │ kysports-cn/kytiyucom │ 200915 │ 1 │
9. │ kysports-cn/kytiyuguanwang │ 199296 │ 1 │
10. │ kysports-cn/kytiyunet │ 195093 │ 1 │
11. │ kysports-cn/kytiyusports │ 194835 │ 1 │
12. │ kysports-cn/kytiyu-game │ 193843 │ 1 │
13. │ kysports-cn/kytiyuwanjia │ 193061 │ 1 │
14. │ kysports-cn/kytiyu-sports │ 192028 │ 1 │
15. │ kysports-cn/kytiyusports-cn.github.io │ 191429 │ 1 │
16. │ kysports-cn/kytiyulogin │ 188351 │ 1 │
17. │ AdguardTeam/AdguardFilters │ 188046 │ 1573 │
18. │ kysports-cn/kytiyuj9 │ 186149 │ 1 │
19. │ kaiyunsports-cn/kaiyunguanwang │ 185329 │ 1 │
20. │ kysports-cn/kytiyuyule │ 184745 │ 1 │
21. │ kysports-cn/kytiyuzhuce │ 184341 │ 1 │
22. │ kaiyunsports-cn/kaiyunqipai │ 177958 │ 1 │
23. │ PlaNFT/PlaNFT-Marketplace-Comments │ 174092 │ 2 │
24. │ lstjsuperman/fabric │ 173979 │ 1 │
25. │ kaiyunsports-cn/kaiyun-sports │ 173920 │ 1 │
26. │ webcompat/web-bugs │ 173816 │ 10129 │
27. │ kaiyunsports-cn/kaiyun-jiuyouj9 │ 170564 │ 1 │
28. │ kysports-cn/kytiyuqipai │ 168474 │ 1 │
29. │ kysports-cn/kytiyuty │ 166750 │ 1 │
30. │ h4sh5/npm-auto-scanner │ 166725 │ 3 │
31. │ kinjalsoftnoesis/Test │ 165565 │ 1 │
32. │ kaiyunsports-cn/kaiyunyule │ 165048 │ 1 │
33. │ kaiyunsports-cn/kaiyunlogin │ 163778 │ 1 │
34. │ kysports-cn/kytiyupc │ 163552 │ 1 │
35. │ kysports-cn/kytiyuhui │ 163212 │ 1 │
36. │ kaiyunsports-cn/kaiyunwanjia │ 163034 │ 1 │
37. │ kaiyunsports-cn/kaiyun-game │ 162699 │ 1 │
38. │ kaiyunsports-cn/kaiyun │ 162452 │ 1 │
39. │ ouyanxia2/hgmgmg │ 162355 │ 52 │
40. │ kysports-cn/kytiyu-jiuyouj9 │ 162273 │ 1 │
41. │ Khan/khan-exercises │ 156952 │ 462 │
42. │ kaiyunsports-cn/kaiyunhui │ 154464 │ 1 │
43. │ No-CQRT/GooGuns │ 151978 │ 1 │
44. │ microsoft/vscode │ 145990 │ 74637 │
45. │ kaiyunsports-cn/kaiyunj9 │ 145782 │ 1 │
46. │ ZeroK-RTS/CrashReports │ 145587 │ 37 │
47. │ kaiyunsports-cn/kaiyunnet │ 145260 │ 1 │
48. │ TingluoHuang/proxy-test │ 142887 │ 3 │
49. │ kaiyunsports-cn/kaiyunzhuce │ 141096 │ 1 │
50. │ kaiyunsports-cn/kaiyunpc │ 140465 │ 1 │
└───────────────────────────────────────┴─────────┴───────┘
50 rows in set. Elapsed: 0.765 sec. Processed 283.80 million rows, 2.74 GB (371.02 million rows/s., 3.58 GB/s.)
The top repository is a demo where a new issue is created every minute — 1.6 million of them from 15 accounts. The second one is an automated PyPI scanner. The third, ghscr/ghscription, has 383 000 issues from 17 452 different authors, which is a different kind of misuse. And there is a whole family of kysports-cn/* repositories with about 200 000 issues each, all from a single account.
The first meaningful result is microsoft/vscode with over 145k issues from over 74k authors. And the issues are real.
Conclusion: if a repository has a high number of issues, maybe issues are created automatically from crash reports.
Let's also add the number of stars to this report:
WITH (event_type = 'IssuesEvent') AND (action = 'opened') AS issue_created
SELECT
repo_name,
sum(issue_created) AS c,
uniqIf(actor_login, issue_created) AS u,
sum(event_type = 'WatchEvent') AS stars
FROM github_events
WHERE event_type IN ('IssuesEvent', 'WatchEvent')
GROUP BY repo_name
ORDER BY c DESC
LIMIT 50
┌─repo_name─────────────────────────────┬───────c─┬─────u─┬──stars─┐
1. │ pddemo/demo │ 1598913 │ 15 │ 11 │
2. │ h4sh5/pypi-auto-scanner │ 396480 │ 4 │ 35 │
3. │ ghscr/ghscription │ 383477 │ 17452 │ 125 │
4. │ koorellasuresh/UKRegionTest │ 379340 │ 4 │ 1 │
5. │ leo424y/heysiri.ml │ 292255 │ 267 │ 6 │
6. │ kysports-cn/kytiyu │ 209933 │ 1 │ 0 │
7. │ kysports-cn/kytiyutiyu │ 201078 │ 1 │ 0 │
8. │ kysports-cn/kytiyucom │ 200915 │ 1 │ 0 │
9. │ kysports-cn/kytiyuguanwang │ 199296 │ 1 │ 0 │
10. │ kysports-cn/kytiyunet │ 195093 │ 1 │ 0 │
11. │ kysports-cn/kytiyusports │ 194835 │ 1 │ 0 │
12. │ kysports-cn/kytiyu-game │ 193843 │ 1 │ 0 │
13. │ kysports-cn/kytiyuwanjia │ 193061 │ 1 │ 0 │
14. │ kysports-cn/kytiyu-sports │ 192028 │ 1 │ 0 │
15. │ kysports-cn/kytiyusports-cn.github.io │ 191429 │ 1 │ 0 │
16. │ kysports-cn/kytiyulogin │ 188351 │ 1 │ 0 │
17. │ AdguardTeam/AdguardFilters │ 188046 │ 1573 │ 4477 │
18. │ kysports-cn/kytiyuj9 │ 186149 │ 1 │ 0 │
19. │ kaiyunsports-cn/kaiyunguanwang │ 185329 │ 1 │ 0 │
20. │ kysports-cn/kytiyuyule │ 184745 │ 1 │ 0 │
21. │ kysports-cn/kytiyuzhuce │ 184341 │ 1 │ 0 │
22. │ kaiyunsports-cn/kaiyunqipai │ 177958 │ 1 │ 0 │
23. │ PlaNFT/PlaNFT-Marketplace-Comments │ 174092 │ 2 │ 0 │
24. │ lstjsuperman/fabric │ 173979 │ 1 │ 5 │
25. │ kaiyunsports-cn/kaiyun-sports │ 173920 │ 1 │ 0 │
26. │ webcompat/web-bugs │ 173816 │ 10129 │ 891 │
27. │ kaiyunsports-cn/kaiyun-jiuyouj9 │ 170564 │ 1 │ 0 │
28. │ kysports-cn/kytiyuqipai │ 168474 │ 1 │ 0 │
29. │ kysports-cn/kytiyuty │ 166750 │ 1 │ 0 │
30. │ h4sh5/npm-auto-scanner │ 166725 │ 3 │ 3 │
31. │ kinjalsoftnoesis/Test │ 165565 │ 1 │ 0 │
32. │ kaiyunsports-cn/kaiyunyule │ 165048 │ 1 │ 0 │
33. │ kaiyunsports-cn/kaiyunlogin │ 163778 │ 1 │ 0 │
34. │ kysports-cn/kytiyupc │ 163552 │ 1 │ 0 │
35. │ kysports-cn/kytiyuhui │ 163212 │ 1 │ 0 │
36. │ kaiyunsports-cn/kaiyunwanjia │ 163034 │ 1 │ 0 │
37. │ kaiyunsports-cn/kaiyun-game │ 162699 │ 1 │ 0 │
38. │ kaiyunsports-cn/kaiyun │ 162452 │ 1 │ 0 │
39. │ ouyanxia2/hgmgmg │ 162355 │ 52 │ 0 │
40. │ kysports-cn/kytiyu-jiuyouj9 │ 162273 │ 1 │ 0 │
41. │ Khan/khan-exercises │ 156952 │ 462 │ 1929 │
42. │ kaiyunsports-cn/kaiyunhui │ 154464 │ 1 │ 0 │
43. │ No-CQRT/GooGuns │ 151978 │ 1 │ 1 │
44. │ microsoft/vscode │ 145990 │ 74637 │ 124831 │
45. │ kaiyunsports-cn/kaiyunj9 │ 145782 │ 1 │ 0 │
46. │ ZeroK-RTS/CrashReports │ 145587 │ 37 │ 15 │
47. │ kaiyunsports-cn/kaiyunnet │ 145260 │ 1 │ 0 │
48. │ TingluoHuang/proxy-test │ 142887 │ 3 │ 0 │
49. │ kaiyunsports-cn/kaiyunzhuce │ 141096 │ 1 │ 0 │
50. │ kaiyunsports-cn/kaiyunpc │ 140465 │ 1 │ 0 │
└───────────────────────────────────────┴─────────┴───────┴────────┘
50 rows in set. Elapsed: 7.555 sec. Processed 838.32 million rows, 15.12 GB (110.97 million rows/s., 2.00 GB/s.)
Now we can distinguish real issues from robot ones. Let's add a cutoff at 1000 stars:
WITH (event_type = 'IssuesEvent') AND (action = 'opened') AS issue_created
SELECT
repo_name,
sum(issue_created) AS c,
uniqIf(actor_login, issue_created) AS u,
sum(event_type = 'WatchEvent') AS stars
FROM github_events
WHERE event_type IN ('IssuesEvent', 'WatchEvent')
GROUP BY repo_name
HAVING stars >= 1000
ORDER BY c DESC
LIMIT 50
┌─repo_name───────────────────────┬──────c─┬─────u─┬──stars─┐
1. │ AdguardTeam/AdguardFilters │ 188046 │ 1573 │ 4477 │
2. │ Khan/khan-exercises │ 156952 │ 462 │ 1929 │
3. │ microsoft/vscode │ 145990 │ 74637 │ 124831 │
4. │ flutter/flutter │ 104125 │ 37056 │ 202194 │
5. │ ron190/jsql-injection │ 93180 │ 67 │ 1820 │
6. │ MicrosoftDocs/azure-docs │ 78482 │ 36662 │ 11493 │
7. │ elastic/kibana │ 73560 │ 6203 │ 20052 │
8. │ cockroachdb/cockroach │ 72859 │ 1776 │ 34656 │
9. │ Microsoft/vscode │ 65111 │ 27038 │ 81998 │
10. │ vueup/vue-quill │ 60003 │ 501 │ 1312 │
11. │ golang/go │ 58071 │ 17866 │ 153423 │
12. │ DigitalPlatDev/FreeDomain │ 56304 │ 38106 │ 116888 │
13. │ godotengine/godot │ 54771 │ 15625 │ 109247 │
14. │ pytorch/pytorch │ 50758 │ 16156 │ 103248 │
15. │ rust-lang/rust │ 49356 │ 11866 │ 118506 │
16. │ home-assistant/core │ 48039 │ 26123 │ 56966 │
17. │ NixOS/nixpkgs │ 47337 │ 10945 │ 24092 │
18. │ wjz304/Redpill_CustomBuild │ 46169 │ 7263 │ 1340 │
19. │ brave/brave-browser │ 43420 │ 6693 │ 26238 │
20. │ kubernetes/kubernetes │ 42121 │ 12483 │ 120717 │
21. │ DaoCloud/public-image-mirror │ 40664 │ 8571 │ 11607 │
22. │ tensorflow/tensorflow │ 40036 │ 22461 │ 229323 │
23. │ grafana/grafana │ 38680 │ 14671 │ 73932 │
24. │ timburgan/timburgan │ 38368 │ 8998 │ 1173 │
25. │ dotnet/runtime │ 38087 │ 9530 │ 18776 │
26. │ spring-projects/spring-boot │ 37874 │ 9092 │ 90796 │
27. │ MiCode/Xiaomi_Kernel_OpenSource │ 36344 │ 1474 │ 10779 │
28. │ Expensify/App │ 35345 │ 387 │ 4671 │
29. │ dart-lang/sdk │ 34719 │ 5376 │ 12544 │
30. │ dotnet/roslyn │ 34434 │ 4777 │ 22987 │
31. │ type-challenges/type-challenges │ 33791 │ 5214 │ 47035 │
32. │ llvm/llvm-project │ 33709 │ 8723 │ 37114 │
33. │ rancher/rancher │ 33617 │ 7311 │ 25742 │
34. │ microsoft/PowerToys │ 33331 │ 22694 │ 136633 │
35. │ Automattic/wp-calypso │ 32764 │ 1484 │ 14720 │
36. │ elastic/elasticsearch │ 32647 │ 7761 │ 75306 │
37. │ pedroslopez/whatsapp-web.js │ 32349 │ 1821 │ 19991 │
38. │ ansible/ansible │ 31960 │ 15343 │ 75461 │
39. │ hq450/fancyss │ 31850 │ 1305 │ 14131 │
40. │ ant-design/ant-design │ 30588 │ 15359 │ 177645 │
41. │ angular/angular │ 30082 │ 14617 │ 122200 │
42. │ CleverRaven/Cataclysm-DDA │ 28935 │ 5217 │ 14113 │
43. │ WordPress/gutenberg │ 28604 │ 5328 │ 12857 │
44. │ facebook/react-native │ 27203 │ 16234 │ 146964 │
45. │ saltstack/salt │ 25961 │ 6784 │ 17105 │
46. │ dotnet/aspnetcore │ 25901 │ 8738 │ 25555 │
47. │ JuliaLang/julia │ 25687 │ 4434 │ 56191 │
48. │ telegramdesktop/tdesktop │ 25605 │ 8922 │ 33724 │
49. │ magento/magento2 │ 24405 │ 9284 │ 14718 │
50. │ ValveSoftware/Dota2-Gameplay │ 23999 │ 16604 │ 1071 │
└─────────────────────────────────┴────────┴───────┴────────┘
50 rows in set. Elapsed: 10.221 sec. Processed 838.32 million rows, 15.12 GB (82.02 million rows/s., 1.48 GB/s.)
Now it looks like a reasonable list of the top repositories.
And for the next report, let's sort by the number of issue authors:
WITH (event_type = 'IssuesEvent') AND (action = 'opened') AS issue_created
SELECT
repo_name,
sum(issue_created) AS c,
uniqIf(actor_login, issue_created) AS u,
sum(event_type = 'WatchEvent') AS stars
FROM github_events
WHERE event_type IN ('IssuesEvent', 'WatchEvent')
GROUP BY repo_name
ORDER BY u DESC
LIMIT 50
┌─repo_name───────────────────────────────────────────────────┬──────c─┬─────u─┬──stars─┐
1. │ microsoft/vscode │ 145990 │ 74637 │ 124831 │
2. │ DigitalPlatDev/FreeDomain │ 56304 │ 38106 │ 116888 │
3. │ flutter/flutter │ 104125 │ 37056 │ 202194 │
4. │ MicrosoftDocs/azure-docs │ 78482 │ 36662 │ 11493 │
5. │ Microsoft/vscode │ 65111 │ 27038 │ 81998 │
6. │ home-assistant/core │ 48039 │ 26123 │ 56966 │
7. │ microsoft/PowerToys │ 33331 │ 22694 │ 136633 │
8. │ tensorflow/tensorflow │ 40036 │ 22461 │ 229323 │
9. │ LeetCode-Feedback/LeetCode-Feedback │ 30282 │ 20810 │ 992 │
10. │ AleoHQ/leo │ 21162 │ 17918 │ 6139 │
11. │ golang/go │ 58071 │ 17866 │ 153423 │
12. │ ghscr/ghscription │ 383477 │ 17452 │ 125 │
13. │ ValveSoftware/Dota2-Gameplay │ 23999 │ 16604 │ 1071 │
14. │ facebook/react-native │ 27203 │ 16234 │ 146964 │
15. │ pytorch/pytorch │ 50758 │ 16156 │ 103248 │
16. │ godotengine/godot │ 54771 │ 15625 │ 109247 │
17. │ ant-design/ant-design │ 30588 │ 15359 │ 177645 │
18. │ ansible/ansible │ 31960 │ 15343 │ 75461 │
19. │ FortAwesome/Font-Awesome │ 19281 │ 14923 │ 90787 │
20. │ grafana/grafana │ 38680 │ 14671 │ 73932 │
21. │ angular/angular │ 30082 │ 14617 │ 122200 │
22. │ 996icu/996.ICU │ 22790 │ 12818 │ 378812 │
23. │ anthropics/claude-code │ 16451 │ 12644 │ 45394 │
24. │ kubernetes/kubernetes │ 42121 │ 12483 │ 120717 │
25. │ docker/for-win │ 14281 │ 12315 │ 2256 │
26. │ hashicorp/terraform │ 20920 │ 11948 │ 50907 │
27. │ rust-lang/rust │ 49356 │ 11866 │ 118506 │
28. │ spyder-ide/spyder │ 19283 │ 11455 │ 10327 │
29. │ npm/npm │ 14490 │ 11403 │ 18142 │
30. │ jlippold/tweakCompatible │ 127457 │ 11278 │ 576 │
31. │ atom/atom │ 15871 │ 11108 │ 81517 │
32. │ laravel/framework │ 18518 │ 11027 │ 39329 │
33. │ NixOS/nixpkgs │ 47337 │ 10945 │ 24092 │
34. │ uBlockOrigin/uAssets │ 23283 │ 10918 │ 5741 │
35. │ vercel/next.js │ 16619 │ 10902 │ 100505 │
36. │ githubschool/open-enrollment-classes-introduction-to-github │ 12094 │ 10730 │ 921 │
37. │ Ultimaker/Cura │ 15781 │ 10603 │ 7130 │
38. │ ContinuumIO/anaconda-issues │ 12995 │ 10544 │ 883 │
39. │ Koenkk/zigbee2mqtt │ 17036 │ 10419 │ 14579 │
40. │ twbs/bootstrap │ 16414 │ 10390 │ 167745 │
41. │ ElemeFE/element │ 17292 │ 10360 │ 63131 │
42. │ Azure/azure-cli │ 17017 │ 10281 │ 4751 │
43. │ qbittorrent/qBittorrent │ 16364 │ 10134 │ 37168 │
44. │ webcompat/web-bugs │ 173816 │ 10129 │ 891 │
45. │ nodejs/node │ 18761 │ 9915 │ 123699 │
46. │ electron/electron │ 16971 │ 9891 │ 114109 │
47. │ expo/expo │ 16439 │ 9847 │ 45252 │
48. │ rails/rails │ 17732 │ 9815 │ 68604 │
49. │ facebook/react │ 14819 │ 9689 │ 290876 │
50. │ huggingface/transformers │ 16311 │ 9678 │ 144801 │
└─────────────────────────────────────────────────────────────┴────────┴───────┴────────┘
50 rows in set. Elapsed: 7.517 sec. Processed 838.32 million rows, 15.12 GB (111.52 million rows/s., 2.01 GB/s.)
:) SELECT repo_name, uniqIf(actor_login, event_type = 'PushEvent') AS u, sum(event_type = 'WatchEvent') AS stars FROM github_events WHERE event_type IN ('PushEvent', 'WatchEvent') AND repo_name != '/' GROUP BY repo_name ORDER BY u DESC LIMIT 50
┌─repo_name───────────────────────────────────────────────────┬────u─┬──stars─┐
1. │ githubschool/open-enrollment-classes-introduction-to-github │ 7869 │ 921 │
2. │ llvm/llvm-project │ 2221 │ 37114 │
3. │ Azure/azure-rest-api-specs │ 1880 │ 3174 │
4. │ github/advisory-database │ 1787 │ 1968 │
5. │ DataDog/documentation │ 1712 │ 598 │
6. │ odoo-dev/odoo │ 1537 │ 194 │
7. │ department-of-veterans-affairs/va.gov-team │ 1130 │ 422 │
8. │ githubschool/on-demand-github-pages │ 1100 │ 165 │
9. │ hackclub/dinosaurs │ 897 │ 1129 │
10. │ ministryofjustice/cloud-platform-environments │ 879 │ 89 │
11. │ cs480-projects/cs480-projects.github.io │ 878 │ 33 │
12. │ elastic/kibana │ 854 │ 20052 │
13. │ betagouv/beta.gouv.fr │ 836 │ 276 │
14. │ Automattic/wp-calypso │ 780 │ 14720 │
15. │ wecode-bootcamp-korea/westagram-frontend │ 730 │ 56 │
16. │ hmcts/cnp-flux-config │ 717 │ 41 │
17. │ DataDog/datadog-agent │ 713 │ 3439 │
18. │ pytorch/pytorch │ 705 │ 103248 │
19. │ hmcts/github-slack-user-mappings │ 687 │ 6 │
20. │ department-of-veterans-affairs/vets-website │ 664 │ 335 │
21. │ Azure/azure-sdk-for-net │ 638 │ 6212 │
22. │ department-of-veterans-affairs/vets-api │ 628 │ 283 │
23. │ elastic/elasticsearch │ 604 │ 75306 │
24. │ bioconda/bioconda-recipes │ 600 │ 2002 │
25. │ lifo/docrails │ 592 │ 356 │
26. │ drshahizan/learn-github │ 578 │ 894 │
27. │ CorporationX/god_bless │ 577 │ 18 │
28. │ microsoftgraph/microsoft-graph-docs │ 575 │ 1252 │
29. │ DataDog/integrations-core │ 574 │ 1102 │
30. │ CorporationX/user_service │ 572 │ 13 │
31. │ IGM-RichMedia-at-RIT/430-Git-Exercise-Part2 │ 569 │ 1 │
32. │ Azure/azure-sdk-for-python │ 568 │ 5784 │
33. │ Automattic/jetpack │ 558 │ 2041 │
34. │ odoo-dev/tutorials │ 546 │ 15 │
35. │ Unity-Technologies/DOTS-training-samples │ 540 │ 1715 │
36. │ grafana/grafana │ 530 │ 73932 │
37. │ tenstorrent/tt-metal │ 525 │ 964 │
38. │ HNGInternship/HNGInternship4 │ 518 │ 45 │
39. │ OneCommunityGlobal/HighestGoodNetworkApp │ 490 │ 65 │
40. │ leanprover-community/mathlib4 │ 468 │ 2488 │
41. │ wecode-bootcamp-korea/git-rebase │ 466 │ 4 │
42. │ tencentyun/qcloud-documents │ 463 │ 2784 │
43. │ HGustavs/LenaSYS │ 457 │ 95 │
44. │ Azure/azure-sdk-for-java │ 439 │ 2732 │
45. │ hnginterns/hngfun │ 437 │ 34 │
46. │ CorporationX/post_service │ 433 │ 2 │
47. │ BitGo/BitGoJS │ 427 │ 429 │
48. │ HNGInternship/HNGFun │ 419 │ 21 │
49. │ ROCm/rocm-libraries │ 416 │ 143 │
50. │ cloudflare/cloudflare-docs │ 414 │ 4325 │
└─────────────────────────────────────────────────────────────┴──────┴────────┘
50 rows in set. Elapsed: 184.555 sec. Processed 6.95 billion rows, 45.98 GB (37.65 million rows/s., 249.16 MB/s.)
The first one is a GitHub training repository. And I'm happy to see the LLVM project in 2nd place. It's really fantastic to have over two thousand people pushing to the repository. Maybe it's just a development model where they are giving access to separate branches for a user or organization?
Repositories with the most people who have push access to the main branch.
SELECT
repo_name,
uniqIf(actor_login, (event_type = 'PushEvent') AND match(ref, '/(main|master)$')) AS u,
sum(event_type = 'WatchEvent') AS stars
FROM github_events
WHERE (event_type IN ('PushEvent', 'WatchEvent')) AND (repo_name != '/')
GROUP BY repo_name
ORDER BY u DESC
LIMIT 50
┌─repo_name───────────────────────────────────────────────────┬────u─┬─stars─┐
1. │ githubschool/open-enrollment-classes-introduction-to-github │ 5603 │ 921 │
2. │ llvm/llvm-project │ 2214 │ 37114 │
3. │ department-of-veterans-affairs/va.gov-team │ 1113 │ 422 │
4. │ cs480-projects/cs480-projects.github.io │ 878 │ 33 │
5. │ ministryofjustice/cloud-platform-environments │ 847 │ 89 │
6. │ betagouv/beta.gouv.fr │ 812 │ 276 │
7. │ githubschool/on-demand-github-pages │ 808 │ 165 │
8. │ elastic/kibana │ 777 │ 20052 │
9. │ Azure/azure-rest-api-specs │ 761 │ 3174 │
10. │ hmcts/cnp-flux-config │ 677 │ 41 │
11. │ hmcts/github-slack-user-mappings │ 642 │ 6 │
12. │ DataDog/documentation │ 611 │ 598 │
13. │ lifo/docrails │ 592 │ 356 │
14. │ drshahizan/learn-github │ 578 │ 894 │
15. │ department-of-veterans-affairs/vets-website │ 575 │ 335 │
16. │ IGM-RichMedia-at-RIT/430-Git-Exercise-Part2 │ 569 │ 1 │
17. │ bioconda/bioconda-recipes │ 554 │ 2002 │
18. │ department-of-veterans-affairs/vets-api │ 538 │ 283 │
19. │ elastic/elasticsearch │ 519 │ 75306 │
20. │ HNGInternship/HNGInternship4 │ 517 │ 45 │
21. │ tencentyun/qcloud-documents │ 456 │ 2784 │
22. │ hnginterns/hngfun │ 437 │ 34 │
23. │ HNGInternship/HNGFun │ 419 │ 21 │
24. │ ndm736/ME433.Kitchen │ 412 │ 0 │
25. │ mks65/list │ 409 │ 0 │
26. │ mks65/euler │ 408 │ 1 │
27. │ Azure/azure-sdk-for-net │ 403 │ 6212 │
28. │ Azure/azure-sdk-for-python │ 398 │ 5784 │
29. │ mks66/line │ 387 │ 0 │
30. │ mks66/picmaker │ 383 │ 3 │
31. │ mks65/dirinfo │ 382 │ 0 │
32. │ Automattic/wp-calypso │ 378 │ 14720 │
33. │ mks66/matrix │ 374 │ 0 │
34. │ CocoaPods/Specs │ 374 │ 7830 │
35. │ mks66/3d │ 373 │ 0 │
36. │ getsentry/sentry │ 369 │ 44316 │
37. │ grafana/grafana │ 362 │ 73932 │
38. │ mks66/curves │ 357 │ 0 │
39. │ NixOS/nixpkgs │ 356 │ 24093 │
40. │ mks66/polygons │ 346 │ 0 │
41. │ mks66/cstack │ 343 │ 0 │
42. │ sourcegraph/handbook │ 342 │ 172 │
43. │ Azure/azure-sdk-for-java │ 339 │ 2732 │
44. │ Pietia1978/kwztutorial │ 336 │ 10 │
45. │ mks65/randfile │ 336 │ 0 │
46. │ mks66/mdl │ 335 │ 0 │
47. │ orka97/Byt10 │ 332 │ 3 │
48. │ yugabyte/yugabyte-db │ 330 │ 7635 │
49. │ TechLabs-Berlin/git-configuration-test │ 328 │ 11 │
50. │ edx/edx-platform │ 328 │ 6617 │
└─────────────────────────────────────────────────────────────┴──────┴───────┘
50 rows in set. Elapsed: 170.261 sec. Processed 5.94 billion rows, 57.67 GB (34.89 million rows/s., 338.70 MB/s.)
Almost the same, with LLVM in 2nd place again.
With a cutoff for the number of stars:
SELECT
repo_name,
uniqIf(actor_login, (event_type = 'PushEvent') AND match(ref, '/(main|master)$')) AS u,
sum(event_type = 'WatchEvent') AS stars
FROM github_events
WHERE (event_type IN ('PushEvent', 'WatchEvent')) AND (repo_name != '/')
GROUP BY repo_name
HAVING stars >= 100
ORDER BY u DESC
LIMIT 50
┌─repo_name───────────────────────────────────────────────────┬────u─┬──stars─┐
1. │ githubschool/open-enrollment-classes-introduction-to-github │ 5603 │ 921 │
2. │ llvm/llvm-project │ 2214 │ 37114 │
3. │ department-of-veterans-affairs/va.gov-team │ 1113 │ 422 │
4. │ betagouv/beta.gouv.fr │ 812 │ 276 │
5. │ githubschool/on-demand-github-pages │ 808 │ 165 │
6. │ elastic/kibana │ 777 │ 20052 │
7. │ Azure/azure-rest-api-specs │ 761 │ 3174 │
8. │ DataDog/documentation │ 611 │ 598 │
9. │ lifo/docrails │ 592 │ 356 │
10. │ drshahizan/learn-github │ 578 │ 894 │
11. │ department-of-veterans-affairs/vets-website │ 575 │ 335 │
12. │ bioconda/bioconda-recipes │ 554 │ 2002 │
13. │ department-of-veterans-affairs/vets-api │ 538 │ 283 │
14. │ elastic/elasticsearch │ 519 │ 75306 │
15. │ tencentyun/qcloud-documents │ 456 │ 2784 │
16. │ Azure/azure-sdk-for-net │ 403 │ 6212 │
17. │ Azure/azure-sdk-for-python │ 398 │ 5784 │
18. │ Automattic/wp-calypso │ 378 │ 14720 │
19. │ CocoaPods/Specs │ 374 │ 7830 │
20. │ getsentry/sentry │ 369 │ 44316 │
21. │ grafana/grafana │ 362 │ 73932 │
22. │ NixOS/nixpkgs │ 356 │ 24093 │
23. │ sourcegraph/handbook │ 342 │ 172 │
24. │ Azure/azure-sdk-for-java │ 339 │ 2732 │
25. │ yugabyte/yugabyte-db │ 330 │ 7635 │
26. │ edx/edx-platform │ 328 │ 6617 │
27. │ BitGo/BitGoJS │ 323 │ 429 │
28. │ Azure/azure-sdk-for-js │ 319 │ 2489 │
29. │ getsentry/sentry-docs │ 317 │ 447 │
30. │ DataDog/datadog-agent │ 314 │ 3439 │
31. │ DaleStudy/leetcode-study │ 294 │ 149 │
32. │ elastic/integrations │ 287 │ 331 │
33. │ sourcegraph/about │ 274 │ 115 │
34. │ microsoft/azure-pipelines-tasks │ 262 │ 2412 │
35. │ mongodb/mongo │ 262 │ 31930 │
36. │ web-platform-tests/wpt │ 259 │ 4813 │
37. │ guardian/frontend │ 259 │ 6591 │
38. │ flutter/flutter │ 258 │ 202194 │
39. │ apple/swift │ 255 │ 81938 │
40. │ sourcegraph/sourcegraph │ 251 │ 13611 │
41. │ dotnet/runtime │ 250 │ 18776 │
42. │ microsoft/Application-Insights-Workbooks │ 241 │ 619 │
43. │ elastic/beats │ 232 │ 11788 │
44. │ PostHog/posthog.com │ 227 │ 782 │
45. │ Azure/azureml-examples │ 224 │ 1891 │
46. │ flutter/engine │ 224 │ 8073 │
47. │ demisto/content │ 220 │ 1271 │
48. │ compsocialscience/summer-institute │ 219 │ 387 │
49. │ hashicorp/consul │ 218 │ 32429 │
50. │ alphagov/govuk-puppet │ 217 │ 146 │
└─────────────────────────────────────────────────────────────┴──────┴────────┘
50 rows in set. Elapsed: 16.060 sec. Processed 6.95 billion rows, 62.64 GB (432.65 million rows/s., 3.90 GB/s.)
:) SELECT repo_name, sum(event_type = 'MemberEvent') AS invitations, sum(event_type = 'WatchEvent') AS stars FROM github_events WHERE event_type IN ('MemberEvent', 'WatchEvent') GROUP BY repo_name HAVING stars >= 100 ORDER BY invitations DESC LIMIT 50
┌─repo_name───────────────────────────────────────────────────┬─invitations─┬──stars─┐
1. │ githubschool/open-enrollment-classes-introduction-to-github │ 10448 │ 921 │
2. │ gatsbyjs/gatsby │ 3168 │ 63800 │
3. │ / │ 3139 │ 5493 │
4. │ openjournals/joss-reviews │ 2624 │ 847 │
5. │ zulip/zulip │ 2094 │ 26411 │
6. │ githubschool/on-demand-github-pages │ 1615 │ 165 │
7. │ MasteringNuxt/NuxtBnB │ 1560 │ 235 │
8. │ pytorch/csprng │ 1295 │ 119 │
9. │ MasteringNuxt/mastering-nuxt-3 │ 1009 │ 186 │
10. │ winglang/wing │ 842 │ 5159 │
11. │ InfiniteFlightAirportEditing/Airports │ 792 │ 242 │
12. │ dotnet/runtime │ 748 │ 18776 │
13. │ appwrite-community/htf24-hackathon-submissions │ 742 │ 166 │
14. │ microsoft/Llama-2-Onnx │ 675 │ 994 │
15. │ zephyrproject-rtos/zephyr │ 637 │ 14209 │
16. │ drshahizan/learn-github │ 572 │ 894 │
17. │ Azure/cognitive-search-vector-pr │ 504 │ 384 │
18. │ leanprover-community/mathlib4 │ 473 │ 2488 │
19. │ tencentyun/qcloud-documents │ 466 │ 2784 │
20. │ pytorch/pytorch │ 457 │ 103248 │
21. │ Automattic/wp-calypso │ 438 │ 14720 │
22. │ microsoft/autogen │ 431 │ 49379 │
23. │ pqrs-org/KE-complex_modifications │ 401 │ 1476 │
24. │ hashicorp/boundary │ 393 │ 4067 │
25. │ oppia/oppia │ 367 │ 7555 │
26. │ leanprover-community/mathlib │ 350 │ 1727 │
27. │ duckduckgo/duckduckhack-docs │ 350 │ 107 │
28. │ MicrosoftDocs/msteams-docs │ 348 │ 346 │
29. │ w3c/web-platform-tests │ 343 │ 1662 │
30. │ duckduckgo/zeroclickinfo-longtail │ 333 │ 104 │
31. │ llvm/llvm-project │ 324 │ 37114 │
32. │ duckduckgo/zeroclickinfo-goodies │ 321 │ 1165 │
33. │ koppi/iso-country-flags-svg-collection │ 318 │ 408 │
34. │ Azure/azure-rest-api-specs │ 315 │ 3174 │
35. │ hngprojects/hng_boilerplate_nextjs │ 305 │ 220 │
36. │ duckduckgo/zeroclickinfo-fathead │ 303 │ 362 │
37. │ cockroachdb/cockroach │ 291 │ 34656 │
38. │ duckduckgo/zeroclickinfo-spice │ 290 │ 648 │
39. │ FightPandemics/FightPandemics │ 287 │ 140 │
40. │ edmcouncil/fibo │ 286 │ 419 │
41. │ duckduckgo/duckduckgo │ 278 │ 2322 │
42. │ DmitryRyumin/INTERSPEECH-2023-Papers │ 267 │ 587 │
43. │ openjdk/skara │ 256 │ 228 │
44. │ PicPay/picpay-desafio-frontend │ 242 │ 221 │
45. │ kufu/smarthr-ui │ 241 │ 1002 │
46. │ magento/magento2 │ 239 │ 14718 │
47. │ NVIDIA/TensorRT-LLM │ 239 │ 11790 │
48. │ programminghistorian/ph-submissions │ 236 │ 163 │
49. │ NVIDIA/NeMo │ 233 │ 15147 │
50. │ compsocialscience/summer-institute │ 232 │ 387 │
└─────────────────────────────────────────────────────────────┴─────────────┴────────┘
50 rows in set. Elapsed: 5.837 sec. Processed 591.25 million rows, 4.29 GB (101.29 million rows/s., 735.54 MB/s.)
:) SELECT repo_name, count() AS forks FROM github_events WHERE event_type = 'ForkEvent' GROUP BY repo_name ORDER BY forks DESC LIMIT 50
┌─repo_name────────────────────────────────┬──forks─┐
1. │ jtleek/datasharing │ 291380 │
2. │ octocat/Spoon-Knife │ 264009 │
3. │ rdpeng/ProgrammingAssignment2 │ 180294 │
4. │ github/gitignore │ 123100 │
5. │ tensorflow/tensorflow │ 115131 │
6. │ firstcontributions/first-contributions │ 114777 │
7. │ twbs/bootstrap │ 113589 │
8. │ TheOdinProject/css-exercises │ 102043 │
9. │ LSPosed/MagiskOnWSA │ 95304 │
10. │ Pierian-Data/Complete-Python-3-Bootcamp │ 95095 │
11. │ SmartThingsCommunity/SmartThingsPublic │ 94620 │
12. │ nightscout/cgm-remote-monitor │ 94148 │
13. │ academicpages/academicpages.github.io │ 92649 │
14. │ barryclark/jekyll-now │ 88427 │
15. │ jwasham/coding-interview-university │ 85208 │
16. │ torvalds/linux │ 79858 │
17. │ facebook/react │ 75700 │
18. │ rdpeng/ExData_Plotting1 │ 72439 │
19. │ github/docs │ 72172 │
20. │ rafaballerini/rafaballerini │ 71098 │
21. │ mmistakes/minimal-mistakes │ 67584 │
22. │ ant-design/ant-design │ 67266 │
23. │ eugenp/tutorials │ 67017 │
24. │ opencv/opencv │ 66660 │
25. │ jlord/patchwork │ 64431 │
26. │ jackfrued/Python-100-Days │ 63248 │
27. │ TheAlgorithms/Python │ 59155 │
28. │ tensorflow/models │ 57512 │
29. │ google/it-cert-automation-practice │ 56993 │
30. │ Yidadaa/ChatGPT-Next-Web │ 56019 │
31. │ donnemartin/system-design-primer │ 55040 │
32. │ EbookFoundation/free-programming-books │ 54685 │
33. │ spring-projects/spring-boot │ 54211 │
34. │ digitalinnovationone/dio-lab-open-source │ 53201 │
35. │ vuejs/vue │ 53112 │
36. │ wesbos/JavaScript30 │ 52470 │
37. │ Snailclimb/JavaGuide │ 52140 │
38. │ mrdoob/three.js │ 52015 │
39. │ django/django │ 51776 │
40. │ spring-projects/spring-framework │ 50877 │
41. │ microsoft/generative-ai-for-beginners │ 50173 │
42. │ kamranahmedse/developer-roadmap │ 50132 │
43. │ RedHatTraining/DO180-apps │ 49841 │
44. │ TheOdinProject/javascript-exercises │ 49798 │
45. │ CyC2018/CS-Notes │ 49763 │
46. │ kubernetes/kubernetes │ 49758 │
47. │ coolsnowwolf/lede │ 49515 │
48. │ laravel/laravel │ 49409 │
49. │ rdpeng/RepData_PeerAssessment1 │ 48407 │
50. │ nvim-lua/kickstart.nvim │ 47537 │
└──────────────────────────────────────────┴────────┘
50 rows in set. Elapsed: 1.248 sec. Processed 175.22 million rows, 1.54 GB (140.43 million rows/s., 1.23 GB/s.)
There are 175 million forks on GitHub — about a third of all repositories. If we filter out repositories whose main purpose is to be forked (course assignments, templates and "fork me" tutorials), the clear winner is Tensorflow, then Bootstrap, then Linux and React.
Let's check the proportion of stars to forks.
SELECT
repo_name,
sum(event_type = 'ForkEvent') AS forks,
sum(event_type = 'WatchEvent') AS stars,
round(stars / forks, 3) AS ratio
FROM github_events
WHERE event_type IN ('ForkEvent', 'WatchEvent')
GROUP BY repo_name
ORDER BY forks DESC
LIMIT 50
┌─repo_name────────────────────────────────┬──forks─┬──stars─┬─ratio─┐
1. │ jtleek/datasharing │ 291380 │ 7725 │ 0.027 │
2. │ octocat/Spoon-Knife │ 264009 │ 9551 │ 0.036 │
3. │ rdpeng/ProgrammingAssignment2 │ 180294 │ 1279 │ 0.007 │
4. │ github/gitignore │ 123100 │ 185238 │ 1.505 │
5. │ tensorflow/tensorflow │ 115131 │ 229323 │ 1.992 │
6. │ firstcontributions/first-contributions │ 114777 │ 55414 │ 0.483 │
7. │ twbs/bootstrap │ 113589 │ 167745 │ 1.477 │
8. │ TheOdinProject/css-exercises │ 102043 │ 2691 │ 0.026 │
9. │ LSPosed/MagiskOnWSA │ 95304 │ 19598 │ 0.206 │
10. │ Pierian-Data/Complete-Python-3-Bootcamp │ 95095 │ 33038 │ 0.347 │
11. │ SmartThingsCommunity/SmartThingsPublic │ 94620 │ 3254 │ 0.034 │
12. │ nightscout/cgm-remote-monitor │ 94148 │ 3310 │ 0.035 │
13. │ academicpages/academicpages.github.io │ 92649 │ 16735 │ 0.181 │
14. │ barryclark/jekyll-now │ 88427 │ 10059 │ 0.114 │
15. │ jwasham/coding-interview-university │ 85208 │ 326155 │ 3.828 │
16. │ torvalds/linux │ 79858 │ 243571 │ 3.05 │
17. │ facebook/react │ 75700 │ 290876 │ 3.842 │
18. │ rdpeng/ExData_Plotting1 │ 72439 │ 390 │ 0.005 │
19. │ github/docs │ 72172 │ 23524 │ 0.326 │
20. │ rafaballerini/rafaballerini │ 71098 │ 3298 │ 0.046 │
21. │ mmistakes/minimal-mistakes │ 67584 │ 15475 │ 0.229 │
22. │ ant-design/ant-design │ 67266 │ 177645 │ 2.641 │
23. │ eugenp/tutorials │ 67017 │ 41702 │ 0.622 │
24. │ opencv/opencv │ 66660 │ 83793 │ 1.257 │
25. │ jlord/patchwork │ 64431 │ 1614 │ 0.025 │
26. │ jackfrued/Python-100-Days │ 63248 │ 197647 │ 3.125 │
27. │ TheAlgorithms/Python │ 59155 │ 231991 │ 3.922 │
28. │ tensorflow/models │ 57512 │ 89928 │ 1.564 │
29. │ google/it-cert-automation-practice │ 56993 │ 1168 │ 0.02 │
30. │ Yidadaa/ChatGPT-Next-Web │ 56019 │ 52076 │ 0.93 │
31. │ donnemartin/system-design-primer │ 55040 │ 336938 │ 6.122 │
32. │ EbookFoundation/free-programming-books │ 54685 │ 312481 │ 5.714 │
33. │ spring-projects/spring-boot │ 54211 │ 90796 │ 1.675 │
34. │ digitalinnovationone/dio-lab-open-source │ 53201 │ 6310 │ 0.119 │
35. │ vuejs/vue │ 53112 │ 251379 │ 4.733 │
36. │ wesbos/JavaScript30 │ 52470 │ 32149 │ 0.613 │
37. │ Snailclimb/JavaGuide │ 52140 │ 168315 │ 3.228 │
38. │ mrdoob/three.js │ 52015 │ 122170 │ 2.349 │
39. │ django/django │ 51776 │ 104961 │ 2.027 │
40. │ spring-projects/spring-framework │ 50877 │ 66830 │ 1.314 │
41. │ microsoft/generative-ai-for-beginners │ 50173 │ 95664 │ 1.907 │
42. │ kamranahmedse/developer-roadmap │ 50132 │ 363470 │ 7.25 │
43. │ RedHatTraining/DO180-apps │ 49841 │ 352 │ 0.007 │
44. │ TheOdinProject/javascript-exercises │ 49798 │ 1805 │ 0.036 │
45. │ CyC2018/CS-Notes │ 49763 │ 170749 │ 3.431 │
46. │ kubernetes/kubernetes │ 49758 │ 120717 │ 2.426 │
47. │ coolsnowwolf/lede │ 49515 │ 36786 │ 0.743 │
48. │ laravel/laravel │ 49409 │ 100499 │ 2.034 │
49. │ rdpeng/RepData_PeerAssessment1 │ 48407 │ 163 │ 0.003 │
50. │ nvim-lua/kickstart.nvim │ 47537 │ 29066 │ 0.611 │
└──────────────────────────────────────────┴────────┴────────┴───────┘
50 rows in set. Elapsed: 6.385 sec. Processed 729.74 million rows, 4.93 GB (114.30 million rows/s., 771.94 MB/s.)
We can see a separation. Some repositories are "for forks" like the "octocat/Spoon-Knife" — they either have fork as a purpose of the repo or some of them represent a template to base something new on. Some repositories are "for stars" — usually it's not software but some text content, like "996icu/996.ICU" and "sindresorhus/awesome".
Let's find out which repositories are the most "for forks" and "for stars".
More stars less forks:
SELECT
repo_name,
sum(event_type = 'ForkEvent') AS forks,
sum(event_type = 'WatchEvent') AS stars,
round(stars / forks, 2) AS ratio
FROM github_events
WHERE event_type IN ('ForkEvent', 'WatchEvent')
GROUP BY repo_name
HAVING (stars > 100) AND (forks > 100)
ORDER BY ratio DESC
LIMIT 50
┌─repo_name────────────────────────┬─forks─┬──stars─┬──ratio─┐
1. │ UsergeTeam/Userge-Assistant │ 105 │ 25033 │ 238.41 │
2. │ M4cs/BabySploit │ 180 │ 32547 │ 180.82 │
3. │ avgupta456/github-trends │ 134 │ 19884 │ 148.39 │
4. │ yoavbls/pretty-ts-errors │ 109 │ 13403 │ 122.96 │
5. │ inkonchain/ink-kit │ 434 │ 40513 │ 93.35 │
6. │ inkonchain/docs │ 456 │ 40656 │ 89.16 │
7. │ symfony/routing │ 111 │ 9287 │ 83.67 │
8. │ astral-sh/ty │ 156 │ 12102 │ 77.58 │
9. │ sebastianbergmann/php-timer │ 110 │ 8229 │ 74.81 │
10. │ gvergnaud/ts-pattern │ 185 │ 13796 │ 74.57 │
11. │ NopeCHALLC/nopecha-extension │ 146 │ 10454 │ 71.6 │
12. │ DigitalPlatDev/FreeDomain │ 1674 │ 116888 │ 69.83 │
13. │ sebastianbergmann/diff │ 118 │ 8206 │ 69.54 │
14. │ tipsy/github-profile-summary │ 330 │ 22390 │ 67.85 │
15. │ inkonchain/node │ 637 │ 40902 │ 64.21 │
16. │ shardeum/shardeum │ 662 │ 41764 │ 63.09 │
17. │ symfony/translation │ 114 │ 7143 │ 62.66 │
18. │ myclabs/DeepCopy │ 150 │ 9359 │ 62.39 │
19. │ total-typescript/ts-reset │ 119 │ 7414 │ 62.3 │
20. │ waydabber/BetterDisplay │ 368 │ 22683 │ 61.64 │
21. │ IdreesInc/Monocraft │ 155 │ 9538 │ 61.54 │
22. │ FelixKratz/SketchyBar │ 167 │ 9984 │ 59.78 │
23. │ aidenybai/react-scan │ 307 │ 18059 │ 58.82 │
24. │ symfony/http-kernel │ 150 │ 8771 │ 58.47 │
25. │ tc39/proposal-pipeline-operator │ 111 │ 6393 │ 57.59 │
26. │ mgdm/htmlq │ 126 │ 7234 │ 57.41 │
27. │ pakastin/open-source-flash │ 128 │ 7325 │ 57.23 │
28. │ doctrine/inflector │ 212 │ 12073 │ 56.95 │
29. │ bensadeh/tailspin │ 120 │ 6792 │ 56.6 │
30. │ JordanSchuetz/LearnCS8-Resume │ 196 │ 11050 │ 56.38 │
31. │ symfony/process │ 142 │ 7972 │ 56.14 │
32. │ tc39/proposal-pattern-matching │ 106 │ 5870 │ 55.38 │
33. │ githubnext/monaspace │ 322 │ 17212 │ 53.45 │
34. │ akavel/up │ 159 │ 8479 │ 53.33 │
35. │ phpDocumentor/ReflectionDocBlock │ 189 │ 10019 │ 53.01 │
36. │ WXRIW/Lyricify-App │ 120 │ 6331 │ 52.76 │
37. │ ulid/spec │ 196 │ 10178 │ 51.93 │
38. │ Stengo/DeskPad │ 134 │ 6921 │ 51.65 │
39. │ sharkdp/hyperfine │ 509 │ 26278 │ 51.63 │
40. │ dandavison/delta │ 551 │ 28040 │ 50.89 │
41. │ ducaale/xh │ 117 │ 5950 │ 50.85 │
42. │ arktypeio/arktype │ 130 │ 6530 │ 50.23 │
43. │ barvian/number-flow │ 103 │ 5158 │ 50.08 │
44. │ chriswalz/bit │ 132 │ 6530 │ 49.47 │
45. │ Jarred-Sumner/bun │ 316 │ 15518 │ 49.11 │
46. │ imputnet/helium │ 104 │ 5097 │ 49.01 │
47. │ xataio/pgroll │ 108 │ 5284 │ 48.93 │
48. │ darrenburns/posting │ 204 │ 9903 │ 48.54 │
49. │ Wilfred/difftastic │ 467 │ 22654 │ 48.51 │
50. │ MightyMoud/sidekick │ 138 │ 6677 │ 48.38 │
└──────────────────────────────────┴───────┴────────┴────────┘
50 rows in set. Elapsed: 5.512 sec. Processed 729.74 million rows, 4.93 GB (132.40 million rows/s., 894.18 MB/s.)
More forks less stars:
SELECT
repo_name,
sum(event_type = 'ForkEvent') AS forks,
sum(event_type = 'WatchEvent') AS stars,
round(forks / stars, 2) AS ratio
FROM github_events
WHERE event_type IN ('ForkEvent', 'WatchEvent')
GROUP BY repo_name
HAVING (stars > 100) AND (forks > 100)
ORDER BY ratio DESC
LIMIT 50
┌─repo_name─────────────────────────────────────────────────────────┬──forks─┬─stars─┬──ratio─┐
1. │ rdpeng/RepData_PeerAssessment1 │ 48407 │ 163 │ 296.98 │
2. │ ibm-developer-skills-network/jbbmo-Introduction-to-Git-and-GitHub │ 40736 │ 169 │ 241.04 │
3. │ LarryMad/recipes │ 38704 │ 191 │ 202.64 │
4. │ cyclic-software/starter-express-api │ 34927 │ 175 │ 199.58 │
5. │ rdpeng/ExData_Plotting1 │ 72439 │ 390 │ 185.74 │
6. │ ServiceNow/devtraining-needit-utah │ 18526 │ 109 │ 169.96 │
7. │ MicrosoftDocs/mslearn-tailspin-spacegame-web │ 40082 │ 253 │ 158.43 │
8. │ yandex-praktikum/backend_test_homework │ 23459 │ 154 │ 152.33 │
9. │ linuxacademy/devops-essentials-sample-app │ 25673 │ 179 │ 143.42 │
10. │ RedHatTraining/DO180-apps │ 49841 │ 352 │ 141.59 │
11. │ rdpeng/ProgrammingAssignment2 │ 180294 │ 1279 │ 140.96 │
12. │ actionsdemos/calculator │ 15062 │ 123 │ 122.46 │
13. │ RedHatTraining/DO288-apps │ 18906 │ 158 │ 119.66 │
14. │ theforage/forage-jpmc-swe-task-1 │ 23970 │ 227 │ 105.59 │
15. │ linuxacademy/cicd-pipeline-train-schedule-git │ 11041 │ 110 │ 100.37 │
16. │ elmas3/mao-seminar │ 10044 │ 103 │ 97.51 │
17. │ Solkaci/Uniswap-Arbitrage-Analysis │ 17025 │ 198 │ 85.98 │
18. │ mate-academy/layout_hello-world │ 9573 │ 113 │ 84.72 │
19. │ aws-samples/aws-elastic-beanstalk-express-js-sample │ 10985 │ 130 │ 84.5 │
20. │ microsoft/SmartHotel360-Website │ 8717 │ 111 │ 78.53 │
21. │ MicrosoftDocs/pipelines-java │ 12461 │ 164 │ 75.98 │
22. │ streamlit/streamlit-example │ 32088 │ 429 │ 74.8 │
23. │ MithunTechnologiesDevOps/maven-web-application │ 12731 │ 177 │ 71.93 │
24. │ yankils/hello-world │ 21709 │ 310 │ 70.03 │
25. │ soyHenry/fe-ct-prepcourse-fs │ 27216 │ 389 │ 69.96 │
26. │ goitacademy/nodejs-homework-template │ 7468 │ 113 │ 66.09 │
27. │ AlreadyBored/basic-js-ds │ 6513 │ 101 │ 64.49 │
28. │ onikiri955/shakrdp │ 7219 │ 114 │ 63.32 │
29. │ jenkins-docs/simple-python-pyinstaller-app │ 7398 │ 118 │ 62.69 │
30. │ alexaorrico/AirBnB_clone_v2 │ 9013 │ 144 │ 62.59 │
31. │ deadlyvipers/dojo_rules │ 13604 │ 219 │ 62.12 │
32. │ catc24/Zaluea │ 7959 │ 132 │ 60.3 │
33. │ RikkaApps/websites │ 16755 │ 281 │ 59.63 │
34. │ elmas3/pull-request-practice │ 6178 │ 107 │ 57.74 │
35. │ romacher/morse-decoder │ 8269 │ 145 │ 57.03 │
36. │ LinkedInLearning/learning-terraform-3087701 │ 7570 │ 135 │ 56.07 │
37. │ jenkins-docs/simple-java-maven-app │ 33479 │ 602 │ 55.61 │
38. │ justinmajetich/AirBnB_clone │ 10499 │ 203 │ 51.72 │
39. │ forcedotcom/sfdx-travisci │ 6078 │ 118 │ 51.51 │
40. │ google/it-cert-automation-practice │ 56993 │ 1168 │ 48.8 │
41. │ AlreadyBored/basic-js │ 13426 │ 277 │ 48.47 │
42. │ mikhama/core-js-101 │ 5450 │ 114 │ 47.81 │
43. │ LSPosed/WSAPatch │ 4981 │ 105 │ 47.44 │
44. │ vagabond-systems/forage-midas │ 11292 │ 242 │ 46.66 │
45. │ ironhack-labs/lab-javascript-basic-algorithms │ 5760 │ 124 │ 46.45 │
46. │ extinside/IPTV │ 6651 │ 151 │ 44.05 │
47. │ ironhack-labs/lab-javascript-vikings │ 6426 │ 150 │ 42.84 │
48. │ jleetutorial/maven-project │ 9888 │ 233 │ 42.44 │
49. │ archway-network/testnets │ 11476 │ 274 │ 41.88 │
50. │ openshift/nodejs-ex │ 6956 │ 171 │ 40.68 │
└───────────────────────────────────────────────────────────────────┴────────┴───────┴────────┘
50 rows in set. Elapsed: 5.712 sec. Processed 729.74 million rows, 4.93 GB (127.75 million rows/s., 862.80 MB/s.)
It's not easy to find some software products here.
The overall proportion between stars and forks:
:) SELECT sum(event_type = 'ForkEvent') AS forks, sum(event_type = 'WatchEvent') AS stars, round(stars / forks, 2) AS ratio FROM github_events WHERE event_type IN ('ForkEvent', 'WatchEvent')
┌─────forks─┬─────stars─┬─ratio─┐
│ 175056730 │ 554401518 │ 3.17 │
└───────────┴───────────┴───────┘
1 rows in set. Elapsed: 0.194 sec. Processed 729.74 million rows, 729.74 MB (3.76 billion rows/s., 3.76 GB/s.)
And it's higher for more popular repositories:
SELECT
sum(stars) AS stars,
sum(forks) AS forks,
round(stars / forks, 2) AS ratio
FROM
(
SELECT
sum(event_type = 'ForkEvent') AS forks,
sum(event_type = 'WatchEvent') AS stars
FROM github_events
WHERE event_type IN ('ForkEvent', 'WatchEvent')
GROUP BY repo_name
HAVING stars > 100
)
┌─────stars─┬────forks─┬─ratio─┐
│ 401635769 │ 91473424 │ 4.39 │
└───────────┴──────────┴───────┘
1 rows in set. Elapsed: 7.012 sec. Processed 729.74 million rows, 4.93 GB (104.07 million rows/s., 702.89 MB/s.)
Total number of issues on GitHub:
:) SELECT count() FROM github_events WHERE event_type = 'IssueCommentEvent'
┌───count()─┐
│ 555525370 │
└───────────┘
1 rows in set. Elapsed: 0.006 sec. Processed 294.18 thousand rows, 294.19 KB (46.19 million rows/s., 46.19 MB/s.)
The top repositories by issue comments:
:) SELECT repo_name, count() FROM github_events WHERE event_type = 'IssueCommentEvent' GROUP BY repo_name ORDER BY count() DESC LIMIT 50
┌─repo_name───────────────────────────────────────────────────┬─count()─┐
1. │ kubernetes/kubernetes │ 2088190 │
2. │ microsoft/winget-pkgs │ 1933560 │
3. │ plutointegtestuser/jsNPMPackage │ 1431548 │
4. │ openshift/release │ 1174664 │
5. │ elastic/kibana │ 1065221 │
6. │ rust-lang/rust │ 1055923 │
7. │ NixOS/nixpkgs │ 974900 │
8. │ Expensify/App │ 953064 │
9. │ apache/spark │ 943695 │
10. │ pytorch/pytorch │ 772964 │
11. │ flutter/flutter │ 738272 │
12. │ cms-sw/cmssw │ 718015 │
13. │ google-test/signclav2-probe-repo │ 689067 │
14. │ google-test/signcla-probe-repo │ 601068 │
15. │ cockroachdb/cockroach │ 599674 │
16. │ openshift/origin │ 595666 │
17. │ brianchandotcom/liferay-portal │ 572225 │
18. │ odoo/odoo │ 559651 │
19. │ home-assistant/core │ 549270 │
20. │ pingcap/tidb │ 543083 │
21. │ microsoft/vscode │ 537443 │
22. │ MicrosoftDocs/azure-docs │ 476003 │
23. │ dotnet/runtime │ 455887 │
24. │ istio/istio │ 447230 │
25. │ golang/go │ 444683 │
26. │ godotengine/godot │ 426189 │
27. │ nodejs/node │ 402021 │
28. │ openshift/openshift-docs │ 397755 │
29. │ llvm/llvm-project │ 383275 │
30. │ tensorflow/tensorflow │ 374269 │
31. │ Automattic/wp-calypso │ 366334 │
32. │ openjournals/joss-reviews │ 354944 │
33. │ tgstation/tgstation │ 352546 │
34. │ Homebrew/homebrew-core │ 351901 │
35. │ ansible/ansible │ 347876 │
36. │ prawn-test-staging-rw/mergequeue_service_test_commit_status │ 337256 │
37. │ elastic/elasticsearch │ 333641 │
38. │ kubernetes/website │ 331524 │
39. │ openshift/console │ 324301 │
40. │ apache/doris │ 324157 │
41. │ facebook/react-native │ 323745 │
42. │ WordPress/gutenberg │ 319317 │
43. │ kubevirt/kubevirt │ 314490 │
44. │ joomla/joomla-cms │ 307338 │
45. │ JuliaLang/julia │ 306063 │
46. │ magento/magento2 │ 305523 │
47. │ owncloud/core │ 297871 │
48. │ everypolitician/everypolitician-data │ 293711 │
49. │ DefinitelyTyped/DefinitelyTyped │ 291421 │
50. │ angular/angular │ 289476 │
└─────────────────────────────────────────────────────────────┴─────────┘
50 rows in set. Elapsed: 0.986 sec. Processed 555.69 million rows, 2.08 GB (563.41 million rows/s., 2.10 GB/s.)
The proportion between issue comments and issues:
SELECT
repo_name,
count() AS comments,
uniq(number) AS issues,
round(comments / issues, 2) AS ratio
FROM github_events
WHERE event_type = 'IssueCommentEvent'
GROUP BY repo_name
ORDER BY count() DESC
LIMIT 50
┌─repo_name───────────────────────────────────────────────────┬─comments─┬─issues─┬────ratio─┐
1. │ kubernetes/kubernetes │ 2088190 │ 126259 │ 16.54 │
2. │ microsoft/winget-pkgs │ 1933560 │ 348890 │ 5.54 │
3. │ plutointegtestuser/jsNPMPackage │ 1431548 │ 19 │ 75344.63 │
4. │ openshift/release │ 1174664 │ 77589 │ 15.14 │
5. │ elastic/kibana │ 1065221 │ 228582 │ 4.66 │
6. │ rust-lang/rust │ 1055923 │ 128958 │ 8.19 │
7. │ NixOS/nixpkgs │ 974900 │ 293816 │ 3.32 │
8. │ Expensify/App │ 953064 │ 80606 │ 11.82 │
9. │ apache/spark │ 943695 │ 48748 │ 19.36 │
10. │ pytorch/pytorch │ 772964 │ 159327 │ 4.85 │
11. │ flutter/flutter │ 738272 │ 154863 │ 4.77 │
12. │ cms-sw/cmssw │ 718015 │ 43630 │ 16.46 │
13. │ google-test/signclav2-probe-repo │ 689067 │ 551089 │ 1.25 │
14. │ google-test/signcla-probe-repo │ 601068 │ 447550 │ 1.34 │
15. │ cockroachdb/cockroach │ 599674 │ 141527 │ 4.24 │
16. │ openshift/origin │ 595666 │ 29809 │ 19.98 │
17. │ brianchandotcom/liferay-portal │ 572225 │ 147171 │ 3.89 │
18. │ odoo/odoo │ 559651 │ 219206 │ 2.55 │
19. │ home-assistant/core │ 549270 │ 107992 │ 5.09 │
20. │ pingcap/tidb │ 543083 │ 58533 │ 9.28 │
21. │ microsoft/vscode │ 537443 │ 154552 │ 3.48 │
22. │ MicrosoftDocs/azure-docs │ 476003 │ 124591 │ 3.82 │
23. │ dotnet/runtime │ 455887 │ 88984 │ 5.12 │
24. │ istio/istio │ 447230 │ 48220 │ 9.27 │
25. │ golang/go │ 444683 │ 65434 │ 6.8 │
26. │ godotengine/godot │ 426189 │ 98203 │ 4.34 │
27. │ nodejs/node │ 402021 │ 55694 │ 7.22 │
28. │ openshift/openshift-docs │ 397755 │ 68349 │ 5.82 │
29. │ llvm/llvm-project │ 383275 │ 119828 │ 3.2 │
30. │ tensorflow/tensorflow │ 374269 │ 62835 │ 5.96 │
31. │ Automattic/wp-calypso │ 366334 │ 98969 │ 3.7 │
32. │ openjournals/joss-reviews │ 354944 │ 10448 │ 33.97 │
33. │ tgstation/tgstation │ 352546 │ 55928 │ 6.3 │
34. │ Homebrew/homebrew-core │ 351901 │ 183072 │ 1.92 │
35. │ ansible/ansible │ 347876 │ 66024 │ 5.27 │
36. │ prawn-test-staging-rw/mergequeue_service_test_commit_status │ 337256 │ 170857 │ 1.97 │
37. │ elastic/elasticsearch │ 333641 │ 107761 │ 3.1 │
38. │ kubernetes/website │ 331524 │ 48611 │ 6.82 │
39. │ openshift/console │ 324301 │ 16149 │ 20.08 │
40. │ apache/doris │ 324157 │ 46362 │ 6.99 │
41. │ facebook/react-native │ 323745 │ 52647 │ 6.15 │
42. │ WordPress/gutenberg │ 319317 │ 65220 │ 4.9 │
43. │ kubevirt/kubevirt │ 314490 │ 16840 │ 18.68 │
44. │ joomla/joomla-cms │ 307338 │ 38567 │ 7.97 │
45. │ JuliaLang/julia │ 306063 │ 40337 │ 7.59 │
46. │ magento/magento2 │ 305523 │ 38903 │ 7.85 │
47. │ owncloud/core │ 297871 │ 29260 │ 10.18 │
48. │ everypolitician/everypolitician-data │ 293711 │ 161780 │ 1.82 │
49. │ DefinitelyTyped/DefinitelyTyped │ 291421 │ 60790 │ 4.79 │
50. │ angular/angular │ 289476 │ 64275 │ 4.5 │
└─────────────────────────────────────────────────────────────┴──────────┴────────┴──────────┘
50 rows in set. Elapsed: 1.394 sec. Processed 555.69 million rows, 4.30 GB (398.59 million rows/s., 3.08 GB/s.)
Kubernetes has the most active discussions among the top repositories — 16.5 comments per issue. In contrast, the repositories with the least comments per issue, the google-test/signcla*-probe-repo pair, are clearly not for discussions.
Now let's find the most active issues...
SELECT
repo_name,
number,
count() AS comments
FROM github_events
WHERE (event_type = 'IssueCommentEvent') AND (action = 'created')
GROUP BY
repo_name,
number
ORDER BY count() DESC
LIMIT 50
┌─repo_name────────────────────────────┬─number─┬─comments─┐
1. │ RahmaNiftaliyev/sourcegraph │ 26 │ 41210 │
2. │ jeremyjia/Games │ 743 │ 8120 │
3. │ vedantmgoyal9/vedantmgoyal9 │ 900 │ 7230 │
4. │ sgmelayu/storybook │ 242 │ 6178 │
5. │ Tiledesk/tiledesk-dashboard │ 94 │ 4842 │
6. │ kubeflow/pipelines │ 5071 │ 4656 │
7. │ cxflowtestuser/VB_3845 │ 48 │ 3873 │
8. │ RahmaNiftaliyev/docs │ 11 │ 3782 │
9. │ BTIGIT03/apache_airflow │ 137 │ 3378 │
10. │ cockpit-project/cockpit │ 7636 │ 3330 │
11. │ fuszenecker/CSharpDemo │ 5 │ 3329 │
12. │ openshift/tektoncd-pipeline-operator │ 494 │ 3307 │
13. │ ahmadsafirun/gradle │ 26 │ 3217 │
14. │ BTIGIT03/apache_airflow │ 218 │ 3216 │
15. │ KyonCN/MaaAssistantArknights │ 45 │ 3023 │
16. │ RahmaNiftaliyev/mui-x │ 127 │ 2971 │
17. │ RahmaNiftaliyev/mui-x │ 138 │ 2888 │
18. │ idaholab/moose │ 17417 │ 2847 │
19. │ cxflowtestuser/VB_3845 │ 9 │ 2812 │
20. │ cxflowtestuser/VB_3845 │ 12 │ 2794 │
21. │ cxflowtestuser/VB_3845 │ 11 │ 2794 │
22. │ cxflowtestuser/VB_3845 │ 13 │ 2779 │
23. │ vedantmgoyal2009/vedantmgoyal2009 │ 900 │ 2746 │
24. │ cxflowtestuser/VB_3845 │ 10 │ 2740 │
25. │ openshift/ovn-kubernetes │ 159 │ 2534 │
26. │ cockpit-project/cockpit │ 7635 │ 2499 │
27. │ cockpit-project/cockpit │ 3455 │ 2499 │
28. │ sauron-demo/sauron-demo │ 190 │ 2482 │
29. │ cxflowtestuser/VB_3845 │ 33 │ 2479 │
30. │ johnperez416/tensorflow │ 1 │ 2476 │
31. │ Jankyboy/go │ 40 │ 2472 │
32. │ cxflowtestuser/VB_3845 │ 7 │ 2455 │
33. │ Jankyboy/hhvm │ 11 │ 2445 │
34. │ RocketSurgeonsGuild/Nuke │ 504 │ 2441 │
35. │ MetaMask/metamask-extension │ 1 │ 2438 │
36. │ D0LLi/confusionRestaurant │ 8 │ 2433 │
37. │ DandelionSprout/adfilt │ 63 │ 2419 │
38. │ AlexRogalskiy/AlexRogalskiy │ 161 │ 2399 │
39. │ DandelionSprout/adfilt │ 7 │ 2395 │
40. │ AdguardTeam/AdguardFilters │ 162262 │ 2392 │
41. │ AlexRogalskiy/AlexRogalskiy │ 107 │ 2392 │
42. │ coinolio/coinolio │ 10 │ 2380 │
43. │ pytorch/test-infra │ 4220 │ 2380 │
44. │ RahmaNiftaliyev/nodejs.org │ 56 │ 2373 │
45. │ cxflowtestuser/VB_3845 │ 51 │ 2369 │
46. │ pytorch/test-infra │ 4232 │ 2366 │
47. │ Abaso007/social-app │ 173 │ 2363 │
48. │ cxflowtestuser/VB_3845 │ 53 │ 2361 │
49. │ Abaso007/node │ 20 │ 2351 │
50. │ cockroachdb/cockroach │ 136669 │ 2347 │
└──────────────────────────────────────┴────────┴──────────┘
50 rows in set. Elapsed: 8.178 sec. Processed 555.69 million rows, 4.34 GB (67.95 million rows/s., 530.99 MB/s.)
The top entries have tens of thousands of comments on a single issue with a very low number. It looks like technical comments made by some script. Nothing interesting here, and actually I did not find these comments on the GitHub website (maybe they have already been deleted).
I aimed to find some "epic bugs". Let's filter out by issue number:
SELECT
repo_name,
number,
count() AS comments
FROM github_events
WHERE (event_type = 'IssueCommentEvent') AND (action = 'created') AND (number > 10)
GROUP BY
repo_name,
number
ORDER BY count() DESC
LIMIT 50
┌─repo_name──────────────────────────────────────────────┬─number─┬─comments─┐
1. │ plutointegtestuser/jsNPMPackage │ 11 │ 95785 │
2. │ plutointegtestuser/jsNPMPackage │ 15 │ 95718 │
3. │ plutointegtestuser/jsNPMPackage │ 13 │ 95577 │
4. │ plutointegtestuser/jsNPMPackage │ 14 │ 95468 │
5. │ plutointegtestuser/jsNPMPackage │ 12 │ 95261 │
6. │ RahmaNiftaliyev/sourcegraph │ 26 │ 41210 │
7. │ mapped/action-vtl │ 212 │ 13058 │
8. │ xbrianlee/liferay-portal │ 553 │ 9085 │
9. │ keycloak/keycloak │ 60 │ 8407 │
10. │ RahmaNiftaliyev/material-ui │ 23 │ 8383 │
11. │ jeremyjia/Games │ 743 │ 8120 │
12. │ spacetelescope/jwst │ 6655 │ 7485 │
13. │ vedantmgoyal9/vedantmgoyal9 │ 900 │ 7230 │
14. │ Omicron666/godot │ 55 │ 7037 │
15. │ sgmelayu/storybook │ 242 │ 6178 │
16. │ spacetelescope/jwst │ 5897 │ 6050 │
17. │ openshift/oc │ 270 │ 5917 │
18. │ xbrianlee/liferay-portal │ 671 │ 5544 │
19. │ openfoodfacts/openfoodfacts-server │ 3767 │ 5098 │
20. │ Tiledesk/tiledesk-dashboard │ 94 │ 4842 │
21. │ openshift/oc │ 814 │ 4710 │
22. │ kubeflow/pipelines │ 5071 │ 4656 │
23. │ openshift/cluster-resource-override-admission-operator │ 29 │ 4562 │
24. │ ingvagabund/kubernetes │ 58 │ 4136 │
25. │ gkd-kit/inspect │ 46 │ 4096 │
26. │ Kitware/CDash │ 80 │ 4005 │
27. │ cxflowtestuser/VB_3845 │ 48 │ 3873 │
28. │ RahmaNiftaliyev/docs │ 11 │ 3782 │
29. │ GuilhermeStracini/POC-GHActions-CI-NetFramework │ 252 │ 3736 │
30. │ GlueOps/glueops-dev │ 193 │ 3664 │
31. │ shyamsantoki/langchain │ 20 │ 3614 │
32. │ BTIGIT03/apache_airflow │ 137 │ 3378 │
33. │ cockpit-project/cockpit │ 7636 │ 3330 │
34. │ openshift/tektoncd-pipeline-operator │ 494 │ 3307 │
35. │ GuilhermeStracini/POC-GHActions-CI-NetFramework │ 256 │ 3267 │
36. │ element-fi/elf-council-frontend │ 724 │ 3266 │
37. │ RahmaNiftaliyev/three.js │ 195 │ 3258 │
38. │ openshift/cluster-kube-apiserver-operator │ 1127 │ 3231 │
39. │ jessejay-ch/FluidFramework │ 58 │ 3229 │
40. │ ahmadsafirun/gradle │ 26 │ 3217 │
41. │ BTIGIT03/apache_airflow │ 218 │ 3216 │
42. │ xbrianlee/liferay-portal │ 369 │ 3201 │
43. │ ivanklee86/rubrical │ 27 │ 3187 │
44. │ D00Med/farlands │ 77 │ 3183 │
45. │ getlantern/forum │ 313 │ 3123 │
46. │ openshift/origin │ 25696 │ 3094 │
47. │ oceanbase/miniob │ 12 │ 3085 │
48. │ openshift/origin │ 18826 │ 3067 │
49. │ sgmelayu/kubernetes │ 1159 │ 3057 │
50. │ KyonCN/MaaAssistantArknights │ 45 │ 3023 │
└────────────────────────────────────────────────────────┴────────┴──────────┘
50 rows in set. Elapsed: 4.056 sec. Processed 555.69 million rows, 4.68 GB (136.99 million rows/s., 1.15 GB/s.)
I checked the first one and it also looks like some script gone out of control — five issues of the same repository with about 95 000 comments each.
Let's also count the number of comment authors and add a threshold:
SELECT
repo_name,
number,
count() AS comments,
uniq(actor_login) AS authors
FROM github_events
WHERE (event_type = 'IssueCommentEvent') AND (action = 'created') AND (number > 10)
GROUP BY
repo_name,
number
HAVING authors >= 10
ORDER BY count() DESC
LIMIT 50
┌─repo_name───────────────────────────────────────────────────┬─number─┬─comments─┬─authors─┐
1. │ xbrianlee/liferay-portal │ 553 │ 9085 │ 13 │
2. │ gkd-kit/inspect │ 46 │ 4096 │ 471 │
3. │ element-fi/elf-council-frontend │ 724 │ 3266 │ 2834 │
4. │ getlantern/forum │ 313 │ 3123 │ 1658 │
5. │ oceanbase/miniob │ 12 │ 3085 │ 163 │
6. │ githubschool/open-enrollment-classes-introduction-to-github │ 927 │ 3005 │ 1362 │
7. │ element-fi/elf-council-frontend │ 384 │ 2926 │ 2666 │
8. │ OpenKore/openkore │ 628 │ 2574 │ 373 │
9. │ element-fi/elf-council-frontend │ 722 │ 2562 │ 2363 │
10. │ D00Med/farlands │ 108 │ 2554 │ 11 │
11. │ ros-infrastructure/roswiki │ 139 │ 2483 │ 1427 │
12. │ DandelionSprout/adfilt │ 63 │ 2419 │ 64 │
13. │ allegro/allegro-api │ 1046 │ 2398 │ 630 │
14. │ AdguardTeam/AdguardFilters │ 162262 │ 2392 │ 34 │
15. │ Kodi-vStream/venom-xbmc-addons │ 2908 │ 2374 │ 12 │
16. │ uBlockOrigin/uAssets │ 20586 │ 2361 │ 398 │
17. │ TencentCloud/tencentcloud-sdk-nodejs │ 160 │ 2325 │ 1654 │
18. │ NixOS/nixpkgs │ 105153 │ 2017 │ 10 │
19. │ gcp/leela-zero │ 78 │ 1978 │ 97 │
20. │ andreiw/RaspberryPiPkg │ 12 │ 1939 │ 137 │
21. │ nervosnetwork/ckb │ 2372 │ 1815 │ 12 │
22. │ ros-infrastructure/roswiki │ 258 │ 1772 │ 1049 │
23. │ quarkusio/quarkus │ 6588 │ 1771 │ 36 │
24. │ ValveSoftware/Proton │ 4450 │ 1747 │ 374 │
25. │ ValveSoftware/halflife │ 387 │ 1716 │ 60 │
26. │ MarlinFirmware/Marlin │ 7076 │ 1695 │ 120 │
27. │ pytorch/pytorch │ 77764 │ 1673 │ 1227 │
28. │ ValveSoftware/Proton │ 37 │ 1657 │ 355 │
29. │ linux-surface/linux-surface │ 91 │ 1655 │ 178 │
30. │ ValveSoftware/Proton │ 3291 │ 1651 │ 311 │
31. │ OpenKore/openkore │ 460 │ 1643 │ 239 │
32. │ dtcxzyw/llvm-opt-benchmark │ 1312 │ 1547 │ 25 │
33. │ finndev/PokeBuddy │ 227 │ 1546 │ 119 │
34. │ OpenNyAI/Jugalbandi-Manager │ 174 │ 1531 │ 10 │
35. │ iiordanov/remote-desktop-clients │ 39 │ 1524 │ 1056 │
36. │ 996icu/996.ICU │ 20 │ 1486 │ 1142 │
37. │ Apostolique/Agar.io-bot │ 380 │ 1482 │ 108 │
38. │ openshift/cluster-network-operator │ 887 │ 1466 │ 10 │
39. │ OpenKore/openkore │ 221 │ 1437 │ 323 │
40. │ npm/registry │ 255 │ 1409 │ 420 │
41. │ acemod/ACE3 │ 3594 │ 1348 │ 424 │
42. │ kubernetes/kubernetes │ 46254 │ 1345 │ 11 │
43. │ OPS-E2E-PPE/E2E_DocFxV3 │ 48 │ 1309 │ 12 │
44. │ BeepIsla/csgo-commend-bot │ 428 │ 1304 │ 114 │
45. │ OPS-E2E-PPE/E2E_DocsBranch_Dynamic │ 11358 │ 1292 │ 11 │
46. │ rust-lang/rust │ 112049 │ 1270 │ 13 │
47. │ istio/istio │ 12276 │ 1269 │ 11 │
48. │ uBlockOrigin/uAssets │ 7636 │ 1265 │ 184 │
49. │ reactjs/rfcs │ 68 │ 1264 │ 333 │
50. │ Fate-Grand-Automata/FGA │ 1347 │ 1256 │ 303 │
└─────────────────────────────────────────────────────────────┴────────┴──────────┴─────────┘
50 rows in set. Elapsed: 6.650 sec. Processed 529.47 million rows, 6.25 GB (79.62 million rows/s., 939.75 MB/s.)
The gems are further down the list now, because the top is taken by airdrop farming — 3266 comments from 2834 different people on a single issue.
SELECT
concat('https://github.com/', repo_name, '/issues/', toString(number)) AS URL,
max(comments),
argMax(authors, comments) AS authors,
argMax(number, comments) AS number,
sum(stars) AS stars
FROM
(
SELECT *
FROM
(
SELECT
repo_name,
number,
count() AS comments,
uniq(actor_login) AS authors
FROM github_events
WHERE (event_type = 'IssueCommentEvent') AND (action = 'created') AND (number > 10)
GROUP BY
repo_name,
number
HAVING authors >= 10
) AS t1
INNER JOIN
(
SELECT
repo_name,
count() AS stars
FROM github_events
WHERE event_type = 'WatchEvent'
GROUP BY repo_name
HAVING stars > 10000
) AS t2 USING (repo_name)
)
GROUP BY repo_name
ORDER BY stars DESC
LIMIT 50
┌─URL─────────────────────────────────────────────────────────────────┬─max(comments)─┬─authors─┬─number─┬─────stars─┐
1. │ https://github.com/flutter/flutter/issues/51752 │ 539 │ 50 │ 51752 │ 832634892 │
2. │ https://github.com/kubernetes/kubernetes/issues/46254 │ 1345 │ 11 │ 46254 │ 536828499 │
3. │ https://github.com/rust-lang/rust/issues/112049 │ 1270 │ 13 │ 112049 │ 441197838 │
4. │ https://github.com/tensorflow/tensorflow/issues/22 │ 633 │ 157 │ 22 │ 423788904 │
5. │ https://github.com/facebook/react-native/issues/4968 │ 512 │ 305 │ 4968 │ 403416180 │
6. │ https://github.com/golang/go/issues/15292 │ 875 │ 162 │ 15292 │ 250539759 │
7. │ https://github.com/microsoft/vscode/issues/191229 │ 1023 │ 722 │ 191229 │ 193238388 │
8. │ https://github.com/facebook/react/issues/13991 │ 541 │ 413 │ 13991 │ 187324144 │
9. │ https://github.com/nodejs/node/issues/5020 │ 471 │ 79 │ 5020 │ 176394774 │
10. │ https://github.com/home-assistant/core/issues/99947 │ 516 │ 255 │ 99947 │ 170784068 │
11. │ https://github.com/angular/angular/issues/5689 │ 307 │ 83 │ 5689 │ 164847800 │
12. │ https://github.com/godotengine/godot/issues/16863 │ 514 │ 90 │ 16863 │ 149122155 │
13. │ https://github.com/pytorch/pytorch/issues/77764 │ 1673 │ 1227 │ 77764 │ 99634320 │
14. │ https://github.com/ant-design/ant-design/issues/13848 │ 1051 │ 577 │ 13848 │ 98415330 │
15. │ https://github.com/vercel/next.js/issues/48748 │ 576 │ 262 │ 48748 │ 93972175 │
16. │ https://github.com/FortAwesome/Font-Awesome/issues/7574 │ 410 │ 328 │ 7574 │ 84885845 │
17. │ https://github.com/996icu/996.ICU/issues/20 │ 1486 │ 1142 │ 20 │ 78792896 │
18. │ https://github.com/electron/electron/issues/21457 │ 276 │ 95 │ 21457 │ 77822338 │
19. │ https://github.com/Microsoft/vscode/issues/224 │ 418 │ 151 │ 224 │ 74044194 │
20. │ https://github.com/bitcoin/bitcoin/issues/26525 │ 242 │ 64 │ 26525 │ 73768240 │
21. │ https://github.com/microsoft/PowerToys/issues/25595 │ 261 │ 94 │ 25595 │ 69682830 │
22. │ https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/2449 │ 440 │ 94 │ 2449 │ 62073200 │
23. │ https://github.com/atom/atom/issues/2956 │ 238 │ 95 │ 2956 │ 55757628 │
24. │ https://github.com/grafana/grafana/issues/2209 │ 299 │ 136 │ 2209 │ 55079340 │
25. │ https://github.com/huggingface/transformers/issues/36979 │ 148 │ 96 │ 36979 │ 51114753 │
26. │ https://github.com/vuejs/vue/issues/2873 │ 214 │ 63 │ 2873 │ 49521663 │
27. │ https://github.com/expo/expo/issues/28656 │ 370 │ 81 │ 28656 │ 47469348 │
28. │ https://github.com/microsoft/TypeScript/issues/16577 │ 227 │ 63 │ 16577 │ 44959662 │
29. │ https://github.com/ansible/ansible/issues/13262 │ 164 │ 94 │ 13262 │ 43993763 │
30. │ https://github.com/cypress-io/cypress/issues/136 │ 480 │ 265 │ 136 │ 41995930 │
31. │ https://github.com/JuliaLang/julia/issues/24990 │ 469 │ 59 │ 24990 │ 41693722 │
32. │ https://github.com/hashicorp/terraform/issues/13022 │ 307 │ 163 │ 13022 │ 41438298 │
33. │ https://github.com/gatsbyjs/gatsby/issues/19618 │ 177 │ 86 │ 19618 │ 40066400 │
34. │ https://github.com/freeCodeCamp/freeCodeCamp/issues/42256 │ 132 │ 35 │ 42256 │ 38882298 │
35. │ https://github.com/webpack/webpack/issues/9802 │ 508 │ 114 │ 9802 │ 37422600 │
36. │ https://github.com/twbs/bootstrap/issues/21943 │ 148 │ 121 │ 21943 │ 36232920 │
37. │ https://github.com/ollama/ollama/issues/2453 │ 197 │ 42 │ 2453 │ 35230020 │
38. │ https://github.com/angular/angular-cli/issues/5618 │ 426 │ 177 │ 5618 │ 34580012 │
39. │ https://github.com/prisma/prisma/issues/2377 │ 353 │ 199 │ 2377 │ 34138146 │
40. │ https://github.com/storybookjs/storybook/issues/9216 │ 169 │ 55 │ 9216 │ 34064292 │
41. │ https://github.com/neovim/neovim/issues/1820 │ 322 │ 21 │ 1820 │ 34056754 │
42. │ https://github.com/NixOS/nixpkgs/issues/105153 │ 2017 │ 10 │ 105153 │ 32694201 │
43. │ https://github.com/microsoft/terminal/issues/1375 │ 315 │ 151 │ 1375 │ 31711095 │
44. │ https://github.com/docker/docker/issues/9176 │ 379 │ 77 │ 9176 │ 31537485 │
45. │ https://github.com/facebook/create-react-app/issues/11756 │ 283 │ 173 │ 11756 │ 30806888 │
46. │ https://github.com/tensorflow/models/issues/9033 │ 150 │ 29 │ 9033 │ 30125880 │
47. │ https://github.com/rails/rails/issues/505 │ 135 │ 20 │ 505 │ 29568324 │
48. │ https://github.com/strapi/strapi/issues/3946 │ 108 │ 87 │ 3946 │ 29044860 │
49. │ https://github.com/elastic/elasticsearch/issues/4915 │ 199 │ 156 │ 4915 │ 28992810 │
50. │ https://github.com/Microsoft/TypeScript/issues/202 │ 272 │ 64 │ 202 │ 27931938 │
└─────────────────────────────────────────────────────────────────────┴───────────────┴─────────┴────────┴───────────┘
50 rows in set. Elapsed: 7.289 sec. Processed 1.11 billion rows, 9.20 GB (152.31 million rows/s., 1.26 GB/s.)
I made this query for you and I hope you will find the most crucial discussions here. Enjoy!
It's rarely the case when a single commit has comments.
Here are the top repositories by the number of commit comments:
SELECT
repo_name,
count() AS comments,
uniq(actor_login) AS authors
FROM github_events
WHERE event_type = 'CommitCommentEvent'
GROUP BY repo_name
ORDER BY count() DESC
LIMIT 50
┌─repo_name─────────────────────────────────┬─comments─┬─authors─┐
1. │ bambootest-bot/githubtest │ 286031 │ 2 │
2. │ iterativv/NostalgiaForInfinity │ 235116 │ 20 │
3. │ NREL/EnergyPlus │ 109119 │ 54 │
4. │ dcos/dcos │ 99251 │ 16 │
5. │ xamarin/xamarin-macios │ 95834 │ 36 │
6. │ ngocnhan2003/ngocnhan2003 │ 71154 │ 1 │
7. │ PaddlePaddle/Paddle │ 57901 │ 61 │
8. │ siggetest/githubtest │ 52165 │ 1 │
9. │ sxadxsx/upptime │ 50593 │ 1 │
10. │ atlase2eintegration/express-simple-server │ 50403 │ 5 │
11. │ w4ctech/front-end-rss │ 46618 │ 1 │
12. │ kwcckw/CogAlg │ 44578 │ 5 │
13. │ JesusFilm/core │ 41452 │ 4 │
14. │ napari/npe2api │ 38224 │ 2 │
15. │ miabot/galleries.csv │ 37328 │ 2 │
16. │ PipecraftNet/v2hot.github.io │ 34273 │ 1 │
17. │ openjdk/jdk │ 33899 │ 287 │
18. │ mozilla/rust │ 33659 │ 82 │
19. │ GuilhermeManteigas/DSC-website │ 31733 │ 1 │
20. │ cypress-io/cypress │ 31540 │ 58 │
21. │ 1024huijia/TestSome │ 31481 │ 1 │
22. │ joschi/asdf-vm-twitter │ 31292 │ 3 │
23. │ manifoldmarkets/manifold │ 29180 │ 15 │
24. │ deadex-ng/lobste.rs-graph │ 28275 │ 1 │
25. │ leetcode-pp/leetcode-pp-node │ 28148 │ 1 │
26. │ microsoft/garnet │ 26128 │ 7 │
27. │ gonlad-x/validator_queue_monitoring │ 25411 │ 1 │
28. │ TrinityCore/TrinityCore │ 25137 │ 1620 │
29. │ doourdo27/linux-command-autobuild │ 24307 │ 1 │
30. │ etheralpha/validatorqueue-com │ 22924 │ 1 │
31. │ juruo888/juruo888.github.io │ 22516 │ 1 │
32. │ akky/statusbase-nuxt │ 22488 │ 1 │
33. │ jupitersh/jupitersh.github.io │ 22434 │ 2 │
34. │ kubernetes/kubernetes │ 21729 │ 362 │
35. │ tgrzesik/express-simple-server │ 21273 │ 6 │
36. │ tahaluindo/Google-Trends-Keywords │ 19472 │ 1 │
37. │ voidfiles/utah-highschool-water-polo │ 19220 │ 1 │
38. │ JuliaLang/julia │ 19180 │ 300 │
39. │ wyattowalsh/wyattowalsh │ 18955 │ 1 │
40. │ skillrecordings/products │ 18900 │ 5 │
41. │ eyelly-wu/hosts │ 18744 │ 1 │
42. │ OrdinaNederland/Stichting-NUTwente │ 18490 │ 1 │
43. │ mozilla-mobile/android-components │ 17004 │ 14 │
44. │ stdlib-js/stdlib │ 16952 │ 18 │
45. │ diarmuidie/headless-nextjs │ 16672 │ 6 │
46. │ yztutu/zhihu-trending-top-search │ 16624 │ 1 │
47. │ yztutu/zhihu-trending-hot-questions │ 16406 │ 1 │
48. │ yztutu/weibo-trending-hot-search │ 16399 │ 1 │
49. │ SJ-Kumar/SJ-Kumar │ 16245 │ 1 │
50. │ rust-lang/rust │ 16139 │ 364 │
└───────────────────────────────────────────┴──────────┴─────────┘
50 rows in set. Elapsed: 0.318 sec. Processed 34.60 million rows, 394.60 MB (108.88 million rows/s., 1.24 GB/s.)
If there are many comments but a small number of comment authors, usually they are comments from the CI robot.
Here are the "hottest" commits:
SELECT
concat('https://github.com/', repo_name, '/commit/', commit_id) AS URL,
count() AS comments,
uniq(actor_login) AS authors
FROM github_events
WHERE (event_type = 'CommitCommentEvent') AND notEmpty(commit_id)
GROUP BY
repo_name,
commit_id
HAVING authors >= 10
ORDER BY count() DESC
LIMIT 50
┌─URL─────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─comments─┬─authors─┐
1. │ https://github.com/cldcvr/codepipes-tutorials/commit/3bcee35cc65ced2c347e7556f6e2ab4e95c6184f │ 1000 │ 11 │
2. │ https://github.com/MiCode/Xiaomi_Kernel_OpenSource/commit/d58d63d69575cfcd235dbc1088c452499b8ddfca │ 976 │ 50 │
3. │ https://github.com/Kile/Killua/commit/e951a3341db6d6f7ac95688dee2a06c91e5923fe │ 866 │ 92 │
4. │ https://github.com/pi-apps/pi-explorer/commit/d42068273513f86364fe4cd9dde7c390bd90ce0c │ 618 │ 189 │
5. │ https://github.com/nixxquality/WebMConverter/commit/c1ac0baac06fa7175677a4a1bf65860a84708d67 │ 476 │ 159 │
6. │ https://github.com/torvalds/linux/commit/6e90b675cf942e50c70e8394dfb5862975c3b3b2 │ 446 │ 177 │
7. │ https://github.com/PolyMC/PolyMC/commit/ccf282593dcdbe189c99b81b8bc90cb203aed3ee │ 441 │ 158 │
8. │ https://github.com/torvalds/linux/commit/8a104f8b5867c682d994ffa7a74093c54469c11f │ 414 │ 181 │
9. │ https://github.com/SEI-ATL/select-best-player/commit/fdc6e0e073fa30acf2041ca0b1c9fecd662d88c8 │ 408 │ 31 │
10. │ https://github.com/mikegithubber/my-first-github-repository/commit/b5ed1a1bc1a737c13818847529fb21ed615e6e66 │ 391 │ 383 │
11. │ https://github.com/twitter/the-algorithm/commit/ec83d01dcaebf369444d75ed04b3625a0a645eb9 │ 389 │ 300 │
12. │ https://github.com/goagent/goagent/commit/e492ed0283f5cde7cf71d7ac47429f64aa48cd13 │ 380 │ 348 │
13. │ https://github.com/egoing2/kt230201-2/commit/000d6e0ea02bae54c5fca656b5195a330e929f2d │ 371 │ 221 │
14. │ https://github.com/therealgliz/blooket-hacks/commit/fef5c9701b665cb4483e5ef69f01dc04477a7ab2 │ 364 │ 211 │
15. │ https://github.com/Marak/colors.js/commit/074a0f8ed0c31c35d13d28632bd8a049ff136fb6 │ 356 │ 244 │
16. │ https://github.com/apple/swift/commit/18844bc65229786b96b89a9fc7739c0fc897905e │ 352 │ 338 │
17. │ https://github.com/git/git/commit/e83c5163316f89bfbde7d9ab23ca2e25604af290 │ 350 │ 323 │
18. │ https://github.com/unixpickle/kahoot-hack/commit/837ffedbc5ed971e9460d47bbe08fade1fa9056f │ 341 │ 163 │
19. │ https://github.com/chromium/chromium/commit/6f47a22906b2899412e79a2727355efa9cc8f5bd │ 338 │ 210 │
20. │ https://github.com/mosh-hamedani/vidly-mvc-5/commit/b727a26e1b4b88abe84a8b48208fec537db2ed43 │ 334 │ 185 │
21. │ https://github.com/shadowsocks/shadowsocks/commit/938bba32a4008bdde9c064dda6a0597987ddef54 │ 309 │ 258 │
22. │ https://github.com/3dshax/ctr/commit/bcb3734b9a26d0fe7ef66f7d3814406fee797303 │ 300 │ 215 │
23. │ https://github.com/ninjamuffin99/Funkin/commit/a083938ac803a91fbb15f03e432dea620f8a3b4a │ 294 │ 22 │
24. │ https://github.com/hklcf/myTV-api/commit/375fa7520455b53fad622c0149a473ebee048e15 │ 282 │ 28 │
25. │ https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/commit/a047be85247755cdbe0acce6f1dafc8beb84f2ac │ 282 │ 248 │
26. │ https://github.com/zhaohmng/-21-/commit/0cc2b2a546ad51fb360ba800c67b057ea3270869 │ 263 │ 199 │
27. │ https://github.com/NecronomiconCoding/NecroBot/commit/26e57d9feac57ab372ac466ef6cd66218faa475c │ 262 │ 75 │
28. │ https://github.com/mlouielu/cn_constitution_2018/commit/646c76a573ad49414e708c091393ddb7c437f286 │ 245 │ 181 │
29. │ https://github.com/DearZh/MonkeyScript/commit/bb2c8fb02ef9e036d21e9fb401a300113f2c4cf9 │ 244 │ 177 │
30. │ https://github.com/kohsuke/sandbox-ant/commit/8ae38db0ea5837313ab5f39d43a6f73de3bd9000 │ 238 │ 67 │
31. │ https://github.com/yuuwill/1024app-android/commit/0b02f1815686fa657d7b4a9583019fe4d203b80a │ 234 │ 147 │
32. │ https://github.com/iride2020/iRide-Token/commit/8f810b3700a24de0ee4f945ef84be7b2f1605610 │ 228 │ 183 │
33. │ https://github.com/j0nathanwither5/myfirstprogram/commit/e2609bb0f43c693ef4eea40e31fa0168e5c40470 │ 227 │ 110 │
34. │ https://github.com/octocat/Spoon-Knife/commit/d0dd1f61b33d64e29d8bc1372a94ef6a2fee76a9 │ 224 │ 163 │
35. │ https://github.com/bitSmiley-protocol/trumeme-tokens/commit/786a6fb0eb8bf238df1b8cfd3977237f0d150b44 │ 210 │ 157 │
36. │ https://github.com/rails/rails/commit/b83965785db1eec019edf1fc272b1aa393e6dc57 │ 206 │ 129 │
37. │ https://github.com/mosh-hamedani/vidly-mvc-5/commit/93f39efe3ef0ce9c25ef09ebef60ad2dfc1fe7f3 │ 203 │ 116 │
38. │ https://github.com/instrumental-finance/airdrop-addresses/commit/256f94b8d3a540b6626fae10890de23289fb264a │ 201 │ 133 │
39. │ https://github.com/pi-apps/pi-explorer/commit/edf6ad486fd36f762c7e4cf83b410cca060d57bd │ 196 │ 25 │
40. │ https://github.com/tmux/tmux/commit/d2b35e19cdd61d163d26c4babccc1550e72a9623 │ 196 │ 188 │
41. │ https://github.com/mosh-hamedani/vidly-mvc-5/commit/acb8c0a0a27255e3f0af85ab2f687e0e9b82b6db │ 191 │ 107 │
42. │ https://github.com/mikegithubber/my-first-github-repository/commit/f38cf543fc7b72dc6ea50f61010827fc8d8d23e9 │ 187 │ 184 │
43. │ https://github.com/getaurora/download/commit/91979ba9b4c84265b1390bfb258c16d492e2e855 │ 185 │ 81 │
44. │ https://github.com/jayphelps/git-blame-someone-else/commit/e5cfe4bb2190a2ae406d5f0b8f49c32ac0f01cd7 │ 183 │ 134 │
45. │ https://github.com/i2ii/i/commit/075295c026c99710c4755c3ce94fe66b817499ba │ 181 │ 62 │
46. │ https://github.com/Grinchiest/operation-bag-of-toys/commit/41615462e4fdc0ceeb4ef1bec693ec3de1125ed2 │ 179 │ 175 │
47. │ https://github.com/udacity/ud843-QuakeReport/commit/14541da929b771249ea8209698d324b61bbeee7e │ 178 │ 102 │
48. │ https://github.com/DuGuQiuBai/Java/commit/0e5c44c62d28371a8d74623b5373ee47ff23509b │ 172 │ 111 │
49. │ https://github.com/Glixerz/Blooket-Hacks/commit/feb68f899878aa558f6902b64a4b338dc9389e9e │ 169 │ 74 │
50. │ https://github.com/londonappbrewery/destini-challenge-completed/commit/69ed867992fc05f13a4fbef452173a956312993d │ 168 │ 131 │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴──────────┴─────────┘
50 rows in set. Elapsed: 11.384 sec. Processed 34.60 million rows, 1.73 GB (3.04 million rows/s., 152.34 MB/s.)
The first entry is a tutorial repository with exactly 1000 comments on one commit — a suspiciously round number for something organic. The Xiaomi kernel sources in 2nd place is where people go to ask for the sources of their own device. Linux is still in the list, and so are many politically loaded discussions.
SELECT
concat('https://github.com/', repo_name, '/pull/', toString(number)) AS URL,
uniq(actor_login) AS authors
FROM github_events
WHERE (event_type = 'PullRequestReviewCommentEvent') AND (action = 'created')
GROUP BY
repo_name,
number
ORDER BY authors DESC
LIMIT 50
┌─URL────────────────────────────────────────────────────────────────────┬─authors─┐
1. │ https://github.com/torvalds/linux/pull/684 │ 76 │
2. │ https://github.com/blondiebytes/calculator-python/pull/2 │ 73 │
3. │ https://github.com/pingcap/docs-cn/pull/12652 │ 57 │
4. │ https://github.com/NixOS/rfcs/pull/49 │ 53 │
5. │ https://github.com/sunpy/sunpy/pull/3391 │ 52 │
6. │ https://github.com/dashbitco/mix_phx_gen_auth_demo/pull/1 │ 51 │
7. │ https://github.com/reactjs/rfcs/pull/2 │ 44 │
8. │ https://github.com/bitcoin/bips/pull/2017 │ 42 │
9. │ https://github.com/rust-lang/rfcs/pull/3463 │ 42 │
10. │ https://github.com/php/php-rfcs/pull/1 │ 40 │
11. │ https://github.com/matrix-org/matrix-doc/pull/1772 │ 40 │
12. │ https://github.com/NixOS/rfcs/pull/166 │ 38 │
13. │ https://github.com/rust-lang/rfcs/pull/3762 │ 38 │
14. │ https://github.com/danielmiessler/SecLists/pull/155 │ 38 │
15. │ https://github.com/pingcap/docs-cn/pull/13312 │ 38 │
16. │ https://github.com/pingcap/docs-cn/pull/11115 │ 37 │
17. │ https://github.com/nrfconnect/sdk-nrf/pull/21339 │ 36 │
18. │ https://github.com/pingcap/docs-cn/pull/11835 │ 36 │
19. │ https://github.com/RikkaApps/websites/pull/79 │ 36 │
20. │ https://github.com/github/site-policy/pull/397 │ 36 │
21. │ https://github.com/symfony/symfony/pull/33553 │ 35 │
22. │ https://github.com/kubernetes/community/pull/306 │ 34 │
23. │ https://github.com/pingcap/docs-cn/pull/8061 │ 34 │
24. │ https://github.com/rust-lang/rfcs/pull/2850 │ 33 │
25. │ https://github.com/pingcap/docs-cn/pull/15667 │ 33 │
26. │ https://github.com/pingcap/docs-cn/pull/12100 │ 33 │
27. │ https://github.com/modelcontextprotocol/modelcontextprotocol/pull/284 │ 33 │
28. │ https://github.com/neu-cs4530-fall2024/Week5-CodeReviewActivity/pull/2 │ 32 │
29. │ https://github.com/rust-lang/rfcs/pull/3810 │ 32 │
30. │ https://github.com/date-fns/date-fns/pull/1671 │ 32 │
31. │ https://github.com/modelcontextprotocol/specification/pull/206 │ 32 │
32. │ https://github.com/pingcap/docs-cn/pull/16664 │ 32 │
33. │ https://github.com/NixOS/nixpkgs/pull/140136 │ 32 │
34. │ https://github.com/woowacourse-precourse/java-racingcar-6/pull/356 │ 32 │
35. │ https://github.com/github/site-policy/pull/582 │ 32 │
36. │ https://github.com/rust-lang/rfcs/pull/3090 │ 32 │
37. │ https://github.com/cardano-foundation/CIPs/pull/380 │ 31 │
38. │ https://github.com/kubernetes/enhancements/pull/4642 │ 31 │
39. │ https://github.com/bontu-fufa/code-review-practice/pull/1 │ 31 │
40. │ https://github.com/github/site-policy/pull/1 │ 31 │
41. │ https://github.com/h-beeen/java-christmas-6-h-beeen/pull/1 │ 31 │
42. │ https://github.com/pingcap/docs-cn/pull/17514 │ 31 │
43. │ https://github.com/symfony/symfony/pull/11882 │ 30 │
44. │ https://github.com/996icu/996.ICU/pull/25509 │ 30 │
45. │ https://github.com/dotnet/designs/pull/92 │ 30 │
46. │ https://github.com/tc39/ecma262/pull/1062 │ 30 │
47. │ https://github.com/rust-lang/rfcs/pull/517 │ 29 │
48. │ https://github.com/matrix-org/matrix-spec-proposals/pull/2545 │ 29 │
49. │ https://github.com/dotnet/designs/pull/281 │ 29 │
50. │ https://github.com/discordjs/discord.js/pull/5106 │ 29 │
└────────────────────────────────────────────────────────────────────────┴─────────┘
50 rows in set. Elapsed: 2.770 sec. Processed 162.48 million rows, 1.70 GB (58.66 million rows/s., 614.42 MB/s.)
The first entry is true insanity. But there are many interesting ones like the proposal to add inline assembly to Rust.
SELECT
actor_login,
count() AS c,
uniq(repo_name) AS repos
FROM github_events
WHERE event_type = 'PushEvent'
GROUP BY actor_login
ORDER BY c DESC
LIMIT 50
┌─actor_login───────────────────┬─────────c─┬───repos─┐
1. │ github-actions[bot] │ 892873545 │ 3400994 │
2. │ LombiqBot │ 79790401 │ 161 │
3. │ renovate[bot] │ 44954646 │ 257283 │
4. │ swa-runner-app[bot] │ 29053308 │ 3403321 │
5. │ pull[bot] │ 26183304 │ 452267 │
6. │ ttgds3asu │ 22465617 │ 74 │
7. │ dependabot[bot] │ 16355204 │ 1329688 │
8. │ censameesss │ 14001046 │ 30 │
9. │ direwolf-github │ 12316283 │ 6229952 │
10. │ B74LABgit │ 9721447 │ 5 │
11. │ renovate-bot │ 9704714 │ 5616 │
12. │ Hall-1910 │ 8960998 │ 1 │
13. │ frdpzk2 │ 7738170 │ 1 │
14. │ commit-b0t │ 6883222 │ 3 │
15. │ mkarmark │ 6492544 │ 3159341 │
16. │ hotspotlab │ 6192364 │ 2 │
17. │ sidkgndfge │ 5670466 │ 12 │
18. │ RayhanZuck │ 5155096 │ 33 │
19. │ SoliSpirit │ 4999471 │ 6 │
20. │ freeukapp │ 4952610 │ 1 │
21. │ OpenLocalizationTest │ 4869867 │ 713 │
22. │ freefastconnect │ 4805998 │ 1 │
23. │ Alexey-Gorulev-kernelics │ 4747250 │ 19 │
24. │ rmayr │ 4620734 │ 18 │
25. │ kpnsec │ 4560256 │ 1 │
26. │ znyt │ 4516648 │ 116 │
27. │ hafsa1319 │ 4511054 │ 434 │
28. │ freecall2019 │ 4287089 │ 1 │
29. │ BurmeseTV │ 3983166 │ 19 │
30. │ radiolo-gical │ 3848170 │ 20 │
31. │ adi224foreverg │ 3820225 │ 15 │
32. │ supermobiteam2 │ 3601240 │ 1 │
33. │ l1nky-1337 │ 3417772 │ 11 │
34. │ vpnsuperapp │ 3397760 │ 2 │
35. │ breakingheatmap │ 3319832 │ 1 │
36. │ lovable-dev[bot] │ 3301244 │ 135980 │
37. │ donaldsouza │ 3215012 │ 4 │
38. │ regro-cf-autotick-bot │ 3057621 │ 2292 │
39. │ algotables │ 3041971 │ 29 │
40. │ happyfish2024 │ 3001821 │ 1 │
41. │ datarods-svc │ 2904363 │ 8 │
42. │ dependabot-preview[bot] │ 2903337 │ 48091 │
43. │ aws-connector-for-github[bot] │ 2867525 │ 826 │
44. │ newstools │ 2837387 │ 1029 │
45. │ Helikopter-Bojowy │ 2766829 │ 4 │
46. │ mdmaid69 │ 2765365 │ 1 │
47. │ CelestiaNFT │ 2748177 │ 3 │
48. │ waqar0345 │ 2746578 │ 1 │
49. │ lorenzo132 │ 2706902 │ 98 │
50. │ MathiasExorde │ 2699227 │ 39 │
└───────────────────────────────┴───────────┴─────────┘
50 rows in set. Elapsed: 13.845 sec. Processed 6.39 billion rows, 41.17 GB (461.81 million rows/s., 2.97 GB/s.)
Obviously most of them are bots. github-actions[bot] alone made 892 million pushes, and direwolf-github has pushed to 6 229 952 repositories. Even the accounts named like people are clearly not.
How can we filter out bots? Let's add a threshold on the number of repositories. Let's also count only those who created at least two issues and gave at least two stars. And also output the favorite repository for every user. And also count only across the top 10k repositories.
SELECT
actor_login,
sum(event_type = 'PushEvent') AS c,
uniqIf(repo_name, event_type = 'PushEvent') AS repos,
sum(event_type = 'IssuesEvent') AS issues,
sum(event_type = 'WatchEvent') AS stars,
anyHeavy(repo_name)
FROM github_events
WHERE (event_type IN ('PushEvent', 'IssuesEvent', 'WatchEvent')) AND (repo_name IN
(
SELECT repo_name
FROM github_events
WHERE event_type = 'WatchEvent'
GROUP BY repo_name
ORDER BY count() DESC
LIMIT 10000
))
GROUP BY actor_login
HAVING (repos < 10000) AND (issues > 1) AND (stars > 1)
ORDER BY c DESC
LIMIT 50
┌─actor_login───────┬──────c─┬─repos─┬─issues─┬─stars─┬─anyHeavy(repo_name)────────────────────┐
1. │ ornicar │ 119500 │ 2 │ 6906 │ 96 │ ornicar/lila │
2. │ uqs │ 94755 │ 1 │ 9 │ 10 │ freebsd/freebsd │
3. │ Gargron │ 73284 │ 2 │ 2906 │ 105 │ tootsuite/mastodon │
4. │ eldy │ 57310 │ 1 │ 2292 │ 16 │ Dolibarr/dolibarr │
5. │ MikhailKasimov │ 54581 │ 1 │ 166 │ 36 │ stamparm/maltrail │
6. │ SergeyZh │ 54571 │ 1 │ 2 │ 2 │ JetBrains/intellij-community │
7. │ sojan-official │ 37065 │ 1 │ 1408 │ 249 │ chatwoot/chatwoot │
8. │ markjaquith │ 34497 │ 2 │ 45 │ 32 │ WordPress/WordPress │
9. │ tonikelope │ 34355 │ 1 │ 362 │ 3 │ tonikelope/megabasterd │
10. │ kroitor │ 33907 │ 1 │ 6149 │ 104 │ ccxt/ccxt │
11. │ alexey-milovidov │ 33825 │ 2 │ 10160 │ 71 │ ClickHouse/ClickHouse │
12. │ frenck │ 33682 │ 7 │ 7196 │ 83 │ home-assistant/core │
13. │ dtzWill │ 32883 │ 3 │ 169 │ 114 │ llvm-mirror/llvm │
14. │ kovidgoyal │ 31942 │ 2 │ 6649 │ 16 │ kovidgoyal/calibre │
15. │ taylorotwell │ 29131 │ 7 │ 4860 │ 124 │ laravel/framework │
16. │ kripken │ 29017 │ 6 │ 2638 │ 6 │ kripken/emscripten │
17. │ SchrodingersGat │ 28931 │ 1 │ 3075 │ 9 │ inventree/InvenTree │
18. │ M66B │ 27550 │ 2 │ 360 │ 10 │ M66B/FairEmail │
19. │ rouault │ 27297 │ 2 │ 3824 │ 5 │ OSGeo/gdal │
20. │ Roshanjossey │ 26668 │ 2 │ 987 │ 37 │ firstcontributions/first-contributions │
21. │ TomasVotruba │ 25954 │ 2 │ 3738 │ 171 │ rectorphp/rector │
22. │ bdraco │ 25881 │ 4 │ 2613 │ 10 │ home-assistant/core │
23. │ fabpot │ 25410 │ 7 │ 7915 │ 33 │ symfony/symfony │
24. │ stamparm │ 24987 │ 2 │ 5367 │ 36 │ stamparm/maltrail │
25. │ timabbott │ 24928 │ 1 │ 8800 │ 11 │ zulip/zulip │
26. │ glenn-jocher │ 24406 │ 3 │ 2268 │ 19 │ ultralytics/ultralytics │
27. │ NickCao │ 24351 │ 1 │ 314 │ 207 │ NixOS/nixpkgs │
28. │ PeterBot │ 23950 │ 1 │ 496 │ 3 │ cdnjs/cdnjs │
29. │ christian-bromann │ 23497 │ 8 │ 5495 │ 112 │ webdriverio/webdriverio │
30. │ mapx- │ 23342 │ 1 │ 10738 │ 4 │ uBlockOrigin/uAssets │
31. │ akien-mga │ 23114 │ 4 │ 21292 │ 63 │ godotengine/godot │
32. │ vitorgalvao │ 22979 │ 3 │ 3268 │ 82 │ caskroom/homebrew-cask │
33. │ veler │ 22637 │ 2 │ 693 │ 11 │ DevToys-app/DevToys │
34. │ balloob │ 22529 │ 6 │ 4252 │ 68 │ home-assistant/core │
35. │ maxlazio │ 22341 │ 1 │ 124 │ 2 │ gitlabhq/gitlabhq │
36. │ josevalim │ 21651 │ 6 │ 9607 │ 5 │ elixir-lang/elixir │
37. │ adeebshihadeh │ 21197 │ 1 │ 1948 │ 6 │ commaai/openpilot │
38. │ fabaff │ 21085 │ 4 │ 1035 │ 5 │ NixOS/nixpkgs │
39. │ charliermarsh │ 21067 │ 5 │ 6613 │ 52 │ astral-sh/uv │
40. │ tmorehouse │ 21011 │ 1 │ 1337 │ 18 │ bootstrap-vue/bootstrap-vue │
41. │ SuperSandro2000 │ 20855 │ 1 │ 1712 │ 1403 │ NixOS/nixpkgs │
42. │ rubenfiszel │ 20169 │ 1 │ 663 │ 10 │ windmill-labs/windmill │
43. │ jerryzh168 │ 19844 │ 4 │ 245 │ 68 │ pytorch/pytorch │
44. │ jsteemann │ 19831 │ 1 │ 215 │ 10 │ arangodb/arangodb │
45. │ ezyang │ 19680 │ 5 │ 3025 │ 7 │ pytorch/pytorch │
46. │ XhmikosR │ 19564 │ 15 │ 2161 │ 1063 │ twbs/bootstrap │
47. │ geraldcombs │ 19557 │ 1 │ 17 │ 41 │ wireshark/wireshark │
48. │ syuilo │ 19470 │ 1 │ 2094 │ 1260 │ misskey-dev/misskey │
49. │ bpasero │ 19443 │ 4 │ 18632 │ 32 │ microsoft/vscode │
50. │ afc163 │ 19343 │ 10 │ 15682 │ 1304 │ ant-design/ant-design │
└───────────────────┴────────┴───────┴────────┴───────┴────────────────────────────────────────┘
50 rows in set. Elapsed: 17.491 sec. Processed 1.57 billion rows, 14.77 GB (89.48 million rows/s., 844.46 MB/s.)
Ok, I see some real people here. I'm definitely sure about at least one of them.
SELECT
lower(substring(repo_name, 1, position(repo_name, '/'))) AS org,
count() AS stars
FROM github_events
WHERE event_type = 'WatchEvent'
GROUP BY org
ORDER BY stars DESC
LIMIT 50
┌─org──────────────────┬───stars─┐
1. │ microsoft/ │ 3909554 │
2. │ google/ │ 2396903 │
3. │ facebook/ │ 1542051 │
4. │ apache/ │ 1114002 │
5. │ sindresorhus/ │ 1077214 │
6. │ facebookresearch/ │ 967632 │
7. │ alibaba/ │ 963413 │
8. │ aplus-framework/ │ 773068 │
9. │ openai/ │ 732591 │
10. │ github/ │ 719612 │
11. │ vuejs/ │ 705660 │
12. │ tensorflow/ │ 575387 │
13. │ freecodecamp/ │ 571033 │
14. │ huggingface/ │ 551411 │
15. │ tencent/ │ 550408 │
16. │ fossasia/ │ 520675 │
17. │ airbnb/ │ 495275 │
18. │ thealgorithms/ │ 477515 │
19. │ kamranahmedse/ │ 465042 │
20. │ donnemartin/ │ 445213 │
21. │ dotnet/ │ 407614 │
22. │ apple/ │ 398957 │
23. │ mozilla/ │ 393944 │
24. │ symfony/ │ 392660 │
25. │ jwasham/ │ 380207 │
26. │ angular/ │ 379427 │
27. │ 996icu/ │ 378921 │
28. │ nvidia/ │ 368376 │
29. │ deepseek-ai/ │ 355350 │
30. │ twitter/ │ 353012 │
31. │ rust-lang/ │ 352110 │
32. │ square/ │ 351762 │
33. │ public-apis/ │ 338225 │
34. │ vercel/ │ 333670 │
35. │ kubernetes/ │ 330935 │
36. │ ebookfoundation/ │ 315748 │
37. │ hashicorp/ │ 315201 │
38. │ karpathy/ │ 306070 │
39. │ vinta/ │ 299867 │
40. │ codecrafters-io/ │ 299837 │
41. │ flutter/ │ 291845 │
42. │ awslabs/ │ 289640 │
43. │ shadowsocks/ │ 285056 │
44. │ docker/ │ 279692 │
45. │ ant-design/ │ 279642 │
46. │ golang/ │ 279480 │
47. │ azure/ │ 274359 │
48. │ netflix/ │ 272932 │
49. │ googlecloudplatform/ │ 272902 │
50. │ spring-projects/ │ 272284 │
└──────────────────────┴─────────┘
50 rows in set. Elapsed: 1.254 sec. Processed 554.52 million rows, 3.39 GB (442.20 million rows/s., 2.71 GB/s.)
You may notice that Microsoft is now well ahead of Google — in the first edition of this article it was the other way round. Actually, it depends on how you count. Maybe you should sum up Tensorflow, Kubernetes, Flutter, Golang, and Chrome for Google; GitHub and DotNet for Microsoft; Facebook Research and React for Facebook.
SELECT
lower(substring(repo_name, 1, position(repo_name, '/'))) AS org,
uniq(repo_name) AS repos
FROM
(
SELECT repo_name
FROM github_events
WHERE event_type = 'WatchEvent'
GROUP BY repo_name
HAVING count() >= 10
)
GROUP BY org
ORDER BY repos DESC
LIMIT 50
┌─org──────────────────┬─repos─┐
1. │ microsoft/ │ 6568 │
2. │ packtpublishing/ │ 3585 │
3. │ aws-samples/ │ 3178 │
4. │ google/ │ 2568 │
5. │ apache/ │ 1650 │
6. │ azure/ │ 1481 │
7. │ openstack/ │ 1438 │
8. │ 51deep/ │ 1325 │
9. │ ibm/ │ 1245 │
10. │ googlecloudplatform/ │ 1242 │
11. │ azure-samples/ │ 1221 │
12. │ facebookresearch/ │ 1214 │
13. │ sindresorhus/ │ 1207 │
14. │ awslabs/ │ 1154 │
15. │ deepcode51/ │ 1085 │
16. │ 66deep/ │ 923 │
17. │ mozilla/ │ 882 │
18. │ deepcode66/ │ 877 │
19. │ nirzaf/ │ 863 │
20. │ apress/ │ 859 │
21. │ adafruit/ │ 846 │
22. │ jenkinsci/ │ 737 │
23. │ oslabs-beta/ │ 680 │
24. │ intel/ │ 624 │
25. │ substack/ │ 606 │
26. │ keijiro/ │ 604 │
27. │ w3c/ │ 598 │
28. │ alibaba/ │ 574 │
29. │ github/ │ 553 │
30. │ mapbox/ │ 544 │
31. │ nvidia/ │ 522 │
32. │ mafintosh/ │ 518 │
33. │ kde/ │ 517 │
34. │ hashicorp/ │ 508 │
35. │ aws/ │ 498 │
36. │ udacity/ │ 493 │
37. │ cyanogenmod/ │ 469 │
38. │ lineageos/ │ 466 │
39. │ hazrat-ali9/ │ 448 │
40. │ cloudflare/ │ 440 │
41. │ esri/ │ 440 │
42. │ unity-technologies/ │ 433 │
43. │ spatie/ │ 425 │
44. │ jetbrains/ │ 420 │
45. │ linkedinlearning/ │ 419 │
46. │ microsoftdocs/ │ 410 │
47. │ nvlabs/ │ 409 │
48. │ shopify/ │ 397 │
49. │ googleapis/ │ 396 │
50. │ apple/ │ 396 │
└──────────────────────┴───────┘
50 rows in set. Elapsed: 2.905 sec. Processed 554.52 million rows, 3.39 GB (190.89 million rows/s., 1.17 GB/s.)
To avoid showing users with a huge number of forked repositories or users with some GitHub misuse, we added a threshold of 10 stars. Microsoft wins by a huge margin. Packt Publishing is quite interesting — they have repositories for articles and books — and AWS Samples is right behind it. Sindre Sorhus, in 13th place, is a single person and also a very notable open-source contributor, mostly for the list of awesome awesomeness.
The size of community that is visible on GitHub.
SELECT
lower(substring(repo_name, 1, position(repo_name, '/'))) AS org,
uniq(actor_login) AS authors,
uniqIf(actor_login, event_type = 'PullRequestEvent') AS pr_authors,
uniqIf(actor_login, event_type = 'IssuesEvent') AS issue_authors,
uniqIf(actor_login, event_type = 'IssueCommentEvent') AS comment_authors,
uniqIf(actor_login, event_type = 'PullRequestReviewCommentEvent') AS review_authors,
uniqIf(actor_login, event_type = 'PushEvent') AS push_authors
FROM github_events
WHERE event_type IN ('PullRequestEvent', 'IssuesEvent', 'IssueCommentEvent', 'PullRequestReviewCommentEvent', 'PushEvent')
GROUP BY org
ORDER BY authors DESC
LIMIT 50
┌─org───────────────────┬─authors─┬─pr_authors─┬─issue_authors─┬─comment_authors─┬─review_authors─┬─push_authors─┐
│ microsoft/ │ 585838 │ 72967 │ 350291 │ 419007 │ 25516 │ 15622 │
│ google/ │ 191343 │ 70099 │ 85815 │ 119940 │ 12935 │ 5572 │
│ facebook/ │ 170390 │ 28034 │ 69665 │ 137568 │ 7423 │ 1634 │
│ apache/ │ 158787 │ 65642 │ 69619 │ 116978 │ 31245 │ 4364 │
│ microsoftdocs/ │ 158703 │ 39941 │ 101751 │ 82374 │ 4442 │ 2199 │
│ azure/ │ 129866 │ 28234 │ 74422 │ 98720 │ 13382 │ 8732 │
│ docker/ │ 118173 │ 10177 │ 50836 │ 90427 │ 2420 │ 410 │
│ dotnet/ │ 105838 │ 14151 │ 63610 │ 79518 │ 5970 │ 1306 │
│ aws/ │ 103583 │ 16072 │ 51068 │ 82546 │ 6464 │ 2997 │
│ home-assistant/ │ 93912 │ 14471 │ 41701 │ 79524 │ 5523 │ 259 │
│ angular/ │ 93439 │ 11794 │ 41931 │ 77236 │ 2751 │ 232 │
│ hashicorp/ │ 93249 │ 17806 │ 47148 │ 70315 │ 4463 │ 1501 │
│ firstcontributions/ │ 85444 │ 83974 │ 794 │ 6163 │ 252 │ 9 │
│ flutter/ │ 79074 │ 6615 │ 40788 │ 63983 │ 2655 │ 447 │
│ tensorflow/ │ 79038 │ 10933 │ 38922 │ 65398 │ 3433 │ 717 │
│ kubernetes/ │ 77455 │ 20008 │ 36636 │ 63281 │ 9429 │ 434 │
│ valvesoftware/ │ 76009 │ 1182 │ 39174 │ 58003 │ 175 │ 73 │
│ alibaba/ │ 74140 │ 9570 │ 48069 │ 50372 │ 2142 │ 1668 │
│ github/ │ 62884 │ 26408 │ 22340 │ 38141 │ 4430 │ 3344 │
│ elastic/ │ 57007 │ 10533 │ 28324 │ 44713 │ 3427 │ 1777 │
│ vuejs/ │ 54951 │ 11556 │ 26521 │ 40225 │ 1775 │ 122 │
│ nextcloud/ │ 54225 │ 5063 │ 28614 │ 44359 │ 1483 │ 516 │
│ grafana/ │ 52921 │ 9100 │ 24996 │ 42160 │ 2989 │ 1054 │
│ vercel/ │ 52424 │ 8553 │ 19894 │ 40068 │ 2052 │ 468 │
│ pytorch/ │ 51309 │ 11161 │ 25065 │ 40130 │ 4649 │ 1221 │
│ firebase/ │ 51253 │ 3956 │ 22425 │ 42414 │ 1224 │ 512 │
│ fortawesome/ │ 50632 │ 801 │ 15961 │ 42281 │ 69 │ 24 │
│ mozilla/ │ 50498 │ 11419 │ 27677 │ 36435 │ 4173 │ 1815 │
│ jetbrains/ │ 49937 │ 34529 │ 10010 │ 21898 │ 1929 │ 1307 │
│ facebookresearch/ │ 49008 │ 6666 │ 28886 │ 34634 │ 1538 │ 1552 │
│ huggingface/ │ 47968 │ 11846 │ 24259 │ 36069 │ 4363 │ 439 │
│ jlord/ │ 46911 │ 45598 │ 914 │ 3300 │ 33 │ 15 │
│ digitalplatdev/ │ 46538 │ 28 │ 46343 │ 4487 │ 0 │ 6 │
│ ibm-epbl/ │ 45375 │ 2841 │ 546 │ 238 │ 4 │ 44450 │
│ awslabs/ │ 45094 │ 12253 │ 24051 │ 31018 │ 4160 │ 3008 │
│ laravel/ │ 44704 │ 15164 │ 20402 │ 34512 │ 2515 │ 62 │
│ googlecloudplatform/ │ 44424 │ 15399 │ 22358 │ 31055 │ 6560 │ 3529 │
│ spring-projects/ │ 43823 │ 10368 │ 24043 │ 31172 │ 1861 │ 168 │
│ homebrew/ │ 43625 │ 20883 │ 15296 │ 30798 │ 6580 │ 117 │
│ ant-design/ │ 43403 │ 5031 │ 25608 │ 31694 │ 1255 │ 232 │
│ npm/ │ 43151 │ 6272 │ 17631 │ 31708 │ 661 │ 156 │
│ ansible/ │ 42586 │ 11323 │ 22131 │ 33207 │ 3488 │ 444 │
│ rust-lang/ │ 42385 │ 13756 │ 22502 │ 32472 │ 5855 │ 327 │
│ learn-co-students/ │ 41761 │ 41528 │ 916 │ 1810 │ 185 │ 687 │
│ nvidia/ │ 41382 │ 7214 │ 23181 │ 31088 │ 3112 │ 2054 │
│ atom/ │ 41149 │ 3811 │ 19623 │ 31162 │ 763 │ 114 │
│ digitalinnovationone/ │ 40537 │ 36650 │ 7793 │ 2561 │ 44 │ 39 │
│ nodejs/ │ 39711 │ 7919 │ 17355 │ 31173 │ 3352 │ 547 │
│ golang/ │ 37400 │ 4680 │ 23030 │ 27414 │ 451 │ 75 │
│ expo/ │ 36062 │ 3340 │ 13150 │ 30431 │ 702 │ 148 │
└───────────────────────┴─────────┴────────────┴───────────────┴─────────────────┴────────────────┴──────────────┘
50 rows in set. Elapsed: 59.258 sec. Processed 8.21 billion rows, 54.05 GB (138.54 million rows/s., 912.13 MB/s.)
Microsoft wins in 3 out of 6 categories. Apache still wins for the number of code reviewers (I thought they are using JIRA, but actually it's not always the case). firstcontributions wins for the number of PR authors — it is the repository that teaches you how to make a pull request by making a pull request to it. Learn-co-students, the previous winner, has dropped to 44th place.
Please take it with a grain of salt. Not every team is using GitHub as their issue tracker or code review system. Linux and Postgres are using maillists. LLVM is using Bugzilla and Phabricator.
I want to get the repositories with the most added and removed code over time. If I do it in a naive way, the multiple forks of cdn.js repository will be on top. By the way, cdn.js is the largest repository in GitHub by size in bytes (it contains all popular JavaScript libraries, the total size is 254GB). To get something interesting, I added a threshold for the number of diff size and also for the ratio of added and removed code (in actively developing codebases, the ratio should be close to one).
SELECT
repo_name,
count() AS prs,
uniq(actor_login) AS authors,
sum(additions) AS adds,
sum(deletions) AS dels
FROM github_events
WHERE (event_type = 'PullRequestEvent') AND (action = 'opened') AND (additions < 10000) AND (deletions < 10000)
GROUP BY repo_name
HAVING (adds / dels) < 10
ORDER BY adds + dels DESC
LIMIT 50
┌─repo_name───────────────────────────────────────────┬────prs─┬─authors─┬─────adds─┬─────dels─┐
│ everypolitician/everypolitician-data │ 150520 │ 19 │ 66775257 │ 71190924 │
│ brianchandotcom/liferay-portal │ 154988 │ 598 │ 46045754 │ 19997392 │
│ elastic/kibana │ 141160 │ 1552 │ 33027034 │ 16015257 │
│ brain-tec/odoo │ 33778 │ 38 │ 31795457 │ 15702210 │
│ googleapis/discovery-artifact-manager │ 8043 │ 25 │ 21408618 │ 20870044 │
│ azure-sdk/azure-sdk-for-js │ 11392 │ 9 │ 25628072 │ 16360994 │
│ code-dot-org/code-dot-org │ 61994 │ 208 │ 26128228 │ 12483851 │
│ azure-sdk/azure-sdk-for-go │ 7625 │ 3 │ 15087520 │ 19003644 │
│ azure-sdk/azure-sdk-for-java │ 7622 │ 9 │ 18935644 │ 14277860 │
│ googleapis/google-api-java-client-services │ 25760 │ 42 │ 19385797 │ 13134487 │
│ redhat-appstudio-qe/devfile-sample-hello-world │ 36007 │ 14 │ 21238359 │ 9143944 │
│ azure-sdk/azure-sdk-for-python │ 6315 │ 7 │ 20533205 │ 9700840 │
│ Azure/azure-sdk-for-python │ 27458 │ 1057 │ 21018560 │ 8371806 │
│ Azure/azure-sdk-for-net │ 31273 │ 2039 │ 21206817 │ 7820622 │
│ NixOS/nixpkgs │ 375905 │ 12332 │ 17750304 │ 10778944 │
│ Azure/azure-sdk-for-java │ 30089 │ 765 │ 21223068 │ 6793308 │
│ dotnet/roslyn │ 40454 │ 883 │ 18973885 │ 8970436 │
│ AzureSDKAutomation/azure-sdk-for-go │ 34757 │ 5 │ 17204216 │ 10413589 │
│ Artur-data-DEV/curso-reactjs-ninja │ 2727 │ 1 │ 13489740 │ 10883652 │
│ rust-lang/rust │ 75684 │ 5754 │ 15104014 │ 9033105 │
│ Azure/azure-sdk-for-js │ 23373 │ 586 │ 16448075 │ 7651714 │
│ Azure/azure-rest-api-specs │ 28782 │ 4595 │ 21373423 │ 2691496 │
│ AzureSDKAutomation/azure-sdk-for-python │ 7730 │ 10 │ 15376669 │ 6205292 │
│ demisto/content │ 38125 │ 1040 │ 17057810 │ 4280802 │
│ aelmanaa/cl-docs │ 97450 │ 1 │ 14585332 │ 6617199 │
│ cockroachdb/cockroach │ 72430 │ 945 │ 13933416 │ 7243259 │
│ raycast/extensions │ 12078 │ 3998 │ 18622230 │ 2310052 │
│ shuyangzhou/liferay-portal │ 9187 │ 163 │ 13198724 │ 7237332 │
│ redhat-appstudio-qe/hacbs-test-project │ 25362 │ 11 │ 11628907 │ 8599446 │
│ awaisab172/JavaScript-Demos │ 6937 │ 2 │ 9605135 │ 10409094 │
│ AzureSDKAutomation/azure-sdk-for-js │ 7053 │ 4 │ 13865653 │ 5654085 │
│ Azure/azure-sdk-for-go │ 19087 │ 365 │ 15150732 │ 3789893 │
│ elastic/elasticsearch │ 83652 │ 2976 │ 12988895 │ 5943821 │
│ AzureSDKAutomation/azure-sdk-for-java │ 8568 │ 2 │ 16158769 │ 2719345 │
│ cms-sw/cmssw │ 40467 │ 1345 │ 12195587 │ 6322577 │
│ kubernetes/kubernetes │ 73582 │ 6613 │ 12278710 │ 6195166 │
│ pytorch/pytorch │ 103588 │ 6420 │ 10902268 │ 6422281 │
│ Azure/azure-powershell │ 15333 │ 1995 │ 11978281 │ 5128995 │
│ redhat-appstudio-qe/hacbs-test-project-konflux-demo │ 17542 │ 2 │ 9257073 │ 7786435 │
│ digicert/reports │ 10845 │ 2 │ 8301174 │ 8296822 │
│ leandrohstein/transporteservico-urbs-data │ 435321 │ 1 │ 8387639 │ 7982044 │
│ microsoft/winget-pkgs │ 266157 │ 5668 │ 12296793 │ 4063471 │
│ tensorflow/tensorflow │ 56360 │ 5062 │ 10602092 │ 5635059 │
│ AzureSDKAutomation/azure-sdk-for-net │ 5997 │ 3 │ 12243298 │ 3808035 │
│ llvm/llvm-project │ 68059 │ 4391 │ 10722892 │ 4780827 │
│ ekapratama93/nginx-bot-blocker │ 1679 │ 3 │ 7717166 │ 7723794 │
│ onkelbeh/gentoo-dev │ 7556 │ 2 │ 8175891 │ 7235372 │
│ liferay-echo/liferay-portal │ 14574 │ 206 │ 9950596 │ 5169928 │
│ odoo/odoo │ 120590 │ 4184 │ 9557798 │ 4944452 │
│ gravitational/teleport │ 39733 │ 550 │ 9342269 │ 4835076 │
└─────────────────────────────────────────────────────┴────────┴─────────┴──────────┴──────────┘
50 rows in set. Elapsed: 6.326 sec. Processed 814.54 million rows, 13.17 GB (128.76 million rows/s., 2.08 GB/s.)
SELECT
repo_name,
count() AS pushes,
uniq(actor_login) AS authors
FROM github_events
WHERE (event_type = 'PushEvent') AND (repo_name IN
(
SELECT repo_name
FROM github_events
WHERE event_type = 'WatchEvent'
GROUP BY repo_name
ORDER BY count() DESC
LIMIT 10000
))
GROUP BY repo_name
ORDER BY count() DESC
LIMIT 50
┌─repo_name───────────────────────┬──pushes─┬─authors─┐
│ unifyai/ivy │ 2525534 │ 152 │
│ CocoaPods/Specs │ 908981 │ 375 │
│ pytorch/pytorch │ 779195 │ 705 │
│ odoo/odoo │ 728111 │ 100 │
│ NixOS/nixpkgs │ 425549 │ 362 │
│ docker-library/docs │ 396372 │ 7 │
│ chromium/chromium │ 375132 │ 6 │
│ Homebrew/homebrew-core │ 325979 │ 69 │
│ microsoft/winget-pkgs │ 264714 │ 21 │
│ openstack/openstack │ 263713 │ 3 │
│ getsentry/sentry │ 256757 │ 410 │
│ llvm/llvm-project │ 253863 │ 2221 │
│ Automattic/wp-calypso │ 246914 │ 780 │
│ grafana/grafana │ 223813 │ 530 │
│ tensorflow/tensorflow │ 221505 │ 167 │
│ airbytehq/airbyte │ 211851 │ 247 │
│ PostHog/posthog │ 199782 │ 252 │
│ cockroachdb/cockroach │ 184064 │ 300 │
│ Homebrew/homebrew-cask │ 181636 │ 35 │
│ elastic/kibana │ 177780 │ 854 │
│ sourcegraph/sourcegraph │ 172198 │ 271 │
│ greatfire/wiki │ 169660 │ 4 │
│ WordPress/gutenberg │ 165155 │ 308 │
│ JetBrains/kotlin │ 164103 │ 214 │
│ gravitational/teleport │ 162611 │ 190 │
│ metabase/metabase │ 158962 │ 158 │
│ MetaMask/metamask-extension │ 153297 │ 281 │
│ voloviktarasewu/darkweb-tourism │ 147426 │ 1 │
│ elastic/elasticsearch │ 137295 │ 604 │
│ edx/edx-platform │ 130303 │ 397 │
│ tata-chka/verified-tor-sites │ 128455 │ 1 │
│ discourse/discourse │ 127600 │ 117 │
│ gradle/gradle │ 127351 │ 117 │
│ ClickHouse/ClickHouse │ 126133 │ 128 │
│ uBlockOrigin/uAssets │ 118690 │ 24 │
│ microsoft/vscode │ 118613 │ 112 │
│ smartcontractkit/chainlink │ 113589 │ 296 │
│ freebsd/freebsd │ 110230 │ 2 │
│ commaai/openpilot │ 103641 │ 52 │
│ dagster-io/dagster │ 101692 │ 87 │
│ arangodb/arangodb │ 100317 │ 84 │
│ chatwoot/chatwoot │ 98775 │ 22 │
│ taosdata/TDengine │ 95142 │ 117 │
│ dart-lang/sdk │ 94817 │ 115 │
│ JetBrains/intellij-community │ 94069 │ 33 │
│ rust-lang/rust │ 93718 │ 56 │
│ nextcloud/server │ 93625 │ 214 │
│ mongodb/mongo │ 92356 │ 316 │
│ servo/servo │ 91187 │ 54 │
│ hashicorp/vault │ 90050 │ 277 │
└─────────────────────────────────┴─────────┴─────────┘
50 rows in set. Elapsed: 4.333 sec. Processed 1.06 billion rows, 6.35 GB (244.10 million rows/s., 1.47 GB/s.)
SELECT
actor_login,
count(),
uniq(repo_name) AS repos,
uniq(repo_name, number) AS prs,
replaceRegexpAll(substringUTF8(anyHeavy(body), 1, 100), '[\r\n]', ' ') AS comment
FROM github_events
WHERE (event_type = 'PullRequestReviewCommentEvent') AND (action = 'created')
GROUP BY actor_login
ORDER BY count() DESC
LIMIT 50
┌─actor_login────────────────────────┬─count()─┬──repos─┬─────prs─┬─comment────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Copilot │ 3944315 │ 301774 │ 1357463 │ There’s a large block of commented-out footer HTML; if it’s no longer needed, removing it will reduc │
│ coderabbitai[bot] │ 3794686 │ 99956 │ 724492 │ _:warning: Potential issue_ **Fix the logical condition in value validation.** The current conditi │
│ hound[bot] │ 2318825 │ 2518 │ 29928 │ Parsing error: 'import' and 'export' may appear only with 'sourceType: module' │
│ github-actions[bot] │ 1211699 │ 15781 │ 165419 │ convention OK │
│ houndci-bot │ 1006942 │ 5926 │ 80376 │ Style/StringLiterals: Prefer double-quoted strings unless you need single quotes to avoid extra back │
│ gemini-code-assist[bot] │ 919712 │ 43527 │ 350828 │  URLに`/`が含まれているかを確認するこのテストケース │
│ sourcery-ai[bot] │ 752355 │ 35576 │ 106290 │ **issue (code-quality):** Raise a specific error instead of the general `Exception` or `BaseExceptio │
│ github-advanced-security[bot] │ 568033 │ 61781 │ 189693 │ ## SQL query built from user-controlled sources This SQL query depends on a [user-provided value](1 │
│ chatgpt-codex-connector[bot] │ 440386 │ 92211 │ 318733 │ **[P1] Existing payments vanish from aggregates when `cancel` is null** The inspection queries now │
│ github-code-scanning[bot] │ 420422 │ 34942 │ 186966 │ ## Use an enum to specify an AWS Region AWS Region is set using a `String`. To explicitly set a pub │
│ houndci │ 341972 │ 3102 │ 28679 │ Prefer double-quoted strings unless you need single quotes to avoid extra backslashes for escaping. │
│ codacy-bot │ 325645 │ 2313 │ 21581 │  Issue found: [Strings must use doublequo │
│ greptile-apps[bot] │ 287874 │ 6332 │ 99054 │ **style:** Using `rglob()` twice creates two lists that are then concatenated - this is inefficient │
│ mateacademy-ai-mentor │ 285022 │ 319 │ 56157 │ Error: The example for GenreSchema uses 'genre_schema_example', which defines the key as 'genre', bu │
│ prisma-cloud-devsecops[bot] │ 262717 │ 19871 │ 34403 │ <h2>morgan 1.0.0 / package.json</h2> <h3>Total vulnerabilities: 1</h3> <table> │
│ amazon-q-developer[bot] │ 206821 │ 1900 │ 49712 │ ****Description:**** The setup_testing_environment fixture duplicates the TESTING configuration set │
│ cursor[bot] │ 201193 │ 25904 │ 98566 │ ### Bug: Sitemap Duplicates Cause SEO Confusion <!-- DESCRIPTION START --> The sitemap contains dup │
│ github-learning-lab[bot] │ 192687 │ 42016 │ 77426 │ Thanks for opening this pull request, @moinali. ```suggestion You can play the game at: https://mo │
│ codeclimate[bot] │ 149797 │ 2279 │ 22161 │ Expected a newline at the end of the file. │
│ cubic-dev-ai[bot] │ 110705 │ 4684 │ 35133 │ <!-- metadata:{"confidence":9,"steps":[{"text":"","toolCalls":[{"toolName":"think","input":{"thought │
│ miyuqapp-preprod-pdx[bot] │ 108862 │ 3 │ 25811 │ ****Description:**** The catch block throws a RuntimeException, which may hide the original SQLExcep │
│ miyuqapp-preprod-iad[bot] │ 101116 │ 3 │ 24839 │ The fix replaces string concatenation with a PreparedStatement using parameterized queries. This add │
│ bridgecrew[bot] │ 95939 │ 1072 │ 9498 │ ```suggestion resource "aws_instance" "web_host" { # ec2 have plain text secrets in user data am │
│ sonatype-lift[bot] │ 77625 │ 2751 │ 10551 │ *JSC_USELESS_CODE:* Suspicious code. This code lacks side-effects. Is there a bug? (at-me [in a rep │
│ devin-ai-integration[bot] │ 77236 │ 6785 │ 34606 │ Fixed in 9999590 — generated a raster PNG (800x420) from the SVG using `rsvg-convert`. `og:image` no │
│ gennady-bars │ 71489 │ 5424 │ 10407 │ Good job that you return the button text in `finally` │
│ codacy-production[bot] │ 68293 │ 1268 │ 4568 │ Codacy found an issue: [Variable expansions must be double-quoted so as to prevent being split into │
│ daosbuild1 │ 68054 │ 5 │ 3114 │ (style) trailing whitespace │
│ greptile-apps-staging[bot] │ 63795 │ 21 │ 12260 │ **`isSpendCapEnabled` is always `false` while entitlements are loading** `entitledFeatures` is an e │
│ accesslint[bot] │ 62943 │ 1755 │ 7007 │ This image is missing a text alternative (`alt` attribute). This is a problem for people using scree │
│ stickler-ci │ 59668 │ 461 │ 8474 │ E265 block comment should start with '# ' │
│ DmitryChitalov │ 57790 │ 1872 │ 11922 │ выполнено │
│ CyrusNajmabadi │ 52710 │ 76 │ 7332 │ > Even if we end up deciding that only labels on loop statements count, I expect we should maintain │
│ claude[bot] │ 48266 │ 2299 │ 18141 │ **Claude finished @rap2hpoutre's task in 16s** —— [View job](https://github.com/mano-sesan/mano/acti │
│ codereviewbot-ai[bot] │ 45622 │ 1385 │ 4702 │ #### Potential In-place Mutation in Filtering The use of `.filter()` on `ownWarehouse` is safe as it │
│ amazon-inspector-frankfurt[bot] │ 44719 │ 2 │ 33559 │ ****Description:**** Potential SQL Injection detected. Untrusted input is being directly included in │
│ MartinHjelmare │ 44353 │ 98 │ 10347 │ ```suggestion from typing import NotRequired, TypedDict ``` │
│ amazon-inspector-n-virginia[bot] │ 43593 │ 29 │ 31511 │ ****Description:**** Potential SQL Injection detected. Untrusted input is being directly included in │
│ amazon-inspector-preprod-pdx[bot] │ 43584 │ 3 │ 28162 │ ****Description:**** Potential SQL Injection detected. Untrusted input is being directly included in │
│ amazon-inspector-oregon[bot] │ 43488 │ 55 │ 32532 │ ****Description:**** Potential SQL Injection detected. Untrusted input is being directly included in │
│ amazon-inspector-preprod-iad[bot] │ 42369 │ 6 │ 26439 │ ****Description:**** Potential SQL Injection detected. Untrusted input is being directly included in │
│ stephentoub │ 41995 │ 119 │ 9959 │ Where is this called? │
│ SuperSandro2000 │ 41847 │ 316 │ 13960 │ Can we symlink? │
│ golangcibot │ 40293 │ 885 │ 8305 │ SA4006: this value of `card` is never used (from `staticcheck`) │
│ ellipsis-dev[bot] │ 39451 │ 1318 │ 18606 │ Consider passing the file path as a parameter to the function instead of hardcoding it. This would m │
│ amazon-inspector-tokyo[bot] │ 39245 │ 2 │ 31986 │ ****Description:**** Potential SQL Injection detected. Untrusted input is being directly included in │
│ amazon-inspector-ohio[bot] │ 39071 │ 3 │ 30176 │ ****Description:**** Potential SQL Injection detected. Untrusted input is being directly included in │
│ amazon-inspector-ireland[bot] │ 39022 │ 1 │ 31310 │ ****Description:**** Potential SQL Injection detected. Untrusted input is being directly included in │
│ eric-wieser │ 38716 │ 143 │ 9379 │ You probably want `EnsureGIL(EnsureGIL const&) = delete;` too (and the assignment version, and perha │
│ reviewbird-inspector-prod-iad[bot] │ 38656 │ 2 │ 13674 │ The vulnerability in the original code was a SQL injection risk. SQL injection occurs when untrusted │
└────────────────────────────────────┴─────────┴────────┴─────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
50 rows in set. Elapsed: 1.388 sec. Processed 162.48 million rows, 38.69 GB (117.04 million rows/s., 27.87 GB/s.)
The most active reviewers are AI agents now: GitHub Copilot at the top, then CodeRabbit and Gemini Code Assist, mixed with the older robots that do style checking and static analysis. But there are real people further down the list: Cyrus Najmabadi, Stephen Toub. You can also find teachers who review code from students.
What are the most popular labels for issues and pull requests?
SELECT
arrayJoin(labels) AS label,
count() AS c
FROM github_events
WHERE (event_type IN ('IssuesEvent', 'PullRequestEvent', 'IssueCommentEvent')) AND (action IN ('created', 'opened', 'labeled'))
GROUP BY label
ORDER BY c DESC
LIMIT 50
┌─label─────────────────────────┬────────c─┐
1. │ dependencies │ 84927970 │
2. │ bug │ 32142812 │
3. │ enhancement │ 26298270 │
4. │ javascript │ 15037072 │
5. │ status │ 9455318 │
6. │ approved │ 6559938 │
7. │ question │ 5942150 │
8. │ lgtm │ 4833677 │
9. │ help wanted │ 4455076 │
10. │ :arrow_heading_down: pull │ 4330646 │
11. │ go │ 4302063 │
12. │ documentation │ 3638120 │
13. │ python │ 3421953 │
14. │ cncf-cla: yes │ 3363746 │
15. │ size/XS │ 3359335 │
16. │ Bug │ 3189267 │
17. │ github_actions │ 3136107 │
18. │ feature │ 2984633 │
19. │ good first issue │ 2964634 │
20. │ ruby │ 2811229 │
21. │ java │ 2600786 │
22. │ cla: yes │ 2392719 │
23. │ size/L │ 2377017 │
24. │ kind/bug │ 2266104 │
25. │ greenkeeper │ 2227166 │
26. │ release-note-none │ 2196107 │
27. │ size/M │ 1998637 │
28. │ Stale │ 1948429 │
29. │ size/S │ 1786491 │
30. │ stale │ 1710902 │
31. │ ci:test:sf - success │ 1600520 │
32. │ codex │ 1544865 │
33. │ ok-to-test │ 1498746 │
34. │ dco-signoff: yes │ 1335152 │
35. │ size/XXL │ 1326455 │
36. │ do-not-merge/work-in-progress │ 1291406 │
37. │ renovate │ 1227425 │
38. │ release-note │ 1178854 │
39. │ in progress │ 1167869 │
40. │ needs-triage │ 1162512 │
41. │ type: bug │ 1142763 │
42. │ do-not-merge/hold │ 1141187 │
43. │ jira/valid-reference │ 1122808 │
44. │ kind/feature │ 1093402 │
45. │ feature request │ 1072972 │
46. │ automerge │ 995183 │
47. │ wontfix │ 942888 │
48. │ ci:test:stable - success │ 915714 │
49. │ CLA Signed │ 913183 │
50. │ needs-ok-to-test │ 901905 │
└───────────────────────────────┴──────────┘
50 rows in set. Elapsed: 1.390 sec. Processed 1.65 billion rows, 18.39 GB (1.19 billion rows/s., 13.23 GB/s.)
The most popular label is now dependencies — the mark of Dependabot, which on its own generates more labelled events than bugs and enhancements combined. There are still more bugs than enhancements, and "javascript" is still the only programming-language related label at the top.
The diversity of bugs and features is overwhelming:
SELECT
arrayJoin(labels) AS label,
count() AS c
FROM github_events
WHERE (event_type IN ('IssuesEvent', 'PullRequestEvent', 'IssueCommentEvent')) AND (action IN ('created', 'opened', 'labeled')) AND ((label ILIKE '%bug%') OR (label ILIKE '%feature%'))
GROUP BY label
ORDER BY c DESC
LIMIT 50
┌─label────────────────────┬────────c─┐
1. │ bug │ 32142812 │
2. │ Bug │ 3189267 │
3. │ feature │ 2984633 │
4. │ kind/bug │ 2266104 │
5. │ type: bug │ 1142763 │
6. │ kind/feature │ 1093402 │
7. │ feature request │ 1072972 │
8. │ bugzilla/valid-bug │ 886332 │
9. │ Feature │ 880796 │
10. │ Type: Bug │ 650358 │
11. │ feature-request │ 638137 │
12. │ type:bug │ 527296 │
13. │ jira/valid-bug │ 402772 │
14. │ type/bug │ 392165 │
15. │ Feature Request │ 354604 │
16. │ new feature │ 320342 │
17. │ bugzilla/severity-high │ 298653 │
18. │ bugzilla/severity-medium │ 286708 │
19. │ C-bug │ 279361 │
20. │ type:feature │ 251609 │
21. │ [Type] Bug │ 251329 │
22. │ type: feature │ 249837 │
23. │ bugfix │ 229863 │
24. │ bugzilla/invalid-bug │ 223515 │
25. │ type-bug │ 180327 │
26. │ Issue-Bug │ 156554 │
27. │ jira/invalid-bug │ 150723 │
28. │ Feature request │ 150620 │
29. │ Type: Feature │ 150084 │
30. │ type: feature request │ 149150 │
31. │ 🐛 bug │ 143749 │
32. │ 🐞 bug │ 133254 │
33. │ BUG │ 130800 │
34. │ type.DocumentationBug │ 127958 │
35. │ new-feature │ 119676 │
36. │ bug report │ 107543 │
37. │ 0.kind: bug │ 103249 │
38. │ New Feature │ 102515 │
39. │ ✨ Feature │ 96577 │
40. │ Type:Bug │ 93663 │
41. │ type.FunctionalityBug │ 91330 │
42. │ kind:bug │ 88174 │
43. │ t/bug │ 87988 │
44. │ type/feature │ 87531 │
45. │ bug fix │ 87361 │
46. │ bugzilla/severity-low │ 86796 │
47. │ doc-bug │ 86366 │
48. │ type.FeatureFlaw │ 81405 │
49. │ 🐛 Bug │ 80276 │
50. │ bugzilla/severity-urgent │ 79163 │
└──────────────────────────┴──────────┘
50 rows in set. Elapsed: 1.365 sec. Processed 1.65 billion rows, 18.39 GB (1.21 billion rows/s., 13.47 GB/s.)
Actually, the "bugzilla" label is not only about bugs.
WITH arrayJoin(labels) AS label
SELECT
sum(label ILIKE '%bug%') AS bugs,
sum(label ILIKE '%feature%') AS features,
bugs / features AS ratio
FROM github_events
WHERE (event_type IN ('IssuesEvent', 'PullRequestEvent', 'IssueCommentEvent')) AND (action IN ('created', 'opened', 'labeled')) AND ((label ILIKE '%bug%') OR (label ILIKE '%feature%'))
┌─────bugs─┬─features─┬──────────────ratio─┐
│ 50967343 │ 12875848 │ 3.9583678682755497 │
└──────────┴──────────┴────────────────────┘
1 rows in set. Elapsed: 1.249 sec. Processed 1.65 billion rows, 18.39 GB (1.32 billion rows/s., 14.72 GB/s.)
Every feature generates 3.96 bugs on average.
:) SELECT count(), repo_name FROM github_events WHERE event_type = 'WatchEvent' GROUP BY repo_name ORDER BY length(repo_name) DESC LIMIT 50
┌─count()─┬─repo_name────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ 1 │ JonathanJonathanJonathanJonathanJonatha/JonathanJonathanJonathanJonathanJonathanJonathanJonathanJonathanJonathanJonathanJonathanJonathanJona │
│ 1 │ Coding-Club-of-IISER-Thiruvananthapuram/summer-projects-beginner-s-web-development-course-building-your-first-webpage-a-personal-about-me-si │
│ 2 │ Directed-Research-in-AI-System-I-25Fall/directed-research-in-ai-system-i-24fall-week2-assignment-programming-best-practices-Module-0-Program │
│ 2 │ GOOMBAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/GOOMBAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA │
│ 47 │ uwuwuwuwuwuwuwuwuwuwuwuwuwuwuwuwuwuwuwu/uwuwuwuwuwuwuwuwuwuwuwuwuwuwuwuwuwuwuwuuwuwuwuwuwuwuwuwuwuwuwuwuwuwuwuwuwuwuwuuwuwuwuwuwuwuwuwuwuwuw │
│ 1 │ UDtUf4aRyjv7nU3YTg056xCJw1ghAJYXgad7oB5/hJJRB7rpa3IXwX8HRsA1B4jCDmlZBY9fAzXZWNPyhrsXYG5kCeC4RPFqKQ4I9sAu1aNzX2G6wAkBjm8BjPfKjdubEqkmeAIkIwgu │
│ 1 │ aarr-secret-CTIKVctcdiCR43yh34wh35rr34F/df06p3f47-93qg4gf-f4353j45kj46k463w4g45j65uj57kj4e5j354uhj46j4w5yh456j4werhytk78k7lk975htrwhrweg2w3y │
│ 1 │ PERIODICTABLEOFELEMENTSYNTHETICOMPOUNDS/data-image-png-base64-iVBORw0KGgoAAAANSUhEUgAAAhIAAAISCAYAAACZPSa-AAAgAElEQVR4nO3dW5bbOrJFUVrD-e-y68 │
│ 1 │ git-git-git-git-git-git-git-git-git-git/git.git.git.git.git.git.git.git.git.git.git.git.git.git.git.git.git.git.git.git.git.git.git.git.git. │
│ 15 │ fkdjfkdkfjskfjsldkfjvndkslakfjgkdlskfjg/fksfekgkslgkkdjfkdlskfjkfkdlslakdjfkekjlklfkjdkslkjfojksjdfaskdjfsalkdfj-laskfjls-kdajfoasjfpoawefjk │
│ 1 │ joooooooooooooooooooooooooooooooooooooj/jooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo │
│ 1 │ xxxzzxczcczxczxczzxcxccccxxvvxcxxzzzzxz/xxxzzxczcczxczxczzzzxcxxcxcxcxcxxxzzxvzxvxyxzxcxzxxzxzzxzxxzzxxxzzxczczxzxvvvvvvczxczxczzzzxcxxxxzxz │
│ 1 │ LongestNamePossibleLongestNamePossibleL/LongestNamePossibleLongestNamePossibleLongestNamePossibleLongestNamePossibleLongestNamePossibleLonge │
│ 1 │ dldadnasjndfsjafajnsdfadnfadfkoafasdklk/fjkdasifjasdiofasiodfasjdogasdgajsdpgasdpgasjkdgo-askdpgspkdgvjaspgpasjidpgjiajisdgpjioasdpigasgpjia │
│ 1 │ a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a/a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a-a- │
│ 1 │ 1ABC2DEF3GHI4JKL5MNO6PQRS7TUV8WXYZ90RON/data-image-png-base64-iVBORw0KGgoAAAANSUhEUgAAAhIAAAISCAYAAACZPSa-AAAgAElEQVR4nO3dW5bbOrJFUVrD-e-y68 │
│ 1 │ ELECTRONICATOMATHEMATICORRELANGEOMITALD/data-image-png-base64-iVBORw0KGgoAAAANSUhEUgAAAhIAAAISCAYAAACZPSa-AAAgAElEQVR4nO3dW5bbOrJFUVrD-e-y68 │
│ 7 │ aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa │
│ 5 │ accounts-inheritance-finders-of-america/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee │
│ 1 │ WWWWWWWWWWWWWWWWWWWWWWWWWWWMWWWWWWWWWWW/WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW │
│ 1 │ helpmetheshadowpeoplearestealingmystuff/helpmetheshadowpeoplearestealingmystuffandidontknowhowtogetitbackthisisabigproblembecauseotherpeople │
│ 1 │ fefiewf688w4f464w8f48wf48fw8t844yu4448d/as-veapofg-pbirsgursiobeNIOESGHILAN-IUB-PASBNGZESBGENJBIGSRNHB-DNI-ODT-PISI-HNJBHSBGINLESBJSEILNEIS- │
│ 4 │ joooooooooooooooooooooooooooooooooooooj/jooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooj │
│ 1 │ DCIT-104-25-26-Programming-Fundamentals/dcit-104-25-26-programming-fundamentals-classroom-3def57-programming_fundamentals_assignment_3-progr │
│ 7 │ strackoverflowwwwwwwwwwwwwwwwwwwwwwwwww/maximum-repository-title-length-is-super-long-but-its-a-free-country-so-go-ahead-and-make-it-longggg │
│ 2 │ xvideos-xnxx-pornhub-xhamster-jerkmate/chengren-chengrenshequ-chengrendizhi-chengrenwangzhan-chengrenpian-chengrenshipin-chengrenshipin-seq │
│ 1 │ COMNETORGINFOBIZUSNAMEDOMAINSCIENCENTAG/https-www.google.com-search-q-Maki-BOT-E2-80-94-Yesterday-at-11-3A32-PM-Role-name-changed-From-ne-1 │
│ 3 │ pwpwpwpwpwpwpwpwpwpwpwpwpwpwpwpwpwpwpwp/pwpwpwpwpwpwpwpwpwpwpwpwpwpwpwpwpwpwpwppwpwpwpwpwpwpwpwpwpwpwpwpwpwpwpwpwpwpwppwpwpwpwpwpwpwpwpwpwp │
│ 2 │ yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy/yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy │
│ 1 │ ALPHANUMERICALDRONTELOGICALCHEMYSTERIOS/https-www.google.com-search-q-Maki-BOT-E2-80-94-Yesterday-at-11-3A32-PM-Role-name-changed-From-ne-1 │
│ 3 │ biubiubiubiubiubiubiubiubiubiubiubiubiu/biubiubiubiubiubiubiubiubiubiubiubiubiubiubiubiubiubiubiubiubiubiubiubiubiubiubiubiubiubiubiubiubiu │
│ 1 │ ACRONYMATHEMATICOMMUNICATIONUMERICALIFO/https-www.google.com-search-q-NAVY-oq-NAVY-aqs-chrome..69i57j0i271l3.2146j0j9-sourceid-chrome-ie-U │
│ 1 │ 1ABC2DEF3GHI4JKL5MNO6PQRS7TUV8WXYZ90RON/https-www.linkedin.com-uas-login-session_redirect-https-3A-2F-2Fwww.linkedin.com-2Ffeed-2Fupdate-2 │
│ 1 │ ELECTRONICATOMATHEMATICORRELANGEOMITALD/https-support.microsoft.com-en-us-topic-remember-passwords-and-fill-out-web-forms-for-internet-exp │
│ 1 │ LOGOIOGOLINGUIAIMIAWIFIPHIMITIOMWONOWEP/https-support.microsoft.com-en-us-topic-remember-passwords-and-fill-out-web-forms-for-internet-exp │
│ 1 │ ALPHANUMERICALDRONTELOGICALCHEMYSTERIOS/https-support.microsoft.com-en-us-topic-remember-passwords-and-fill-out-web-forms-for-internet-exp │
│ 1 │ verifyreportlabreportsmbmlabcomresult/GetByBarcode-clientCode-MBMLAB-resultId-1000019289670U6MPC55AWBQHRDP7L8YC5G3N14MM5FWR67UCJCP9J3LSP5N │
│ 1 │ LANGUAGESUNIFIPHINGEOMETRICALPHABETICAL/https-support.microsoft.com-en-us-topic-remember-passwords-and-fill-out-web-forms-for-internet-exp │
│ 1 │ ALPHANUMERICALDRONTELOGICALCHEMYSTERIOS/https-www.science.gov-scigov-desktop-en-service-link-runSearch-fullRecord-https-3A-2F-2Fgithub.com │
│ 1 │ 1ABC2DEF3GHI4JKL5MNO6PQRS7TUV8WXYZ90RON/https-login.live.com-login.srf-wa-wsignin1.0-rpsnv-13-ct-1617864377-rver-6.7.6631.0-wp-MBI_SSL-wre │
│ 1 │ A1B2C3D4E5F6G7H89IJ0K1L2M3N4O5P6Q7R8S9T/https-www.google.com-search-q-NAVY-oq-NAVY-aqs-chrome..69i57j0i271l3.2146j0j9-sourceid-chrome-ie-U │
│ 1 │ A1B2C3D4E5F6G7H89IJ0K1L2M3N4O5P6Q7R8S9T/https-www.google.com-search-q-https-3A-2F-2Fwww.bing.com-2Fsearch-3Fq-3D-2524-25C2-25A4-25D8-258B- │
│ 1 │ LANGUAGESUNIFIPHINGEOMETRICALPHABETICAL/https-accounts.google.com-signin-oauth-consent-authuser-0-part-AJi8hAMVntzos7OLNO3NkoJYtIH-QjU-PmW │
│ 1 │ ACRONYMATHEMATICOMMUNICATIONUMERICALIFO/https-www.google.com-search-q-https-3A-2F-2Fwww.bing.com-2Fsearch-3Fq-3D-2524-25C2-25A4-25D8-258B- │
│ 1 │ LANGUAGESUNIFIPHINGEOMETRICALPHABETICAL/https-github.com-ALPHANUMERICALDRONTELOGICALCHEMYSTERIOS-https-studio.youtube.com-channel-UCao_6EW │
│ 1 │ 1ABC2DEF3GHI4JKL5MNO6PQRS7TUV8WXYZ90RON/https-www.google.com-search-q-NAVY-oq-NAVY-aqs-chrome..69i57j0i271l3.2146j0j9-sourceid-chrome-ie-U │
│ 1 │ ALPHABETAPHIDELTATHETASIGMAEPSILONLAMDA/https-www.google.com-search-q-https-3A-2F-2Fwww.bing.com-2Fsearch-3Fq-3D-2524-25C2-25A4-25D8-258B- │
│ 1 │ 1ABC2DEF3GHI4JKL5MNO6PQRS7TUV8WXYZ90RON/https-www.reddit.com-r-UzGXxQu51yT8nFzUuSf1W-comments-mmjgpf-httpsgithubcomsolveforcehttpswwwreddi │
│ 1 │ https-facebook-com-post-NikaKilasonia/https-facebook.comprofile-php.github.io-permalink.php-story_fbid-184489347245943-id-120007054636829- │
│ 1 │ LOGOIOGOLINGUIAIMIAWIFIPHIMITIOMWONOWEP/https-discord.com-connections-youtube-state-1741b8e0b735e7ff2e2d812723c5c994-code-4-0AY0e-g6hW-Y9X │
└─────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
50 rows in set. Elapsed: 2.890 sec. Processed 554.52 million rows, 3.39 GB (191.87 million rows/s., 1.17 GB/s.)
The most favorite of the insane is the "132 e's". Did you miss the story?
:) SELECT repo_name, count() FROM github_events WHERE event_type = 'WatchEvent' AND repo_name LIKE '%_/_%' GROUP BY repo_name ORDER BY length(repo_name) ASC LIMIT 50
┌─repo_name─┬─count()─┐
│ e/m │ 1 │
│ x/x │ 2 │
│ 2/t │ 1 │
│ 2/1 │ 1 │
│ k/k │ 1 │
│ f/f │ 38 │
│ 2/2 │ 4 │
│ s/s │ 3 │
│ f/mb │ 1 │
│ 9H/F │ 2 │
│ f/hy │ 1 │
│ 1k/Y │ 1 │
│ 8l/b │ 1 │
│ f/pq │ 165 │
│ i/AI │ 2 │
│ cv/t │ 33 │
│ Xe/x │ 297 │
│ nd/e │ 1 │
│ tj/n │ 19588 │
│ Xe/h │ 7 │
│ 69/a │ 3 │
│ as/a │ 431 │
│ gc/e │ 3 │
│ f/r3 │ 6 │
│ 3x/n │ 3 │
│ 7f/h │ 2 │
│ hf/q │ 8 │
│ 9H/Z │ 1 │
│ l/za │ 1 │
│ 6/qj │ 4 │
│ mg/i │ 7 │
│ 19/x │ 2 │
│ 5-7/- │ 1 │
│ Xe/ln │ 42 │
│ rr/rr │ 361 │
│ 520/a │ 7 │
│ tj/es │ 278 │
│ d7n/t │ 1 │
│ 9xN/v │ 1 │
│ d3/d3 │ 75736 │
│ b/hex │ 11 │
│ 51c/0 │ 1 │
│ 8l/ff │ 43 │
│ MD4/I │ 2 │
│ Vh9/Z │ 1 │
│ f/fka │ 25 │
│ kjk/w │ 7 │
│ 51c/5 │ 1 │
│ ddm/- │ 1 │
│ rf/nd │ 52 │
└───────────┴─────────┘
50 rows in set. Elapsed: 2.860 sec. Processed 554.52 million rows, 3.39 GB (193.89 million rows/s., 1.19 GB/s.)
I'm surprised that tj/n is a real thing. f/pq is also a real thing and also related to node.js. Maybe node.js developers are addicted to short names?
:) SELECT repo_name, count() FROM github_events WHERE body ILIKE '%ClickHouse%' GROUP BY repo_name ORDER BY count() DESC LIMIT 50
┌─repo_name──────────────────────────────────────┬─count()─┐
│ ClickHouse/ClickHouse │ 175958 │
│ apache/incubator-gluten │ 11064 │
│ oap-project/gluten │ 8877 │
│ PostHog/posthog │ 8468 │
│ ClickHouse/clickhouse-docs │ 8182 │
│ yandex/ClickHouse │ 7411 │
│ ClickHouse/clickhouse-java │ 4631 │
│ Altinity/clickhouse-operator │ 4301 │
│ getsentry/snuba │ 4096 │
│ traceon/ClickHouse │ 4074 │
│ tyjdf/ghfdg │ 4068 │
│ dtyjns/dtfg │ 3302 │
│ ClickHouse/clickhouse-go │ 2838 │
│ kokizzu/ClickHouse │ 2804 │
│ bitnami/containers │ 2421 │
│ ClickHouse/click-ui │ 2282 │
│ htydf/tzyfnd88 │ 2205 │
│ airbytehq/airbyte │ 2192 │
│ SigNoz/signoz │ 2150 │
│ ibis-project/ibis │ 2113 │
│ duyet/clickhouse-monitoring │ 2096 │
│ AlexAkulov/clickhouse-backup │ 1974 │
│ ClickHouse/clickhouse-jdbc │ 1940 │
│ oxidecomputer/omicron │ 1925 │
│ RTHSA/DTYJHN │ 1903 │
│ Altinity/clickhouse-backup │ 1879 │
│ Mu-L/ClickHouse │ 1868 │
│ skirdey/ClickHouse │ 1863 │
│ open-telemetry/opentelemetry-collector-contrib │ 1832 │
│ yjndsrt/cn │ 1708 │
│ Altinity/ClickHouse │ 1707 │
│ apecloud/helm-charts │ 1662 │
│ tensorzero/tensorzero │ 1572 │
│ truecharts/charts │ 1566 │
│ langfuse/langfuse │ 1562 │
│ unkeyed/unkey │ 1507 │
│ ClickHouse/clickhouse-connect │ 1415 │
│ sfafsfafasfa/rhz │ 1292 │
│ cloudquery/cloudquery │ 1289 │
│ sergeybataev/homelab │ 1245 │
│ datafuselabs/databend │ 1233 │
│ getsentry/sentry │ 1216 │
│ great-expectations/great_expectations │ 1204 │
│ trinodb/trino │ 1201 │
│ metabase/metabase │ 1173 │
│ grafana/clickhouse-datasource │ 1168 │
│ mymarilyn/clickhouse-driver │ 1162 │
│ ClickHouse/dbt-clickhouse │ 1159 │
│ sentry-kubernetes/charts │ 1111 │
│ getsentry/self-hosted │ 1111 │
└────────────────────────────────────────────────┴─────────┘
50 rows in set. Elapsed: 12.839 sec. Processed 10.36 billion rows, 2.60 TB (806.68 million rows/s., 202.46 GB/s.)
There are 30 897 of those. Excluding ClickHouse's own repositories, the most active are Apache Gluten, then PostHog, the JDBC driver, the Kubernetes operator and Sentry Snuba.
Repositories with ClickHouse-related comments, sorted by stars:
SELECT
repo_name,
sum(event_type = 'WatchEvent') AS num_stars,
sum(body ILIKE '%ClickHouse%') AS num_comments
FROM github_events
WHERE (body ILIKE '%ClickHouse%') OR (event_type = 'WatchEvent')
GROUP BY repo_name
HAVING num_comments > 0
ORDER BY num_stars DESC
LIMIT 50
┌─repo_name─────────────────────────────┬─num_stars─┬─num_comments─┐
1. │ sindresorhus/awesome │ 450318 │ 1 │
2. │ 996icu/996.ICU │ 378812 │ 1 │
3. │ kamranahmedse/developer-roadmap │ 363470 │ 1 │
4. │ donnemartin/system-design-primer │ 336938 │ 1 │
5. │ vinta/awesome-python │ 292819 │ 2 │
6. │ awesome-selfhosted/awesome-selfhosted │ 226992 │ 4 │
7. │ avelino/awesome-go │ 171577 │ 12 │
8. │ golang/go │ 153423 │ 30 │
9. │ n8n-io/n8n │ 138806 │ 11 │
10. │ 521xueweihan/HelloGitHub │ 137952 │ 7 │
11. │ microsoft/vscode │ 124831 │ 24 │
12. │ nodejs/node │ 123699 │ 1 │
13. │ kubernetes/kubernetes │ 120717 │ 2 │
14. │ rust-lang/rust │ 118506 │ 7 │
15. │ langgenius/dify │ 115341 │ 18 │
16. │ open-webui/open-webui │ 103675 │ 3 │
17. │ pytorch/pytorch │ 103248 │ 39 │
18. │ iluwatar/java-design-patterns │ 102937 │ 1 │
19. │ bitcoin/bitcoin │ 101330 │ 1 │
20. │ deepseek-ai/DeepSeek-V3 │ 97427 │ 1 │
21. │ langflow-ai/langflow │ 93672 │ 54 │
22. │ supabase/supabase │ 93270 │ 21 │
23. │ denoland/deno │ 92312 │ 3 │
24. │ spring-projects/spring-boot │ 90796 │ 23 │
25. │ Significant-Gravitas/Auto-GPT │ 84899 │ 3 │
26. │ python/cpython │ 82384 │ 6 │
27. │ ruanyf/weekly │ 78248 │ 21 │
28. │ nestjs/nest │ 77855 │ 1 │
29. │ immich-app/immich │ 77109 │ 3 │
30. │ ansible/ansible │ 75461 │ 12 │
31. │ elastic/elasticsearch │ 75306 │ 9 │
32. │ nomic-ai/gpt4all │ 74468 │ 2 │
33. │ grafana/grafana │ 73932 │ 339 │
34. │ openclaw/openclaw │ 73358 │ 1 │
35. │ fffaraz/awesome-cpp │ 73177 │ 4 │
36. │ louislam/uptime-kuma │ 73066 │ 3 │
37. │ scikit-learn/scikit-learn │ 72681 │ 5 │
38. │ zed-industries/zed │ 70563 │ 47 │
39. │ lobehub/lobe-chat │ 69128 │ 1 │
40. │ rails/rails │ 68604 │ 6 │
41. │ oven-sh/bun │ 67016 │ 16 │
42. │ openai/openai-cookbook │ 67001 │ 1 │
43. │ spring-projects/spring-framework │ 66830 │ 4 │
44. │ langchain-ai/langchain │ 66771 │ 99 │
45. │ astral-sh/uv │ 65510 │ 19 │
46. │ prometheus/prometheus │ 64729 │ 14 │
47. │ GrowingGit/GitHub-Chinese-Top-Charts │ 63208 │ 1 │
48. │ modelcontextprotocol/servers │ 61513 │ 15 │
49. │ TryGhost/Ghost │ 59542 │ 23 │
50. │ minio/minio │ 59511 │ 7 │
└───────────────────────────────────────┴───────────┴──────────────┘
50 rows in set. Elapsed: 13.568 sec. Processed 10.69 billion rows, 2.60 TB (787.80 million rows/s., 191.74 GB/s.)
Even the "996.ICU" repository has something about ClickHouse. I cannot find it on the GitHub website, but I can do it with our dataset!
:) SELECT * FROM github_events WHERE body ILIKE '%ClickHouse%' AND repo_name = '996icu/996.ICU' \G
Row 1:
──────
file_time: 2019-03-28 05:00:00
event_type: IssuesEvent
actor_login: garyelephant
repo_name: 996icu/996.ICU
created_at: 2019-03-28 05:52:55
updated_at: 2019-03-28 05:52:54
action: opened
comment_id: 0
body: > 去了解和学习一个真正的大数据项目吧，一技在手，走遍天下。
这里有一个基于Spark的项目，可以让我们不写spark代码，用最简单的配置，迅速跑起来流式streaming或离线的数据处理或分析的spark程序，大家可以玩一玩。它有丰富的数据输入，输出插件，比如kafka, elasticsearch, mongodb, mysql, hdfs, hive，clickhouse，TiDB 还可以直接用sql做数据处理。如果觉得功能不够还可以开发自己的插件，挺方便的。目前有微博，新浪，永辉超市等多家公司在线上使用。
* 项目地址：https://github.com/InterestingLab/waterdrop
* 文档地址: https://interestinglab.github.io/waterdrop/
* 附一篇用waterdrop流式处理kafka数据写入ES的介绍：
https://interestinglab.github.io/waterdrop/#/zh-cn/case_study/3
* 用Spark支持分布式，大规模的数据写入ES：
https://interestinglab.github.io/waterdrop/#/zh-cn/case_study/3
path:
position: 0
line: 0
ref:
ref_type: none
creator_user_login: garyelephant
number: 7029
title: 除了进ICU，大数据从业人员还能做什么？
labels: []
state: open
locked: 0
assignee:
assignees: []
comments: 0
author_association: NONE
closed_at: 1970-01-01 00:00:00
merged_at: 1970-01-01 00:00:00
merge_commit_sha:
requested_reviewers: []
requested_teams: []
head_ref:
head_sha:
base_ref:
base_sha:
merged: 0
mergeable: 0
rebaseable: 0
mergeable_state: unknown
merged_by:
review_comments: 0
maintainer_can_modify: 0
commits: 0
additions: 0
deletions: 0
changed_files: 0
diff_hunk:
original_position: 0
commit_id:
original_commit_id:
push_size: 0
push_distinct_size: 0
member_login:
release_tag_name:
release_name:
review_state: none
1 rows in set. Elapsed: 0.091 sec. Processed 2.98 million rows, 125.24 MB (32.74 million rows/s., 1.37 GB/s.)
It's a job offer from "Waterdrop" company that is using ClickHouse.
:) SELECT body, count() FROM github_events WHERE notEmpty(body) AND length(body) < 100 GROUP BY body ORDER BY count() DESC LIMIT 50
┌─body──────────────────────────────────────────────────────┬─count()─┐
│ Automated test pull request │ 5206013 │
│ LGTM │ 2773899 │
│ Body of PR │ 1836825 │
│ autogenerated pr │ 1678276 │
│ done │ 1166280 │
│ @dependabot merge │ 1082066 │
│ Done │ 1054629 │
│ @dependabot rebase │ 1016942 │
│ Superseded by #6. │ 971673 │
│ Thanks! │ 971086 │
│ Superseded by #7. │ 921323 │
│ Please provide me the recommendations │ 918786 │
│ Superseded by #9. │ 860678 │
│ The following labels could not be found: `dependencies`. │ 851016 │
│ Superseded by #10. │ 843990 │
│ Superseded by #8. │ 835120 │
│ Superseded by #5. │ 819063 │
│ Superseded by #11. │ 802994 │
│ Superseded by #4. │ 734053 │
│ Superseded by #12. │ 727338 │
│ Superseded by #13. │ 666758 │
│ Superseded by #16. │ 642441 │
│ Superseded by #17. │ 640718 │
│ Superseded by #14. │ 638254 │
│ +1 │ 636444 │
│ /lgtm │ 635659 │
│ 👍 │ 631968 │
│ Superseded by #15. │ 626455 │
│ Superseded by #18. │ 625390 │
│ Superseded by #20. │ 622050 │
│ Done. │ 620751 │
│ Superseded by #21. │ 620623 │
│ Superseded by #19. │ 618263 │
│ Superseded by #3. │ 606447 │
│ Automatically generated by Netlify CMS │ 601732 │
│ Superseded by #22. │ 584079 │
│ test │ 554672 │
│ Superseded by #23. │ 554260 │
│ lgtm │ 542202 │
│ Superseded by #24. │ 523661 │
│ Superseded by #2. │ 508115 │
│ LGTM! │ 507034 │
│ fixed │ 502635 │
│ ok │ 497166 │
│ /retest │ 496159 │
│ @dependabot squash and merge │ 477233 │
│ Superseded by #25. │ 476545 │
│ :+1: │ 470878 │
│ <samp> ↴│ 454685 │
│↳Pipelines successfully started running 1 pipeline(s).<br>↴│ │
│↳</samp> │ │
│ Automated commit │ 432339 │
└───────────────────────────────────────────────────────────┴─────────┘
50 rows in set. Elapsed: 12.839 sec. Processed 10.36 billion rows, 2.60 TB (806.68 million rows/s., 202.46 GB/s.)
(among comments shorter than 100 bytes)
The list is machinery now: bodies of automatically opened pull requests, Dependabot instructions, and an endless "Superseded by #N." series from bots that keep replacing their own pull requests. What people contribute is still short and positive: LGTM, done, Thanks!, +1, 👍.
If we roll a dice and SELECT 50 random repositories with probability proportional to the number of stars, what will we get?
:) SELECT repo_name FROM github_events WHERE event_type = 'WatchEvent' ORDER BY rand() LIMIT 50
┌─repo_name─────────────────────────────────┐
│ samoshkin/tmux-config │
│ kubernetes/kubernetes │
│ jbranchaud/til │
│ dotchen/LearningByCheating │
│ leemunroe/responsive-html-email-template │
│ Unity-Technologies/ml-agents │
│ Rayhahah/AdvancedAnim │
│ freach/kubernetes-security-best-practice │
│ seanprashad/leetcode-patterns │
│ theogf/Turkie.jl │
│ anydb-rs/anydb │
│ aplus-framework/one │
│ 2crashtestorc28/g4v8t83nwx5y │
│ metafizzy/packery │
│ getdnsapi/stubby │
│ liuxinyu95/unplugged │
│ IbraOO7/AppKlinik │
│ codlab/android_screen │
│ shiyunbo/django-smartdoc │
│ technogirls/technogirls.github.io │
│ afarhan/sbitx │
│ testableapple/how-they-automate-on-mobile │
│ mohammedghanemi08/My-Personal-web-page │
│ srawlins/allocation_stats │
│ namtk/ManiNetCluster │
│ mrdulin/blog │
│ Maxxum69/lambda-packager │
│ jaredLunde/masonic │
│ MIT-LCP/mimic-code │
│ schmittjoh/metadata │
│ alexichepura/bevy_garage │
│ GDQuest/gdscript-docs-maker │
│ HT524/500LineorLess_CN │
│ devopsacademyau/academy │
│ EugeneMeles/laravel-react-i18n │
│ AlexanderKoch-Koch/low_cost_robot │
│ paypal/junodb │
│ codigoencasa/builderbot │
│ buildermare/Sui-Modules │
│ ongres/stackgres │
│ oomol-lab/pdf-craft │
│ Pythagora-io/gpt-pilot │
│ asottile/dead │
│ Unitech/pm2 │
│ junghyun0783/spring2 │
│ public-apis/public-apis │
│ l2xin/CommonUtils │
│ facebook/react-native │
│ vinta/awesome-python │
│ mihaip/dex-method-counts │
└───────────────────────────────────────────┘
50 rows in set. Elapsed: 0.325 sec. Processed 554.92 million rows, 4.99 GB (1.71 billion rows/s., 15.36 GB/s.)
This is the true "face" of GitHub. Run a few queries and you can estimate and taste the Octoverse this way.
Follow the instruction.
You can download the dataset as a file and then insert into ClickHouse or any other DBMS.
Here is the dataset in ClickHouse Native format (73 GB).
Alternatively you can download the same dataset in tab-separated format (85 GB).
Note that these two files are a snapshot taken in February 2022, so they are considerably smaller than the live dataset. The demo server is updated every hour and contains everything.
To decompress the dataset from .xz it's recommended to install and use the pixz (parallel xz) tool. The ordinary xz tool can be used as well.
To insert the downloaded dataset into ClickHouse, create a table:
$ clickhouse-client
ClickHouse client version 20.11.3.3 (official build).
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 20.11.3 revision 54442.
:)
CREATE TABLE github_events
(
file_time DateTime,
event_type Enum('CommitCommentEvent' = 1, 'CreateEvent' = 2, 'DeleteEvent' = 3, 'ForkEvent' = 4,
'GollumEvent' = 5, 'IssueCommentEvent' = 6, 'IssuesEvent' = 7, 'MemberEvent' = 8,
'PublicEvent' = 9, 'PullRequestEvent' = 10, 'PullRequestReviewCommentEvent' = 11,
'PushEvent' = 12, 'ReleaseEvent' = 13, 'SponsorshipEvent' = 14, 'WatchEvent' = 15,
'GistEvent' = 16, 'FollowEvent' = 17, 'DownloadEvent' = 18, 'PullRequestReviewEvent' = 19,
'ForkApplyEvent' = 20, 'Event' = 21, 'TeamAddEvent' = 22),
actor_login LowCardinality(String),
repo_name LowCardinality(String),
created_at DateTime,
updated_at DateTime,
action Enum('none' = 0, 'created' = 1, 'added' = 2, 'edited' = 3, 'deleted' = 4, 'opened' = 5, 'closed' = 6, 'reopened' = 7, 'assigned' = 8, 'unassigned' = 9,
'labeled' = 10, 'unlabeled' = 11, 'review_requested' = 12, 'review_request_removed' = 13, 'synchronize' = 14, 'started' = 15, 'published' = 16, 'update' = 17, 'create' = 18, 'fork' = 19, 'merged' = 20),
comment_id UInt64,
body String,
path String,
position Int32,
line Int32,
ref LowCardinality(String),
ref_type Enum('none' = 0, 'branch' = 1, 'tag' = 2, 'repository' = 3, 'unknown' = 4),
creator_user_login LowCardinality(String),
number UInt32,
title String,
labels Array(LowCardinality(String)),
state Enum('none' = 0, 'open' = 1, 'closed' = 2),
locked UInt8,
assignee LowCardinality(String),
assignees Array(LowCardinality(String)),
comments UInt32,
author_association Enum('NONE' = 0, 'CONTRIBUTOR' = 1, 'OWNER' = 2, 'COLLABORATOR' = 3, 'MEMBER' = 4, 'MANNEQUIN' = 5),
closed_at DateTime,
merged_at DateTime,
merge_commit_sha String,
requested_reviewers Array(LowCardinality(String)),
requested_teams Array(LowCardinality(String)),
head_ref LowCardinality(String),
head_sha String,
base_ref LowCardinality(String),
base_sha String,
merged UInt8,
mergeable UInt8,
rebaseable UInt8,
mergeable_state Enum('unknown' = 0, 'dirty' = 1, 'clean' = 2, 'unstable' = 3, 'draft' = 4, 'blocked' = 5),
merged_by LowCardinality(String),
review_comments UInt32,
maintainer_can_modify UInt8,
commits UInt32,
additions UInt32,
deletions UInt32,
changed_files UInt32,
diff_hunk String,
original_position UInt32,
commit_id String,
original_commit_id String,
push_size UInt32,
push_distinct_size UInt32,
member_login LowCardinality(String),
release_tag_name String,
release_name String,
review_state Enum('none' = 0, 'approved' = 1, 'changes_requested' = 2, 'commented' = 3, 'dismissed' = 4, 'pending' = 5),
PROJECTION max_file_time
(
SELECT max(file_time)
)
)
ENGINE = MergeTree
ORDER BY (event_type, repo_name, created_at)
And insert it with the command:
pixz -d < github_events.native.xz | clickhouse-client --query "INSERT INTO github_events FORMAT Native"
You can transform the datasets between various formats with the clickhouse-local tool. Example to convert data from Native to JSONEachRow (aka jsonlines aka ndjson):
xz -d < github_events.native.xz | clickhouse-local --input-format Native --output-format JSONEachRow --query "SELECT * FROM table" --structure "event_type Enum('CommitCommentEvent' = 1, 'CreateEvent' = 2, 'DeleteEvent' = 3, 'ForkEvent' = 4, 'GollumEvent' = 5, 'IssueCommentEvent' = 6, 'IssuesEvent' = 7, 'MemberEvent' = 8, 'PublicEvent' = 9, 'PullRequestEvent' = 10, 'PullRequestReviewCommentEvent' = 11, 'PushEvent' = 12, 'ReleaseEvent' = 13, 'SponsorshipEvent' = 14, 'WatchEvent' = 15, 'GistEvent' = 16, 'FollowEvent' = 17, 'DownloadEvent' = 18, 'PullRequestReviewEvent' = 19, 'ForkApplyEvent' = 20), actor_login LowCardinality(String), repo_name LowCardinality(String), created_at DateTime, updated_at DateTime, action Enum('none' = 0, 'created' = 1, 'added' = 2, 'edited' = 3, 'deleted' = 4, 'opened' = 5, 'closed' = 6, 'reopened' = 7, 'assigned' = 8, 'unassigned' = 9, 'labeled' = 10, 'unlabeled' = 11, 'review_requested' = 12, 'review_request_removed' = 13, 'synchronize' = 14, 'started' = 15, 'published' = 16, 'update' = 17, 'create' = 18, 'fork' = 19), comment_id UInt64, body String, path String, position Int32, line UInt32, ref LowCardinality(String), ref_type Enum('none' = 0, 'branch' = 1, 'tag' = 2, 'repository' = 3), creator_user_login LowCardinality(String), number UInt32, title String, labels Array(LowCardinality(String)), state Enum('none' = 0, 'open' = 1, 'closed' = 2), locked UInt8, assignee LowCardinality(String), assignees Array(LowCardinality(String)), comments UInt32, author_association Enum('NONE' = 0, 'CONTRIBUTOR' = 1, 'OWNER' = 2, 'COLLABORATOR' = 3, 'MEMBER' = 4, 'MANNEQUIN' = 5), closed_at DateTime, merged_at DateTime, merge_commit_sha String, requested_reviewers Array(LowCardinality(String)), requested_teams Array(LowCardinality(String)), head_ref LowCardinality(String), head_sha String, base_ref LowCardinality(String), base_sha String, merged UInt8, mergeable UInt8, rebaseable UInt8, mergeable_state Enum('unknown' = 0, 'dirty' = 1, 'clean' = 2, 'unstable' = 3, 'draft' = 4), merged_by LowCardinality(String), review_comments UInt32, maintainer_can_modify UInt8, commits UInt32, additions UInt32, deletions UInt32, changed_files UInt32, diff_hunk String, original_position UInt32, commit_id String, original_commit_id String, push_size UInt32, push_distinct_size UInt32, member_login LowCardinality(String), release_tag_name String, release_name String, review_state Enum('none' = 0, 'approved' = 1, 'changes_requested' = 2, 'commented' = 3, 'dismissed' = 4, 'pending' = 5)"
This is an option to import the dataset into ClickHouse without a separate download and decompression step.
It requires minimum ClickHouse version 20.12 to support xz decompression.
Create a foreign table that will read data from a URL:
CREATE TABLE github_events_url
ENGINE = URL('https://clickhouse-public-datasets.s3.amazonaws.com/github_events_v2.native.xz');
Create the destination table and insert data:
CREATE TABLE github_events ENGINE = MergeTree ORDER BY (event_type, repo_name, created_at) AS SELECT * FROM github_events_url;
This section explains how to recreate the dataset from the raw data (about 5 TB json.gz). You don't have to follow these steps because we have already prepared a preprocessed structured dataset for download (just 65 GB).
There is a wonderful project called GH Archive. It provides dumps of events on GitHub to download. Data is split into files per hour like https://data.gharchive.org/2015-01-01-15.json.gz.
You can download all the files with the command:
wget --continue https://data.gharchive.org/{2011..2026}-{01..12}-{01..31}-{0..23}.json.gz
If you see the Argument list too long message, split to multiple commands by years.
If download fails, just run the command again (--continue will skip already downloaded files).
The total number of files is about 135 000 and the total size is about 5 TB.
Typical download speed is ~16 MB/sec. The download will take several days.
There is no point parallelizing the download within one server, because rate limits will apply and the effective download speed will remain the same. You can parallelize by using different servers... but it's better not to bother because it is just a few terabytes and you just need to wait from several days to a week at most. Prepare a server with at least 6 terabytes of space, run the download, leave for the weekend and that's it.
The raw dataset is a bunch of gzipped files containing data in JSON Lines format (JSON object on every line). The structure of the JSON object is the same as what the GitHub API provides. Every JSON object contains one event and all the related info. For example, let's look at the PullRequestEvent:
gzip -cd < 2020-11-13-9.json.gz | grep -F 'PullRequestEvent' | head -n1 | jq '.'
{
"id": "14180283452",
"type": "PullRequestEvent",
"actor": {
"id": 10810283,
"login": "direwolf-github",
"display_login": "direwolf-github",
"gravatar_id": "",
"url": "https://api.github.com/users/direwolf-github",
"avatar_url": "https://avatars.githubusercontent.com/u/10810283?"
},
"repo": {
"id": 312523815,
"name": "direwolf-github/ephemeral-ci-ff8a7037",
"url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037"
},
"payload": {
"action": "opened",
"number": 1,
"pull_request": {
"url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/pulls/1",
"id": 520447878,
"node_id": "MDExOlB1bGxSZXF1ZXN0NTIwNDQ3ODc4",
"html_url": "https://github.com/direwolf-github/ephemeral-ci-ff8a7037/pull/1",
"diff_url": "https://github.com/direwolf-github/ephemeral-ci-ff8a7037/pull/1.diff",
"patch_url": "https://github.com/direwolf-github/ephemeral-ci-ff8a7037/pull/1.patch",
"issue_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/issues/1",
"number": 1,
"state": "open",
"locked": false,
"title": "Direwolf review apps test branch-b928a7ac",
"user": {
"login": "direwolf-github",
"id": 10810283,
"node_id": "MDQ6VXNlcjEwODEwMjgz",
"avatar_url": "https://avatars0.githubusercontent.com/u/10810283?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/direwolf-github",
"html_url": "https://github.com/direwolf-github",
"followers_url": "https://api.github.com/users/direwolf-github/followers",
"following_url": "https://api.github.com/users/direwolf-github/following{/other_user}",
"gists_url": "https://api.github.com/users/direwolf-github/gists{/gist_id}",
"starred_url": "https://api.github.com/users/direwolf-github/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/direwolf-github/subscriptions",
"organizations_url": "https://api.github.com/users/direwolf-github/orgs",
"repos_url": "https://api.github.com/users/direwolf-github/repos",
"events_url": "https://api.github.com/users/direwolf-github/events{/privacy}",
"received_events_url": "https://api.github.com/users/direwolf-github/received_events",
"type": "User",
"site_admin": false
},
"body": "Direwolf review apps test branch-b928a7ac",
"created_at": "2020-11-13T08:59:59Z",
"updated_at": "2020-11-13T08:59:59Z",
"closed_at": null,
"merged_at": null,
"merge_commit_sha": null,
"assignee": null,
"assignees": [],
"requested_reviewers": [],
"requested_teams": [],
"labels": [],
"milestone": null,
"draft": false,
"commits_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/pulls/1/commits",
"review_comments_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/pulls/1/comments",
"review_comment_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/pulls/comments{/number}",
"comments_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/issues/1/comments",
"statuses_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/statuses/d48d634ff6ea6973baa91724836e797232cd4424",
"head": {
"label": "direwolf-github:branch-b928a7ac",
"ref": "branch-b928a7ac",
"sha": "d48d634ff6ea6973baa91724836e797232cd4424",
"user": {
"login": "direwolf-github",
"id": 10810283,
"node_id": "MDQ6VXNlcjEwODEwMjgz",
"avatar_url": "https://avatars0.githubusercontent.com/u/10810283?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/direwolf-github",
"html_url": "https://github.com/direwolf-github",
"followers_url": "https://api.github.com/users/direwolf-github/followers",
"following_url": "https://api.github.com/users/direwolf-github/following{/other_user}",
"gists_url": "https://api.github.com/users/direwolf-github/gists{/gist_id}",
"starred_url": "https://api.github.com/users/direwolf-github/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/direwolf-github/subscriptions",
"organizations_url": "https://api.github.com/users/direwolf-github/orgs",
"repos_url": "https://api.github.com/users/direwolf-github/repos",
"events_url": "https://api.github.com/users/direwolf-github/events{/privacy}",
"received_events_url": "https://api.github.com/users/direwolf-github/received_events",
"type": "User",
"site_admin": false
},
"repo": {
"id": 312523815,
"node_id": "MDEwOlJlcG9zaXRvcnkzMTI1MjM4MTU=",
"name": "ephemeral-ci-ff8a7037",
"full_name": "direwolf-github/ephemeral-ci-ff8a7037",
"private": false,
"owner": {
"login": "direwolf-github",
"id": 10810283,
"node_id": "MDQ6VXNlcjEwODEwMjgz",
"avatar_url": "https://avatars0.githubusercontent.com/u/10810283?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/direwolf-github",
"html_url": "https://github.com/direwolf-github",
"followers_url": "https://api.github.com/users/direwolf-github/followers",
"following_url": "https://api.github.com/users/direwolf-github/following{/other_user}",
"gists_url": "https://api.github.com/users/direwolf-github/gists{/gist_id}",
"starred_url": "https://api.github.com/users/direwolf-github/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/direwolf-github/subscriptions",
"organizations_url": "https://api.github.com/users/direwolf-github/orgs",
"repos_url": "https://api.github.com/users/direwolf-github/repos",
"events_url": "https://api.github.com/users/direwolf-github/events{/privacy}",
"received_events_url": "https://api.github.com/users/direwolf-github/received_events",
"type": "User",
"site_admin": false
},
"html_url": "https://github.com/direwolf-github/ephemeral-ci-ff8a7037",
"description": null,
"fork": false,
"url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037",
"forks_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/forks",
"keys_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/keys{/key_id}",
"collaborators_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/collaborators{/collaborator}",
"teams_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/teams",
"hooks_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/hooks",
"issue_events_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/issues/events{/number}",
"events_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/events",
"assignees_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/assignees{/user}",
"branches_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/branches{/branch}",
"tags_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/tags",
"blobs_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/git/blobs{/sha}",
"git_tags_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/git/tags{/sha}",
"git_refs_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/git/refs{/sha}",
"trees_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/git/trees{/sha}",
"statuses_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/statuses/{sha}",
"languages_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/languages",
"stargazers_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/stargazers",
"contributors_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/contributors",
"subscribers_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/subscribers",
"subscription_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/subscription",
"commits_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/commits{/sha}",
"git_commits_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/git/commits{/sha}",
"comments_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/comments{/number}",
"issue_comment_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/issues/comments{/number}",
"contents_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/contents/{+path}",
"compare_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/compare/{base}...{head}",
"merges_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/merges",
"archive_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/{archive_format}{/ref}",
"downloads_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/downloads",
"issues_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/issues{/number}",
"pulls_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/pulls{/number}",
"milestones_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/milestones{/number}",
"notifications_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/notifications{?since,all,participating}",
"labels_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/labels{/name}",
"releases_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/releases{/id}",
"deployments_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/deployments",
"created_at": "2020-11-13T08:59:50Z",
"updated_at": "2020-11-13T08:59:55Z",
"pushed_at": "2020-11-13T08:59:59Z",
"git_url": "git://github.com/direwolf-github/ephemeral-ci-ff8a7037.git",
"ssh_url": "[email protected]:direwolf-github/ephemeral-ci-ff8a7037.git",
"clone_url": "https://github.com/direwolf-github/ephemeral-ci-ff8a7037.git",
"svn_url": "https://github.com/direwolf-github/ephemeral-ci-ff8a7037",
"homepage": null,
"size": 0,
"stargazers_count": 0,
"watchers_count": 0,
"language": null,
"has_issues": true,
"has_projects": true,
"has_downloads": true,
"has_wiki": true,
"has_pages": false,
"forks_count": 0,
"mirror_url": null,
"archived": false,
"disabled": false,
"open_issues_count": 1,
"license": null,
"forks": 0,
"open_issues": 1,
"watchers": 0,
"default_branch": "master"
}
},
"base": {
"label": "direwolf-github:master",
"ref": "master",
"sha": "e445f055327daff1e9f9909dbfd062c461646601",
"user": {
"login": "direwolf-github",
"id": 10810283,
"node_id": "MDQ6VXNlcjEwODEwMjgz",
"avatar_url": "https://avatars0.githubusercontent.com/u/10810283?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/direwolf-github",
"html_url": "https://github.com/direwolf-github",
"followers_url": "https://api.github.com/users/direwolf-github/followers",
"following_url": "https://api.github.com/users/direwolf-github/following{/other_user}",
"gists_url": "https://api.github.com/users/direwolf-github/gists{/gist_id}",
"starred_url": "https://api.github.com/users/direwolf-github/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/direwolf-github/subscriptions",
"organizations_url": "https://api.github.com/users/direwolf-github/orgs",
"repos_url": "https://api.github.com/users/direwolf-github/repos",
"events_url": "https://api.github.com/users/direwolf-github/events{/privacy}",
"received_events_url": "https://api.github.com/users/direwolf-github/received_events",
"type": "User",
"site_admin": false
},
"repo": {
"id": 312523815,
"node_id": "MDEwOlJlcG9zaXRvcnkzMTI1MjM4MTU=",
"name": "ephemeral-ci-ff8a7037",
"full_name": "direwolf-github/ephemeral-ci-ff8a7037",
"private": false,
"owner": {
"login": "direwolf-github",
"id": 10810283,
"node_id": "MDQ6VXNlcjEwODEwMjgz",
"avatar_url": "https://avatars0.githubusercontent.com/u/10810283?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/direwolf-github",
"html_url": "https://github.com/direwolf-github",
"followers_url": "https://api.github.com/users/direwolf-github/followers",
"following_url": "https://api.github.com/users/direwolf-github/following{/other_user}",
"gists_url": "https://api.github.com/users/direwolf-github/gists{/gist_id}",
"starred_url": "https://api.github.com/users/direwolf-github/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/direwolf-github/subscriptions",
"organizations_url": "https://api.github.com/users/direwolf-github/orgs",
"repos_url": "https://api.github.com/users/direwolf-github/repos",
"events_url": "https://api.github.com/users/direwolf-github/events{/privacy}",
"received_events_url": "https://api.github.com/users/direwolf-github/received_events",
"type": "User",
"site_admin": false
},
"html_url": "https://github.com/direwolf-github/ephemeral-ci-ff8a7037",
"description": null,
"fork": false,
"url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037",
"forks_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/forks",
"keys_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/keys{/key_id}",
"collaborators_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/collaborators{/collaborator}",
"teams_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/teams",
"hooks_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/hooks",
"issue_events_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/issues/events{/number}",
"events_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/events",
"assignees_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/assignees{/user}",
"branches_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/branches{/branch}",
"tags_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/tags",
"blobs_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/git/blobs{/sha}",
"git_tags_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/git/tags{/sha}",
"git_refs_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/git/refs{/sha}",
"trees_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/git/trees{/sha}",
"statuses_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/statuses/{sha}",
"languages_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/languages",
"stargazers_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/stargazers",
"contributors_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/contributors",
"subscribers_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/subscribers",
"subscription_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/subscription",
"commits_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/commits{/sha}",
"git_commits_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/git/commits{/sha}",
"comments_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/comments{/number}",
"issue_comment_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/issues/comments{/number}",
"contents_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/contents/{+path}",
"compare_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/compare/{base}...{head}",
"merges_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/merges",
"archive_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/{archive_format}{/ref}",
"downloads_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/downloads",
"issues_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/issues{/number}",
"pulls_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/pulls{/number}",
"milestones_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/milestones{/number}",
"notifications_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/notifications{?since,all,participating}",
"labels_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/labels{/name}",
"releases_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/releases{/id}",
"deployments_url": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/deployments",
"created_at": "2020-11-13T08:59:50Z",
"updated_at": "2020-11-13T08:59:55Z",
"pushed_at": "2020-11-13T08:59:59Z",
"git_url": "git://github.com/direwolf-github/ephemeral-ci-ff8a7037.git",
"ssh_url": "[email protected]:direwolf-github/ephemeral-ci-ff8a7037.git",
"clone_url": "https://github.com/direwolf-github/ephemeral-ci-ff8a7037.git",
"svn_url": "https://github.com/direwolf-github/ephemeral-ci-ff8a7037",
"homepage": null,
"size": 0,
"stargazers_count": 0,
"watchers_count": 0,
"language": null,
"has_issues": true,
"has_projects": true,
"has_downloads": true,
"has_wiki": true,
"has_pages": false,
"forks_count": 0,
"mirror_url": null,
"archived": false,
"disabled": false,
"open_issues_count": 1,
"license": null,
"forks": 0,
"open_issues": 1,
"watchers": 0,
"default_branch": "master"
}
},
"_links": {
"self": {
"href": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/pulls/1"
},
"html": {
"href": "https://github.com/direwolf-github/ephemeral-ci-ff8a7037/pull/1"
},
"issue": {
"href": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/issues/1"
},
"comments": {
"href": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/issues/1/comments"
},
"review_comments": {
"href": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/pulls/1/comments"
},
"review_comment": {
"href": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/pulls/comments{/number}"
},
"commits": {
"href": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/pulls/1/commits"
},
"statuses": {
"href": "https://api.github.com/repos/direwolf-github/ephemeral-ci-ff8a7037/statuses/d48d634ff6ea6973baa91724836e797232cd4424"
}
},
"author_association": "OWNER",
"active_lock_reason": null,
"merged": false,
"mergeable": null,
"rebaseable": null,
"mergeable_state": "unknown",
"merged_by": null,
"comments": 0,
"review_comments": 0,
"maintainer_can_modify": false,
"commits": 1,
"additions": 1,
"deletions": 0,
"changed_files": 1
}
},
"public": true,
"created_at": "2020-11-13T09:00:00Z"
}
It's quite a rich JSON. It contains the data about the pull request itself and all the associated objects (actor — who has modified the pull request, the author of the pull request, the repository where the pull request exists, etc.)
There are multiple ways to analyze a bunch of .json.gz files without any preprocessing. For example, you can use the jq tool, though it will be very inefficient.
A more efficient way to analyze data without preprocessing is by using the clickhouse-local tool. It is like a replacement for awk, sed, grep with the ClickHouse SQL engine:
clickhouse-local --query "
SELECT count() FROM file('*.json.gz', LineAsString, 'data String')
WHERE JSONExtractString(data, 'actor', 'login') = 'alexey-milovidov'"
Here we parse input data as a single String field per line and then process it with JSON functions.
clickhouse-local will do all the heavy-lifting:
— select files by wildcard pattern;
— decompress gzip data on the fly (with zlib-ng);
— parallelize data processing by files and by chunks within each file;
— process JSON with the help of simdjson — the fastest JSON library in existence;
Data processing will be limited by disk speed. If we assume 1 GB/sec sequential read speed, it will take about 20 minutes per query. You may think this is okay, but it isn't. It is just the best possible speed of processing raw .json.gz data, but we can do better if we preprocess our data.
If we want to analyze our data faster, we have to transform it from semistructured JSON format to a set of well-structured relational tables. The principle: structured is always more efficient than semistructured.
First let's find out what data we have by reading the GitHub API docs.
There are a bunch of event types:
— CommitCommentEvent;
— CreateEvent;
— DeleteEvent;
— ForkEvent;
— GollumEvent;
— IssueCommentEvent;
— IssuesEvent;
— MemberEvent;
— PublicEvent;
— PullRequestEvent;
— PullRequestReviewCommentEvent;
— PushEvent;
— ReleaseEvent;
— SponsorshipEvent;
— WatchEvent;
And the list is incomplete... while analyzing the data I have found a few "secret" or obsolete event types.
Every event type has its own properties.
What is the best way to structure all this data?
We can consider two options:
1. Every event type in its own table.
For example, WatchEvent is about stars assignment. We can create a table for these events. And when we need to calculate the number of stars per repository, the graph of stars or top starred repositories per year, we will just do SELECT from this table. But if we need to compare the ratio of stars and the number of pull request contributors, we will have to JOIN several tables.
2. All event types in a single table.
The table works like a "discriminated union". It will contain a field named event_type to discriminate various events. It will contain all columns that are common for different events like "event date and time", "actor name", "repository name". It will also contain the union of all different properties of different event types. We will have a wide table with many columns, but some columns will be very sparse (filled with values only for certain rare event types).
I can strongly recommend the second option: all events in one table.
The table will contain many columns... and that's exactly what column-oriented databases are for! Sparse columns will compress well, so you don't have to worry that they are sparse. As an added benefit, you can avoid doing queries with JOINs when you need to analyze multiple event types at once.
I analyzed the properties of all event types and ended up with the following table structure in ClickHouse:
CREATE TABLE github_events
(
file_time DateTime,
event_type Enum('CommitCommentEvent' = 1, 'CreateEvent' = 2, 'DeleteEvent' = 3, 'ForkEvent' = 4,
'GollumEvent' = 5, 'IssueCommentEvent' = 6, 'IssuesEvent' = 7, 'MemberEvent' = 8,
'PublicEvent' = 9, 'PullRequestEvent' = 10, 'PullRequestReviewCommentEvent' = 11,
'PushEvent' = 12, 'ReleaseEvent' = 13, 'SponsorshipEvent' = 14, 'WatchEvent' = 15,
'GistEvent' = 16, 'FollowEvent' = 17, 'DownloadEvent' = 18, 'PullRequestReviewEvent' = 19,
'ForkApplyEvent' = 20, 'Event' = 21, 'TeamAddEvent' = 22),
actor_login LowCardinality(String),
repo_name LowCardinality(String),
created_at DateTime,
updated_at DateTime,
action Enum('none' = 0, 'created' = 1, 'added' = 2, 'edited' = 3, 'deleted' = 4, 'opened' = 5, 'closed' = 6, 'reopened' = 7, 'assigned' = 8, 'unassigned' = 9,
'labeled' = 10, 'unlabeled' = 11, 'review_requested' = 12, 'review_request_removed' = 13, 'synchronize' = 14, 'started' = 15, 'published' = 16, 'update' = 17, 'create' = 18, 'fork' = 19, 'merged' = 20),
comment_id UInt64,
body String,
path String,
position Int32,
line Int32,
ref LowCardinality(String),
ref_type Enum('none' = 0, 'branch' = 1, 'tag' = 2, 'repository' = 3, 'unknown' = 4),
creator_user_login LowCardinality(String),
number UInt32,
title String,
labels Array(LowCardinality(String)),
state Enum('none' = 0, 'open' = 1, 'closed' = 2),
locked UInt8,
assignee LowCardinality(String),
assignees Array(LowCardinality(String)),
comments UInt32,
author_association Enum('NONE' = 0, 'CONTRIBUTOR' = 1, 'OWNER' = 2, 'COLLABORATOR' = 3, 'MEMBER' = 4, 'MANNEQUIN' = 5),
closed_at DateTime,
merged_at DateTime,
merge_commit_sha String,
requested_reviewers Array(LowCardinality(String)),
requested_teams Array(LowCardinality(String)),
head_ref LowCardinality(String),
head_sha String,
base_ref LowCardinality(String),
base_sha String,
merged UInt8,
mergeable UInt8,
rebaseable UInt8,
mergeable_state Enum('unknown' = 0, 'dirty' = 1, 'clean' = 2, 'unstable' = 3, 'draft' = 4, 'blocked' = 5),
merged_by LowCardinality(String),
review_comments UInt32,
maintainer_can_modify UInt8,
commits UInt32,
additions UInt32,
deletions UInt32,
changed_files UInt32,
diff_hunk String,
original_position UInt32,
commit_id String,
original_commit_id String,
push_size UInt32,
push_distinct_size UInt32,
member_login LowCardinality(String),
release_tag_name String,
release_name String,
review_state Enum('none' = 0, 'approved' = 1, 'changes_requested' = 2, 'commented' = 3, 'dismissed' = 4, 'pending' = 5)
) ENGINE = MergeTree ORDER BY (event_type, repo_name, created_at);
A few details:
1. Table is ordered by event_type. This means that when you SELECT with the condition on the event type, it will process as efficiently as if the selected events were located in a separate table.
2. Next to event_type we have repo_name. When you SELECT data for a certain repo, it will process almost instantly because only a few contiguous ranges of data will be read — only the ranges that contain data for this repo. Basically, ORDER BY expression in a table is the most efficient index possible. It's also worth noting that data sorted by repository name will compress better due to better data locality.
3. Next to repo_name we have created_at — the time of event. We put it as the last column in the ORDER BY expression, because proper data ordering will improve compression.
4. ENGINE = MergeTree is used as the recommended choice for medium to large sized datasets.
5. We use Enum data types when we know the set of values in advance.
6. We don't use Nullable data types, because they are less efficient than just using empty string / zeros for absence of values.
7. LowCardinality(String) is used to apply additional dictionary compression when strings are expected to repeat frequently. Data is always compressed in ClickHouse. But LowCardinality acts better than generic compression, because it allows to process data without decompression.
8. We use Array data type for labels, assignees, etc. It's quite convenient to have this option.
No external tools are needed: ClickHouse reads the .json.gz files itself and does the whole transformation in SQL.
INSERT INTO github_events
WITH parseDateTimeBestEffort(replaceRegexpOne(_file,
'^(\\d{4}-\\d{2}-\\d{2})-(\\d{1,2})\\.json\\.gz$', '\\1 \\2:00:00')) AS file_time
SELECT
file_time,
type,
COALESCE(actor.login, actor_attributes.login),
COALESCE(repo.name, repository.owner || '/' || repository.name),
created_at,
COALESCE(payload.updated_at, payload.comment.updated_at, payload.issue.updated_at, payload.pull_request.updated_at),
payload.action,
payload.comment.id,
COALESCE(payload.review.body, payload.comment.body, payload.issue.body, payload.pull_request.body, payload.release.body),
payload.comment.path,
payload.comment.position,
payload.comment.line,
payload.ref,
payload.ref_type,
COALESCE(payload.comment.user.login, payload.issue.user.login, payload.pull_request.user.login),
COALESCE(payload.issue.number, payload.pull_request.number, payload.number),
COALESCE(payload.issue.title, payload.pull_request.title),
arrayMap(x -> tupleElement(x, 'name'), payload.issue.labels || payload.pull_request.labels),
COALESCE(payload.issue.state, payload.pull_request.state),
COALESCE(payload.issue.locked, payload.pull_request.locked),
COALESCE(payload.issue.assignee.login, payload.pull_request.assignee.login),
arrayMap(x -> tupleElement(x, 'login'), payload.issue.assignees || payload.pull_request.assignees),
COALESCE(payload.issue.comments, payload.pull_request.comments),
COALESCE(payload.review.author_association, payload.issue.author_association, payload.pull_request.author_association),
COALESCE(payload.issue.closed_at, payload.pull_request.closed_at),
payload.pull_request.merged_at,
payload.pull_request.merged_commit_sha,
arrayMap(x -> tupleElement(x, 'login'), payload.pull_request.requested_reviewers),
arrayMap(x -> tupleElement(x, 'name'), payload.pull_request.requested_teams),
payload.pull_request.head.ref,
payload.pull_request.head.sha,
payload.pull_request.base.ref,
payload.pull_request.base.sha,
payload.pull_request.merged,
payload.pull_request.mergeable,
payload.pull_request.rebaseable,
payload.pull_request.mergeable_state,
payload.pull_request.merged_by.login,
payload.pull_request.review_comments,
payload.pull_request.maintainer_can_modify,
payload.pull_request.commits,
payload.pull_request.additions,
payload.pull_request.deletions,
payload.pull_request.changed_files,
payload.comment.diff_hunk,
payload.comment.original_position,
payload.comment.commit_id,
payload.comment.original_commit_id,
payload.size,
payload.distinct_size,
startsWith(payload.member, '{') ? JSONExtractString(payload.member, 'login') : payload.member,
payload.release.tag_name,
payload.release.name,
payload.review.state
FROM file('*.json.gz', JSONEachRow,
'
id Nullable(String),
type Nullable(String),
actor Tuple(
login Nullable(String)),
actor_attributes Tuple(
login Nullable(String)),
repo Tuple(
name Nullable(String)),
repository Tuple(
owner Nullable(String),
name Nullable(String)),
created_at DateTime,
payload Tuple(
updated_at Nullable(DateTime),
action Nullable(String),
ref Nullable(String),
ref_type Nullable(String),
number Nullable(Int64),
size Nullable(Int64),
distinct_size Nullable(Int64),
comment Tuple(
updated_at Nullable(DateTime),
id Nullable(String),
body Nullable(String),
path Nullable(String),
position Nullable(String),
line Nullable(String),
user Tuple(
login Nullable(String)),
diff_hunk Nullable(String),
original_position Nullable(String),
commit_id Nullable(String),
original_commit_id Nullable(String)),
issue Tuple(
updated_at Nullable(DateTime),
closed_at Nullable(DateTime),
body Nullable(String),
user Tuple(
login Nullable(String)),
number Nullable(Int64),
title Nullable(String),
labels Array(Tuple(name Nullable(String))),
state Nullable(String),
locked Nullable(String),
assignee Tuple(
login Nullable(String)),
assignees Array(Tuple(
login Nullable(String))),
comments Nullable(Int64),
author_association Nullable(String)),
pull_request Tuple(
updated_at Nullable(DateTime),
closed_at Nullable(DateTime),
merged_at Nullable(DateTime),
merged_commit_sha Nullable(String),
body Nullable(String),
user Tuple(
login Nullable(String)),
number Nullable(Int64),
title Nullable(String),
labels Array(Tuple(name Nullable(String))),
state Nullable(String),
locked Nullable(String),
assignee Tuple(
login Nullable(String)),
assignees Array(Tuple(
login Nullable(String))),
comments Nullable(Int64),
review_comments Nullable(Int64),
author_association Nullable(String),
requested_reviewers Array(Tuple(
login Nullable(String))),
requested_teams Array(Tuple(
name Nullable(String))),
head Tuple(
ref Nullable(String),
sha Nullable(String)),
base Tuple(
ref Nullable(String),
sha Nullable(String)),
merged Nullable(String),
mergeable Nullable(String),
rebaseable Nullable(String),
maintainer_can_modify Nullable(String),
mergeable_state Nullable(String),
merged_by Tuple(
login Nullable(String)),
commits Nullable(Int64),
additions Nullable(Int64),
deletions Nullable(Int64),
changed_files Nullable(Int64)),
review Tuple(
body Nullable(String),
author_association Nullable(String),
state Nullable(String)),
release Tuple(
body Nullable(String),
tag_name Nullable(String),
name Nullable(String)),
member Nullable(String)
)
')
SETTINGS date_time_input_format = 'best_effort', input_format_allow_errors_ratio = 0.01, input_format_allow_errors_num = 1000
Save it as load.sql and run clickhouse-client --queries-file load.sql.
A few details:
1. The file table function accepts a wildcard, so it reads every hourly file, decompresses gzip on the fly and parses JSON with simdjson, in parallel across the files and across chunks within every file. If you prefer not to download the archive first, replace file with url and pass the address of the file — everything else stays the same.
2. The third argument is the structure of the input: only the fields we are interested in. Everything else in the JSON is skipped. Nested objects are described with Tuple, and arrays of objects with Array(Tuple(...)). All fields are Nullable, because most of them are absent for most of the event types.
3. COALESCE takes the first field that is present. The format of the archive changed over the years, and the same logical value can live in different places — for example, the actor login is actor.login in the modern format and actor_attributes.login in the oldest files.
4. arrayMap(x -> tupleElement(x, 'name'), ...) turns an array of objects into an array of strings, which is what the labels, assignees and requested_reviewers columns are.
5. The destination columns are not Nullable, and NULLs are converted into default values (zeros and empty strings) on insertion — exactly what we decided in the previous section.
6. _file is a virtual column with the name of the file being read. The name contains the hour, and that's where file_time comes from.
7. date_time_input_format = 'best_effort' allows to parse the DateTime fields in various formats, including ISO-8601. The input_format_allow_errors settings let the load survive the handful of malformed records that the archive contains.
Data loading is limited by decompression and JSON parsing, and it uses all your cores. Previous versions of this article used the jq tool for this step and it was the bottleneck; it is not needed anymore.
Congratulations, now we are able to recreate the dataset from the raw data! (Assuming the GHArchive continues to exist).
If you want to continuously update the database, put one line in cron:
* * * * * ubuntu cd /home/ubuntu && ./update.sh >> update.log 2>&1
And update.sh runs the same INSERT as above, with two differences: it reads a single hour straight from a URL instead of local files, and it works out which hour to read from the data that is already loaded:
clickhouse-client --query_id github_events_update --query "
INSERT INTO github_events
WITH
(SELECT max(file_time) + INTERVAL 1 HOUR FROM github_events) AS time,
format(
'https://clickhouse-public-datasets.s3.amazonaws.com/gharchive/original/{}-{}.json.gz',
formatDateTime(time, '%Y-%m-%d'), toHour(time)) AS url
SELECT
time,
type,
COALESCE(actor.login, actor_attributes.login),
... the same expressions as above ...
FROM url((SELECT url), auto,
'
... the same structure as above ...
')
SETTINGS date_time_input_format = 'best_effort', input_format_allow_errors_ratio = 0.01, input_format_allow_errors_num = 1000
"
There is no separate download step at all — the URL points to a mirror of the GH Archive files, and one hour is inserted per run. If the next hour has not been published yet, the query fails, nothing is inserted, and the run a minute later tries again. The max(file_time) lookup is instant because of the max_file_time projection in the table definition, and the fixed --query_id makes ClickHouse refuse to start a second copy while the previous one is still running, which is what you want from a job that fires every minute.
Install the pixz tool for faster xz compression and run one of the following commands:
clickhouse-client --progress --max_threads 1 --query "SELECT * FROM github_events FORMAT TSV" | pixz > github_events.xz clickhouse-client --progress --max_threads 1 --query "SELECT * FROM github_events FORMAT Native" | pixz > github_events.native.xz
Connect with clickhouse-client:
clickhouse-client --secure --host play.clickhouse.com --user explorer
HTTPS interface: https://play.clickhouse.com/ (port 443)
Minimal web UI: https://play.clickhouse.com/play?user=play
Keep in mind that this dataset is actually very small for ClickHouse. There are multiple companies that use distributed multi-petabyte ClickHouse setups for various heavy duty tasks, e.g. to analyze a significant share of all internet traffic in real time. This article is not intended to advertise ClickHouse, but at least now you know where to look.
We encourage you to create your own research and tools based on the dataset. This article is open-source and the content is available under the CC-BY-4.0 license or Apache 2 license. Attribution is required. You can propose changes, extend the article, and share ideas by pull requests and issues on GitHub repository. Please notify us about interesting usages of the dataset. We also encourage the application of the dataset for DBMS benchmarks.
If you need to cite this article, please do it as follows:
"Milovidov A., 2020. Everything You Ever Wanted To Know About GitHub (But Were Afraid To Ask), https://ghe.clickhouse.tech/"
The authors don't own any rights to the dataset. The dataset includes material that may be subject to third party rights. The query results from the dataset are published under section 107 of the Copyright Act of 1976; allowance is made for "fair use" for purposes such as criticism, comment, news reporting, teaching, scholarship, education and research. Fair use is a use permitted by copyright statute that might otherwise be infringing.
This website does not use cookies. Discuss at Hacker News.
© 2020-2026 Alexey Milovidov