pzhdfy's recent timeline updates pzhdfy's recent timeline updatespzhdfy's repos on GitHub
Java · 6 watchers
incubator-druid
Apache Druid (Incubating) - Column oriented distributed data store ideal for powering interactive applications
Python · 2 watchers
ChinaUnicom-NetSpeed-Client
A cross-platform and open source replacement of China Unicom Net Speed Client
Python · 1 watchers
Xunlei-FastDick
迅雷快鸟
CSS · 0 watchers
2048
A small clone of 1024 (https://play.google.com/store/apps/details?id=com.veewo.a1024)
0 watchers
982092332-qq.com
test_module
Objective-C · 0 watchers
BatteryQuery
此项目用于iOS设备的电池信息查询（请注意！由于此项目动用了私有框架，因此几乎上不了App Store！）
HTML · 0 watchers
bigdata-ecosystem
BigData Ecosystem Dataset
JavaScript · 0 watchers
blink
Blink Mobile Shell for iOS (Mosh based)
C · 0 watchers
BSI
C · 0 watchers
cachemaster
Cachemaster is similar to VMTOUCH, but with more functions. Such as kick page cache, warmup/readahead data, lock data in mem, stat page cache, stat page cache in realtime mode, by file or directory~ Personally, I think the most usefule function is real-time-statistic of page cache. You can see the page cache thrashing when kernel do page reclaiming.
0 watchers
charybdefs
ScyllaDB fault injection filesystem
C++ · 0 watchers
ClickHouse
ClickHouse is a free analytic DBMS for big data.
0 watchers
cmake-examples
Useful CMake Examples
0 watchers
converthouse
A distributed scheduler for CK using Prophet
0 watchers
datasketches-cpp
Core C++ Sketch Library
CoffeeScript · 0 watchers
dispatch-proxy
Combine internet connections, increase your download speed
Java · 0 watchers
doris
Apache Doris is an easy-to-use, high performance and unified analytics database.
0 watchers
doris-shade
Apache Doris Shade Repository
Python · 0 watchers
gcrawler
A lightweight crawler framework using gevent.
0 watchers
grpc-java
The Java gRPC implementation. HTTP/2 based RPC
C++ · 0 watchers
HeiBaiQiWithAI
HeiBaiQiWithAI writen in QT4
0 watchers
hev-socks5-tunnel
A lightweight, fast and reliable tun2socks
JavaScript · 0 watchers
How-To-Ask-Questions-The-Smart-Way
本文原文由知名 Hacker Eric S. Raymond 所撰寫，教你如何正確的提出技術問題並獲得你滿意的答案。
Objective-C · 0 watchers
IOKitBrowser
Hardware Browser for iOS
0 watchers
jvm-tools
Small set of tools for JVM troublshooting, monitoring and profiling.
0 watchers
linux-insides-zh
linux内核揭密
JavaScript · 0 watchers
loveyue
loveyue系列1到8的源码
Python · 0 watchers
luigi
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
0 watchers
lyq
小米Xiaomi路由器 BE6500 Pro 解锁SSH和科学上网教程
Python · 0 watchers
mortar-luigi
Mortar extensions for Luigi
Rust · 0 watchers
paimon-rust
Apache Paimon Rust The rust implementation of Apache Paimon.
0 watchers
pcstat
Page Cache stat: get page cache stats for files on Linux
Swift · 0 watchers
Potatso
Potatso is an iOS client that implements Shadowsocks proxy with the leverage of NetworkExtension framework in iOS 9.
HTML · 0 watchers
pzhdfy.github.io
0 watchers
rust-in-databend
Rust 培养提高计划, 感谢 Databend 社区 支持https://github.com/datafuselabs/databend
Python · 0 watchers
salary_ke
岁岁有余
0 watchers
shadowsocks-java
shadowsocks server base on netty4 , tcp & udp full support,
Python · 0 watchers
simDHT
A very simple DHT crawler, was written in Python.
VimL · 0 watchers
spf13-vim
The ultimate vim distribution
0 watchers
termtunnel
Cross-platform terminal tunnel tool
0 watchers
test
test2
0 watchers
useful-scripts
🐌 useful scripts for making developer's everyday life easier and happier, involved java, shell etc.
VimL · 0 watchers
vimrc
The ultimate Vim configuration: vimrc
0 watchers
vmtouch
Portable file system cache diagnostics and control
0 watchers
vTemplate
v2ray的模板们
Java · 0 watchers
zeus2
Hadoop job work flow schedule
 |
pzhdfy🏢 快手 / 大数据架构师 V2EX member #86161, joined on 2014-12-11 11:20:38 +08:00 |
北京
pzhdfy
hadoop生态开源贡献者
| 国区 icloud 2T+music 拼车（年/半年/季付）最后一个车位 拼车 • pzhdfy • May 13, 2025 |
| chrome 浏览 v2ex, 关闭从列表页点开的新标签页，结果同时关闭新标签页和列表页 问与答 • pzhdfy • Dec 2, 2024 |
| office 365 拼车 ,到期 2027/07/14, 一年 50，两年 80，最后一个车位 拼车 • pzhdfy • Aug 30, 2024 |
| office 365 拼车 ,到期 2027/07/14, 一年 50，两年 80 二手交易 • pzhdfy • Aug 28, 2024 |
| 国区 icloud 2T+music 拼车（年/半年/季付） iCloud • pzhdfy • Oct 12, 2023 • Lastly replied by pzhdfy | 3 |
| office 365 拼车 40/人/年，到期 2022/07/14,可 1 年可 2 年 二手交易 • pzhdfy • Jul 16, 2020 • Lastly replied by softliumin110 | 2 |
| [北京] 快手大数据架构团队急招（欢迎热衷技术，参与开源的小伙伴）
1 酷工作 • pzhdfy • Oct 30, 2019 • Lastly replied by pzhdfy
|
4 |
| [北京] 快手大数据架构团队持续招人（欢迎热衷技术，开源的小伙伴） 酷工作 • pzhdfy • May 25, 2019 • Lastly replied by ffbh | 35 |
pzhdfy's recent replies
Feb 18, 2024 Replied to a topic by FeifeiJin › 程序员 › 超大型文件比较，内存不足，只能分页读区再匹配，但头都秃了，也没想到优化的方式，朋友们帮帮忙啊。 |
这不是大数据经典处理方法吗
将 PersonListA.csv 通过 name hash 拆分为 10 个，PersonListA_1.csv,PersonListA_2.csv...,PersonListA_10.csv （或者更多，每个文件能载入内存就行）
规则是每行数据通过 hash(name)%10 来确定放到哪个文件
将 PersonListB.csv 也是一样的原理，生成 PersonListB_1.csv,PersonListB_2.csv...,PersonListB_10.csv
这样 PersonListA_1.csv 只会根 PersonListB_1.csv 有相同 name 的数据，
所以只需要 10 组文件对比就行
将 PersonListA.csv 通过 name hash 拆分为 10 个，PersonListA_1.csv,PersonListA_2.csv...,PersonListA_10.csv （或者更多，每个文件能载入内存就行）
规则是每行数据通过 hash(name)%10 来确定放到哪个文件
将 PersonListB.csv 也是一样的原理，生成 PersonListB_1.csv,PersonListB_2.csv...,PersonListB_10.csv
这样 PersonListA_1.csv 只会根 PersonListB_1.csv 有相同 name 的数据，
所以只需要 10 组文件对比就行
Oct 12, 2023 Replied to a topic by pzhdfy › iCloud › 国区 icloud 2T+music 拼车（年/半年/季付） |
满了
Select Language