- Machine Learning With Go
- Daniel Whitenack
- 181字
- 2021-07-08 10:37:29
Pachyderm jargon
Think about versioning data in Pachyderm kind of like versioning code in Git. The primitives are similar:
- Repositories: These are versioned collections of data, similar to having versioned collections of code in Git repositories
- Commits: Data is versioned in Pachyderm by making commits of that data into data repositories
- Branches: These lightweight points to certain commits or sets of commits (for example, master points to the latest HEAD commit)
- Files: Data is versioned at the file level in Pachyderm, and Pachyderm automatically employs strategies, such as de-duplication, to keep your versioned data space efficient
Even though versioning data with Pachyderm feels similar to versioning code with Git, there are some major differences. For example, merging data doesn't exactly make sense. If there are merge conflicts on petabytes of data, no human could resolve these. Furthermore, the Git protocol would not be space efficient in general for large sets of data. Pachyderm uses its own internal logic to perform the versioning and work with versioned data, and the logic is both space efficient and processing efficient in terms of caching.
推薦閱讀
- Kibana Essentials
- Learning Selenium Testing Tools with Python
- Scientific Computing with Scala
- Azure Serverless Computing Cookbook
- CRYENGINE Game Development Blueprints
- Webpack實戰：入門、進階與調優（第2版）
- 小程序從0到1：微信全棧工程師一本通
- JavaScript從入門到精通（視頻實戰版）
- Oracle SOA Suite 12c Administrator's Guide
- Apache Solr for Indexing Data
- Learning GraphQL and Relay
- Illustrator CS6中文版應用教程（第二版）
- 大象：Thinking in UML（第二版）
- Implementing DevOps with Ansible 2
- Learning RxJava