rhetenor/awesome-awesomeness
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

update lists

Browse Source
This commit is contained in:
Jonas Zeunert
2025-07-18 22:22:32 +02:00
parent 55bed3b4a1
commit 5916c5c074
3078 changed files with 331679 additions and 357255 deletions
Download Patch File Download Diff File Expand all files Collapse all files

251
html/storage.html
View File

@@ -1,251 +0,0 @@
<h1 id="awesome-storage">Awesome Storage</h1>
<p>A curated list of storage open source tools. Backups, redundancy,
sharing, distribution, encryption, etc.</p>
<h2 id="distributed-file-systems">Distributed file systems</h2>
<p>Systems that expose a traditional file system interface with POSIX
semantics.</p>
<ul>
<li><p><a href="https://ceph.com/ceph-storage/file-system/">Ceph file
system</a> - Ceph’s file system runs on top of the same object storage
system that provides object storage and block device interfaces. The
Ceph metadata server cluster provides a service that maps the
directories and file names of the file system to objects stored within
RADOS clusters. The metadata server cluster can expand or contract, and
it can rebalance the file system dynamically to distribute data evenly
among cluster hosts. This ensures high performance and prevents heavy
loads on specific hosts within the cluster.</p></li>
<li><p><a href="http://leo-project.net/">LeoFS</a> - LeoFS is a highly
available, distributed, eventually consistent object/blob store. If you
are searching a storage system that is able to store huge amount and
various kind of raw data in its native format, LeoFS is suitable for
that.</p></li>
<li><p><a href="http://www.xtreemfs.org/">XtreemFS</a> - XtreemFS is a
general purpose storage system and covers most storage needs in a single
deployment. It is open-source, requires no special hardware or kernel
modules, and can be mounted on Linux, Windows and OS X.</p></li>
<li><p><a href="http://orangefs.com/">OrangeFS</a> - OrangeFS is a
software based scale-out parallel storage system. It is ideal for large
storage problems faced by HPC, BigData, Streaming Video, Genomics and
Bioinformatics.</p></li>
<li><p><a href="https://www.beegfs.io/">BeeGFS</a> - BeeGFS is the
leading parallel cluster file system, developed with a strong focus on
performance and designed for very easy installation and management. If
I/O intensive workloads are your problem, BeeGFS is the
solution.</p></li>
<li><p><a href="https://www.gluster.org/">Gluster</a> - Gluster is a
scalable network filesystem. Using common off-the-shelf hardware, you
can create large, distributed storage solutions for media streaming,
data analysis, and other data- and bandwidth-intensive tasks. Gluster is
free.</p></li>
<li><p><a href="http://pl.atyp.us/hekafs.org/">HekaFS</a> - CloudFS is a
distributed fileysystem specifically intended to solve problems faced by
a cloud provider when offering filesystem access “as a service” to cloud
users (a.k.a. tenants).</p></li>
<li><p><a href="https://juicefs.com/en/">JuiceFS</a> - JuiceFS is an
open-source cloud-native distributed file system with high performance,
commonly utilized in AI &amp; machine learning for training, inference,
model distribution, as well as in big data processing and
analysis.</p></li>
<li><p><a href="https://lizardfs.com/">LizardFS</a> - LizardFS Software
Defined Storage is a distributed, parallel, scalable, fault-tolerant,
Geo-Redundant and highly available file system.</p></li>
<li><p><a href="https://github.com/chrislusf/seaweedfs">SeaweedFS</a>
SeaweedFS is a simple and highly scalable distributed file system.
SeaweedFS can transparently integrate with the cloud. With hot data on
local cluster, and warm data on the cloud with O(1) access time,
SeaweedFS can achieve both fast local access time and elastic cloud
storage capacity.</p></li>
<li><p><a href="https://github.com/scality/Zenko">Zenko</a> Zenko is the
open source multi-cloud data controller: own and keep control of your
data on any cloud.</p></li>
<li><p><a href="https://github.com/cubefs/cubefs">CubeFS</a> CubeFS is a
new generation cloud-native storage that supports access protocols such
as S3, HDFS, and POSIX.</p></li>
<li><p><a href="https://github.com/moosefs/moosefs">MooseFS</a> MooseFS
is a Petabyte Open Source Network Distributed File System. It is easy to
deploy and maintain, highly reliable, fault tolerant, highly performing,
easily scalable and POSIX compliant.</p></li>
</ul>
<h2 id="file-sharing">File sharing</h2>
<ul>
<li><p><a href="https://www.linshare.org/">Linshare</a> - Linshare is an
Open Source secure file sharing application intended to cover your
business security and file transfer needs. If confidentiality and
traceability are paramount for your business file transfers, then
LinShare is your solution and better yet it’s free ! We also offer high
quality support services provided by our IT teams to suit your network
at best.</p></li>
<li><p><a href="https://www.seafile.com">Seafile</a> - Seafile is an
open source file sync&amp;share solution designed for high reliability,
performance and productivity. Sync, share and collaborate across devices
and teams. Build your team’s knowledge base with Seafile’s built-in Wiki
feature.</p></li>
<li><p><a href="https://www.projectsend.org/">ProjectSend</a> -
ProjectSend is a self-hosted application (you can install it easily on
your own VPS or shared web hosting account) that lets you upload files
and assign them to specific clients that you create yourself! Secure,
private and easy. No more depending on external services or e-mail to
send those files!</p></li>
<li><p><a href="https://afterlogic.org/aurora-files">Aurora Files</a> -
Aurora Files is an open-source file storage platform for small teams and
personal users.</p></li>
<li><p><a href="https://owncloud.org/">ownCloud</a> - With over 50
million users worldwide, ownCloud is the market-leading open source
software for cloud-based collaboration platforms. As an alternative to
Dropbox, OneDrive and Google Drive, ownCloud offers real data security
and privacy for you and your data.</p></li>
<li><p><a href="https://nextcloud.com/">Nextcloud</a> - Nextcloud Files
enables enterprises to take back control over their data, ensuring
compliance, security and instant availability wherever their employees
are.</p></li>
<li><p><a href="https://pydio.com/">Pydio</a> - Modern file management
platform built according to your business needs and regulations, Pydio
is open-source software deployed on your servers or wherever you
decide.</p></li>
<li><p><a href="http://www.youtransfer.io/">YouTransfer</a> -
YouTransfer is a simple but elegant self-hosted file transfer &amp;
sharing solution. It is an alternative to paid services like Dropbox and
WeTransfer by offering similar features but without limitations, price
plans and a lengthy privacy policy. You remain in control of your
files.</p></li>
</ul>
<h2 id="backups-replication-distribution">Backups, replication,
distribution</h2>
<p>Software that handles backup and high availability of information,
independent from the storage itself.</p>
<ul>
<li><p><a href="http://amanda.zmanda.com/">Amanda</a> - Amanda is the
most popular open source backup and recovery software in the world.
Amanda protects more than a million servers and desktops running various
versions of Linux, UNIX, BSD, Mac OS-X and Microsoft Windows operating
systems worldwide.</p></li>
<li><p><a href="https://backuppc.github.io/backuppc/">BackupPC</a> -
BackupPC is a high-performance, enterprise-grade system for backing up
Linux, Windows and macOS PCs and laptops to a server’s disk. BackupPC is
highly configurable and easy to install and maintain.</p></li>
<li><p><a href="https://github.com/vdbsh/backy">backy</a> - Tiny
multiprocessing utility for file backups.</p></li>
<li><p><a href="https://www.bacula.org/">Bacula</a> - Bacula is a set of
Open Source, computer programs that permit you (or the system
administrator) to manage backup, recovery, and verification of computer
data across a network of computers of different kinds.</p></li>
<li><p><a href="https://www.bareos.org/en/">Bareos</a> - Bareos is a
100% open source fork of the backup project from bacula.org. The fork is
in development since late 2010, it has a lot of new features.</p></li>
<li><p><a href="https://github.com/borgbackup/borg">Borg</a> -
BorgBackup (short: Borg) is a deduplicating backup program. Optionally,
it supports compression and authenticated encryption.</p></li>
<li><p><a href="https://github.com/gilbertchen/duplicacy">Duplicacy</a>
- Duplicacy is a new generation cross-platform cloud backup tool based
on the idea of Lock-Free Deduplication.</p></li>
<li><p><a href="https://www.duplicati.com/">Duplicati</a> - Free backup
software to store encrypted backups online. For Windows, macOS and
Linux.</p></li>
<li><p><a href="https://www.elkarbackup.org/">ElkarBackup</a> -
ElkarBackup is a free open-source backup solution based on
RSync/RSnapshot.</p></li>
<li><p><a href="https://github.com/kopia/kopia">Kopia</a> - Kopia is a
simple, cross-platform tool for managing encrypted backups in the cloud.
It provides fast, incremental backups, secure, client-side end-to-end
encryption, compression and data deduplication.</p></li>
<li><p><a href="https://restic.net/">Restic</a> - Open Source easy to
use single binary backup tool with encryption, dedupe, and supports
object storage backends.</p></li>
<li><p><a href="https://www.urbackup.org/">UrBackup</a> - UrBackup is an
easy to setup Open Source client/server backup system, that through a
combination of image and file backups accomplishes both data safety and
a fast restoration time.</p></li>
<li><p><a href="https://www.zenko.io/">Zenko</a> - Zenko is Scality’s
Open Source Multi-Cloud Data Controller. Zenko lets you be in control of
your data and leverage the efficiency of private and public
clouds.</p></li>
<li><p><a href="https://github.com/gui-text/backup-anything">Backup
Anything</a> - Backup Anything are 100% open-source scripts for creating
backup of databases and cloud buckets. Can be used on CI/CD or
local.</p></li>
</ul>
<h2 id="s3-compatible-file-servers">S3 compatible file servers</h2>
<p>File servers that expose S3 compatible APIs</p>
<ul>
<li><p><a href="https://min.io/">MinIO</a> - A high performance,
distributed object storage server, designed for large-scale data
infrastructure. It is an ideal S3-compatible replacement for Hadoop HDFS
and scale out NAS for machine learning and other big data
workloads.</p></li>
<li><p><a href="https://ceph.com/ceph-storage/file-system/">Ceph file
system</a> - Ceph’s file system runs on top of the same object storage
system that provides object storage and block device interfaces. The
Ceph metadata server cluster provides a service that maps the
directories and file names of the file system to objects stored within
RADOS clusters. The metadata server cluster can expand or contract, and
it can rebalance the file system dynamically to distribute data evenly
among cluster hosts. This ensures high performance and prevents heavy
loads on specific hosts within the cluster.</p></li>
<li><p><a href="http://leo-project.net/">LeoFS</a> - LeoFS is a highly
available, distributed, eventually consistent object/blob store. If you
are searching a storage system that is able to store huge amount and
various kind of raw data in its native format, LeoFS is suitable for
that.</p></li>
<li><p><a href="https://riak.com/products/riak-s2/">RIAK S2</a> - Riak®
S2 is a highly available, scalable, easy-to-operate object storage
software solution that’s optimized for holding videos, images, and other
files. It provides simple but powerful storage for large objects built
for private, public, and hybrid clouds.</p></li>
<li><p><a href="https://www.openio.io/">OpenIO</a> - OpenIO SDS is an
open source object storage solution ideal for Big Data, HPC and AI. With
its distributed grid architecture and unique self-healing technology,
OpenIO scales easily without mandatory data rebalancing, while
delivering consistent high performance. OpenIO is S3 compatible and can
be deployed on-premise or cloud-hosted, on any hardware that you
choose.</p></li>
<li><p><a href="https://docs.openstack.org/swift/latest/">Openstack
Swift</a> - Openstack Swift is an autonomous object storage system part
of the OpenStack Project. Swift is feature complete, is designed to
integrate well with the OpenStack ecosystem, and emphasizes
configurability and extensibility. It supports the S3 API but also
provides its own, more complete, API which already benefits from the
work done by its substantial community.</p></li>
<li><p><a href="https://garagehq.deuxfleurs.fr/">Garage</a> - Garage is
a hacker-oriented object storage software: it runs on every machine and
through the regular Internet. The key enablers to achieve these
properties are its consensus-less design and its Rust close-to-the-metal
implementation. Garage is S3 compatible and honors the same consistency
properties as Amazon S3. It can host static websites and stands as a
backend for any S3-compatible applications like Nextcloud.</p></li>
<li><p><a href="https://github.com/treeverse/lakeFS/">lakeFS</a> -
lakeFS is an open source tool that transforms your object storage into a
Git-like repository. It enables you to manage your data lake the way you
manage your code.</p></li>
<li><p><a href="https://github.com/chrislusf/seaweedfs">SeaweedFS</a>
SeaweedFS is a simple and highly scalable distributed file system.
SeaweedFS can transparently integrate with the cloud. With hot data on
local cluster, and warm data on the cloud with O(1) access time,
SeaweedFS can achieve both fast local access time and elastic cloud
storage capacity.</p></li>
<li><p><a href="https://github.com/cubefs/cubefs">CubeFS</a> CubeFS is a
new generation cloud-native storage that supports access protocols such
as S3, HDFS, and POSIX.</p></li>
</ul>
<h2 id="cloud-sync-engine">Cloud Sync Engine</h2>
<p>Starting in Windows 10, version 1709, Windows provides the cloud
files API. This API consists of several native Win32 and WinRT APIs that
formalize support for cloud sync engines, and handles tasks such as
creating and managing placeholder files and directories. Users of this
API are typically sync providers and to some extent, Windows
applications.</p>
<ul>
<li><a href="https://github.com/styletronix/cfapiSync">cfapiSync</a> -
Working c# Example implementig a Cloud Sync Engine on Windows 10 / 11
based on the cloud files API. This is very early alpha.</li>
</ul>
<h2 id="contributions-welcome">Contributions welcome</h2>
<p>If you wish to contribute to this list, just fork, make your changes
and send me a pull request, I’ll be happy to review all of your
suggestions :)</p>
<h2 id="check-out-also">Check out also</h2>
<ul>
<li><a
href="https://github.com/okhosting/awesome-cyber-security/">Awesome
Cyber Security</a> - A curated list of cyber security resources and
tools.</li>
</ul>