Java 基于磁盘的 HashMap

Disk based HashMap

提问by synic

Does Java have (or is there a library available) that allows me to have a disk based HashMap? It doesn't need to be atomic or anything, but it will be accessed via multiple threads and shouldn't crash if two are accessing the same element at the same time.

Java 是否具有（或是否有可用的库）允许我拥有基于磁盘的 HashMap？它不需要是原子的或任何东西，但它将通过多个线程访问，如果两个线程同时访问同一个元素，则不应崩溃。

Anyone know of anything?

有人知道吗？

采纳答案by BalusC

Either properties filesor Berkeley DBmight be what you're looking for. The java.util.Propertiesitself implements java.util.Mapand provides methods to loadfrom and storeto a file. The Berkeley DB is often been recommended as a lightweight key-value pair datastore.

无论是属性文件或Berkeley DB的可能是你在找什么。它java.util.Properties本身实现java.util.Map并提供了load从store文件到文件的方法。Berkeley DB 通常被推荐为轻量级键值对数据存储。

回答by Paul Sasik

Sounds like you need something close to a lightweight db. Have you looked at/considered Java DB?A light db with a single, indexed table would basically be a disk-based, thread-safe hash map.

听起来你需要一些接近轻量级数据库的东西。您是否看过/考虑过Java DB？具有单个索引表的轻型数据库基本上是基于磁盘的线程安全哈希映射。

回答by Alfred

Project Voldemortis also a really fast/scalable/replication "Hashmap". It is used at LinkedIn an performance is also pretty good:

伏地魔项目也是一个非常快速/可扩展/复制的“Hashmap”。用在LinkedIn上，性能也不错：

A quote from their site:

来自他们网站的报价：

Here is the throughput we see from a single multithreaded client talking to a single server where the "hot" data set is in memory under artificially heavy load in our performance lab:

Reads: 19,384 req/sec
Writes: 16,559 req/sec

以下是我们从单个多线程客户端与单个服务器对话时看到的吞吐量，其中“热”数据集在我们性能实验室中人为负载的内存中：

读取：19,384 请求/秒
写入：16,559 请求/秒

回答by Andrejs

JDBM2is exactly what you are asking. It provides a HashMap backed up by disk storage (among other maps). Its fast, thread-safe and the API is really simple.

JDBM2正是您要问的。它提供了由磁盘存储（以及其他映射）备份的 HashMap。它快速、线程安全，而且 API 非常简单。

回答by Quartz

MapDB

地图数据库

MapDB provides concurrent TreeMap and HashMap backed by disk storage or off-heap-memory. It is a fast, scalable and easy to use embedded Java database engine. It is packed with features such as transactions, space efficient serialization, instance cache and transparent compression/encryption. It also has outstanding performance rivaled only by native embedded db engines.

MapDB 提供由磁盘存储或堆外内存支持的并发 TreeMap 和 HashMap。它是一种快速、可扩展且易于使用的嵌入式 Java 数据库引擎。它具有事务、空间高效序列化、实例缓存和透明压缩/加密等功能。它还具有只有本机嵌入式数据库引擎才能媲美的出色性能。

http://www.mapdb.org/

http://www.mapdb.org/

jdbm2

jdbm2

Embedded Key Value Java database.

嵌入式键值 Java 数据库。

https://code.google.com/p/jdbm2/

https://code.google.com/p/jdbm2/

回答by Jesús Zazueta

So the year is now 2016. And if anyone's looking to tackle this problem, I found out that the low level environments API in Xodusfrom JetBrains works for this same purpose, using their computeInTransactionstore lambdas.

所以现在是 2016 年。如果有人想解决这个问题，我发现来自 JetBrains 的Xodus中的低级环境 API使用他们的computeInTransaction商店 lambdas 也用于同样的目的。

Granted, it's not as slick as having a pure Mapinstance, but it worked for my use case.

当然，它不像拥有一个纯Map实例那么灵巧，但它适用于我的用例。

Another recent option is to use H2's MVStorestorage enginewhich does the same thing, but I think it's more tailored towards the database itself.

最近的另一个选择是使用H2 的MVStore存储引擎，它做同样的事情，但我认为它更适合数据库本身。

Cheers!

干杯!

回答by leventov

Chronicle Mapimplements ConcurrentMapand persists data to disk via mapping it's memory to a file.

Chronicle MapConcurrentMap通过将其内存映射到文件来实现并将数据持久化到磁盘。

Chronicle Map is conceptually very similar to MapDB (provides similar builder API and Mapinterface), but Chronicle Map is times fasterthan MapDB and has much better concurrency (Chronicle Map uses highly striped multi-level spin locks).

Chronicle Map 在概念上与 MapDB 非常相似（提供类似的构建器 API 和Map接口），但 Chronicle Map比 MapDB快几倍并且具有更好的并发性（Chronicle Map 使用高度条带化的多级自旋锁）。

回答by Stuart Cardall

In 2018the lightest persistent key valuestore is the H2 Database with it's MVStore:

在2018最轻的持久性key value存储是H2数据库与它的MVStore：

The MVStore is a persistent, log structured key-value store. It is planned to be the next storage subsystem of H2, but it can also be used directly within an application, without using JDBC or SQL.

• MVStore stands for "multi-version store".

• Each store contains a number of maps that can be accessed using the java.util.Map interface.

• Both file-based persistence and in-memory operation are supported.

• It is intended to be fast, simple to use, and small.

• Concurrent read and write operations are supported.

• Transactions are supported (including concurrent transactions and 2-phase commit).

• The tool is very modular. It supports pluggable data types and serialization, pluggable storage (to a file, to off-heap memory), pluggable map implementations (B-tree, R-tree, concurrent B-tree currently), BLOB storage, and a file system abstraction to support encrypted files and zip files.

MVStore 是一个持久的、日志结构的键值存储。它计划成为 H2 的下一个存储子系统，但它也可以直接在应用程序中使用，无需使用 JDBC 或 SQL。

• MVStore 代表“多版本商店”。

• 每个商店都包含许多可以使用 java.util.Map 接口访问的地图。

• 支持基于文件的持久性和内存中操作。

• 它旨在快速、易于使用且体积小。

• 支持并发读写操作。

• 支持事务（包括并发事务和两阶段提交）。

• 该工具非常模块化。它支持可插拔数据类型和序列化、可插拔存储（到文件、堆外内存）、可插拔映射实现（当前 B 树、R 树、并发 B 树）、BLOB 存储和文件系统抽象支持加密文件和zip文件。

h2-mvstorehas no dependencies and version 1.4.200 is a single jar of 0.3 Mb

h2-mvstore没有依赖项，版本 1.4.200 是一个0.3 Mb 的jar

I also looked at:

我还看了：

• MapDB(13 megdependencies)
• chronicle-map(5.5 megdependencies - fast optionally distributed)
• lmdbjava(2 megjava dependencies + lmdbC library) - fastest implementation but not thread safeout of the box.

• MapDB（13 meg依赖项）
• 编年史地图（5.5 meg依赖项 - 快速可选地分发）
• lmdbjava（2 megjava 依赖项 + lmdbC 库）- 最快的实现，但不是thread safe开箱即用的。