"Matrix multiplication's associativity and non-commutativity properties provide a natural definition for a [cryptographic hash](https://en.wikipedia.org/wiki/Cryptographic_hash_function) / digest / summary of an ordered list of elements while preserving concatenation operations. Due to the non-commutativity property, lists that differ in element order result in a different summary. Due to the associativity property, arbitrarily divided adjacent sub-lists can be summarized independently and combined to find the summary of their concatenation in one operation. This definition provides exactly the properties needed to define a list, and does not impose any unnecessary structure that could cause two equivalent lists to produce different summaries. The name *Merklist* is intended to be reminicent of other hash-based data structures like [Merkle Tree](https://en.wikipedia.org/wiki/Merkle_tree) and [Merklix Tree](https://www.deadalnix.me/2016/09/24/introducing-merklix-tree-as-an-unordered-merkle-tree-on-steroid/)."
"This definition of a hash of a list of elements is pretty simple:\n",
"\n",
"* A **list element** is an arbitrary buffer of bytes. Any length, any content. Just bytes.\n",
"* A **list**, then, is a sequence of such elements.\n",
"* The **hash of a list element** is the cryptographic hash of its bytes, formatted into a square matrix with byte elements. (More details later.)\n",
"* The **hash of a list** is reduction by matrix multiplication of the hashes of all the list elements in the same order as they appear in the list.\n",
"* The **hash of a list with 0 elements** is the identity matrix.\n",
"\n",
"This construction has a couple notable concequences:\n",
"\n",
"* The hash of a list with only one item is just the hash of the item itself.\n",
"* You can calculate the hash of any list concatenated with a copy of itself by matrix multiplication of the the hash with itself. This works for single elements as well as arbitrarily long lists.\n",
"* A list can have multiple copies of the same list item, and swapping them does not affect the list hash. Consider how swapping the first two elements in `[1, 1, 2]` has no discernible effect.\n",
"* The hash of the concatenation of two lists is the matrix multiplication of their hashes.\n",
"* Concatenating a list with a list of 0 elements yields the same hash.\n",
"The function `hash_m/1` takes a buffer of bytes as its first argument, and returns the sha512 hash of the bytes formatted as an 8×8 2-d array of 8-bit unsigned integers with wrapping overflow. **We define this hash to be the hash of the list element.** Based on a shallow wikipedia dive, someone familiar with linear algebra might say it's a [matrix ring](https://en.wikipedia.org/wiki/Matrix_ring), $R_{256}^{8×8}$. Not coincidentally, sha512 outputs 512 bits = 64 bytes = 8 * 8 array of bytes, how convenient. (In fact, that might even be the primary reason why I chose sha512!)"
"8×8 seems big compared to 3×3 or 4×4 matrixes, but otherwise everything looks normal. The values are as random as you might expect a cryptographic hash to be, and range from 0-255."
"Ok so we've got our element hashes, how do we combine them to construct the hash of a list? We defined the hash of the list to be reduction by matrix multiplication of the hash of each element:"
"* [Associativity](#Associativity) - Associativity enables you to reduce a computation using any partitioning because all partitionings yield the same result. Addition is associative $(1+2)+3 = 1+(2+3)$, subtraction is not $(5-3)-2\\neq5-(3-2)$. ([Associative property](https://en.wikipedia.org/wiki/Associative_property))\n",
"* [Non-Commutativity](#Non-Commutativity) - Commutativity allows you to swap elements without affecting the result. Addition is commutative $1+2 = 2+1$, but division is not $1\\div2 \\neq2\\div1$. And neither is matrix multiplication. ([Commutative property](https://en.wikipedia.org/wiki/Commutative_property))\n",
"Upon consideration, these are the exact properties that one would want in order to define the hash of a list of items. Non-commutativity enables the order of elements in the list to be well defined, since swapping different elements produces a different hash. Associativity enables calculating the hash of the list by performing the reduction operations in any order, and you still get the same hash.\n",
"Lets sanity-check that these properties can hold for the construction described above. For setup, lets consider the hashes of three elements, $he_1$, $he_2$, and $he_3$:"
"If it's associative, we should get the same hash if we rearrange the parenthesis to indicate reduction in a different operation order. $((he_1 × he_2) × he_3) = (he_1 × (he_2 × he_3))$"
"The security of the design of merklist leans on existing cryptographic hash function by the requiring that values that go into the final list digest came from some real hashed preimage, and any implementation can require that the primage of every element must be provided. This means that it's not possible to choose an *arbitrary* element hash as an input to the reduction operation, limiting the blast radius that any specific element can have on the final list digest to the bit fixing that is possible by asic bitcoin miners today. So then the question is if bit fixing is enough to degrade the preimage resistance of the merklist design, and if that is impossible to overcome. (Challenges anyone?)"
]
},
{
"cell_type": "markdown",
"id": "9515603f-f8b5-482a-ae1b-ed504358b860",
"metadata": {},
"source": [
"## Unknowns\n",
"\n",
"This appears to me to be a reasonable way to define the hash of a list. The mathematical description of a list aligns very nicely with the properties offered by matrix multiplication. But is it appropriate to use for the same things that a Merkle Tree would be? The big questions are related to the valuable properties of hash functions:\n",
"* Given a merklist summary but not the elements, is it possible to produce a different list of elements that hash to the same summary? (~preimage resistance)\n",
"Matrixes are well-studied objects, perhaps some of these questions already have an answer. If *you* know something about deriving a preimage of a [matrix ring](https://en.wikipedia.org/wiki/Matrix_ring) $R_{256}^{8×8}$ using just multiplication, I would be very interested to know. Given the simplicity of the definition, I wouldn't be surprised if there was already a big wrench in the cogs here."
"*If this construction has appropriate security properties*, it seems to be a better merkle tree in all respects. Any use of a merkle tree could be replaced with this, and it could enable use-cases where merkle trees aren't useful. Some examples of what I think might be possible:\n",
"* Using a Merklist with a sublist summary tree structure enables creating a $O(1)$-sized 'Merklist Proof' that can verify the addition and subtraction of any number of elements at any single point in the list using only $O(log(N))$ time and $O(log(N))$ static space. As a bonus the proof generator and verifier can have totally different tree structures and can still communicate the proof successfully.\n",
"* Using a Merklist summary tree you can create a consistent hash of any ordered key-value store (like a btree) that can be maintained incrementally inline with regular node updates, e.g. as part of a [LSM-tree](https://en.wikipedia.org/wiki/Log-structured_merge-tree). This could facilitate verification and sync between database replicas.\n",
"* The sublist summary tree structure can be as dense or sparse as desired. You could summarize down to pairs of elements akin to a merkle tree, but you could also summarize a compressed sublist of hundreds or even millions of elements with a single hash. Of course, calculating or verifying a proof of changes to the middle of that sublist would require rehashing the whole sublist, but this turns it from a fixed structure into a tuneable parameter.\n",
"* If all possible elements had an easily calculatable inverse, that would enable \"subtracting\" an element by inserting its inverse in front of it. That would basically extend the group from a ring into a field, and might have interesting implications.\n",
