sdlc.dev

Computer science fundamentals for working engineers path · Limits and everyday tools · Topic 12 of 13

Hashing and checksums

Not started

A hash function takes input of any size and returns a fixed-size value, its hash or digest. The same input always gives the same output, and a good hash spreads different inputs evenly over all possible outputs. Beyond that, hash functions are built for very different jobs, and using one built for the wrong job is a classic mistake: a fast table hash where you need security, or a slow cryptographic hash in a hot loop.

Why it matters: Hash tables, sharding, caches, deduplication, download checks and password storage all use hash functions, but they need very different kinds.

  1. 1Fixed-size fingerprints

    A hash function maps input of any length to a fixed-size value. Equal inputs always give equal hashes; different inputs usually give different ones.

  2. 2Checksums catch accidents

    CRC-32 and the Internet checksum detect bit flips and truncation cheaply, but anyone can forge a matching value on purpose.

  3. 3Cryptographic hashes resist attackers

    SHA-256 is designed so that finding two inputs with the same hash, or an input for a given hash, is infeasible.

  4. 4Passwords need slow hashes

    Store passwords with a deliberately slow, salted function such as Argon2, scrypt or bcrypt, never with a bare fast hash.

1

How it works

The explanation, with examples you can run.

Three jobs, three kinds of hash

JobExamplesWhat it must do
Hash tables, sharding, Bloom filtersFNV, MurmurHash, xxHash, SipHashVery fast; spread keys evenly
Detect accidental corruptionCRC-32, CRC-32C, the Internet checksum, Adler-32Cheap; catch bit flips and bursts
Integrity and identity against attackersSHA-256, SHA-3, BLAKE3Infeasible to find collisions or preimages

Plus a fourth, special case: password hashing (Argon2id, scrypt, bcrypt, PBKDF2), deliberately slow and salted.

Fast hashes and checksums in a few lines

⌘/Ctrl + Enter
OutputNot run yet

Press Run to see what your code prints, in order.

Runs in this browser, in an isolated frame with no access to this page. console.log, info, warn and error are shown; code runs as a classic script, so use an async function for await.

Node printed 4f9f2cab 5c9f4122: changing one letter changed most of the bits. The buckets came out between 985 and 1,010 keys each, close to the ideal 1,000. And the CRC-32 of "123456789" is cbf43926, the standard check value every correct implementation must produce.

A table hash only needs to be fast and well spread, but if attackers choose the keys (HTTP parameters, JSON field names) they can pick ones that all collide and turn O(1) lookups into O(n). Languages defend against this "hash flooding" with randomly seeded hashes such as SipHash, which is why Python's and Rust's string hashes differ between runs.

Checksums

Networks and disks flip bits. A checksum stored or sent with the data lets the receiver recompute it and detect most corruption. TCP, UDP and IPv4 use the 16-bit Internet checksum (RFC 1071), Ethernet and zip use CRC-32, and storage systems such as databases, ZFS and Kafka checksum their blocks and records, often with CRC-32C, which modern CPUs compute in hardware. Checksums are not security: CRCs are linear, so an attacker can change data and fix up the CRC in a few operations.

Cryptographic hashes

A cryptographic hash such as SHA-256 (FIPS 180-4) is designed so that nobody can feasibly find two inputs with the same digest (collision resistance) or an input that produces a given digest (preimage resistance). That makes the digest a trustworthy identifier: Git commits, package lock files, container image digests and content-addressed storage all name data by its hash. MD5 and SHA-1 are broken for collision resistance and shouldn't be used for new security purposes. In browsers and Node, crypto.subtle.digest("SHA-256", bytes) computes one.

To prove that a message came from someone holding a key, not just that it is unchanged, use a MAC such as HMAC-SHA-256 or a digital signature.

2

Check yourself

3 questions. Get them all right to finish this topic.

  1. 1.A download page lists a SHA-256 hash next to the file. What does matching it prove?
  2. 2.Why are CRC-32 checksums not used to protect against tampering?
  3. 3.Why is SHA-256 alone a poor choice for storing passwords?

0 of 3 right so far. Get all three to finish this topic.

Go deeper