I Thought I Noticed a Glitch in the Matrix

The problem with randomness - visualised.

I recently ran a command that was supposed to simulate something like the scrolling charcters from the Matrix movie which I used to test some terminal refresh rates and suddenly noticed a pattern in the output and thought my eyes were tricking me. I ctrl-c the command and scrolled back to see what had happened. Most people know that random is very difficult to simulate from deterministic starting points especially in computers, but I thought output from urandom should have been more random than this. I was running following command

od -x /dev/urandom | tr "\n" " " | tr -d " "

And started noticing patterns each time I ran it. By scrolling back through the output I spotted this example (and on different runs the same type of pattern would appear)

The numbers 17 had enough space around them and the line width was wide enough that they created a pattern visible by even my failing human eyes. Taking a closer look there is actually a diagonal line of multiple 14172 appear. Looking deeper, if you follow the diagonal line further back you can see it actually started as 14166 and looking forward ends in 14200. You could make an assumption that urandom might use a time as a seed and that time element is leaking into a bias output, so I worked through the problem.....

​071b8e763adfac23219bfd3ed25a16c76141666203cd696cd82c187971d8334bf0291fb99141666408e563962ad6d93422771ebb9cf084a571416666071f3acedf35789e39d13fc06f9aa8fef14166700e6707d83321f05c9715a318880909cef14166720d787340e9951379c6adb05af95a4912d1416674059029fffb4da993ae1
2edea1a769976f141667600aeb03fce602f275db91984fd859bba5141670009cebd1117f442e464b4b43d012e0c18214167020c513a4cc1f476f7b0dcd3d3accfbf215141670405aa08599927b937f947945b2360c835514167060fa44d569e63f3abf25cbd5e539d57f8b1416710049773e0bb8aef2aac95447e7f9fee87114167
120acb673c4a50a949439cd17d442f935781416714080fc5ac7124e0280798d0505f49d706a1416716044aa9ebb4e38c94547f2157b68e9b0c41416720002dfd7117494197d7f86c9155206d5eb141672205c38de2f18e11112502f78b31cc797f714167240d17f194aa42c1531d5c5d401fbbf1431141672602f928444f71b109b
1339ca42fa6a3f571416730001612f3f8f9065fa398972420021712014167320cb2c8684a881f156d909925f858ea6bc141673405060733a0ccca3ee049832b30d42435a141673600526e8a590ce19647e0faaa4124c975f1416740073036c50bfae94debdab1e65cd692d91141674203aedd06718a5feb3518fb4aca6ed88f9141
674401262b548edcf3c1e1673b61ab8fad82914167460558f8b07dc453f6e836d76005126d8471416750007551e155bbdf331e303b1befb5bdaf11416752087d8ae0e01d7e0492a2b88c5f283183e14167540ab5a395ab6c868d9e5625351fc1de4c214167560653c939cec10ffc1e3e084acf60a36b814167600ba31588ad609ed
76b02d66da328e14bc1416762040391b32524c38fa8988eddcae0e591814167640378f0ad593b79d0be927bb45eaded66f1416766025c432c2f13b3b32c6e8a1c112b3025314167700e56d89428b05490eda4d5ba12c7b6963141677203489ce4de2df9931dd6a8d949ce766a61416774090f0ce25021cb2adaed61353475e3da31
4167760287ccdccc8b3ae5cce78bd5e532eeba11417000072c67fa1ef09033f4aa4ae7492bd052814170020d5307ba0b7264153f4b85bfada7f921714170040b3dcbd7a2fff9a9524e55f4c94ac17ac141700609c45deb5e17e8e9b0a91e824da8bd14414170100b7cc4d6ed120d417f1b467d04d2d0d7914170120d3df142692a9
8170becc815a7207ea9914170140f6962934eed108d4e4dd6b4e06e5f79714170160dfe8d842b890fffce24dc2eccabcaae6141702005846126699af7ca86edc1295f999037614170220af7b142a4f9735c913698c45d12872d61417024036b918a4ffbdae092396e00a69d5721a141702609a26909cbd754c7cd0bddd4c202eb84
8141703009ce9cab4f0210c929a7e35814dd2b056141703205a5b0c7bd8df6dffb1160ec3672e0d0614170340c418bb6d70b17af1cb93f5c284ed2478141703608039bb4a3d76bfb5382634acb86508b814170400093a601ff5ee9ee9efce87eaba52be6414170420d65e3e9a629504167cabe7c0e032a111141704406ce6b72e95
87154b7d309bdc3f4e727e1417046006d2bee8493b9426762e21fe6c6d6fc5141705001ff30cee9e9d80945d5a933f8ee31467141705200db8b88213d660e491c7d258b1e8c94414170540c0a9e608d3873c8fbc201a419739a18814170560972f05f067ec0c915476c267b946d37714170600ae44a430e5fef12f71e46730280af
6e2141706205f1d0b5ace923c5c51b7f3ec9f58fb1514170640bfc0efcef4ab404b286e2ea122e295af14170660e80cc5cf8855ecd2bb2e63f11e858915141707000a27c7d1673642bfb288fdfca178e8cd141707204fa6b0a3640e20ac614b3ce18a1abc4614170740b02ee5bfaa219ee7b12956dca6f917d91417076093e35a8c
4cbd02bef154cba8d806979714171000c5c0334aa6e05d4efe7ddcce009b5e78141710206a8edef14202d9f336aa16ffeb6b17f814171040321c95430396eb819fe607d0123c7eb8141710608a50211d564374cf71167e8172bb3b77141711008987da35be9cc7302f5d723ff6f1cb0d1417112085272c8b2884e261b7e563fde0d
0e324141711404ed3b419a22ff8a57a97518aa5db2a4114171160bbf53dd4ac7ceb9ab14062ad2b67346914171200062198937df21f6c8534cccfe72e4bcc141712205bf09219d14c89c544f8046acd8c8bef14171240e0e68519a2510767ea37e3503ce837d514171260293ec97e88742becc325ee32c198afdc14171300244f41
2f9779e516a68f8d996082fc5814171320470313d7783a3c9001288fa837531e4514171340512a445b013b40113bea66003bb186c11417136079d0fedac1e2c5b91681da0550f4a57714171400261cd216de38accbe87bb9f7b07d0e8a14171420341673d26dd649370f34a299b0249747141714406b5c98734b5a622032aeab387
8cab64f1417146085681972a190f0e17cc1c185a242e57b1417150029160b0310c2d9065295f43b479085d11417152056ce740bf05c20364edabdaa2a36b34c141715406280c6323b3778d1a0d570c6d943aa3a1417156029cae4f7ffc55acd75a2634622557d901417160044853dcfd674975d01c11d45fa81e4b2141716207d86
2fed5a2863349e2cb39f45ac4723141716401be36b1a5982dd22486a5e82ac8827d31417166069871724dee4f9714fa7c331e9f8d336141717009b9ee8c6f5cb0c25f00d262f48f5714114171720391f0797f25e49141cddca3f26d50b9d1417174055804ab1f03517070954c38aa1905bc014171760a509ebf92ce95ad9c6d0c26
6e35770eb141720009181afc058f7f9a9ff4228f0b146ba46141720204595aa32558872faa1ed0b2b7818b2db141720406d37b0bb35e005f72bf4b631fb274367141720606d6c91eb58f05b9f0a05f773978173d8141721004ff1a3f0ec55f71761b902196b6e81691417212096c2b4f3220ca8ea322d5ecde88e8aa61417214020
7cdf40dd56e9935e0020919d8e7192141721606d185d8f0e58cb5582789d6affdb9f1614172200386c4e50ed08ae019e4834de99da253414172220d68b5d790265889796b64177969ea828141722407d3aa4da272b1de0687ea750cb06a6c514172260dad46f90360d9bacf74a5928db9cdf2014172300c1a8715d48c65fb656c49
108d495c93c14172320561e32bc2e496a3cfacfd3ca0d87061514172340306f7f75af8741a0740d4f6497f00ef6141723608d22a66e0bfd0c4c2e0639929143e297141724004087fddaf1641f39dd7c43375a937bee14172420c464e8fbd422881541293bacb02271b014172440dc4e3581492768aed64a55cacd73491714172460
44981bf0185c2bd0bcdec8eb998af583141725007b60b32ad39566cd3abea719e87688ff141725204ba730b181214b96446b4690a0f8a11d14172540f35b1ce7cbe55ad85194a609c338b1b014172560fd88efedb5c5767bfa9b5959a9f5b51314172600a00d189bbdf3dc40f00f540675e86e5414172620e0a828ae8541a0b27c5
8e57765188ad814172640b7752c937f85f828dca88c24970637fc14172660ca06f48f912ff1420968e982fae9903a141727000cb34e0faf99d08d223859aecc1b111114172720a26afd5c7c5340e49ae4a7fa2fa8569714172740a35b109e11c7ae2a3e42b14cb8fcbe8b141727604d858f4aece6110bdae3c76d7c14282f141730
00ee34cef1c557478e32afecf9a9c685b81417302046c8d810119368bc4267049323886179141730400ec9b0643a609a6de4350db71df3885c14173060ea1784b15efc0373aa8e60ed66594f76141731000575d7322c825471a193f7c9d1a3242b141731209d8b4b95379502362ca8cb4b96b9b57214173140775a3c4eec99f0c36
9a5862b5c45b3d114173160709d4cd1cc79127badc580c40d6ffb82141732004369135741af45f603167da1253abdd214173220def72c6fe84aebc7c79835f97d2ce61614173240bc5d43306c12b1f886ccfc563f1fa47e141732601d9a941d141d78947ffef19339db55e5141733001fffcd089ecc1bf3d2a286c0d44f7c9e1417
332021f781f6980c8a46656b03b36a0e75521417334068382536e73f5dc883b148fda669da0f14173360a2833bc4f46b49bc91392dac7eb52bd614173400c514631cb6e7323916f5161a4aea1d1414173420402481659f2041e98cf48a06b70dff541417344052a5b5265682d297356937d081da7078141734604b947dfe9e97860
b558710613f9fbefd14173500bed2522e77ddd575716be369570946f514173520fd78f782344bd1a1579929a262c43ca614173540b958aa5671113beb89b82df86c5e5e6f141735609a67d8d2994886f27dbdaa99a9e597c91417360074a7ed15f473fb28ab41eca6db3cbff9141736201118aa01176aa3f5590e5cacb144f7db14
173640d93beb93b54b653a4c855df32e73623e14173660af32ba66337150d2321e4ed13d929a4a14173700b2720e7f5b73b4c7288c97f67146dc96141737203796572263f7e380425131471d174b161417374059c6d92ebda109d02e97d60ca892cc0d141737609d4d3f49a1ff06fce5bfd0a92b28c6f6141740007bd47fb019d56
376b6826d036a9bd470141740200bb8cd353780e2031ca646ed62cf9f5814174040a98f5109961cd5eab022411ab5f076d3141740608a2be14fe818722db65afc5dc5718f1214174100beeac289fc3c91023f2c67a0aa7c3e5214174120f084e1df027327e04312e3663a772e581417414065e41f2e56d96ad733559fe03eb13b1e
1417416032a88c741c9abdfe23069bf808d2746b14174200bae98e91d18dd387b1950b86054c159514174220def2b09a03da284eb09a92151d4b1c7d14174240bc6b029de30166fe28f7ea9bd1ee7c8b1417426069b8e76ebdb7dbc8353d561b7c31b67c14174300000862ce57e4ca6e104f6870f15ee0c1141743205ce0bcbf273
7f4b783b78b3e449c2fcd14174340e5592b1a0a743dde067bbdbbb8add96814174360d5b2f168ab477ff8fd8e2e06015c98a51417440011965e9a6e7d4a1cab1debccd623b19814174420872a5afa2fd752b546be2645dfad6aa914174440e6ea8e63115b9a67a55d11cdeec517a014174460d440bbc0c7b40cbc20806757e60981

I have known about the problem with randomness for over 30 years when I used to work with fractal seeds to generate 8bit landscapes but would have thought it would have thought that random in 64bit linux would have gotton better over that time? Occording to the man page for urandom, as a general rule, /dev/urandom should be used for everything except long-lived GPG/SSL/SSH keys and now I thought I could see why.

In the above there were 40 characters in each block the first 6 are counting, the next two are from 20 40 60 00 leaving 32 characters, then it struck me - by printing in hex I am adding the first 8 characters, but didnt notice it as the original command removed the newlines and spaces. Running od -x normally produced the expected output where I spotted the 00 20 40 60. Isnt it funny how spotting a pattern can both be right and so wrong:

0000340 6a7b 8476 5beb eeae a298 382a d9f6 41e5
0000360 9770 3013 a20c c535 a604 853e f979 5576
0000400 7dbc ef46 5f24 0845 830d b014 6d78 2746
0000420 f94b 3e24 1a8f 18e6 2df4 240b 92ea 4277
0000440 4221 c590 4b52 c1a1 6b96 0e9f e3a3 0c46
0000460 981c a09c 81f5 ed63 c072 c48a 90d3 a463
0000500 05a3 668e a068 4274 2df9 a14a 1021 da98