# [[Data analysis comes at the expense of data processing]] ![[Data analysis comes at the expense of data processing.svg]] Making sense of information is often at odds with [[Ingest|ingesting]] it in the first place. ## [[Database|Databases]] There is a tradeoff in database design. A database can: - be good at analytical use cases ([[Online Analytical Processing|OLAP]]) that require the aggregation of data - be good at observability use cases ([[Online Transaction Processing|OTLP]]) that require the real-time retrieval of information - ... or it can be mediocre at both. But it can't be really good at both. This is because data analysis requires a [[Column-oriented database]] to be able to quickly [[Aggregating monitoring data|aggregate]] over columns of data. Meanwhile, it's essential in [[Observability]] use cases that data is retrieved as close to real-time as possible and reproduced as faithfully as possible (for example, the full, raw log line). Examples of databases that prioritise data analysis are [[ClickHouse]], [[Amazon Redshift]], [[Apache Druid]], and [[Vertica]]. Examples of databases that prioritise retrieval ([[Row-oriented database|row-oriented databases]]) are [[Postgres]], [[MySQL]], [[MariaDB]], [[SQLite]], [[Oracle Database]], [[Cassandra]], and [[MongoDB]]. %% # Excalidraw Data ## Text Elements Get average of all values in this column ^rV9TzuGQ Return rows that include this string ^X2R9Hi8M THIS ^ncpJEsCi or ^Lv2rjtoC THAT ^BfXDbV6I ## Embedded Files 5f6616867fd6530134afff13f4c1454bc8b891fe: [[icon-database-table.svg]] ## Drawing ```compressed-json N4KAkARALgngDgUwgLgAQQQDwMYEMA2AlgCYBOuA7hADTgQBuCpAzoQPYB2KqATLZMzYBXUtiRoIACyhQ4zZAHoFAc0JRJQgEYA6bGwC2CgF7N6hbEcK4OCtptbErHALRY8RMpWdx8Q1TdIEfARcZgRmBShcZQUebQBGAGZtAAYaOiCEfQQOKGZuAG1wMFAwMogSbggAEQAFAEkALVIjAHlSAFV65gApADMhAE56UxSARyp+cthEKsJ9aKQpyExu Z3iAdgAWVJ4tjcTBngA2AFYtrfjB5YgYNZ5EgA5UwePBx63H48SeR/iePjFSAUEjqbg8f6pY7HAHHLZnHiDQbbG6SBCEZTScGQlLQ2Hw06I5FbG7WZTBbgpG7MKCkNgAawQAGE2Pg2KQqrTrMw4LhArl0mVIJpcNh6co6UIOMQWWyORIuRweXyclBBeU+oR8PgAMqwCkSQQedUCWkMhAAdVBknB1LNjL1MAN6CNlRuksxHHC+TQ8RubF52DUd19K SpQIgEuEcHqxB9qAKAF0bn1yNlY9wOEJtTdCNKsFVcGl3cJpV7mPGszmI2EEMRuKdHttHj8ATdGCx2Fw0N920xWJwAHKcMTcf4HDYpC6T3PMaqZKB17h9AhhG6aUvEACiwWyuXjSZuQjgxFwi/rvo2p02v0bx02JIjRA49Mz2fwNzZYqXaBX+DXNZRFAQjxhAiDSnmygmhAmrBBmEinH00LxMcXwbH0xBnIkKRJFsuB9ARSR9Fs2DxFs5yaNgjya I8gzxH0Sw1u44gJkCYB+mx8RAsmEbYHScBvtqxQAL5TKU5SVBIPAAErVAAgkyQiJPQzitBstTHPQpxGMcTLOMQABW0EzCxECBNgUQcOSjFChAqxoM47zaE8xxTi216uW8gK2SGqDOACGzaJOKQbPETaHIkmwhTcILEGCPY7B8blhh8SSNtcEZohiWK+k5GyDFOpynBsTanMc+WPrZZLOuGtk0nSjKyuyVQAMQICkkWRdBIpilGUoyqyzUKuQSq8v yaoplqur6qZrr1naDWWta3CVeU9Xmo6zoQHN0EepI5bxrV5QBqKwZjmGNx9TGcaFDxtmprg6YXqgVYfhGebEAWEi4PEu2bgd3DidM8AsYkQKiTWCA/qgV5bIiPDbMcfadpw3AbN55QdgOHDDhwo6+qhjwQgC7yzvOwTnsuq4IOum47lkqoHnd5THqelOXte6NfPEYXwp+eavmgr2fmw37PX+YQiWJ73PWZABqgwACpGEIADiACKxkg5yWATRG9l+ WFcTFVsKQAjh7zlXCNy+eszxkXD3xHFO2FbBltmxfFqBhac2hwuRkU8KcqLopiapoKcR2QNVLFR9t9rMoN8roG1HVJL966iuKkrSk1yfQCNyrjdBsHTU6s2sm6gGLVacU2mgq2motm0V8aJZ+Pt3qUv6gZnaGcdXbGTMpmmCDwS9765vmBsQLgPB/f1ANC5PkPQ/8nzYZHiSNww/ZduChzI9juP47wYUFYSJxkwu0MSzTEYbv19N7nkhRsUDEmyw A0jAbC1BQAAJGA9BSAAFk5YnFaPSHoX85L1CgAADRNMDWY31SB0ioGxYS3EjwnjPGvK8N4mzo2vDvZ8gsJ7Vlsl+Rk4tqY3EXJgcO6BVYICgKgXAHZFioDYH0Dh2pUAAB0OD0AIEIcIqA8yoHUIQZgqA9C+H0Fwd0lBFa6yqKw9hnCmDcN4fw/AQiRFiIkVImRciFFCCUSXTgUAdSECMCxHgcc+g2IAGKPS1L5HejCoBySIMobs6Bgh9D1rZDsUB zAED8RiQJ0AAzQT0LkXAeYmDj2FhGdkGI8wEDUUwjRbCOFcOUAgHhfCCAGOEaI3wJiODSMkLI+RrJLHKIjLgIQUA2DSXCPYlitJxH8y9AA0OOVvbaCDlLYoQMKiy0VvEAB9RJA6iZAgektRnBCCtIrPoYwhCtEGLULWqD0DmUstZaCBt1hIm0FsJ4YUib3mhNhDYNs1goUGNoV45xyp7BOCFM2MVlpoBOM8K2lxiopFojhRIwdMrDOYYiMZHVaLQ mvBCGEyJSRWRqgtc0edWpXFCtgeemdeo5wGnKTkhcxqqhLlNFuVQdo4sZLXL2O91oOhmgyyu80Ix7SXqgOOJ0gywHOgPSU11h4Rgek9QSb1bIfS+ugXAiQF5li7mgD+0BtZoDBkKCGdUobPQ8okUKdEd5Y33mgJsR8uwnxYvEIqlwLh4mvhTW+9CH5013IzW6uC2YEM5r8M4oUvgDIoek6hotaFU3/AgCZZQpmSXQAgmSgwAGEEeKAw5pkfHnO4I 8X2MI0WJB+MVR4fwkYRltk4lICQUIpGKt8EKVsAV1zHOcbQhIyIfBNTC2yWUw4NjjjHbu1dcVJ3xcieIRLupZz6rnCdw1uTUoFJNbU9LDTcuguypabaG5MoQBul0W726enVQKnup0RX90uuKoevqpWjzSSveV09CxbFVcQflEa1qGrHFcJI3wziPBtajIFELQM4xHPaw4pwTVFtdWw91sbaZP29fuN+QoShsUgEmiAmgYBhXwN4betQEDxEHHFZw pxmAUGkqcZBkATKFnQWwTBWHsFCmZpAVm+DnqbEDU2T4xMw2ypFmLGNAFbK5okF04CpBakYLkeoM8hi8zYF8J9OpDT6qQV2qo9Rsm2EiEU2x5TkhVPCPU5pkpZjUC6astY3IdiHHgmcW4jx+AvEMN1tEgJVRgmhMxkwCJ7g/OxI6QJG4iSogpNIM+qhx1SBZI4Dkwz6A5MmdQEpuplmODWaEFpuzDmoKknaZ07prm0B9PvtQlJQzsrMPiGM048bs PytlnjOAPQtzMCZIQbNOs8k3ANihZ4+wOqwlxPsa1Va1jltSChc4RNflOp3p7euqByKdrKnCXEQcQ6NfBH28oI60Bxx3XiiQLUCXTuJQ/OdZKrvoEVEXGla6y5bUZWO5lgKtsHqPdtE9vLhBnorKO2yQq+7ewuhGQeN00CHkfY9Mez0f24bfd9Bj7c1Xg+XolgQf6G4bEnJ8S+kGxy/Eg3a/9IVBhPFCnHWR5MkN0JQ56tDDMMOI/fjh6ZVQCNEZ I1sMjFGqM0bo9jvnzG0EYIgFgnBEZePs29oQrmV4vjnFE/juV5QaHIak9MdLEBFbzJ1PpiguTmEm7N052xPS3Mpg8/oTxK0fNMPCwFhAIToLhMifgT3Cp4nRZsckr08W0cvqSyltLeSJCm/qObsrHSumsCq9I0g/Snz1bhWOFrbXE2ywADL0B4KQAyHSmSDYVOokbY4C0JCOA68txDGyhrmw5RsYzESPAhabcqnxcKtq9psIKLYnGBxO5AAdIyDu tKxbHA9z2IA3fI2v2dpLNzL9eyuoLkBS6A++3VBOLLNtsoTof4Htk+XnsFb3a9MOxXRnvYj7jMEn2R4JxUTHSrjifu/VHoTtDHBnsEbDCBTglNTtBtwD8DhBsOVCVIhirnfKhtKM/D6q/n6nxmOGrr8PAbiKFNrpQrrpAPrmzobkxsbs1CopblQRyE7s5g7uBgwVAO4i7l5m7hGD4oHkEt7nvrvKQKFlEv4hFsHrxKHnFgliQRAJkv4LHtbtQa0u VqnkwRnlnnVoMrnr6PnmUPqu1p/FUAAEJ9AILVCaByzHD1DV4va176xjj3idpTjxBTi4iHD7CNgvIOS/AfIHDOxTiDA3KuwgYRgbZDpBRwxhRXiHaDpApT6zwL4Q5rQJzL6pydQZwPab79Tb5UoqirpSp0qcqbptw/a7qsoA6FHHrFHX6g6dx44XoZL36+TOFP7Hgv4Jhv7Sqo5ibvQ/6zwbD/7nro7bRE6oC0TAoIzgERgWpgaq5xHTFQZ4wsSf D3gtjloYy4Zzg3zkG1blCPxoHoavyYFK54Iq4CZELwGmxlREFDFkGSY7GUFx7oCm5ySKwW5W5VDPGvEsEuaOLua5BsGu77pcG+YiFe4+7IxCEB6glB5RbiFJKSGf7SGyHZL4DvHx4AIvHQRtIp6Va9KZ73EQDPgIANYxGjLjK6HSwdYC6EaPDEZwCkbkaUZQDUa0b0bWGzysaTB2EEw7AwiHBHCNhuz5STE+RrChRBSeQNo4QQrgpxGhE9hxDlpv LOEBHlRlTkTREjJnAtZEz04HBOJJCYpnLnZL6Lopy3YzokrZxb5mkFzLq5H8EH4VFA5VFJE1x/bn7NzOlH7lA351F35XpNGw62Tw6Sr3Qf7dGvqfQzy4CPADF1Gaoy68DgzUgjE/CmzOGRyVphJ7wzFhRxzzE049jwHvA9rBHyqbFurbGoHbgHFhkswnEBpEJ/AQq0TXGAGElRoG4ElwBsB5iHGsRYZFBYZgBRxlApBsTcZgDDlYbODYTXInBkTv CJBvBhimzPI4Zzm1pfAXxExXDbwrlkSTnLDTmbkwjaBKlvAqnwhXhwhT5lDODalBy6kmqwFJCTmK7UKhBQAsj6Au4yB1i1B9kCg652h8hQCGEfR6YapsQYAHHjwQBdY9Z9YDYnkwSiwgRrC1ok7wH5SRS95wwhSbBoXKC4ACQ6oXnwglphgQhuwFQFopkRg5DECQUQSOYwVYZwVc5QAIUprSRpoZpZpoUuLYCYXnYfIpABFJAk7wb7luwkVkVjjJ CSWhRt5Ni0S3lbCMXH7gVyScloi4CIk3DMV6UYIGWyx8hy7GX4AbgUDdkF4yxVA/x/yALAJgIQLHBQIwJwKILsmWVsZ5q+hFR1pwgwiNolQnDrG3CvJxA4TXivAwjfD5TNjD6bbdrhFIgXAlpUXEialNbvIQh3j0W950QOo7xnb1HH6LQpHtRpEb7WlZG2k74Om0rrrelX5unmin6cFVUbTtWumQB+mHSXrCpBktESoPrhko5SFTzRmFiDDxnxiJ narJl6qpnQylWlq5VTG5mBLQoFm7VFnexwbIhcxtjvSVms53E1noHc7tFYGnG4FKlCnZl64CyRl65dnVkRi9n9kHhsQzlChjmjnHkA1nnHAXlPIOwXBmwQY4aBSdROJ7Arl/Dlpwig1DmbnpXbCZWBE5Uog4ZXBjLN6rnlqSU8zFQflcafjfm/n/nnhAX9kfWmjgWsWODsXEHGXShs3QWoCaqZAvwIWkAKzKxqyazCUYWHTOQholoAjApJU8wKXk WoDJCIiuSSU4QGkNqIjaVJG6X6UhBGVMXSimVsbmUsZWVMU2Vsb2W6HgB3SzxwBwB6j4KAzFDQBojZABZHZTAMCEAIAUCGGPY2kUrXYETh19DqgQCiXoKqjwJZB6jVW2mpHpxR0x3jTx36BB2ZELqh0vY5HFy+3p1x2Lj6CuIFHlxcqunR0iAZ2l2J1dUelF210l0J0X79VVzlDF25CZ3SQ1H8pHQ12x092l2tCNGirN3D08Wl2uLO6An/bu3d3T 1ZCz2MHp5OKT111ZBW48GEl8Fp0t0j1t361mWG0fVD1b36Bbgm0G2GUW0BWb2t36Cm2W4rVmSbhR3MB8Ssj4BIJApuxjLwGbBYQuzbAnbbTf3agACaMBYYF5AINyoU0KSIZwvtRgbABgbtYSxi4IbWF9T9fdi85679/UUdEoJAPxju7t5DxAeoCAyt4DNDoCbAn019uAmgwQ315QNDz2UyhhrIsspAygIoAAFBCM8mfBI+I9QAKp2gAJTQRdLKDZ h8hVBCOiMPBUi8DYQyOaMyO1qnAKN4NL0N2Mhj0RKcCVhR7v4o5dL5jJYc1TI5DsOcPcA1bRZEDK3uMRipZe3Vb4n+jtJEluP4l4N2AGQIAWTMA6ipZwDMOsOpYcPdm+2igRKMCKwYP4BYMoKzSZAWT7zRYgQdL6CKwrU3FfXXWPoGA6h5MWOBIoFPjflyT5PpOZOyoiTgD6owRajhCAzYLCRAA= ``` %%