356 private links
A collection of antipatterns and how to avoid them
Time management can be useful
- Organizations don't use that much data.
Of queries that scan at least 1 MB, the median query scans about 100 MB. The 99.9th percentile query scans about 300 GB.
but 99.9% of real world queries could run on a single large node.
I did the analysis for this post using DuckDB, and it can scan the entire 11 GB Snowflake query sample on my Mac Studio in a few seconds.
When we think about new database architectures, we’re hypnotized by scaling limits. If it can’t handle petabytes, or at least terabytes, it’s not in the conversation. But most applications will never see a terabyte of data, even if they’re successful. We’re using jackhammers to drive finish nails.
As an industry, we’ve become absolutely obsessed with “scale”. Seemingly at the expense of all else, like simplicity, ease of maintenance, and reducing developer cognitive load
Years it takes to get to 10x:
10% -> ~ 24y
50% -> ~ 5.7y
200% -> ~ 2.10y
Scaling is also a luxurious issue in many cases: it means the business runs well.
- Hardware is getting really, really good
In the last decade:
SSDs got ~5.6x cheaper, 30x more on a single SSD and 11x faster in sequential reads and 18x in radom reads.
CPUs core count went up 2.6x, price went down at least 5x per core, each Turin core is also probably 2x-2.5x faster.
Distributed systems are also overkill as hardware progresses faster.
#define MAKE_U32_FROM_TWO_U16(high, low) ( ((uint32_t)(high) << 16) | ((uint32_t)(low) & 0xFFFF) )
Piccalilli shares links!
When we look around in our field, everyone in Tech seems to focus on one thing: "How can we adopt AI in our tooling and in our processes?"
So it is a proof of a bubble. Everyone is enthusiasts but it doesn't solve real use cases.
A rightful question can be: "How can we set up our engineers for long-term career success?"
Jens Meiert ask pertinent questions to solve this big up question.
What can be done reasonably well with AI today? (And tomorrow? And the day after tomorrow?)
How are our engineers affected by AI?
- Are our engineers using AI?
- How are our engineers using AI?
- What are realistic expectations for our engineers in terms of AI use and proficiency?
- Are we setting clear expectations for use of and proficiency with AI in our job descriptions as well?
- Do we document and anchor these expectations in our competency and skill matrixes?
- Are we watching the AI market, and are we evaluating tooling?
- While the AI market is in flux—which it may be for some time—, do we have enough flexibility (budget, processes, room for errors) to test AI tooling?
- If our engineers leave the company, would they find a new job—or would their profile make them less interesting?
- If they would not necessarily find a new job, what extra skills and experience do they need?
- How can we make our engineers ready for the AI age?
As you can tell, we cannot have all those answers yet—this is precisely why this is so important to get on top of, and it’s also the reason why I say “start answering.”
Now, everyone’s a prize exhibit in the FAANG zoo, because mastering this tangled mess is what opens their gates. Being just a CRUD monkey doesn’t feel fun anymore. If this is what “progress” looks like, I don’t want any part of it.
The technologies to build for 10 years ago dramatically improved!
As mentionned by LeHollandaisVolant, one thing the article doesn't mention is that:
- 1 the pages are more interactive
- 2 the data changes in real time
Hi, I’m Rob Weychert. I make art and design, obsess over film and music, hoard trivial archival data, and share it all on this here website. Enjoy your stay.
This manifest critics modern (bloated, unreliable and worsening over time) softwares.
It favorises self-reliant programming (few features and simplicity, minimum amount of dependencies, write your own tools). It has benefits such as learning, improving skills, simpler code, simpler tools, easy modification and deployment.
An organization of developers dedicated to writing free and open source software for everyone.
Ideas for creative projects. Lean and have fun.
“What I cannot create, I do not understand”
One ting to consider: KISS. The program can crash or panic for a lot of code path. Implement only the necessary!
KISS for maintainability: "Nothing in Rust forces us to get fancy. You can write straightforward code in Rust just like in any other language. But in code reviews, I often see people trying to outsmart themselves and stumble over their own shoelaces. They use all the advanced features at their disposal without thinking much about maintainability."
Here an real life example.
But if simplicity is so obviously “better,” why isn’t it the norm? Because achieving simplicity is hard!
Even in Rust: abstractions are never zero cost for developers.
Often, simple code can be optimized by the compiler more easily and runs faster on CPUs. That’s because CPUs are optimized for basic data structures and predictable access patterns. And parallelizing work is also easier when that is the case. All of that works in our favor when our code is simple.
Most of the code you’ll write for companies will be application code, not library code. That’s because most companies don’t make money writing libraries, but business logic. There’s no need to get fancy here. Application code should be straightforward to keep your fellow developers in mind.
Tips:
- start small
- avoid optimization early
- delay refactoring: we have limited information at the time of writing our first prototype.
- write code for humans
- The right abstractions guide you to do the right thing
As a nuclear engineer, I have never been asked to show my portfolio of reactor designs I maintain in my free time, I have never been asked to derive the six-factor formula, the quantization of angular momentum, Brehmsstrahlung, or to whiteboard gas centrifuge isotopic separation, water hammer, hydrogen detonation, or cross-section resonance integrals.
There's something deeply wrong with an industry that presumes you're a fraud unless repeatedly and performatively demonstrated otherwise and treats the hiring process as a demented form of 80s-era fraternity hazing.
The table is available as PDF https://dr-knz.net/programming-levels/prog-skill-matrix.pdf
Take a moment and think; what does everybody needs here?
What are the things this team struggle with?
How can I write up a doc, show what is wrong and propose a solution?
What are developer's blocks?
All these practices (documentation, tests, CI) are valuable, but sometimes they just mount up until you’re blocked.
Either you are new to the project and you just feel overwhelmed or you’ve been working on the project for a while, but you run out of stream and get stuck. it may be due to overwork or a lack of motivation.
How to unblock?
- Take time with learning. Sometimes, the docs or tests need to be read to get an idea of the externals. Eventually looking at the source code and building up a mental model of how it fall fits together. "If you think education is expensive, try ignorance"
- Realise when you're tired
- Work incrementally: implement it with the minimum effort. Circle back round to improve the tests, docs, etc...
- Write a prototype: Concern yourself only with the happy path. If you’re trying to learn about a dependency, it’s sometimes easier to write a quick prototype of using the dependency, possibly in an empty repository.
- Start with draft documentation: avoid premature docs polishing. Capture why you did things a particular way. Provide basic usage instructions.
- Release early, release often