- Miscellaneous

Maybe ORM/ODM are not dead? Yet...

Fri, 06 Mar 2026 00:00:00 +0000

So, let’s pick up where we left off. A couple of weeks ago, I wrote about how I took a 4-year-old fever dream—an ODM for Cassandra and ScyllaDB called coodie—and let an AI build the whole thing while I sipped my morning coffee.

It was a fun experiment. But then, a funny coincidence happened (or maybe the algorithm just has a sick sense of humor).

Right after I hit publish and started feeling good about my newfound “prompt engineer” status, I was listening to an episode of the pythonbytes podcast discussing Michael Kennedy’s recent post, Raw+DC: The ORM pattern of 2026?. The overarching thesis of their discussion? ORMs and ODMs are fundamentally dead. They are a relic of the past, bloated, abstraction-heavy, and ultimately, absolute performance killers.

I actually fired off a Twitter thread in response to it. And honestly, at first, I had to concede. They make a really good point. I spend my days deep in the ScyllaDB trenches, where we fight for every single microsecond. Putting a thick Python abstraction layer on top of a highly optimized driver usually sounds like a brilliant way to turn a sports car into a tractor.

But it got me thinking. How bad was coodie? Was my AI-generated Beanie-wannabe actually a performance disaster waiting to happen?

Giving it a test run

Like any respectable developer looking for an excuse to avoid real work, I decided to put my money where my AI-generated mouth is. Instead of sitting at my desk, I just offloaded the whole task to the Copilot agent from my phone to run some extensive benchmarks.

I didn’t just want to compare coodie to existing solutions like cqlengine. I wanted to establish an absolute performance floor. I wanted to test it against the Raw+DC pattern (Python dataclasses + hand-written CQL with prepared statements) to see exactly how much the “ORM tax” was really costing us.

The test spun up a local ScyllaDB node, hammered it with various workloads—inserts, reads, conditional updates, batch operations—and fetched the results back.

The finally surprising results

I had the agent run the script. I fully expected coodie to be heavily penalized. We all accept slower performance in exchange for autocomplete, declarative schemas, and not writing raw CQL strings.

I stared at the results the agent sent back to my screen. Then I told it to clear the cache, restart the Scylla container, and run it again just to be sure.

The results were genuinely surprising, and running them actually highlighted a few spots where we could squeeze out even more performance, leading straight to PR #190 to apply those lessons learned.

Here is the breakdown of what the agent and I found across the board.

Three-way Benchmark Results (scylla driver)

Benchmark	Raw+DC (µs)	coodie (µs)	cqlengine (µs)	coodie vs Raw+DC	coodie vs cqlengine
single-insert	456	485	615	1.06×	0.79× ✅
insert-if-not-exists	1,180	1,170	1,370	~1.00×	0.85× ✅
insert-with-ttl	448	469	640	1.05×	0.73× ✅
get-by-pk	461	520	665	1.13×	0.78× ✅
filter-secondary-index	1,370	2,740	8,530	2.00× 🟠	0.32× ✅
filter-limit	575	627	1,220	1.09×	0.51× ✅
count	904	1,500	1,590	1.66× 🟡	0.94× ✅
partial-update	409	960	542	2.35× 🔴	1.77× ❌
update-if-condition (LWT)	1,140	1,620	1,340	1.42× 🟡	1.21× ❌
single-delete	941	925	1,190	~1.00×	0.78× ✅
bulk-delete	872	921	1,200	1.06×	0.77× ✅
batch-insert-10	596	634	1,700	1.06×	0.37× ✅
batch-insert-100	42,800	1,960	52,900	0.05× 🚀	0.04× ✅
collection-write	448	485	679	1.08×	0.71× ✅
collection-read	478	508	689	1.06×	0.74× ✅
collection-roundtrip	939	1,060	1,380	1.13×	0.77× ✅
model-instantiation	0.671	2.02	12.1	3.01× 🔴	0.17× ✅
model-serialization	10.1	2.05	4.56	0.20× 🚀	0.45× ✅

Summary of Findings

Near Parity on the Hot Paths: For standard reads (get-by-pk), inserts, deletes, and basic limited queries, coodie is operating with a completely negligible overhead compared to writing raw CQL by hand (usually hovering around 5-13% tax). It also outperforms cqlengine across the board on these operations.

Pydantic V2 is a beast: We need to remember that Pydantic V2 is basically a blazing-fast Rust engine wearing a Python trench coat. The data validation, object instantiation, and serialization happen at the speed of Rust, keeping the overhead minimal from the start. Just look at the serialization and batch performance multipliers!

The AI kept it lean: The code the LLM generated wasn’t building massive ASTs or doing unnecessary query translations. coodie essentially formats a clean dictionary and hands it straight to the underlying Scylla driver to do its native magic. It gets out of the way.

Lessons Learned (PR #190): Even with great initial numbers, running the full benchmark suite revealed bottlenecks on things like partial-update and count. We realized we were doing redundant data validation during read operations when fetching data we already knew was valid straight from the database. By bypassing the extra validation pass and loading the raw rows more directly into the Pydantic models (using model_construct() for DB data), we can shave off the remaining overhead.

Wrapping up

So, maybe ORM/ODM are not dead? Yet…

If the final overhead of using an ODM on hot paths is a measly 5-13%, but in exchange I get full type-safety, declarative schemas, and zero boilerplate, I am taking the ODM every single time.

It seems my digital sidekick built something that is actually production-ready, and with a little bit of human-driven optimization, it screams.

You can check out the code, star it, or run your own tests over at github.com/fruch/coodie.

PRs are welcome. יאללה, let’s see how fast we can make it.

Coodie: A 4-year-old idea brought to life by AI (and some coffee)

Tue, 24 Feb 2026 00:00:00 +0000

So, let’s rewind a bit. About four years ago, I was looking at Beanie — this really nifty, Pydantic-based ODM for MongoDB. And as someone who spends an unhealthy amount of time deep in the trenches of ScyllaDB and Cassandra, I had a thought: “Where is my Beanie? I want a hoodie, but for Cassandra.” And thus, the name was born: coodie = cassandra + beanie (hoodie). Catchy, right?

Like any respectable developer, I rushed to GitHub, created the repository fruch/coodie, maybe wrote a highly ambitious README.md, set up a pyproject.toml, and then… absolutely nothing. Crickets. Life happened. I had CI pipelines to fix, Scylla drivers to maintain, conferences like PyConIL and EuroPython to attend, and live-tweeting to do. coodie just sat there, gathering digital dust, a glorious monument to my good intentions and severe lack of free time.

Fast forward to today. The AI era.

We are living in a weird timeline where LLMs are everywhere, threatening to take our jobs while simultaneously failing to center a div. People are talking about AI coding assistants non-stop. I myself was quite happy with my usual workflow, but I figured — why not take my 4-year-old fever dream for a walk in the park and feed it to the machine?

I threw the basic concept at an AI. I gave it some ground rules: it has to use Pydantic v2, it needs to be backed by scylla-driver, and it needs to feel as magical as Beanie.

And holy crap. It actually worked.

I spent four years procrastinating on this, and a glorified autocomplete bot basically bootstrapped the whole thing while I was sipping my morning coffee. I’m not sure if I should be proud of my newfound status as a “prompt engineer,” or slightly terrified that my open-source street cred is now partially owned by a matrix of weights and biases.

The sweet setup we have now

Despite the AI doing the heavy lifting, the architecture actually makes a lot of sense. Here is what coodie brings to the table:

Pydantic v2 all the way: Define your documents with full type-checking support.
Declarative schemas: You just annotate fields with PrimaryKey, ClusteringKey, Indexed, or Counter.
Automatic table management: You call sync_table() and it creates or evolves the table idempotently. No more writing raw CQL just to add a column.
Sync and Async APIs: Because why choose? You can use coodie.aio if you are riding the asyncio train, or coodie.sync if you want blocking code.
Chainable QuerySets: Doing await Product.find(brand="Acme").limit(10).all() just feels right.

Here is a quick taste of what the AI and I managed to cook up:

from typing import Annotated
from uuid import UUID, uuid4
from pydantic import Field
from coodie import Document, init_coodie, PrimaryKey

class Product(Document):
    id: Annotated[UUID, PrimaryKey()] = Field(default_factory=uuid4)
    name: str
    price: float = 0.0

    class Settings:
        keyspace = "my_ks"

And then querying it is a breeze:

# async
products = (
    await Product.find()
    .filter(brand="Acme")
    .order_by("price")
    .limit(20)
    .all()
)

Wrapping up

The AI works out of the box with almost zero friction. It’s very impressive, and honestly, a bit surreal to see an idea you abandoned years ago suddenly have a passing CI suite, complete with tests and GitHub Actions (which are still very tidy to look at, by the way).

You can check out the code, star it, or fork it over at github.com/fruch/coodie.

PRs are welcome. Preferably written by humans, but honestly… I guess I can’t really enforce that anymore, can I? יאללה, let’s see how it goes.

Takes from pyconil 2022

Tue, 28 Jun 2022 19:52:00 +0000

Takes from pyconil 2022

First in person conference I attend since Covid-19, was fun to meet some of the familiar faces and meet a few new ones. I’ve tried live tweeting, but seem like I’m not very good at it. (I’m better at hijacking others live tweets)

Day 1

Keynote

Keynote speaker couldn’t make it, and was replaced by a panel of the known experts in python While Sim Zacks asks them popular questions about python

@alonisser and I were a bit surprised, and live tweeted the panel session:

Hidden cost of your favorite function

@mayaGBar was talking about reason to profile, and the importance of looking under the hood of the utilities and functions we use and demonstrated using a profiler comparing different numpy function to achieve the same goal.

my live tweet

Hidden Hacks in Linters for Better & More Secure Code

@gemanor explained about a use case he and his team need to build lint checkers for block different personas form doing things that shouldn’t.

even that he introduce him self as non-pythonista (“alien” in his words), he managed to showcase a high level python craftsmanship, and a fun talk overall. he’s my MVP for this day

my live tweet

What happens when you import a module?

@reuvenmlerner with very interesting talk about how exactly the import system works while showing how to implement some parts of it.

while drinking the morning coffee, we had a talk with Reuven (me and someone I don’t remember his name..) about possible use-cases for using importlib.

one was for decryption of the sourcecode before executing it, to safeguard the patented algorithms and company secrets.

and my use case of using it for testing script that were meant to be run as main module. which is a nice way to avoid using subprocess, so mocks can be used: https://github.com/scylladb/scylla/pull/10809/files#diff-a9fc368799be893485a61ba81b9b736b2f3f9918a228351e7d6abe7293011584R104

my live tweet

Off road profiling - when the automated profilers just don’t cut it.

@ShacharSh goes over how to manually profile some bits of code, and how the project he built manprof can help with that.

that was last talk for the day, and I’ve could really concentrate on tweeting about it, but it was very nice to see Shachar talk, and improvise punchlines at the right time like when someone’s phone rang during his talk.

It was a shame that his package isn’t yet public (said something about legal didn’t yet improve, which always makes me a bit sad, that those processes are never straight forward)

הדרן עלך יום ראשון

let see how it goes tomorrow

Day 2

Keynote

This day start with me entering to main hall to see Bitya face staring at me (A former colleague of mine that gave a lecture in pyconil few years back)

Then the [@Yampeleg] started his colorful keynote, show the behind the scenes of competitive machine learning contexts and how he played with ML tools to “rig” a bit the results. and moved to more examples of what you can gain with mixing a bit of different models.

@alonisser (צייצינו המקצועי) with more details in his live tweet

BTW [@Yampeleg] is asking for forgivness about what he did. as someone that doesn’t have keggle account I’m automatically grant him one

How to Lift Your Tech Debt Curse with the Magic of the Open-Close Principle

@AmramDor talked about tech debt, start with a kind of “spoken word” show telling the horrors of how a project degrade during its lifecycle. then he moved to showing a practical code examples of how refactoring can take place and a few tips and guidelines on do it smoothly.

One thing I had missing, in real life every thing is a balance between few options, and each has its pros and cons. talking a bit about the cons of each refactoring practice was something I would have loved to hear, for example splitting a function into multiple method in multiple classes and maybe multiple files, might hinder readability of the code (just example, with proper names it might do the opposite)

live tweet

Optimizing Code Performance for Python Internals

@YonatanGoldsch2 started yet another talk about what seems to be profiling (which that was more than handful of those) but then supersized me with a deeper talk that shad light into the inner working of dicts, that I wasn’t aware of and for sure would make me think next time I’m deleting attributes from objects.

live tweet

JSON - The Fine Print

@tebeka talks are always a pleasure to attend, and this one was interesting to see how much can we learn about something as commonly used as json.

in the end I didn’t learn anything surprisingly new, but Miki exampled a few of the things people should remember when reading/parsing json, for example handling datatime of bytes.

BTW I know that the json parser have advanced features, like hooks Miki showed. but I never ever used them, I probably should start.

live tweet

Overcoming Concurrency Issues in Web Applications

@be_haki gave a great and funny talk, on a subject very dear to my heart, since I was facing almost the exact same things a few years back. (and I’m work for @scylladb a DB company for the last 3 years)

unfortunately for me, we weren’t using SQL DB, but couchbase and mongodb and almost none of the suggested solutions fit those DBs, since they by design don’t give similar guarantees or functions.

After the talk when I talked with Haki, he was surprised how many people attend (also said that in the beginning of the talk) I wasn’t surprised, this topic is sound so trivial (once explained) but it so many times forgotten or overlooked and solutions to some of those type of problems are not easy to implement on different types of DBs, you need a higher the usual understanding of the specific DB you are working with.

(I think I should take this one to our experts to see their suggestions)

I was so enjoying the talk, I’ve forgotten about tweeting, Alon is better tweeter (and he should get a raise for that)

alon’s live tweet

Taking Github Actions for a walk in the park

Tue, 29 Dec 2020 00:00:00 +0000

Taking Github Actions for a walk in the park

Travis started to scare me a bit

People were talking about github actions for a while now, lot of hype around it. I myself was quite happy with the things I’ve managed to get done with Travis.

But then came in this announcement from travis: “The new pricing model for travis-ci.com”. I was reading via lot of the backlash in tweeter about it, and after carefully reading the announcement itself, I was starting to think that the work I did for scylla-driver was counting on Travis too much.

Recently I’ve start seeing long queues for job triggers in travis, and I was starting to have second thoughts about picking travis. (not so long ago, less than a year ago)

The sweet setup I had in Travis

As a recap for scylla-driver, since it’s a c based package, heavily using cython. We are building python wheels for windows, mac, linux, ranging from python2.7 to python3.9, and pypy. (also using the Travis experimental aarch64 and ppc64le support)

For building the wheels we are using cibuildwheel, which is a nifty tool that magically keep all know how, on how building wheels correctly on multiple platforms. In linux, it’s using the official docker images provided by the [Python Packaging Authority ] for manylinux2014. And on windows and mac equivalents with the correct set to needed tool to get you wheel happily containing all the need dynamic libraries it depended on

cibuildwheel also comes with readily available examples for all the major CI system out there

cibuildwheel is also helping you in running unittest ontop of each built wheel (since in my case the setup is a bit complex, getting it installed and compile across the board was hard, we we’re skipping some unittest on some platforms)

Travis was also running our integration test suite, and running it on our PRs, and automatically uploading all the wheels and source distribution to Pypi.

I was working on this setup for long week, with a tight deadline for it to be ready for europython2020, thanks to @ultrabug

Github actions

For a while I was a bit unsure if it would be a match to what I had in travis. But one day our own Jenkins server was down for maintenance, and a big portion of my day was clear case of that.

I’ve started with copy-pasting cibuildwheel github actions example, and worked my way from there.

When I’ve started adding all the different unittest and platforms, I was struggling a bit with how the matrix is working, and how conditional steps works.

It ended up something like this, almost identical to what I had in travis:

name: Build and upload to PyPi

on: [push, pull_request]

env:
  ...

jobs:
  build_wheels:
    name: Build wheels $ ($)
    if: contains(github.event.pull_request.labels.*.name, 'test-build') || github.event_name == 'push' && endsWith(github.event.ref, 'scylla')
    runs-on: $
    strategy:
      fail-fast: false
      matrix:
        include:
          - os: ubuntu-18.04
            platform: x86_64

          - os: ubuntu-18.04
            platform: i686

          - os: ubuntu-18.04
            platform: PyPy

          - os: windows-latest
            platform: win32

          - os: windows-latest
            platform: win64

          - os: windows-latest
            platform: PyPy

          - os: macos-latest
            platform: all

          - os: macos-latest
            platform: PyPy

We had a nice twist that we didn’t have before, but now we can control from github PR which part of my pipeline would run using label, since github actions is so tightly integrated you can lookup almost anything related to the PR.

Github Actions - UI

This is how the final thing looks like:

logging is a bit weird to watch while it’s running, but after the fact the log are very tidy to look at

Late surprise - cross compile using qemu

With almost perfect timing I’ve run into this https://github.com/joerick/cibuildwheel/pull/482 by Pavel Savchenko

I was trying all kind of things to get aarch64 support, but this PR figure it out quite nicely, with one magic step:

- name: Set up QEMU
id: qemu
uses: docker/setup-qemu-action@v1
with:
    platforms: all

For reference the full workflow

Now we can have docker using qemu to run docker instances based on different cpu architecture, it’s horribly slow but it’s working.

And nice side effect I’ve learned how to run it locally, so I can debug those things. Something I haven’t figured when running with Travis, apparently travis was using the same trick on their experimental support for those architectures.

Summary

Github actions works out of the box, with almost zero fraction, almost anything I could think about I’ve found a readily available step I can pickup, very impressing.

One thing I haven’t yet tried out, is using self hosted runner, hopefully I won’t need it. Cause who want to babysit anther server on his own, or worse, a bunch of them.

How much fun was EuroPython 2020

Wed, 29 Jul 2020 00:00:00 +0000

How much fun was EuroPython 2020

#pyconil got canceled

This year I’ve finally got enough courage and will, and I had 2 submissions for #pyconil. COVID-19 had other plans, and #pyconil was canceled

I’ve told @ultrabug about this (Numberly CTO, Alexys Jacob), after a few weeks he surprised me with telling me he’s gonna present scylla-driver in europython2020, the shard-aware driver we were working on in the last 6 months.

At the time it wasn’t yet ready nor publish. (Also found out that Numberly were sponsoring europython for years now) Took me a few seconds to figure that he just set me deadline without my consent…

Fast-forwarding a bit, and scylla-driver initial release came out: https://pypi.org/project/scylla-driver/3.21.0/

And my tickets for europython2020 were booked…

Discord is fun

Few days before the date, I’ve got an email with instruction to connect to the discord of europython2020, since it’s my first COVID-19 online conference, I’ve registered and login straight away.

It was really nice to be start to start talking with people and meet them a few days before

Each track got its own chat room, backed with zoom webinar and online youtube stream each talk got its own chat room, so people can promote their and answer question before and after the talks. Sponsors had rooms too, and all the sprints had rooms (we’ll get to that later on)

Talk about timing, day before europython2020, this press release came out https://www.scylladb.com/press-release/discord-chooses-scylla-core-storage-layer/

Talks — Day 1

The opening keynote was cancelled cause of technical difficulties, that lost touch with whom was supposed to talk. (it rescheduled to next day)

Elias Mistler was talking about “How to write multi-paradigm code” show casing and coding live a sudoku solver, with functional and object originated approaches, and how to mix them both. The code is available as notebook
Bojan Miletic was talking about “Django Testing on Steroid: pytest + Hypothesis” — shows the basic usage of hypothesis, I’m hear about hypothesis all over the place, maybe it will bring me one step close to use is in one of my projects
Martin Christen gave a very cool demo of “pyRT - Computer Graphics in Jupyter Notebooks for Fun and Teaching” - I love Jupyter notebook visual trick, and I love 3d. So this one I enjoyed a lot, Martin was quite a performer and funny. I strongly recommend trying out the demo
Bernat Gabor gave an excellent talk “Lessons from the Trenches: rewriting and re-releasing virtualenv” about a big refactoring virtualenv had, packaging is a subject close to my heart. This is an excellent lesson on how to approach major refactor and a reminder that it would always take more time than you’ll estimate it. I must say that this change, from a POV of a heavy user of virtualenv, wasn’t even noticed. When he was demoing a few things I’ve realized that I’ve seen it in logs, but didn’t know the big story happing behind the scenes.
In the next one, Philip Jones was building a ASGI server from scratch, show the details of the ASGI protocol — “An ASGI Server from scratch”
“A deep dive and comparison of Python drivers for Cassandra and Scylla” — Alexys great slides (which I had a peek at few days before)with lot of emojis, going down to the details of how cassandra/scylla hash ring works, and how scylla shard share nothing architecture is use by the scylla-driver fork we work on for the last few months. And he finished with data from Numberly production before/after moving to shard aware client, 2x-2.5x speed up in response time I was live tweeting the whole thing into our company slack…
“Elegant Exception Handling”
“Best practices for production-ready Docker packaging”
“Pluggable Architecture”
“Writing Zenlike Python”

Social — the missing `e` in my whisky

During the whole day I was playing cat and mouse with people, trying to bring them into the open track, to have a face to face meeting, only later after all the sessions end, some people came in, most of them were the part of the organizing team. Every one where show their beer or whisky that they were drinking. Keith Gaughan was laughing at the ice in my whisky, and also thought me a real whisky is spelled with whiskey with an e in it.

Marc-Andre Lemburg was talking about the challenges they have as organizers, one thing led to anther, and I ask how can I help. He said they need a hand with handling the tweeter account, and that he’ll hook me tomorrow with whom handle it.

After a few more rounds of whisky, I call it a day.

Talks — Day 2

TODO:

Sprints

packaging -
hypothesis -
terminusdb-client -

Introducing pytest-elk-reporter

Mon, 24 Jun 2019 00:00:00 +0000

pytest-elk-reporter

tl;td: https://github.com/fruch/pytest-elk-reporter

History

Few years back I’ve wrote a post about how I’ve connected python based test to ELK setup - “ELK is fun”, it was using an xunit xml, parsing it and sending it via Logstash.

Over time I’ve learn a lot about ElasticSearch and it’s friend Kibana, using them as a tool to handle logs. and also as a backend for a search component on my previous job.

So now I know logstash isn’t needed for reporting test result, posting straight into elasticsearch is easier and gives you better control, ES is doing anything “automagiclly” anyhow nowadays.

What can it do ?

collect as much information as you can about one test results and running environment and send it to elasticsearch.
- data like git commit/sha from current directory
- jenkins environment variables, like job id and username triggered the job.
- cool tracebacks which pytest excels at.
extend - let user configure (like target address and credentials) it from code, and add data of their own to each session or test.

Tricks I’ve picked on the way

Every time I start a new package, I learn a few new things, here’s a few I’m happy about this time:

Created the package with cookiecutter-pytest-plugin
I’m using pre-commit
- black - instead of the old friend autopep8.
- pylint in both python27 and python36 to keep the code clean of silly mistakes.
Travis

build and test with Tox on all possible python versions.

also uploading wheels to pypi based on tags.

thanks to setuptools_scm version no need to edit files for setting version number.
Appveyor

I also cover windows own little issue (non found so far).
Codecov

really like that it can collect from multiple run, for example both linux and windows tests run, which helps to show the total coverage.

got me to 93% line coverage (and going down as I add more code).

The bad stuff - a.k.a things I need to improve

Documentation - I’ve mostly wrote test, zero docs.
Name - it’s a bit boring, and might conflict with other plugins.
Kibana Dashboards examples

I should really include some sample kibana dashboard to show the strength of it, it’s own test data is a bit boring.
auto generating and upload of docs

I’ve tried using Travis Deploy to github Pages, didn’t exactly nailed, or got it working as I expect, maybe I should try readthedocs.
pytest-xdist

I still need to test it with xdist, it’s going to make thing a more tricker, as usual.

What you should next

Go grab it, git it a star, fork it.

I’m waiting for your PRs, and your stories on how it’s help you tame a whole bunch of tests.

    # forgot to show how to install :\
    pip install pytest-elk-reporter

KISS - Keep it shorter short

Fri, 08 Jul 2016 00:00:00 +0000

Today on work on a python related mailing list someone sent this snippet of code he didn’t wrote. and that was causing some of the server he worte/manged some troubles.

Here’s the code:

    def deploy_file(full_path_to_file, full_path_in_server):
        if full_path_to_file is None or full_path_in_server is None or full_path_to_file == "" or full_path_in_server == "":
            print_error_exit(" wrong input for deploy_file")
        try:
            post_status = None
            with open(full_path_to_file, mode="rb") as f:
                print ("Deploying...:  %s" % full_path_in_server)
                file_content = f.read()
                post_status = requests.put(url= full_path_in_server, data=file_content)
                if post_status.status_code == 200 or post_status.status_code == 201 or post_status.status_code == 202:
                    print ("Success:  %s" % full_path_in_server)
                    return {"uploaded": file_to_upload }
                else:
                    print_error_exit ("Deployment to %s failed. status_code = %s, returned_text = %s" % (full_path_in_server, post_status.status_code, post_status.text))
        except Exception as ex:
            print (ex)
            print_error_exit ("Posting with url %s failed. status: %s" % (full_path_in_server, post_status))

First thought I hade was, the one wrote it is must be coming from C (or maybe Java), so let go over what’s wrong here.

Input validation

I don’t know what’s print_error_exit() is doing exactly, let’s assume it’s raising expection (hopefully it’s not calling os.exit())
So assertion with the proper text would be alsmot identical.

Checking both for None, and for empty string is futile, just assert, like this:

        assert full_path_to_file, "parameter can't be empty"
        assert full_path_in_server, "parameter can't be empty"

Using requests to stream file

Requests is cool, alsmot the defacto standard when HTTP client is need in python. it has very good documention, clearly someone didn’t read it.streaming uploads
Multiple error paths

Lot of different error handling code, one for checking the http statuses and one for catching expections.

My version

    def deploy_file(full_path_to_file, full_path_in_server):
        assert full_path_to_file, "parameter can't be empty"
        assert full_path_in_server, "parameter can't be empty"

        res = None
        try:
            with open(full_path_to_file, mode="rb") as f:
                logging.info("Deploying...:  %s", full_path_in_server)

                res = requests.put(url= full_path_in_server, data=f)

                assert res.status_code in [200, 201, 202]
                logging.info("Success:  %s", full_path_in_server)

                return {"uploaded": file_to_upload }

        except Exception as ex:
            logging.expection("deploy_file faild") # so we'll have a nice print in the log

            # casue I don't know what print_error_exit() does, we'll call it.
            # even that I would prefer to just raise the expection up, i.e. "raise ex"
            print_error_exit ("Deployment to %s failed. status_code = %s, returned_text = %s" % 
                (full_path_in_server, res.status_code if res else None, res.text if res else None))

Taming the logging formatter

Thu, 06 Nov 2014 00:00:00 +0000

We love logging, and we use it all around our code. (it’s the standard/built-in/batteries included)

We have those hugh json objects flying around, we want to log them.

pprint.pformat comes handy for those cases:

import pprint
my_large_data = [ dict(data="my large inforamtion", data1="dicts, dicts, dicts,", data3=dict(a="??", b=2, c=3, d="***")),
                  dict(data="my large inforamtion", data1="how we love big dicts ?", data3=dict(a="??", b=2, c=3, d="***")),
                  dict(data="my large inforamtion", data1="how we love big dicts ? don't we ?,", data3=dict(a="???", b=2, c=3, d="***")),
                ]

logger.error("Mary \n had \n a \n little \n lamb")
logger.error(pprint.pformat(my_large_data))
logging.error(pprint.pformat(my_large_data))

but when used with complex formatter the output goes like that:

^14/11/06 00:58:39 !ERROR <t:MainThread T:root M:log.py F:<module> L:74 > Mary
 had
 a
 little
 lamb
^14/11/06 00:58:39 !ERROR <t:MainThread T:root M:log.py F:<module> L:75 > [{'data': 'my large inforamtion',
  'data1': 'dicts, dicts, dicts,',
  'data3': {'a': '??', 'b': 2, 'c': 3, 'd': '***'}},
 {'data': 'my large inforamtion',
  'data1': 'how we love big dicts ?',
  'data3': {'a': '??', 'b': 2, 'c': 3, 'd': '***'}},
 {'data': 'my large inforamtion',
  'data1': "how we love big dicts ? don't we ?,",
  'data3': {'a': '???', 'b': 2, 'c': 3, 'd': '***'}}]
^14/11/06 00:58:39 !ERROR <t:MainThread T:root M:log.py F:<module> L:76 > [{'data': 'my large inforamtion',
  'data1': 'dicts, dicts, dicts,',
  'data3': {'a': '??', 'b': 2, 'c': 3, 'd': '***'}},
 {'data': 'my large inforamtion',
  'data1': 'how we love big dicts ?',
  'data3': {'a': '??', 'b': 2, 'c': 3, 'd': '***'}},
 {'data': 'my large inforamtion',
  'data1': "how we love big dicts ? don't we ?,",
  'data3': {'a': '???', 'b': 2, 'c': 3, 'd': '***'}}]

Kinda, not really helpful:

confusing - and you loose track of where this print came from really fast (keep in mind I’m talking about humongous json object)
filtering is (almost) impossible (yeah, not without some extra regex kung-fu)

Skimmed through the logging documentations, google it a bit think someone already solved those things and I could copy paste, came out empty handed.

So here how it can be fixed:

import sys
import logging


class MultiLineFormatter(logging.Formatter):

    def format(self, record):
        """
        split the message into different line and prepend it with the formatting
        :param record:
        :return: the formatted string
        """
        output = []

        if self.usesTime():
            record.asctime = self.formatTime(record, self.datefmt)

        data = dict(**record.__dict__)

        message = record.getMessage()
        for msg in message.split('\n'):
            data['message'] = msg
            output += [self._fmt % data]

        s = "\n".join(output)

        if record.exc_info:
            # Cache the traceback text to avoid converting it multiple times
            # (it's constant anyway)
            if not record.exc_text:
                record.exc_text = self.formatException(record.exc_info)
        if record.exc_text:
            if s[-1:] != "\n":
                s = s + "\n"
            try:
                s = s + record.exc_text
            except UnicodeError:
                # Sometimes filenames have non-ASCII chars, which can lead
                # to errors when s is Unicode and record.exc_text is str
                # See issue 8924.
                # We also use replace for when there are multiple
                # encodings, e.g. UTF-8 for the filesystem and latin-1
                # for a script. See issue 13232.
                s = s + record.exc_text.decode(sys.getfilesystemencoding(),
                                               'replace')

        return s

# and two examples how to configure the root logger to use it

import logging
import logging.config

logger = logging.getLogger()
logging.basicConfig()

def configure_with_api():
    handler = logging.StreamHandler()
    formatter = MultilineFormatter(fmt='^%(asctime)s !%(levelname)s <t:%(threadName)s T:%(name)s M:%(filename)s F:%(funcName)s L:%(lineno)d > %(message)s',
                                  datefmt='%y/%m/%d %H:%M:%S')
    handler.formatter = formatter
    logger.addHandler(handler)

def configure_with_dictConfig():

    logging.config.dictConfig( {
        'version': 1,
        'disable_existing_loggers': True,
        'formatters': {
            'multiline': {
                '()' : MultilineFormatter,
                'format': '^%(asctime)s !%(levelname)s <t:%(threadName)s T:%(name)s M:%(filename)s F:%(funcName)s L:%(lineno)d > %(message)s',
                'datefmt': '%y/%m/%d %H:%M:%S'
            },
        },
        'handlers': {
            'console':{
                'class':'logging.StreamHandler',
                'formatter': 'multiline'
            },
        },
        'loggers': {
            '': {
                'handlers':['console'],
                'propagate': True,
                'level':'INFO',
            },
        }
    }
    )

and now our output is much much clearer:

^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:74 > Mary
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:74 >  had
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:74 >  a
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:74 >  little
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:74 >  lamb
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:75 > [{'data': 'my large inforamtion',
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:75 >   'data1': 'dicts, dicts, dicts,',
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:75 >   'data3': {'a': '??', 'b': 2, 'c': 3, 'd': '***'}},
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:75 >  {'data': 'my large inforamtion',
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:75 >   'data1': 'how we love big dicts ?',
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:75 >   'data3': {'a': '??', 'b': 2, 'c': 3, 'd': '***'}},
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:75 >  {'data': 'my large inforamtion',
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:75 >   'data1': "how we love big dicts ? don't we ?,",
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:75 >   'data3': {'a': '???', 'b': 2, 'c': 3, 'd': '***'}}]
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:76 > [{'data': 'my large inforamtion',
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:76 >   'data1': 'dicts, dicts, dicts,',
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:76 >   'data3': {'a': '??', 'b': 2, 'c': 3, 'd': '***'}},
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:76 >  {'data': 'my large inforamtion',
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:76 >   'data1': 'how we love big dicts ?',
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:76 >   'data3': {'a': '??', 'b': 2, 'c': 3, 'd': '***'}},
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:76 >  {'data': 'my large inforamtion',
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:76 >   'data1': "how we love big dicts ? don't we ?,",
^14/11/06 00:54:51 !ERROR <t:MainThread T:root M:log.py F:<module> L:76 >   'data3': {'a': '???', 'b': 2, 'c': 3, 'd': '***'}}]

Disclaimer: use with caution, written past midnight :)

ELK is fun

Thu, 30 Oct 2014 00:00:00 +0000

As I do a lot of testing and CI in my day job, I took ELK for a test drive. (ELK stands for ElasticSearch, LogStash and Kibana)

I’ve played around with Logstash before, but it’s first time i’m trying Kibana.

I’ve look for a quick way to setup it, and there where gazillion hits in Docker Hub (I’ve forgot which one I’ve actually installed :)) EDIT: it was cyberabis/docker-elkauto.

That went quick quickly, with a small change that I’ve used port 81

Next thing I wanted to feed it with some data, good thing someone else added some key=value files to our CI just a few days ago (Thanks Mike). And I had first real data in to play with in matter of minutes

That’s was not enough for me, I wanted all the information from my tests. Seem like the junit-xml output has all the information I wanted. I hated the idea of running something after the tests are running, I wanted it to run automatic on any test I’ll run with py.test.

Took me some time to figure it out which py.test hook fits the best, and here’s how it can be done:

    # content of conftest.py
    def collect_ec_data():
        """
        if we run in EC, collect all data we need
        if not in EC, collect only commit and commit_sha
        :return: dict
        """

        # lets assume you don't really want to know what we save in our setup :)
        return data_collected

    def send_to_elk(item):
        """
        send `item` to Logstash port
        :param item: dict
        :return: None
        """
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        s.connect((TCP_IP, TCP_PORT))
        logger.info("Sending to %s:%d", TCP_IP, TCP_PORT)
        s.sendall(json.dumps(item) + '\n')
        s.close()

    def find_test_data(junit_xml_filename):
        """
        flatten all the data from junit based xml file into a list of dicts

        :param junit_xml_filename: the output from py.test --junit-xml
        :return:
        """

        tests_list = []

        xml_test = open(junit_xml_filename, 'r').read()

        et = ElementTree.fromstring(xml_test)
        for test in et.iter('testcase'):

            item = dict(**test.attrib)
            outcome = None
            failure = test.find("failure")
            error = test.find("error")
            skipped = test.find("skipped")

            if failure is not None:
                item['failure_message'] = failure.text
                outcome = 'failure'
            elif skipped is not None:
                item['skip_message'] = skipped.attrib['message']
                outcome = 'skipped'
            elif error is not None:
                item['error_message'] = error.text
                outcome = 'error'
            else:
                outcome = 'success'

            item['outcome'] = outcome
            item['run_time'] = item['time']
            # time is a built in in Logstash
            del item['time']

            tests_list.append(item)

        # filter by group, and merge result
        cumulative_keys = ['outcome', 'run_time']
        tests_list_grouped = []
        for key, group in groupby(tests_list, lambda x: (x['name'], x['classname'])):
            item = dict()
            for thing in group:
                # merge needed values
                if item.has_key('outcome'):
                    item['outcome'] += " & " + thing['outcome']

                if item.has_key('run_time'):
                    item['run_time'] += thing['run_time']

                # overwrite the rest
                for key in thing.keys():
                    if key not in cumulative_keys or not item.has_key(key):
                        item[key] = thing[key]
            tests_list_grouped.append(item)

        return tests_list_grouped

    def pytest_unconfigure():

        junit_file = pytest.config.option.xmlpath
        ec_data = collect_ec_data()
        log_dir = os.environ.get('LOG_DEST_DIR', './')

        if junit_file:  # if found xml file, send all of it
            tests_list = find_test_data(junit_file)
            for i, test in enumerate(tests_list):
                item = dict(**ec_data)
                item.update(test)

                # save data to a file
                with open(os.path.join(log_dir, 'job_info%d.json' % i), 'w+') as f:
                    f.write(json.dumps(item, sort_keys=True,
                                       indent=4, separators=(',', ': ')))

                send_to_elk(item)

Just for reference, to show how simple is the logstash configuration, almost untouched:

input {
        tcp { port => 3333 type => "text event"}
        tcp { port => 3334 type => "json event" codec => json_lines {} }
}
filter {
        if [createTime] {
                date {
                        match => [ "createTime", "ISO8601" ]
                        target => "@timestamp"
                }
        }

}
output { elasticsearch { host => localhost } }

ELK is a great combo, me like a lot…

TheLinuxGameTome is dying, lets save it

Fri, 05 Apr 2013 00:00:00 +0000

I’ve read this story on Slashdot, and thought I might take the database as a django practice build a real web site. Who want to join me ?

- Miscellaneous

Maybe ORM/ODM are not dead? Yet...

Giving it a test run

The finally surprising results

Three-way Benchmark Results (scylla driver)

Summary of Findings

Wrapping up

Coodie: A 4-year-old idea brought to life by AI (and some coffee)

Fast forward to today. The AI era.

The sweet setup we have now

Wrapping up

Takes from pyconil 2022

Takes from pyconil 2022

Day 1

Keynote

Day 2

Keynote

Taking Github Actions for a walk in the park

Taking Github Actions for a walk in the park

Travis started to scare me a bit

The sweet setup I had in Travis

Github actions

Github Actions - UI

Late surprise - cross compile using qemu

Summary

How much fun was EuroPython 2020

How much fun was EuroPython 2020

#pyconil got canceled

Discord is fun

Talks — Day 1

Social — the missing e in my whisky

Talks — Day 2

Sprints

Introducing pytest-elk-reporter

pytest-elk-reporter

History

What can it do ?

Tricks I’ve picked on the way

The bad stuff - a.k.a things I need to improve

What you should next

KISS - Keep it shorter short

My version

Taming the logging formatter

ELK is fun

TheLinuxGameTome is dying, lets save it

Social — the missing `e` in my whisky