Kalshi has some good engineers building their platform - recently reliability has been very high from the exchange standpoint (altho it can always be better) and I think they have overcome a lot of tech debt from the early days (when decisions were made without fully considering future volume - standard for a startup). I think there is still a general understaffing and they still do not take certain bugs as seriously as they should (given the sheer amount of money some have at risk - but that is steadily improving as well).
Overall, I think a lot of people underestimate the difficulty of building a financial market at a scale they are at now - its not some simple distributed system where they can spin up infinite runtimes to handle additional volume - there are a lot of bottlenecked processes due to requirements around a single source of truth - you cannot just multi-thread that up - that creates a shell game of resource locks and contentions that will lead to a dumpster fire.
Kalshi has some good engineers building their platform - recently reliability has been very high from the exchange standpoint (altho it can always be better) and I think they have overcome a lot of tech debt from the early days (when decisions were made without fully considering future volume - standard for a startup). I think there is still a general understaffing and they still do not take certain bugs as seriously as they should (given the sheer amount of money some have at risk - but that is steadily improving as well).
Overall, I think a lot of people underestimate the difficulty of building a financial market at a scale they are at now - its not some simple distributed system where they can spin up infinite runtimes to handle additional volume - there are a lot of bottlenecked processes due to requirements around a single source of truth - you cannot just multi-thread that up - that creates a shell game of resource locks and contentions that will lead to a dumpster fire.
I meaaan if they have the volume there's no reason why they couldn't shard things by region/etc.
I meaaan if they have the volume there's no reason why they couldn't shard things by region/etc.
Can't wait to unlock that cross-shard arbitrage lol
Can't wait to unlock that cross-shard arbitrage lol
more markets more dollars
more markets more dollars
Are you suggesting a setup where not all market participants can trade with all other participants within a market? That seems suboptimal.
Are you suggesting a setup where not all market participants can trade with all other participants within a market? That seems suboptimal.
pretty much lol
as long as the markets are saturated it's not really an issue
pretty much lol
as long as the markets are saturated it's not really an issue
queue position nightmare, among other issues
queue position nightmare, among other issues
would literally be no different from how the current market works
would literally be no different from how the current market works
It's already done with most fintech for regulatory reasons (global vs USA type ish)
It's already done with most fintech for regulatory reasons (global vs USA type ish)
I've never built an exchange, but AWS has really big machines available. Seems like vertical scaling is possible for a long way if the engineering is good
I've never built an exchange, but AWS has really big machines available. Seems like vertical scaling is possible for a long way if the engineering is good
Where are kalshi servers at? What’s the best spot for a virtual machine?
Where are kalshi servers at? What’s the best spot for a virtual machine?
us-east-2
us-east-2
Thank you bro. They’re confirmed at us-east-2?
Thank you bro. They’re confirmed at us-east-2?
Is there documentation for that
Is there documentation for that
dns
dns
A, B, or C?
A, B, or C?
Imagine if Visa/Mastercard was like "oh, you don't have enough money on that shard - transaction denied!" - unless you are trying to prove my point around people underestimating building a financial market at scale...
Imagine if Visa/Mastercard was like "oh, you don't have enough money on that shard - transaction denied!" - unless you are trying to prove my point around people underestimating building a financial market at scale...
It changes - you will need to measure yourself.
It changes - you will need to measure yourself.
how can it change
how can it change
One core/thread/memory space can only have so much throughput - you will be limited by at least one of those no matter your architecture - an orderbook can only have single source of truth - you can't scale that out
One core/thread/memory space can only have so much throughput - you will be limited by at least one of those no matter your architecture - an orderbook can only have single source of truth - you can't scale that out
Server restarts for one - there are weekly maintenance windows and occasional meltdowns - plus I assume there are failovers that we don't even notice.
Server restarts for one - there are weekly maintenance windows and occasional meltdowns - plus I assume there are failovers that we don't even notice.
hmm interesting thank you
hmm interesting thank you
well, I'd imagine you can do a lot with an instance of the top tier at 1920 CPUs and 32 TB of memory. I suppose the DB is the real bottleneck, they always seem to be
well, I'd imagine you can do a lot with an instance of the top tier at 1920 CPUs and 32 TB of memory. I suppose the DB is the real bottleneck, they always seem to be
I doubt the DB comes into play - most likely they don't even write/read to/from the DB directly from the transaction threads - they likely use some sort of pub/sub queue or message bus they write transactions to for other runtimes to consume (one of which likely persists it) - the orderbook state is almost certainly just held in runtime memory. The additional CPUs can help with multiple markets but I doubt a single market orderbook is spread out across multiple CPUs/threads (some CPUs may be tied to multiple orderbooks). Given the sheer volume, I am guessing that they would even have issues with using a dynamic assignment process for orders - more efficient to dedicate a thread to an orderbook rather than having it pick up new qork from a queue and have to load state from a memory space.
I doubt the DB comes into play - most likely they don't even write/read to/from the DB directly from the transaction threads - they likely use some sort of pub/sub queue or message bus they write transactions to for other runtimes to consume (one of which likely persists it) - the orderbook state is almost certainly just held in runtime memory. The additional CPUs can help with multiple markets but I doubt a single market orderbook is spread out across multiple CPUs/threads (some CPUs may be tied to multiple orderbooks). Given the sheer volume, I am guessing that they would even have issues with using a dynamic assignment process for orders - more efficient to dedicate a thread to an orderbook rather than having it pick up new qork from a queue and have to load state from a memory space.
IIRC one of the kalshi engineers shared that the matching engine for the entire exchange is a single thread
IIRC one of the kalshi engineers shared that the matching engine for the entire exchange is a single thread
the chat history in this channel seems to have been truncated so i can't find the message, unfortunately
the chat history in this channel seems to have been truncated so i can't find the message, unfortunately
Makes some sense - on top of orderbook there is also balance to consider - there likely may be some ways to handle those two datapoints but there would be trade offs
Makes some sense - on top of orderbook there is also balance to consider - there likely may be some ways to handle those two datapoints but there would be trade offs
I found this talk from a Jane Street guy interesting in terms of exhcange architecture: https://www.janestreet.com/tech-talks/building-an-exchange/
I found this talk from a Jane Street guy interesting in terms of exhcange architecture: https://www.janestreet.com/tech-talks/building-an-exchange/
if you enjoyed that one, I'd also recommend this podcast episode also from JS: https://signalsandthreads.com/state-machine-replication-and-why-you-should-care/ where they discuss their sequencer which rips through millions of messages per sec on a single core
if you enjoyed that one, I'd also recommend this podcast episode also from JS: https://signalsandthreads.com/state-machine-replication-and-why-you-should-care/ where they discuss their sequencer which rips through millions of messages per sec on a single core
yeah, i think the ME has to keep every user's balance in memory since it is constantly having to prune orders when a user's balance can't cover it, and it would be prohibitively slow to keep calling out to some other service to get a user's balance
yeah, i think the ME has to keep every user's balance in memory since it is constantly having to prune orders when a user's balance can't cover it, and it would be prohibitively slow to keep calling out to some other service to get a user's balance
we also know that settlements involving a lot of users are slow. from my notes, james shared some details about factors that contribute to ME latency a while back:
> there's just a lot of users whose positions and balances we need to update. sometimes there's additional computation around collateral return.
>
> it's not slow for any single user, (which is why you don't notice latency spikes when we regularly settle markets with a small number of traders), but you should do some back of the envelope calculations for how long you think it should take to do all of this cleanup for a single user and then multiply it by the number of users you think are trading on some of these markets 🙂
>
> Beyond that, it's not just adds and subtracts, there's unfortunately some additional bookkeeping we need to do around some of the other features we offer
we also know that settlements involving a lot of users are slow. from my notes, james shared some details about factors that contribute to ME latency a while back:
> there's just a lot of users whose positions and balances we need to update. sometimes there's additional computation around collateral return.
>
> it's not slow for any single user, (which is why you don't notice latency spikes when we regularly settle markets with a small number of traders), but you should do some back of the envelope calculations for how long you think it should take to do all of this cleanup for a single user and then multiply it by the number of users you think are trading on some of these markets 🙂
>
> Beyond that, it's not just adds and subtracts, there's unfortunately some additional bookkeeping we need to do around some of the other features we offer