fig (aka:[phil])

cute!!

i really gotta fix that date counter!!!

it goes back to end of jan 2025, bit over 13mo

(full backfill cominggggggg but still a lot of steps to get there)

stickers? no i’ll be bringing googly eyes to atmosphereconf

(got the pi before the last ram-driven price hike, but only just got a 2TB$$$ NVMe to go with it. constellation index is over 500GB!

i now have hardware backups on hand for most critical components, which also means benchmarking hw. no backup modem but i’m looking at a second ISP hookup 👀)

a raspberry pi 5 with active cooling and two different sized googly eyes on a wooden table next to the packaging from an “nvme BASE” from pimoroni. you can see screws and the pcie cable from the base installed underneath.

newest member

(at current dram and ssd prices this hardware approach has become actually very expensive 😭)

[cropped] created their first [at://app.bsky.feed.post]! [cropped] removed their last [at://app.bsky.graph.list, at://app.bsky.graph.listitem] [cropped] created their first [at://app.bsky.feed.repost]!

buuuut it's doing the thing!?!!

*statelessly* detecting peoples first and last records per-collection just from firehose commits muah ha ha

every time i check one thing off for finishing lightrail, i also add three more todos.

(realistically this is going fast but ughhh)

minimalist ascii art of a light rail vehicle viewed head-on. original text a few messages up thread. it says "LIGHT RAIL" in the front windshield, and feels like it's really bearing down on you.

oh wow these are both so nice!!

for posterity, Jay's rendered like this for me in monospace

roland thank you for putting the raw ascii in alt text, i'm going to use that. (might need to crop the width a bit). can i credit you on that page?

yes!

i sometimes feel like being in one of the "fine" groups makes people over-confident in how much the whole thing can survive The Event. that growing subsets can live past it is real and cool (and often totally discounted by critics), but yeah.

yeah absolutely

i always feel like there are three distinct differently impacted groups in this conversation

1. normie bluesky users: they will get mostly screwed
2. atproto and community microblogers: we'll be fine
3. other apps: honestly, fine

shrinking the "mostly" for group 1 is the long and hard work

A picture of the LA Metro light rail leaving Los Angeles. Note the 3 trains all coupled together

A picture of the LA Metro light rail on a bridge

A picture of the LA Metro light rail crossing a street overhead

so good news and bad news.

bad news: that isn't light rail
good news: it's CLOSE to being light rail

light rail is made up of these little 2-unit trains, all chained together. they look like your locomotive in the image, sliced in half. they don't carry other cars, just themselves

ohhh or did you mean listRepos

oh yeah and getRepo as a redirect like relays do? should be easy enough! (i expect most times it’ll be beside/behind a relay that would handle that endpoint anyway)

right yep. well enumerating all accounts is just key scan on the accounts prefix so no prob :)

still thinking through inactive lifecycle: on reactivation of a matching-but-excluded account, is the expectation that apps will catch it on a rescan, like tap? (esp for slow collections like profiles)

i haven’t decided yet whether to do that just by parallel iterating+merging dids from every matching collection, or by additional indexes per segment prefix.

(multi-collection version will parallel merge anyway)

inactive like deactivated status?

i think both doable. deactivated state is in there already so it can filter on the way out with a point lookup per id. or drop the account from the index and resync on reactivate.

wildcard i really want to do! or at least nsid prefix by segment

very rough still but getting there. we've got CAR-slice helpers for walking a whole repo, a prefix, an explicit range, or a single key, *proving the slice completeness* in the processsss

docs.rs/repo-stream/...

i think it levels off at 2.1x because the initial CAR loading and validation (all of which can't be skipped) ends up dominating.

but anything that speeds up big working with big CARs is a compounding win for backfill, because releasing resources faster means more available concurrency.

repo collection set extraction from CAR (table) results summary: empty and tiny CARs don't benefit using seek but aren't slower, but then there's a big win at the `little` (279K) CAR bench (1.9x speedup) and an even bigger 3.1x speedup for midsize (5.0M). then large and huge CARs (149 and 449M respectively) level off slightly lower again at 2.1x speedup. (i guess the initial load and validation times start to dominate) full table in ascii: bench | size | naive | seeking | speedup --------+------+---------+---------+--------- empty | 329 | 630ns | 629ns | 1.0x tiny | 3.4K | 5.18us | 5.04us | 1.0x little | 279K | 173us | 91.5us | 1.9x midsize| 5.0M | 6.94ms | 2.23ms | 3.1x large | 149M | 437ms | 208ms | 2.1x huge | 449M | 1.33s | 626ms | 2.1x

richer mst-walking apis are coming in the next repo-stream release, like `.seek(key)` for skipping parts of the tree you don't need!

<clicks in to profile>: yep, checks out

(thanks!)

ascii art train thing that says "Lightrail": an engine pulling one train car. (is this light rail train?) "this is an atproto backfill-by-collection assister. available endpoints: - /xrpc/com.atproto.sync.listReposByCollection - /xrpc/com.atproto.sync.getRepoStatus source: https://tangled.org/microcosm.blue/lightrail

progress

(if you reply tell me this depicted rail is not light, you are obligated to include alternate more-correct ascii art in your response thank you)

oh yes finally!

XRPC requests between atproto servers are authenticated using JWTs. There are some inconsistencies in how OAuth permissions, PDS proxy headers, and JWTs all represent the "audience" of these tokens.

This proposal gives background and describes a rough solution.

Looking for rapid feedback!

finally completed the dashboard redesign to not necessitate anymore scrolling, let me know if things are broken or look weird. I love compact design

hell yeah you're up there, not quite beating the top-content-from-humans repos i think

i'm considering giving in and using its `collections` array for lightrail backfill (like collectiondir does), but that is unfortunately sketchy

com.atproto.repo.describeRepo doesn't have the repo's `rev` :sob:

diagram: two large clouds with a thin "compressed firehose" arrow joining them the top cloud (blue, "the cloud") contains many PDSs with arrows toward a "relay" box labelled "subscribeRepos", which has a thick arrow out to "entangler". the "compressed firehose" arrow joining the clouds starts from "the cloud"'s entangler box, and lands in the second large "microcosm HQ" cloud on another box labelled "entangler" microcosm HQ's entangler then has a thick arrow to a large "rainbow" box, with thick arrows flowing out to "constellation", "UFOs", and "spacedust" boxes

i think there's a mostly-unexplored design space here between massive client fan-out and point-to-point links.

eg. i want to run my own relay, but not from home, and ideally not with the full weight of multiple full subscribeRepos coming into my home.