* fix(profiling): Ingest profile file path
`ingest-profiles` is now using vroomrs to ingest profiles instead of writing
through vroom. For self-hosted, we need to make sure filestore for profiles is
properly configured so vroom can find the ingested profiles.
* feat: move profiling data to seaweedfs
* feat: review from Sentry
* Apply suggestions from code review
Co-authored-by: Burak Yigit Kaya <byk@sentry.io>
* ref: volume migration tests
* ref: execute file creation from vroom container
* fix: brainfart
* debug
* hack
* more debug
* now I know what I'm missing out
* explicitly state feature complete
* try to pull vroom image
* should only run when COMPOSE_PROFILES is feature complete
* using run invoked weed instead of empty shell
* execute the upload script from vroom container
* execute apt command as root
* gonna sleep
* missing endgroup
* missing sh
* directly execute s3cmd and do 'wc' outside out the container
* why did other test start failing
* manual cleanup
* vroom is not a persistent volume
* what about not removing the seaweed volume
---------
Co-authored-by: Reinaldy Rafli <github@reinaldyrafli.com>
Co-authored-by: Burak Yigit Kaya <byk@sentry.io>
* Remove symbolicator volume creation command
* Remove symbolicator volume
* Create sentry-symbolicator on docker compose up
* Pass volume name to remove_command
* Remove sentry-symbolicator from unit test
* feat: Use S3 node store with garage
* lol, fix bash
* moar bash
* lol
* hate bash
* fix moar bash
* Add healthcheck to garage service
Co-authored-by: Reinaldy Rafli <aldy505@proton.me>
* revert +x
* fix healthcheck, fix config
* add env var for garage size
* use better compression level
* simpler garage config
* add migration support
* feat: use seaweedfs as nodestore backend (#3842)
* feat: seaweedfs as s3 nodestore backend
* fix: 'server' was missing for seaweed
* feat: remove minimum volume free space
* feat: specify hostname on ip
* fix: grpc port on seaweed should be `-{service}.port.grpc` instead of `-{service}.grpcPort`
* fix: wrong access key & secret key; use localhost for internal comms
* fix: create index directory
* test: add sentry-seaweedfs volume into expected volumes
* debug: aaaaaaaaaaaaaaaaaaaaaaarrrrggggggghhhhhhhhhhhhhhh
* test: correct ordering for expected volumes
* chore: seaweedfs healthcheck to multiple urls
See https://stackoverflow.com/a/14578575/3153224
* chore: add swap for arm64 runners
* ci: debug memory issues for arm64 runners
* ci: turn off swapfile first
Turns out the arm64 runners already have 3GB of swap
* feat: nodestore config update behind a prompt/flag
* feat: set s3 lifecycle policy
* fix: seaweed is a busybox
* fix: try xml policy
* fix: go back to simplified json
* Revert "fix: go back to simplified json"
This reverts commit 2f1575dfe33db6f781b09d09b01f5382716b8826.
* chore: reword debug lifecycle policy
* fix: don't pollute APPLY_AUTOMATIC_CONFIG_UPDATES variable
---------
Co-authored-by: Reinaldy Rafli <github@reinaldyrafli.com>
In this version, there's a new `--pull` argument for `docker compose run` which we will start leveraging, especially with `sentry-admin` command.
Should come with a slight speed boost.
Since we download JS SDKs in a for loop which invokes a separate docker container for each `curl` run, we seem to be triggering some sort of a DoS protection. And rightfully so as the old method causes TCP and TLS churn although we advertise we support HTTP/1.1 and HTTP/2.
This patch does a few things:
1. Uses `curl`s globbing support to download all files in one go, maxing TCP and TLS reuse. This should fix the DoS protection
2. Uses `curl`'s `--compress` option to make things even more efficient
3. Uses `curl`'s `--create-dirs` to save 1 docker container run per version for creating the directory
4. Removes the `-I` `HEAD` checks in favor of a `-f` fail option combined with `|| true` which makes curl fail and not write the output on a non-200 response while still allowing the script to succeed
5. To make sure the above approach works, it adds a file size test, requiring all downloaded files to be larger than 1kB
Hopefully fixes https://github.com/getsentry/sentry/issues/22715#issuecomment-2458066842
### Legal Boilerplate
Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.
* Reference paths relative to the current script or project root
Before this PR:
- some scripts change the current working directory and use relative paths
- different approaches are taken to know which directory a script is running in
- paths are sometimes relative, sometimes absolute, sometimes traversing directories
After this PR:
- scripts do neither change nor care much about the current working directory
- a unified approach determines the directory of the current script
- paths are always relative to the project root
This should resolve an issue I already tried to fix with https://github.com/getsentry/self-hosted/pull/1798,
where the contents of `./sentry` were not copied
into the built container image,
thus `enhance-image.sh` did not apply.
Co-authored-by: Amin Vakil <info@aminvakil.com>
In this PR we fix our usage of jq:
1. Use the container. Before we were accidentally using jq installed in the environment
1. Rewrite generate_breadcrumb_json to read log file all at once
1. Build docker image for jq in unit tests
* fix failing unit tests
* fix unit test pipeline
* add newline to unit-test file
* preserve semantics from docker volume test
* add error-handling tests
* reorganize file structure of unit tests
* add comment for using _file format in ensure relay credentials test
* add error handling unit tests
* use unit-test directory
* stack trace should be from the unit test directory
* Slightly better fail signal
* Clearer start as well
Co-authored-by: Chad Whitacre <chadwhitacre@sentry.io>
