* fix(profiling): Ingest profile file path
`ingest-profiles` is now using vroomrs to ingest profiles instead of writing
through vroom. For self-hosted, we need to make sure filestore for profiles is
properly configured so vroom can find the ingested profiles.
* feat: move profiling data to seaweedfs
* feat: review from Sentry
* Apply suggestions from code review
Co-authored-by: Burak Yigit Kaya <byk@sentry.io>
* ref: volume migration tests
* ref: execute file creation from vroom container
* fix: brainfart
* debug
* hack
* more debug
* now I know what I'm missing out
* explicitly state feature complete
* try to pull vroom image
* should only run when COMPOSE_PROFILES is feature complete
* using run invoked weed instead of empty shell
* execute the upload script from vroom container
* execute apt command as root
* gonna sleep
* missing endgroup
* missing sh
* directly execute s3cmd and do 'wc' outside out the container
* why did other test start failing
* manual cleanup
* vroom is not a persistent volume
* what about not removing the seaweed volume
---------
Co-authored-by: Reinaldy Rafli <github@reinaldyrafli.com>
Co-authored-by: Burak Yigit Kaya <byk@sentry.io>
* feat: Use S3 node store with garage
* lol, fix bash
* moar bash
* lol
* hate bash
* fix moar bash
* Add healthcheck to garage service
Co-authored-by: Reinaldy Rafli <aldy505@proton.me>
* revert +x
* fix healthcheck, fix config
* add env var for garage size
* use better compression level
* simpler garage config
* add migration support
* feat: use seaweedfs as nodestore backend (#3842)
* feat: seaweedfs as s3 nodestore backend
* fix: 'server' was missing for seaweed
* feat: remove minimum volume free space
* feat: specify hostname on ip
* fix: grpc port on seaweed should be `-{service}.port.grpc` instead of `-{service}.grpcPort`
* fix: wrong access key & secret key; use localhost for internal comms
* fix: create index directory
* test: add sentry-seaweedfs volume into expected volumes
* debug: aaaaaaaaaaaaaaaaaaaaaaarrrrggggggghhhhhhhhhhhhhhh
* test: correct ordering for expected volumes
* chore: seaweedfs healthcheck to multiple urls
See https://stackoverflow.com/a/14578575/3153224
* chore: add swap for arm64 runners
* ci: debug memory issues for arm64 runners
* ci: turn off swapfile first
Turns out the arm64 runners already have 3GB of swap
* feat: nodestore config update behind a prompt/flag
* feat: set s3 lifecycle policy
* fix: seaweed is a busybox
* fix: try xml policy
* fix: go back to simplified json
* Revert "fix: go back to simplified json"
This reverts commit 2f1575dfe33db6f781b09d09b01f5382716b8826.
* chore: reword debug lifecycle policy
* fix: don't pollute APPLY_AUTOMATIC_CONFIG_UPDATES variable
---------
Co-authored-by: Reinaldy Rafli <github@reinaldyrafli.com>
* Revert "fix(vroom): Explicitly set PROFILES_DIR for upcoming change (#3759)"
This reverts commit e07445d6be41793165316a3e077ebec343740530.
It also very importantly changes where we mount the profiles volume which fixes the issue. Our theory is as follows:
1. Vroom Dockerfile had a line doing `mkdirp /var/lib/sentry-profiles` at image build time. This makes the directory owned by `root`
2. When we mount over that directory, and change permissions we can store the permissions changes _in_ the directory but not the directory itself
3. So when we start the vroom image with the new mount, the contents are owned by `vroom` but the main directory is still owned by `root`. This is also why [this approach](a23a4e3952) worked as the entrypoint script did this at the start of every container instance.
---------
Co-authored-by: Burak Yigit Kaya <byk@sentry.io>
Sentry Admin Script always fail because of missing import of lib script.
### Legal Boilerplate
Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.
* Reference paths relative to the current script or project root
Before this PR:
- some scripts change the current working directory and use relative paths
- different approaches are taken to know which directory a script is running in
- paths are sometimes relative, sometimes absolute, sometimes traversing directories
After this PR:
- scripts do neither change nor care much about the current working directory
- a unified approach determines the directory of the current script
- paths are always relative to the project root
This should resolve an issue I already tried to fix with https://github.com/getsentry/self-hosted/pull/1798,
where the contents of `./sentry` were not copied
into the built container image,
thus `enhance-image.sh` did not apply.
Co-authored-by: Amin Vakil <info@aminvakil.com>
This is the MVP of error monitoring of the self-hosted installer. We have:
- Added a prompt for users to opt into error reporting
- Rewritten the traceback code to give a full stack trace
- Hooked up sending errors to our self-hosted instance
Fixes#740
Co-authored-by: Chad Whitacre <chadwhitacre@sentry.io>
Co-authored-by: hubertdeng123 <hubertdeng123@gmail.com>
We will use Change Data Capture to stream WAL updates from postgres into clickhouse so that features like issue search will be able to join event data and metadata (from postgres) through Snuba.
This requires the followings:
A logical replicaiton plugin to be installed in postgres (https://github.com/getsentry/wal2json)
A service to run that streams from the replication log to Kafka (https://github.com/getsentry/cdc)
Datasets in Snuba.
This PR is preparing postgres to stream updates via the replication log.
The idea is to
download the the replication log plugin binary during install.sh
mount a volume with the binary when starting postgres
providing a new entrypoint to postgres that ensures everything is correctly configured.
There is a difference between how this is set up and how we do the same in the development environment.
In the development environment we download the library from the entrypoint itself and store it in a persistent volume, so we do not have to download it every time.
Unfortunately this does not work here as the postgres image is postgres:9.6 while it is postgres:9.6-alpine. This one does not come with either wget or curl. I don't think installing that in the entrypoint would be a good idea, so the download happens in install.sh. I actually think this way is safer so we never depend on connectivity for postgres to start properly.
We used to build local images for Sentry services to be able to
include required plugins in the image. With this change we instead
do this in a custom entrypoint script and use the volume `/data`
to store the plugins permanently.
This should resolve many issues people have around building local
images and pushing them to places like private repositories or swarm
clusters.
This is not 100% compatible with the old way but it should still be
a mostly transparent change to many folks.
* ref(requirements): Add min CPU requirement, relax soft RAM
Adds minimum of 4 CPU cores requirement as anything below will perform quite poorly even on lower loads. Relaxes the soft RAM requirement from 8000 MB to 7800 MB as even when there is 8 GB RAM installed, the system reserves some of it to itself and under reports the amount.
* pass on CI with soft limit
Fixes#823. In `install.sh` we build a local sentry image that is used by many services from using the build context under the `./sentry` directory. To avoid building this image multiple times, we also give it a specific name which is referred from multiple services. The issue is, we also run `docker-compose build --parallel` which creates a race condition when building this image as `docker-compose` doesn't check whether the image is already there or not. This is the root cause of all these random failures: an unsurprising race condition.
This is in preparation to make the PY3 version the default for Docker images and self-hosted. It is part **4/5**:
1. ~~Add `-py2` variants for the Python 2 build tags and introduce the `SENTRY_PYTHON2` env variable usage~~ (getsentry/sentry#22460)
2. ~~Switch getsentry/onpremise to Python 3 by default*, introducing the `SENTRY_PYTHON2` env var for Py2 builds via the `-py2` suffix~~ (getsentry/onpremise#763)
3. ~~Move the unsuffixed version of the builds to Python 3~~ (getsentry/sentry#22466)
4. **Remove the `SENTRY_PYTHON3` env var support and `-py3` prefix usage from getsentry/onpremise**
5. Remove tagging of `-py3` builds from getsentry/sentry
This is in preparation to make the PY3 version the default* for Docker images and self-hosted. It is part **2/5**:
1. ~~Add `-py2` variants for the Python 2 build tags and introduce the `SENTRY_PYTHON2` env variable usage~~ (getsentry/sentry#22460)
2. __Switch getsentry/onpremise to Python 3 by default*, introducing the `SENTRY_PYTHON2` env var for Py2 builds via the `-py2` suffix__
3. Move the unsuffixed version of the builds to Python 3
4. Remove the `SENTRY_PYTHON3` env var support and `-py3` prefix usage from getsentry/onpremise
5. Remove tagging of `-py3` builds from here
_* this will only happen when item 3 above gets landed_
We kept getting issue reports that we traced down to `git-bash` which doesn't seem to play nice with Docker for Windows with bind mounts. This PR uses the existence of `$MSYSTEM` to detect `git-bash` and exit early with a relevant message.
Co-authored-by: Chad Whitacre <chadwhitacre@sentry.io>
This is to ensure clean shutdown of Celery, with fully drained queues. This is needed as versions may change the event format and not be backwards compatible. FWIW this is a hacky workaround without a strong guarantee that the queues will be empty. Ideally we'd shutdown everything first, spin up the workers and check for queues being drained every second or so.
This fixes a serious bug in install.sh where it ignored externaly set env variable values such as `SENTRY_IMAGE` in favor of the ones defined in `.env`, essentially making all our e2e tests usless.
