Mount a certificate folder to local ca storage in containers,
and add update command to cron image's entrypoint.
Result of poking and prodding from getsentry/sentry#26851
Fixes#1009 by partially reverting #1002. We need to make a 21.6.2 release soon and I didn't have time to dig into why Kafka upgrades were failing so reverting for safety for now.
This PR is a try to update most middlewares used by Sentry to latest stable versions.
[As mentioned in the forum](https://forum.sentry.io/t/middleware-version-compatibility/14353/2) I didn't update Postgresql & Clickhouse due to known issues.
I also :
- changed versions to immutable tags (MAJOR.MINOR.PATCH semver versions when possible).
- changed nginx to the Alpine variant
We will use Change Data Capture to stream WAL updates from postgres into clickhouse so that features like issue search will be able to join event data and metadata (from postgres) through Snuba.
This requires the followings:
A logical replicaiton plugin to be installed in postgres (https://github.com/getsentry/wal2json)
A service to run that streams from the replication log to Kafka (https://github.com/getsentry/cdc)
Datasets in Snuba.
This PR is preparing postgres to stream updates via the replication log.
The idea is to
download the the replication log plugin binary during install.sh
mount a volume with the binary when starting postgres
providing a new entrypoint to postgres that ensures everything is correctly configured.
There is a difference between how this is set up and how we do the same in the development environment.
In the development environment we download the library from the entrypoint itself and store it in a persistent volume, so we do not have to download it every time.
Unfortunately this does not work here as the postgres image is postgres:9.6 while it is postgres:9.6-alpine. This one does not come with either wget or curl. I don't think installing that in the entrypoint would be a good idea, so the download happens in install.sh. I actually think this way is safer so we never depend on connectivity for postgres to start properly.
Add basic healthchecks on Zookeeper & Kafka containers to have a view on container status. These checks are quite basic because I have no knowledge at all on these components.
Co-authored-by: Sébastien Pierre <spi@dfakto.com>
This is a stop-gap solution to #918 until we figure out the negative DNS caching issue inside `relay`. This may also be due to Docker Compose making some assumptions/optimizations/limiting regarding cross-container access unless they are explicitly linked via the `depends_on` key.
This change ensures that the Snuba consumer and replacer start
to fill in the new table. It should be applied once we have
backfilled data and are ready to cut over to the new storage.
Depends on https://github.com/getsentry/snuba/pull/1801
These were looked over when they were added. This is not a big deal as running `docker-compose up -d` spins up all services but this fix is for correctness sake, especially for folks using this repo as a basis for more complex setups.
We used to build local images for Sentry services to be able to
include required plugins in the image. With this change we instead
do this in a custom entrypoint script and use the volume `/data`
to store the plugins permanently.
This should resolve many issues people have around building local
images and pushing them to places like private repositories or swarm
clusters.
This is not 100% compatible with the old way but it should still be
a mostly transparent change to many folks.
This is in preparation to make the PY3 version the default for Docker images and self-hosted. It is part **4/5**:
1. ~~Add `-py2` variants for the Python 2 build tags and introduce the `SENTRY_PYTHON2` env variable usage~~ (getsentry/sentry#22460)
2. ~~Switch getsentry/onpremise to Python 3 by default*, introducing the `SENTRY_PYTHON2` env var for Py2 builds via the `-py2` suffix~~ (getsentry/onpremise#763)
3. ~~Move the unsuffixed version of the builds to Python 3~~ (getsentry/sentry#22466)
4. **Remove the `SENTRY_PYTHON3` env var support and `-py3` prefix usage from getsentry/onpremise**
5. Remove tagging of `-py3` builds from getsentry/sentry
This is in preparation to make the PY3 version the default* for Docker images and self-hosted. It is part **2/5**:
1. ~~Add `-py2` variants for the Python 2 build tags and introduce the `SENTRY_PYTHON2` env variable usage~~ (getsentry/sentry#22460)
2. __Switch getsentry/onpremise to Python 3 by default*, introducing the `SENTRY_PYTHON2` env var for Py2 builds via the `-py2` suffix__
3. Move the unsuffixed version of the builds to Python 3
4. Remove the `SENTRY_PYTHON3` env var support and `-py3` prefix usage from getsentry/onpremise
5. Remove tagging of `-py3` builds from here
_* this will only happen when item 3 above gets landed_
@xbenjii reported the following error on #629:
>You requested maxclients of 10000 requiring at least 10032 max file descriptors.
Increasing this limit by default makes sense to make Redis more available to heaveier loads.
Fixes#502 and applies the suggestions from there:
- Number of partitons=1 (from 40)
- Log retention to 1 day (from 7 days)
These settings should be more suited towards the scale this repo is intended for.
NOTE: The partition count change will only affect new installs unless `sentry-kafka` and related volumes are cleaned.
Closes#616, supersedes #651
Adds an option to reduce max memory usage of Clickhouse server. Sets it to 30% of all available RAM as the default.
Co-authored-by: Renaud Chaput <renchap@gmail.com>
This change allows one to override any Sentry service image, mostly for testing purposes. It also removes the SENTRY_VERSION variable as docker-compose makes it very hard to cascade default values for these. Next step is to have integration tests in getsentry/snuba and getsentry/relay (and possibly for getsentry/symbolicator) for getsentry/onpremise using this PR.
Also related: #596.
Without the consumer-group option, transactions consumer and events consumer will compete for messages on the same topic and usually events win, which is the first one. This may cause some data loss for performance and make it seem not work.
Should address https://forum.sentry.io/t/perfomance-tracing-for-sentry-itself/10405/5?u=byk
I would like to be able to customize the configuration for my Sentry 10 symbolicator instance, which this change allows me to easily do.
See related: https://github.com/getsentry/symbolicator/issues/245
Co-authored-by: Burak Yigit Kaya <ben@byk.im>
We regularly prune old Google Cloud Build images and also GCB registry is not accessible to everyone all the time (firewall settings, being in China, etc.)
Fixes#445.
* feat: Add snuba outcomes consumers to setup
* fix: Rename all references of snuba-consumer
* ref: Rename back to snuba-consumer
* fix: Change auto-offset-reset
* fix: Attempt to fix CI
This follows the best-practice of mounting the config folder as a volume and removes the need to rebuild sentry images for config changes. Partially addresses #314.
