Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MSET error with AWS Redis on ElasticCache implementation #1098

Closed
rumeau opened this issue May 24, 2021 · 21 comments
Closed

MSET error with AWS Redis on ElasticCache implementation #1098

rumeau opened this issue May 24, 2021 · 21 comments
Labels
bug
Milestone
2.8.0

Comments

@rumeau
Copy link

rumeau commented May 24, 2021 •
edited
Loading

How Shlink is set-up

  • Shlink Version: 2.7.0
  • PHP Version: 8.0.6
  • How do you serve Shlink: Self-hosted Docker image
  • Database engine used: MySQL 5.7.22

Summary

Recently Shlink started reporting intermitence on our self-hosted service. We are using AWS ECS to deploy using official shlink docker image, mysql on RDS and Redis on ElasticCache.

We are using an API endpoint in order to generate short links, but its not wokring as expected. According to logs the only visible error is:

Shlink.ERROR - Predis\NotSupportedException: Cannot use 'MSET' over clusters of connections. in /etc/shlink/vendor/predis/predis/src/Connection/Aggregate/PredisCluster.php:134

Seems to be a problem with predis implementation.

Stack trace is below. Any help is greatly appreciated

1621877034521,[2021-05-24T17:23:54.520840+00:00] [18ec5525-6c25-4d84-a080-329638785be8] Shlink.ERROR - Predis\NotSupportedException: Cannot use 'MSET' over clusters of connections. in /etc/shlink/vendor/predis/predis/src/Connection/Aggregate/PredisCluster.php:134
1621877034521,Stack trace:
1621877034521,#0 /etc/shlink/vendor/predis/predis/src/Connection/Aggregate/PredisCluster.php(215): Predis\Connection\Aggregate\PredisCluster->getConnection(Object(Predis\Command\StringSetMultiple))
1621877034521,#1 /etc/shlink/vendor/predis/predis/src/Client.php(331): Predis\Connection\Aggregate\PredisCluster->executeCommand(Object(Predis\Command\StringSetMultiple))
1621877034521,#2 /etc/shlink/vendor/predis/predis/src/Client.php(314): Predis\Client->executeCommand(Object(Predis\Command\StringSetMultiple))
1621877034521,"#3 /etc/shlink/vendor/doctrine/cache/lib/Doctrine/Common/Cache/PredisCache.php(78): Predis\Client->__call('mset', Array)"
1621877034521,"#4 /etc/shlink/vendor/doctrine/cache/lib/Doctrine/Common/Cache/CacheProvider.php(99): Doctrine\Common\Cache\PredisCache->doSaveMultiple(Array, 0)"
1621877034521,"#5 /etc/shlink/vendor/doctrine/cache/lib/Doctrine/Common/Cache/Psr6/CacheAdapter.php(305): Doctrine\Common\Cache\CacheProvider->saveMultiple(Array, 0)"
1621877034521,"#6 /etc/shlink/vendor/doctrine/cache/lib/Doctrine/Common/Cache/Psr6/CacheAdapter.php(215): Doctrine\Common\Cache\Psr6\CacheAdapter->doSaveMultiple(Array, 0)"
1621877034521,#7 /etc/shlink/vendor/doctrine/cache/lib/Doctrine/Common/Cache/Psr6/CacheAdapter.php(91): Doctrine\Common\Cache\Psr6\CacheAdapter->commit()
1621877034521,#8 /etc/shlink/vendor/doctrine/persistence/lib/Doctrine/Persistence/Mapping/AbstractClassMetadataFactory.php(258): Doctrine\Common\Cache\Psr6\CacheAdapter->getItems(Array)
1621877034521,#9 /etc/shlink/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(300): Doctrine\Persistence\Mapping\AbstractClassMetadataFactory->getMetadataFor('Shlinkio\Shlink...')
1621877034521,#10 /etc/shlink/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(1290): Doctrine\ORM\EntityManager->getClassMetadata('Shlinkio\Shlink...')
1621877034521,#11 /etc/shlink/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php(396): Doctrine\ORM\UnitOfWork->getCommitOrder()
1621877034521,#12 /etc/shlink/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(383): Doctrine\ORM\UnitOfWork->commit(NULL)
1621877034521,#13 /etc/shlink/vendor/doctrine/orm/lib/Doctrine/ORM/Decorator/EntityManagerDecorator.php(181): Doctrine\ORM\EntityManager->flush(NULL)
1621877034521,#14 /etc/shlink/module/Core/src/Visit/VisitsTracker.php(81): Doctrine\ORM\Decorator\EntityManagerDecorator->flush()
1621877034521,#15 /etc/shlink/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php(254): Shlinkio\Shlink\Core\Visit\VisitsTracker->Shlinkio\Shlink\Core\Visit{closure}(Object(Doctrine\ORM\EntityManager))
1621877034521,#16 /etc/shlink/vendor/doctrine/orm/lib/Doctrine/ORM/Decorator/EntityManagerDecorator.php(69): Doctrine\ORM\EntityManager->transactional(Object(Closure))
1621877034521,#17 /etc/shlink/module/Core/src/Visit/VisitsTracker.php(84): Doctrine\ORM\Decorator\EntityManagerDecorator->transactional(Object(Closure))
1621877034521,"#18 /etc/shlink/module/Core/src/Visit/VisitsTracker.php(34): Shlinkio\Shlink\Core\Visit\VisitsTracker->trackVisit(Object(Closure), Object(Shlinkio\Shlink\Core\Model\Visitor))"
1621877034521,"#19 /etc/shlink/module/Core/src/Action/AbstractTrackingAction.php(57): Shlinkio\Shlink\Core\Visit\VisitsTracker->track(Object(Shlinkio\Shlink\Core\Entity\ShortUrl), Object(Shlinkio\Shlink\Core\Model\Visitor))"
1621877034521,"#20 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(41): Shlinkio\Shlink\Core\Action\AbstractTrackingAction->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))"
1621877034521,"#21 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))"
1621877034521,#22 /etc/shlink/vendor/akrabat/ip-address-middleware/src/IpAddress.php(132): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
1621877034521,"#23 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(41): RKA\Middleware\IpAddress->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))"
1621877034521,"#24 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))"
1621877034521,#25 /etc/shlink/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(84): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
1621877034521,"#26 /etc/shlink/vendor/mezzio/mezzio-router/src/Route.php(85): Laminas\Stratigility\MiddlewarePipe->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))"
1621877034521,"#27 /etc/shlink/vendor/mezzio/mezzio-router/src/RouteResult.php(97): Mezzio\Router\Route->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))"
1621877034521,"#28 /etc/shlink/vendor/mezzio/mezzio-router/src/Middleware/DispatchMiddleware.php(30): Mezzio\Router\RouteResult->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))"
1621877034521,"#29 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(41): Mezzio\Router\Middleware\DispatchMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))"
1621877034521,"#30 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))"
1621877034521,#31 /etc/shlink/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(84): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
1621877034521,"#32 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Laminas\Stratigility\MiddlewarePipe->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))"
1621877034521,#33 /etc/shlink/vendor/laminas/laminas-stratigility/src/Middleware/PathMiddlewareDecorator.php(48): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
1621877034521,"#34 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Laminas\Stratigility\Middleware\PathMiddlewareDecorator->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))"
1621877034521,#35 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(54): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
1621877034521,#36 /etc/shlink/vendor/mezzio/mezzio-router/src/Middleware/ImplicitHeadMiddleware.php(76): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
1621877034521,"#37 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(41): Mezzio\Router\Middleware\ImplicitHeadMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))"
1621877034521,"#38 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))"
1621877034521,#39 /etc/shlink/vendor/mezzio/mezzio-router/src/Middleware/RouteMiddleware.php(50): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
1621877034521,"#40 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(41): Mezzio\Router\Middleware\RouteMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))"
1621877034521,"#41 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))"
1621877034521,#42 /etc/shlink/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(84): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
1621877034521,"#43 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Laminas\Stratigility\MiddlewarePipe->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))"
1621877034521,#44 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(54): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
1621877034521,#45 /etc/shlink/vendor/shlinkio/shlink-common/src/Middleware/CloseDbConnectionMiddleware.php(27): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
1621877034521,"#46 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(41): Shlinkio\Shlink\Common\Middleware\CloseDbConnectionMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))"
1621877034521,"#47 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))"
1621877034521,#48 /etc/shlink/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(84): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
1621877034521,"#49 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Laminas\Stratigility\MiddlewarePipe->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))"
1621877034521,#50 /etc/shlink/vendor/laminas/laminas-stratigility/src/Middleware/PathMiddlewareDecorator.php(48): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
1621877034521,"#51 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Laminas\Stratigility\Middleware\PathMiddlewareDecorator->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))"
1621877034521,#52 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(54): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
1621877034521,#53 /etc/shlink/vendor/laminas/laminas-stratigility/src/Middleware/ErrorHandler.php(144): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
1621877034521,"#54 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(41): Laminas\Stratigility\Middleware\ErrorHandler->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))"
1621877034521,"#55 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))"
1621877034521,#56 /etc/shlink/vendor/shlinkio/shlink-common/src/Middleware/ContentLengthMiddleware.php(28): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
1621877034521,"#57 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(41): Shlinkio\Shlink\Common\Middleware\ContentLengthMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))"
1621877034521,"#58 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))"
1621877034521,#59 /etc/shlink/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(84): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
1621877034521,"#60 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(61): Laminas\Stratigility\MiddlewarePipe->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))"
1621877034521,#61 /etc/shlink/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(84): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
1621877034521,"#62 /etc/shlink/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(73): Laminas\Stratigility\MiddlewarePipe->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\EmptyPipelineHandler))"
1621877034521,#63 /etc/shlink/vendor/mezzio/mezzio-swoole/src/Event/RequestHandlerRequestListener.php(121): Laminas\Stratigility\MiddlewarePipe->handle(Object(Laminas\Diactoros\ServerRequest))
1621877034521,#64 /etc/shlink/vendor/mezzio/mezzio-swoole/src/Event/EventDispatcher.php(41): Mezzio\Swoole\Event\RequestHandlerRequestListener->__invoke(Object(Mezzio\Swoole\Event\RequestEvent))
1621877034521,#65 /etc/shlink/vendor/mezzio/mezzio-swoole/src/SwooleRequestHandlerRunner.php(142): Mezzio\Swoole\Event\EventDispatcher->dispatch(Object(Mezzio\Swoole\Event\RequestEvent))
1621877034521,#66 {main}
@acelaya
Copy link
Member

acelaya commented May 24, 2021

Thanks! I'll check.

Just to double check. I guess you mean Elasticache, not Elasticsearch, right?

@acelaya acelaya added bug and removed question labels May 24, 2021
@rumeau
Copy link
Author

rumeau commented May 24, 2021

Thanks! I'll check.

Just to double check. I guess you mean Elasticache, not Elasticsearch, right?

Right. Sorry about that.

@rumeau rumeau changed the title MSET error with AWS Redis on ElasticSearch implementation MSET error with AWS Redis on ElasticCache implementation May 24, 2021
@acelaya
Copy link
Member

acelaya commented May 30, 2021

Hey @rumeau, how do you provide the config to Shlink in order to set-up redis?

@mgoeppl
Copy link

mgoeppl commented May 31, 2021

Hi!

I've got the same problem with a redis host on another machine in my local network.
The redis config looks like this:
redis://:PASSWORD@IPADDRESS:6379
It is passed to the docker container as an ENV variable.


Creating fresh database if needed...

Updating database...

Generating proxies...


In PredisCluster.php line 134:

                                                   

  Cannot use 'MSET' over clusters of connections.  

                                                   


orm:generate-proxies [--em EM] [--filter FILTER] [--] [<dest-path>]



Fatal error: Uncaught Predis\NotSupportedException: Cannot use 'MSET' over clusters of connections. in /etc/shlink/vendor/predis/predis/src/Connection/Aggregate/PredisCluster.php:134
�
Stack trace:

#0 /etc/shlink/vendor/predis/predis/src/Connection/Aggregate/PredisCluster.php(215): Predis\Connection\Aggregate\PredisCluster->getConnection(Object(Predis\Command\StringSetMultiple))

#1 /etc/shlink/vendor/predis/predis/src/Client.php(331): Predis\Connection\Aggregate\PredisCluster->executeCommand(Object(Predis\Command\StringSetMultiple))

#2 /etc/shlink/vendor/predis/predis/src/Client.php(314): Predis\Client->executeCommand(Object(Predis\Command\StringSetMultiple))

#3 /etc/shlink/vendor/doctrine/cache/lib/Doctrine/Common/Cache/PredisCache.php(78): Predis\Client->__call('mset', Array)

#4 /etc/shlink/vendor/doctrine/cache/lib/Doctrine/Common/Cache/CacheProvider.php(99): Doctrine\Common\Cache\PredisCache->doSaveMultiple(Array, 0)

#5 /etc/shlink/vendor/doctrine/cache/lib/Doctrine/Common/Cache/Psr6/CacheAdapter.php(305): Doctrine\Common\Cache\CacheProvider->saveMultiple(Array, 0)

#6 /etc/shlink/vendor/doctrine/cache/lib/Doctrine/Common/Cache/Psr6/CacheAdapter.php(215): Doctrine\Common\Cache\Psr6\CacheAdapter->doSaveMultiple(Array, 0)

#7 /etc/shlink/vendor/doctrine/cache/lib/Doctrine/Common/Cache/Psr6/CacheAdapter.php(223): Doctrine\Common\Cache\Psr6\CacheAdapter->commit()

#8 [internal function]: Doctrine\Common\Cache\Psr6\CacheAdapter->__destruct()
�

}

  thrown in /etc/shlink/vendor/predis/predis/src/Connection/Aggregate/PredisCluster.php on line 134

@rumeau
Copy link
Author

rumeau commented May 31, 2021

Hey @rumeau, how do you provide the config to Shlink in order to set-up redis?

As an environment variable on an Elastic Container Service Task

REDIS_SERVERS: tcp://xxxxxxxxx.ikva66.0001.euw3.cache.amazonaws.com:6379

Greetings

@ikaruswill
Copy link

I have the exact same issue but not on AWS Redis, but rather, redis:6.2.4-alpine. My instance of Shlink is hosted at home on a bare-metal, mixed-arch Kubernetes cluster of amd64 & arm64 machines.

The error shows up in the logs at different points of the startup sequence.
For example:

Creating fresh database if needed...
Updating database...
Generating proxies...

In PredisCluster.php line 134:
...
<Container restarts>

Then on the next restart:

[2021-06-08T00:53:48.963785+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 13
[2021-06-08T00:54:22.387930+00:00] [64c3a81b-0f6d-488e-88b7-f30786a1a49b] Shlink.ERROR - Predis\NotSupportedException: Cannot use 'MSET' over clusters of connections. in /etc/shlink/vendor/predis/predis/src/Connection/Aggregate/PredisCluster.php:134

I'm using shlink:2.7.1 and only observed this problem when upgrading from shlink:2.6.2.

On further observation. Shlink starts after all those errors but as I keep refreshing the shlinkweb interface, on some refreshes, we'll get this message.

Something went wrong while loading short URLs :(

My suspicion is it has something to do with concurrency issues and redis, while running multiple instances of shlink.

@acelaya acelaya added this to the 2.9.0 milestone Aug 4, 2021
@acelaya acelaya mentioned this issue Aug 4, 2021
Closed
@acelaya
Copy link
Member

acelaya commented Aug 5, 2021 •
edited
Loading

This issue is happening because the abstraction library used by Shlink for caching, does not really support Redis Clusters (even though they depend on a library that does support clusters, predis).

I'm moving to a different one that supposedly fully supports clusters. In the meantime I would suggest using a single redis instance or not using redis at all (Shlink only takes advantage of redis when working with a cluster of Shlink instances).

@acelaya acelaya mentioned this issue Aug 5, 2021
Merged
@acelaya
Copy link
Member

acelaya commented Aug 5, 2021

I have just merged a refactoring of the caching logic, using another library. Theoretically, it should solve this problem, but I'm still testing it.

If anyone here is open to test it, I can publish an alpha version (would be same as v2.8.0 + the caching changes).

I'll continue trying to verify it myself in the meantime.

@ikaruswill
Copy link

Thanks for spending time to look at this. On my side, even when a single redis instance is specified, it still encounters the MSET error, that's the strange part. I use Shlink in a high-availability setup in K8s at home so redis is crucial.

I'm up for testing it, just drop the tag I should pull.

@acelaya
Copy link
Member

acelaya commented Aug 5, 2021

Yep. I was seeing the comments and looks like the library thinks it's working against a cluster even when a single instance is provided.

Maybe I'm passing some incorrect value to the config. I'll double check.

In any case, you can test with shlinkio/shlink:2.9.0-alpha.1 as soon as this build finishes (around 1h) https://github.com/shlinkio/shlink/runs/3254543356?check_suite_focus=true

@ikaruswill
Copy link

Alrighty. I'll have to test this in 8h as it's just past midnight here. Will get back to you after I've tested it.

@acelaya
Copy link
Member

acelaya commented Aug 6, 2021 •
edited
Loading

What version of redis are you guys using? I'm asking because I cannot reproduce the issue with Shlink 2.8.0, which should still be affected by this problem.

I'm testing with a simple docker-compose file like this, with redis 6.2.4:

version: '3'

services:
    shlink:
        image: shlinkio/shlink:2.8.0
        # image: shlinkio/shlink:2.9.0-alpha.1
        ports:
            - "8080:8080"
        links:
            - shlink_redis
        environment:
            LC_ALL: C
            SHORT_DOMAIN_HOST: localhost:8080
            SHORT_DOMAIN_SCHEMA: http
            REDIS_SERVERS: tcp://shlink_redis:6379

    shlink_redis:
        image: redis:6.2.4-alpine
        ports:
            - "6380:6379"

And this is the output I get when running docker-compose up:

{11:14}~/Descargas/shlink-with-redis ➭ docker-compose up
Creating shlink-with-redis_shlink_redis_1 ... done
Creating shlink-with-redis_shlink_1       ... done
Attaching to shlink-with-redis_shlink_redis_1, shlink-with-redis_shlink_1
shlink_1        | Creating fresh database if needed...
shlink_redis_1  | 1:C 06 Aug 2021 09:16:17.616 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
shlink_redis_1  | 1:C 06 Aug 2021 09:16:17.616 # Redis version=6.0.5, bits=64, commit=00000000, modified=0, pid=1, just started
shlink_redis_1  | 1:C 06 Aug 2021 09:16:17.616 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
shlink_redis_1  | 1:M 06 Aug 2021 09:16:17.618 * Running mode=standalone, port=6379.
shlink_redis_1  | 1:M 06 Aug 2021 09:16:17.618 # Server initialized
shlink_redis_1  | 1:M 06 Aug 2021 09:16:17.618 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
shlink_redis_1  | 1:M 06 Aug 2021 09:16:17.618 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
shlink_redis_1  | 1:M 06 Aug 2021 09:16:17.618 * Ready to accept connections
shlink_1        | Updating database...
shlink_1        | Generating proxies...
shlink_1        | Clearing entities cache...
shlink_1        | [2021-08-06T09:16:18.852963+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 16
shlink_1        | [2021-08-06T09:16:18.853092+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 17
shlink_1        | [2021-08-06T09:16:18.853298+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 18
shlink_1        | [2021-08-06T09:16:18.853707+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 19
shlink_1        | [2021-08-06T09:16:18.854021+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 20
shlink_1        | [2021-08-06T09:16:18.854407+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 21
shlink_1        | [2021-08-06T09:16:18.854786+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 22
shlink_1        | [2021-08-06T09:16:18.855144+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 23
shlink_1        | [2021-08-06T09:16:18.855661+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 24
shlink_1        | [2021-08-06T09:16:18.856153+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 25
shlink_1        | [2021-08-06T09:16:18.856667+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 26
shlink_1        | [2021-08-06T09:16:18.857164+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 27
shlink_1        | [2021-08-06T09:16:18.857586+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 28
shlink_1        | [2021-08-06T09:16:18.858198+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 29
shlink_1        | [2021-08-06T09:16:18.858865+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 30
shlink_1        | [2021-08-06T09:16:18.859248+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 31
shlink_1        | [2021-08-06T09:16:18.860073+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 0
shlink_1        | [2021-08-06T09:16:18.860919+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 1
shlink_1        | [2021-08-06T09:16:18.861343+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 2
shlink_1        | [2021-08-06T09:16:18.862762+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 4
shlink_1        | [2021-08-06T09:16:18.862866+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 3
shlink_1        | [2021-08-06T09:16:18.864859+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 6
shlink_1        | [2021-08-06T09:16:18.865253+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 7
shlink_1        | [2021-08-06T09:16:18.868687+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 11
shlink_1        | [2021-08-06T09:16:18.870490+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 5
shlink_1        | [2021-08-06T09:16:18.870760+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 10
shlink_1        | [2021-08-06T09:16:18.873032+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 12
shlink_1        | [2021-08-06T09:16:18.873350+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 15
shlink_1        | [2021-08-06T09:16:18.873650+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 13
shlink_1        | [2021-08-06T09:16:18.874534+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 9
shlink_1        | [2021-08-06T09:16:18.875708+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 14
shlink_1        | [2021-08-06T09:16:18.876249+00:00] [NULL] Access.NOTICE - Worker started in /etc/shlink with ID 8

@ikaruswill
Copy link

ikaruswill commented Aug 6, 2021 via email

@acelaya
Copy link
Member

acelaya commented Aug 6, 2021 •
edited
Loading

I'm digging in the code of the library I use for redis integration (Predis), to see what are the conditions for them to throw that exception, just in case it's related with some metadata the library gets directly from the redis server.

@acelaya
Copy link
Member

acelaya commented Aug 6, 2021

@ikaruswill if you get the same error with the alpha version, it would be amazing if you could provide some reproducible config in the form of docker-compose, docker command, or kubernetes deployment.

I'm trying several combinations and never get that error, so I'm clearly doing something differently.

@ikaruswill
Copy link

Just tested the alpha image shlinkio/shlink:2.9.0-alpha.1, the problem seems to have gone away. 👍

I'm able to reproduce the issue in shlinkio/shlink:2.7.1, where even with just the first instance starting, the MSET error is triggered.

I also tested shlinkio/shlink:2.8.0, and it seems to already be working fine.

@ikaruswill
Copy link 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: shlink
  labels:
    app: shlink
    tier: backend
spec:
  replicas: 2
  selector:
    matchLabels:
      app: shlink
      tier: backend
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: shlink
        tier: backend
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: shlink
                tier: backend
            topologyKey: kubernetes.io/hostname
      containers:
      - image: shlinkio/shlink:2.7.1
        imagePullPolicy: IfNotPresent
        name: shlink
        env:
        - name: SHORT_DOMAIN_HOST
          value: placeholder.com
        - name: SHORT_DOMAIN_SCHEMA
          value: https
        - name: GEOLITE_LICENSE_KEY
          valueFrom:
            secretKeyRef:
              name: shlink
              key: geolite_license_key
        - name: DB_HOST
          value: db-postgresql
        - name: DB_PORT
          value: '5432'
        - name: DB_DRIVER
          value: postgres
        - name: DB_USER
          valueFrom:
            secretKeyRef:
              name: shlink
              key: db_user
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: shlink
              key: db_password
        - name: REDIS_SERVERS
          value: tcp://shlink-redis:6379
        - name: AUTO_RESOLVE_TITLES
          value: 'true'
        startupProbe:
          tcpSocket:
            port: http
          failureThreshold: 6
          periodSeconds: 5
        livenessProbe:
          tcpSocket:
            port: http
        readinessProbe:
          tcpSocket:
            port: http
        ports:
        - name: http
          containerPort: 8080
        resources:
          requests:
            memory: 128Mi
            cpu: 100m
          limits:
            memory: 128Mi
            cpu: 200m
      restartPolicy: Always

@acelaya Here's my deployment configuration with the MSET issue in k8s for 2.7.1, with sensitive information redacted + secret references (Hope I didn't miss out any secrets). Maybe it'll help.

@acelaya
Copy link
Member

acelaya commented Aug 6, 2021

Yes! That definitely helps. I'll try to set-up some minikube and apply this deployment.

I also see you are using postgres, which is kind-of related, as all the redis errors reported here are thrown by the database abstraction layer, which uses redis to cache some stuff.

I'll also try to include a postgres in my PoC and see how it goes.

@acelaya
Copy link
Member

acelaya commented Aug 6, 2021

Just tested the alpha image shlinkio/shlink:2.9.0-alpha.1, the problem seems to have gone away.

I'm able to reproduce the issue in shlinkio/shlink:2.7.1, where even with just the first instance starting, the MSET error is triggered.

I also tested shlinkio/shlink:2.8.0, and it seems to already be working fine.

I missed this!

Thanks for all the investigation @ikaruswill. All your colaboration has been super helpful 🙂

I'll try with v2.7.1, and if I can reproduce it with it, I'll close this issue and assume it was a bug on some of the libraries implied here.

@acelaya
Copy link
Member

acelaya commented Aug 6, 2021

Confirmed. The docker-compose PoC above can be used to reproduce the issue with 2.7.1, but not with 2.8.0, so I'll consider this fixed and part of 2.8.0.

Thanks!

@acelaya acelaya closed this as completed Aug 6, 2021
@acelaya acelaya modified the milestones: 2.9.0, 2.8.0 Aug 6, 2021
@ikaruswill
Copy link

Thanks for your work on this @acelaya , I'm glad to have been able to contribute to this awesome project 😎. Cheers!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
2.8.0
Development

No branches or pull requests
4 participants
@rumeau @acelaya @ikaruswill @mgoeppl