Added passthrough pulp mode #372

sbernhard · 2019-06-05T11:50:25Z

This change implements a passthrough mode on a pulpnode (=capsule) to pulp master using the already existing squid proxy.

The advantage is, that many customers don't want to duplicate the pulp content but still have separated subnets. Therefore, we configured a squid proxy on the capsule to route all requests for pulp content (rpm, apt) to pulp master (=katello)

For using this at katello:

install the katello-ca-consumer-latest.noarch.rpm of the pulp master on the content host (instead of the capsule ca consumer)
register via the subscription manager
set the proxy in yum.conf OR in apt.conf to use the proxy like
proxy=http://capsule.dev:3129

ehelms · 2019-06-05T13:21:22Z

This is an interesting idea. I have a few questions to help me think on it.

This couples the squid deployed to help with lazy content syncing to a squid used for passthrough proxying. What if I want one but not the other?
In puppet-foreman_proxy_content, a reverse proxy is deployed through Apache that in theory could route back to the main server. What is the advantage of squid in this case? Given it requires using a separate port as well?
How does one deploy this configuration to a proxy without deploying all of Pulp given it seems tightly coupled to puppet-pulp?

sbernhard · 2019-06-05T21:07:43Z

Thanks Eric for you question and your interest.
To 1: I added the passthrough mode as a addition - means, the pulp streamer scenario for lazy content syncing should still work.
To 2: I guess, because of SSL certificates to have a secure connection AND to authenticate on pulp (using the client cert) a reverse proxy can not be used. Additionally, I prefered to have the squid solution as its possible to verify that only requests of a given network to pulp master is allowed. Don't know if this is possible with apache.
3. it is coupled with pulp. I think, it might be possible to use passthrough pulp without having the rest of the pulp scenario but this is MUCH more effort and I guess, the katello-agent can not be used as it still requires qpid and (I think) some pulp services (am I right?)

ehelms · 2019-06-06T00:36:53Z

I think I am not grasping the concept here. Let me recap my thoughts and you can steer me back on course. I interpreted this as a lightweight way to present content on a proxy so that hosts isolated could get content from the main server without having to have a full blown Pulp configured and syncing content.

sbernhard · 2019-06-06T05:33:29Z

This is correct. During installation, all pulp node services are still configured. If you want, you can still sync the content to the pulp node (capsule) - maybe only parts. In the end, the client decides if the passthrough mode is used (= proxy configuration on the hosts is configured) or the pulp node. Of course, it doesn't make sense to sync all the content to the pulp node and then use the passthrough mode. The intention is to use the passthrough mode and don't sync the content so that the content (rpms, debs) doesn't need to be duplicated.

…

---- Eric Helms schrieb ---- I think I am not grasping the concept here. Let me recap my thoughts and you can steer me back on course. I interpreted this as a lightweight way to present content on a proxy so that hosts isolated could get content from the main server without having to have a full blown Pulp configured and syncing content. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#372?email_source=notifications&email_token=AGCOEFOHMRT3QE4BN5CDQMTPZBL2PA5CNFSM4HTWEVCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXBM2KA#issuecomment-499305768>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AGCOEFLEV7EHV2O37K43QXTPZBL2PANCNFSM4HTWEVCA>.

sbernhard · 2019-07-16T11:15:23Z

ping @ehelms

sbernhard · 2019-09-24T15:49:30Z

ping @ehelms / @ekohl

ekohl

What's the general design behind this? Looking at the configuration, this would still deploy a full Pulp instance, correct? Is that really needed?

ekohl · 2019-10-21T11:26:39Z

manifests/init.pp

@@ -517,6 +533,11 @@
  Optional[String] $ldap_bind_dn = $pulp::params::ldap_bind_dn,
  Optional[String] $ldap_bind_password = $pulp::params::ldap_bind_password,
  String $ldap_remote_user_attribute = $pulp::params::ldap_remote_user_attribute,
+  Boolean $enable_passthrough_pulp = $pulp::params::enable_passthrough_pulp,
+  Stdlib::Port $passthrough_pulp_http_port = $pulp::params::passthrough_pulp_http_port,
+  Optional[String] $passthrough_pulp_allowed_net = $pulp::params::passthrough_pulp_allowed_net,


You can enforce the data type with https://github.com/puppetlabs/puppetlabs-stdlib/blob/master/REFERENCE.md#stdlibipaddressv4cidr. You may want to consider https://github.com/puppetlabs/puppetlabs-stdlib/blob/master/REFERENCE.md#stdlibipaddressv6cidr as well.

@ekohl See #372 (comment) regarding your answer about "full pulp instance deployment".

ekohl · 2019-10-21T11:27:16Z

manifests/squid.pp

@@ -47,7 +71,7 @@
    order          => '60',
    config_entries => {
      'cache_peer'          => "${streamer_host} parent ${streamer_port} 0 no-digest no-query originserver name=PulpStreamer",
-      'cache_peer_access'   => 'PulpStreamer allow all',
+      'cache_peer_access'   => 'PulpStreamer allow localhost',


This looks like a general thing that can be applied to all installations, correct? Only Apache/Pulp connect to squid.

yes. I think so.

sbernhard · 2020-01-27T12:53:59Z

can you please have a look at it again @ekohl ?

ehelms · 2020-04-28T23:27:35Z

@sbernhard We have had a hard time wrapping our brains around this change and how it plays into the architecture. With Pulp 3 integration appearing in Katello releases, and these components no longer being present. Is this a change you still want to persue?

sbernhard · 2021-05-11T12:05:37Z

@ehelms The passthrough mode is feature often used by our customers. They like it a lot because they use a foreman proxy to make sure provisioning works in a specific network but don't waste storage as they just forward the requests to the rpm/deb packages.

For pulp3, we want to have similar / the same approach. Therefore, we will work on theforeman/puppet-foreman_proxy_content#205, too.

ekohl · 2021-05-11T12:13:03Z

This module is just for Pulp 2. No new features will be accepted in this module. It's only here for Katello 3.18 until that's EOL. https://github.com/theforeman/puppet-pulpcore is used for Pulp 3 and is completely different. For example, squid is not present.

theforeman-bot added Needs testing Not yet reviewed labels Jun 5, 2019

sbernhard mentioned this pull request Jun 5, 2019

Added passthrough pulp mode theforeman/puppet-foreman_proxy_content#205

Closed

sbernhard force-pushed the passthrough_pulp branch from efe5fbe to 1e1ee92 Compare June 5, 2019 12:05

Added passthrough pulp mode

6cdfb71

sbernhard force-pushed the passthrough_pulp branch from 1e1ee92 to 6cdfb71 Compare June 6, 2019 08:03

ekohl reviewed Oct 21, 2019

View reviewed changes

zjhuntin added the Waiting on contributor label Apr 16, 2020

ehelms mentioned this pull request Jul 7, 2020

Fixes #30316: Move bootstrap RPM generation from puppet-certs theforeman/puppet-foreman_proxy_content#272

Merged

ekohl closed this May 11, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added passthrough pulp mode #372

Added passthrough pulp mode #372

sbernhard commented Jun 5, 2019

ehelms commented Jun 5, 2019

sbernhard commented Jun 5, 2019

ehelms commented Jun 6, 2019

sbernhard commented Jun 6, 2019 via email

sbernhard commented Jul 16, 2019

sbernhard commented Sep 24, 2019

ekohl left a comment

ekohl Oct 21, 2019

sbernhard Oct 21, 2019

ekohl Oct 21, 2019

sbernhard Oct 21, 2019

sbernhard commented Jan 27, 2020

ehelms commented Apr 28, 2020

sbernhard commented May 11, 2021 •

edited

Loading

ekohl commented May 11, 2021

Added passthrough pulp mode #372

Added passthrough pulp mode #372

Conversation

sbernhard commented Jun 5, 2019

ehelms commented Jun 5, 2019

sbernhard commented Jun 5, 2019

ehelms commented Jun 6, 2019

sbernhard commented Jun 6, 2019 via email

sbernhard commented Jul 16, 2019

sbernhard commented Sep 24, 2019

ekohl left a comment

Choose a reason for hiding this comment

ekohl Oct 21, 2019

Choose a reason for hiding this comment

sbernhard Oct 21, 2019

Choose a reason for hiding this comment

ekohl Oct 21, 2019

Choose a reason for hiding this comment

sbernhard Oct 21, 2019

Choose a reason for hiding this comment

sbernhard commented Jan 27, 2020

ehelms commented Apr 28, 2020

sbernhard commented May 11, 2021 • edited Loading

ekohl commented May 11, 2021

sbernhard commented May 11, 2021 •

edited

Loading