Helm Chart for Kubernetes Deployment

[briantopping]

I'm interested in putting together a Helm chart for deploying FreeIPA in Kubernetes. Here's a few reasons why I think this would be a good idea for the community:

• Basic deployments on k8s can be very difficult to plan and execute because of the variety of target configurations (AWS, Azure, GCE, bare metal, etc). As unified repository, the differences and similarities of these deployments will become more apparent for all developers through working code.
• As IPA is critical infrastructure that must be secure, a known good deployment can reduce risk and document best practice.
• A reference deployment can provide the basis that others can be more successful with their own deployment, whether they are deploying directly from Helm or just using the Helm chart as a known working reference.
• For IPA developers, this effort would provide a means of quickly deploying clusters in Kubernetes, hopefully removing barriers for the core team members that wish to know more about this use case.

#154 (comment) is an example of the kind of knowledge that could be embodied in a Helm chart. I have been considering such a pattern for my own production deployment, but I have been concerned that it opens other challenges that I am not prepared for. I think this project would provide an open forum for debating such patterns, making sure that the investment by core team members to answer specialized questions rolled back into deployable artifacts.

Of course, this project would "stand on the shoulders of giants". None of it would be possible without the great work that's already went into containerizing FreeIPA, so thanks again for that.

Thoughts? I would like to see the effort eventually under the umbrella of this GitHub organization, so makes sense to open up discussion early.

[adelton]

@briantopping, certainly, whatever we can do to show people reasonable, working, and secure deployments of containerized FreeIPA is helpful. The initial focus of this project and repo was on barebone docker-based setups, then we've added support for Atomic host, and we've shown freeipa-server-openshift*.json for deployments on OpenShift. I know that people have struggled with docker-compose but hopefully we made things easier recently (#254).

In the last couple of months, I've been adding and hardening the tests for basic docker-based setups in Travis CI, so anything for which we could have automated tests / CI would be a plus.

Is there anything that makes sense to add to this repository, or would separate repo be easier?

[briantopping]

That's exciting, thanks @adelton. The primary Helm repo is at https://github.com/helm/charts, and that one comes pre-configured for the client to install from. Other repositories are not unheard of, adding the repository is an additional step that must be taken before a chart can be used and can be cumbersome if it is only for one product. I'm open for anything tho.

That sounds like great progress on the system, congrats! I've learned from your Docker code and also from the efforts to deploy it in Kubernetes. It certainly could not have happened without your work. I have seen that there were efforts for OpenShift, but admittedly the JSON format has caused me to avoid looking inside. It's something I need to do as there is no sense having needlessly divergent deployment strategies.

I've most recently been looking at https://libreswan.org/wiki/IKEv1_XAUTH_with_FreeOTP_and_FreeIPA and it's struck me that it might be good to isolate the IPA client build and the FreeIPA deployment that uses it. There's a lot of momentum in Kubernetes service authentication (Istio,
https://spiffe.io/ in the general case), but PAM and LDAP are going to be prominent for a long time and a base container that can more easily support traditional service authentication would promote adoption.

Knowing that this is interesting, I'll start putting something together and we'll revisit as it's working!

[adelton]

For the IPA client container inspiration, look at the client branch in this repo. Granted, I haven't worked with that for ages, so I'm not sure whether it is still working.
For a more complete docker-compose based setup of FreeIPA container and IPA client containers used for authentication of web application, see https://github.com/adelton/webauthinfra / https://pagure.io/webauthinfra.

[briantopping]

@adelton I was in airports the last few days so just getting a chance to look over your generous notes. My first go at the container before writing above was to take the CentOS build for the server and strip out the server parts. It looks like they use different strategies to get the container booted in a manner that dbus is usable. I am not clear if dbus is even needed on the client though. This is expected as you have been evolving this over years and there's not a big community yet.

What I guess I am wondering: is some "grand unification" possible? By that I mean creating a core container that the server build is based on and is known to work for clients. Admittedly, this is a gratuitous optimization, things work as they are today. On the other hand, if there's a single core container that "just works" for a baseline deployment of server or client, such a developer knows that if what they are doing doesn't work, it's probably their fault (which is actually a great confidence builder).

I don't know the ins and outs of the container and client. In non-container environments, everything just works. I'm kind of pushing the use cases in the effort of getting to that same level of reliability. Doing the bootstrap in Go is something that comes to mind so it can be more (shall we say) "deterministic".

Indeed this is all just speculation at this point and looking for ways to strengthen the foundation in the process of solving existing problems. I'm learning here, but also looking to contribute with patterns that are scalable to other projects over time.

[adelton]

From my experience, the server and client are completely separate things and are best to be treated as separate.

For server, you don't assume any further modifications of that image. It's supposed to be an application which makes some services available on network ports, and you don't care about the internals and it just works. Think of PostgreSQL container -- it's a single-purpose container. The server container is systemd-based, and it is not going to change any time soon. So the behaviour of the container is driven by .services that we define because we need to have systemd (and dbus and such) up and running. The server takes multiple minutes to start the first time when it runs ipa-server-install (or ipa-replica-install).

For the client, on the other hand, you assume that it will be a building block of some additional application. That application will likely not be running under systemd (in the container), so that already gives you completely different modes of operation. You likely expect fast startup time for the client container.

[briantopping]

That's all been very helpful to understand the scripts that are there and consolidate them a bit for the needs I have. I'm in more of a Kubernetes environment and I've created a dependency on the use of StatefulSet.spec.template.spec.containers[].lifecycle.postStart. This seems to reduce the code quite a bit as the postStart will only be called one time, and otherwise the container can always use ENTRYPOINT /usr/sbin/init. It's definitely a twisty rabbit hole of choices that are available. You can see (quite literally) the gist of it at https://gist.github.com/briantopping/617c249004e0389d685fd1dc987c50e1. After I get the IPA join user into a secret, I will publish it as a repo.

One thing I have noticed is ipa-client-install tries to run systemctl restart rhel-domainname.service, which ends up failing due to no dbus being available. I maybe I need to unlink it from a wants directory somewhere to break a dependency.

I'm wondering if you could explain https://github.com/adelton/webauthinfra/blob/0f8604b4d71007f51888571a137521c5938c162c/src/init-data#L24, it seems that it is intended to run when the container is already set up and just restarting. If I read it correctly, it's copying files from /data to the root filesystem, which doesn't make much sense to me. At least in a StatefulSet, things seem to work without it.

EDIT: I see, it is copying the files back to the root volume. I got used to the pattern of the current server build where the data volume was symlinked from root. I'll be good to reconcile everything, leaving the rest of the comment as a status update, no response necessary ☺️

[adelton]

It's definitely a twisty rabbit hole of choices

Is this about the IPA server container or about IPA client?

I see, it is copying the files back to the root volume. I got used to the pattern of the current server build where the data volume was symlinked from root.

The main reason was that https://github.com/adelton/webauthinfra is a developer and development / testing environment. So after the initial setup is done (and /data and thus the persistent volume populated), you likely want to be able to restart with the environment that got set up, over and over again, without accumulating the changes in the volume.

[briantopping]

Thanks Jan, I did reason that out after staring at it some. I see now how the Dockerfile is setting up the /data-template, which has to be done that way so the symlinks can be baked into the image at build time. Then the template is copied to /data one single time and when the container is booted, the links are magically pointing to the right place. This fits perfectly with the pattern above using StatefulSet.spec.template.spec.containers[].lifecycle.postStart. It seems like four files need to be captured from /etc after the client install for minimal functionality, krb5.keytab, sssd/sssd.conf, ipa/defaults.conf and ipa/ca.crt.

Very excited for this! Thanks again for your help!

[briantopping]

One thing I just considered: It might be very interesting if the IPA installers were able to check for an existing symlink before installing any given file, then put the file in the place the symlink pointed to in case one existed. In this manner, the volume-data-list process could be run in the Docker build and when the installer was run, no post-processing would be necessary.

Combined with a StatefulSet in Kubernetes, a rather natural setup process would follow: 1) deploy SS and PVC; 2) launch interactive terminal; 3) install manually. When the installers run, they put the files in the correct spot because symlinks exist from the Docker image build. When the instance is rebooted, everything just works.

[adelton]

Are you talking about IPA server or client?

[briantopping]

Seems like that technique would work for either client or server, both for initial server or replica install.

[rojopolis]

@briantopping I see that this issue was closed in November. Was any progress toward a Helm Chart made? I also need to deploy FreeIPA into a Kubernetes environment and would love to manage it with Helm. I'm happy to contribute back to the OSS effort if you have a starting point.

[clayrisser]

Any update?

[briantopping]

@rojopolis @codejamninja Reopening this now that I have some mileage with the scripts. I'll start posting what I have in a separate repo that we can work on together until it's stable. The good news is I've been using it for a few months and don't feel like I'm going to be doing folks a disservice by sharing them. The bad news is I haven't started the templates at all yet, they are just manifests.

Here's where I know they need work:

• Getting the manifests into Helm templates (obviously).
• Logging needs to be fixed. It's currently just filling up the PV with noise and they aren't propagating logs like good pods should. We may need https://bugzilla.redhat.com/show_bug.cgi?id=1390191.
• The images in Quay aren't pushing with tags that allow users to select a specific version and hold that version. FreeIPA container repository tags are dangerous for orchestrated production use #246 addresses most of this.
• Need to work out some of the PVC stuff. I did those by hand.

Something that would be a "nice to have" is to build it with Operator Pattern out of the box. If anyone else feels strongly about this and has done it before, I would love to help, but I'm not sure I'm ready to lead on that front.

I'll add a followup as soon as I have a repo together! I think we can make pretty short work of this problem and bring the sources back to this repo.

[clayrisser]

@briantopping can you please share your scripts. I have a lot of helm experience and would find them extremely useful. I would be willing to contribute.

[briantopping]

Yes I’d like to take a cut at it first. I was impressed by the csi-ceph layout and will probably emulate that unless you have strong opinions about it.

[adelton]

Of the issues listed in #255 (comment), are any of them generic FreeIPA containerization issues that should be handled in this repository. For example, you mention "Logging needs to be fixed. It's currently just filling up the PV with noise and they aren't propagating logs like good pods should." but I'm not sure if you talk about your Kubernetes/Helm setup or what freeipa-container does in general.

[briantopping]

@adelton It may have to do with the specifics around how kubernetes expects logs, I think it's just out to console. I'm guessing the container may need to be parameterized if the logs to console are undesirable outside of a k8s environment. That's not difficult. I was mostly making mental notes, apologies if that was out of context.

[clayrisser]

@briantopping do you have an eta on when you'll post the scripts? I don't really care if they're nice, I just kinda want to dig through what you have.

I'm ok with the csi-ceph layout, although I think it would be nice to add support for rancher charts.

I have a nice generator for helm charts I use a lot.

https://www.npmjs.com/package/generator-helm-chart

You can also take a look at my personal helm charts if you need inspiration.

https://github.com/codejamninja/charts

[briantopping]

I was thinking this weekend? Def not trying to be clingy with them. Thrilled you have interest and want to learn by seeing your impressions instead of not really engaging with it and only appreciating where they end up. I hope that makes sense.

[clayrisser]

I'm sorry @briantopping I didn't get you. Looking forward to this weekend!

[briantopping]

@codejamninja I am moving with this as promised and its at the top of my stack until it’s done or client work starts, which I don’t have scheduled at the moment. As I review my work from some months ago in the context of what I know now, a straight Helm template isn’t going to cut it. The good news is I have a design that I am prototyping, the bad news is what I’ve done in the past isn’t really usable without documentation that will be worthless in short order. So I don’t have anything for you besides this right now. Will post here when I do!

[briantopping]

Update from here: Spent most of this week in discovery mode on census-instrumentation/opencensus-proto#200. At this point, I'm going to get back into the actual controller as a Go/dep bulld at this point. Hopefully there's some resolution there, I'd like to release this using Bazel.

[rshutt]

Been a couple of months here since last update and since Im a glutton for punishment and I just deployed 1.15 onto a CentOS pacemaker, corosync, pcsd cluster with GFS2/DRBD underlying for Local storageclass PVCs... hahaha. Definitely interested and perhaps even could help a little!

[briantopping]

Yah sorry about that. Summer came and I ended up with endless projects on the house.

I have started and pushed the operator, it looks good so far, but it's not... quite... there... Hmm.

[rshutt]

Yeah you'll have that :). Nothing public to read short of this thread? I kinda have an itch to at least get something deployed. I don't know that I'd want to have a single container... Maybe a nice statefulset indexed like they do those in the prom operator with types for ldaps and dnses and certmongers and so on and flags for all of the relevant tunables :). Lots of work man.

[briantopping]

Please do feel free to help with the repo! https://github.com/briantopping/freeipa-operator. It has some whizzy Go in there, but the point is to make it so the declarative aspects of creating a stable FreeIPA deployment are separated from the programatic ones of understanding (for instance) what's already been deployed. It's a pretty tricky situation, I certainly do not want all the fancy operator poo if it ends up wiping out my corporate identity system!

Eventually, the operator should provide for:

• Scaling instances, starting with zero instances and protecting the last instance
• Managing intentional version migration rather than random upgrades because the cluster was rebooted
• Backups

@ShuttR: I agree with all the other stuff you're proposing as well!! My manual deployment of FreeIPA is as a SS, I just need to get the YAMLs working in the operator context..

[rshutt]

I'll TAL man! Maybe this weekend!

…

On Wed, Jul 24, 2019 at 5:34 PM Brian Topping ***@***.***> wrote: Please do feel free to help with the repo! https://github.com/briantopping/freeipa-operator. It has some whizzy Go in there, but the point is to make it so the declarative aspects of creating a stable FreeIPA deployment are separated from the programatic ones of understanding (for instance) what's already been deployed. It's a pretty tricky situation, I certainly do not want all the fancy operator poo if it ends up wiping out my corporate identity system! Eventually, the operator should provide for: - Scaling instances, starting with zero instances and protecting the last instance - Managing intentional version migration rather than random upgrades because the cluster was rebooted - Backups — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#255?email_source=notifications&email_token=AMIHSGOULOSQTIEYXN5CG5DQBDDGHA5CNFSM4GFBL2T2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2XVXPQ#issuecomment-514808766>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMIHSGJBBNOKRVKFPO47K3DQBDDGHANCNFSM4GFBL2TQ> .

[rshutt]

@abbra Heya. Read through this today: https://www.freeipa.org/page/Troubleshooting/PrivilegeSeparation

Should the httpd even be trying to open /etc/krb5.keytab? Somethings horked me thinks :( . Probably hostname/dns conjanglement. But this is what it is unless I hack some records into CoreDNS. It may simply be easier to use Host Networking... Then most of this goes away since DNS is what it is for the host. But it will eat up ALL of those ports as INADDR_ANY tho if Im not mistaken :( And that's a lot of ports!

EDIT - But oh wow, if I use host networking, we're going to be in a different situation. Then I believe I couldn't enroll the actual Kubelet server since it would be the same hostname and ergo identity as the master itself. Perhaps I could play games with keys and mounting them in both the Kubelet and the container, but that sounds iffy too...

[rshutt]

@ShuttR @briantopping any chance you could share the code?

@codejamninja Howdy. I have shared pretty much all I have so far above :) You'll have to ask @briantopping for his repo wherein he has some skeletons of a controller using kubebuilder and some templates for the primitives. Hell, I'm just trying to find an architecture that works inside of Kubernetes and doesn't require so much fudging that it's basically a rube goldberg machine to begin with.

Part of me is really starting to wonder how things are going with KubeVirt and if this isn't really the best possible test case in my lab for that tech :)

[clayrisser]

@ShuttR I'm sure many have tried and failed to get freeipa working on kubernetes. Honestly, I'm quite sure it's doable. It's just a matter of pushing through. I have no idea if this was encouraging or discouraging. Hopefully, it's encouraging.

[briantopping]

@codejamninja I have been using FreeIPA as a StatefulSet for about a year now. It's just creating the operator for it...

[rshutt]

I mean all that's really left for me is the PTR record... and right now on the local IPA DNS, it's nxdomain.

I just straced and captures the entire thing from soup to nuts (fyi, kubectl plugins are great. Kubectl sniff actually spins up tcpdump in a remote container and it's a plugin availavble with krew)

-rw-rw-r--.  1 console console 3315682472 Aug  6 02:34 install.strace.txt
-rw-r--r--  1 [redacted]  [redacted]/[redacted]  11507145 Aug  5 22:31 install.pcap

Well there's some stuff to sift :P I have work work to do, so not tonight for me :P

[rshutt]

@briantopping Yeah I have no idea how Im on the failbus here...

The latest, I got it to fwd/rev everything by dynamically adding an --ip-address argument in my shim script. So at this point the hostname is 100% copacetic (ish), but I still seem to have a broken gss auth module in httpd :( . 100.64 is me using CGnat for the Pod network. 192.168.101 is my internal IPv4 :(

[root@ipa01 /]# for ipaddr in $(dig @localhost ipa01.randallman.net A +short); do echo -n "${ipaddr}: "; dig @localhost -x $ipaddr +short; done
192.168.101.40: ipa01.randallman.net.
100.64.0.85: ipa01.randallman.net.

[briantopping]

Slooooowwww down... This is not working code yet! Y'all asked for a link to WIP, I provided it. It still needs work. If you need something that works out of the box, it's not there yet, at all...

[rshutt]

Im good... Just truckin along :P I had it working as a statefulset but there were still fwd/rev issues. I know i can get it there. Strange that there are no forward/reverse issues now and it's more not working :) . Im definitely not seeing httpd interact with gssproxy when it starts or is hit. I think that's what is supposed to happen... I do see the kt in the /var/.../gssproxy dir so grr...

After that, Ill switch over to making it stampable and getting into the controller.

[rshutt]

[Tue Aug 06 07:30:30.139166 2019] [:info] [pid 4162] Connection to child 1 established (server ipa01.randallman.net, client 100.64.0.94)
[Tue Aug 06 07:30:30.139339 2019] [:debug] [pid 4162] nss_engine_init.c(1934): SNI: Found nickname Server-Cert for vhost: ipa01.randallman.net
[Tue Aug 06 07:30:30.142021 2019] [:debug] [pid 4162] nss_engine_init.c(1956): SNI: Successfully paired vhost ipa01.randallman.net with nickname: Server-Cert
[Tue Aug 06 07:30:30.149658 2019] [:debug] [pid 4162] nss_engine_kernel.c(96): SNI request for ipa01.randallman.net
[Tue Aug 06 07:30:30.149671 2019] [:info] [pid 4162] Initial (No.1) HTTPS request received for child 1 (server ipa01.randallman.net:443)
[Tue Aug 06 07:30:30.149849 2019] [authz_core:debug] [pid 4162] mod_authz_core.c(809): [client 100.64.0.94:55180] AH01626: authorization result of Require valid-user : denied (no authenticated user yet), referer: https://ipa01.randallman.net/ipa/xml
[Tue Aug 06 07:30:30.149857 2019] [authz_core:debug] [pid 4162] mod_authz_core.c(809): [client 100.64.0.94:55180] AH01626: authorization result of <RequireAny>: denied (no authenticated user yet), referer: https://ipa01.randallman.net/ipa/xml
[Tue Aug 06 07:30:30.149876 2019] [auth_gssapi:debug] [pid 4162] mod_auth_gssapi.c(857): [client 100.64.0.94:55180] URI: /ipa/json, no main, no prev, referer: https://ipa01.randallman.net/ipa/xml
[Tue Aug 06 07:30:30.150580 2019] [auth_gssapi:error] [pid 4162] [client 100.64.0.94:55180] GSS ERROR gss_acquire_cred[_from]() failed to get server creds: [Unspecified GSS failure.  Minor code may provide more information ( SPNEGO cannot find mechanisms to negotiate)], referer: https://ipa01.randallman.net/ipa/xml
[Tue Aug 06 07:30:30.150764 2019] [core:debug] [pid 4162] util_cookies.c(129): [client 100.64.0.94:55180] AH00009: ap_cookie: user '(null)' removed cookie: 'ipa_session=;Max-Age=0;path=/ipa;httponly;secure;', referer: https://ipa01.randallman.net/ipa/xml
[Tue Aug 06 07:30:30.150771 2019] [headers:debug] [pid 4162] mod_headers.c(823): AH01502: headers: ap_headers_output_filter()
[Tue Aug 06 07:30:30.151146 2019] [:info] [pid 4162] Connection to child 1 closed (server ipa01.randallman.net:443, client 100.64.0.94)

Getting into the code, it's not hitting gssproxy...

https://github.com/modauthgssapi/mod_auth_gssapi/blob/4a22af70389e071e451e8bb21cf67e3b72212291/src/mod_auth_gssapi.c#L227

Now we're cooking with gas I think!

But it's way too late :)

[abbra]

I think we get the 'SPNEGO cannot find mechanisms to negotiate' from gss_acquire_cred_from() processing in SPNEGO because it is second call to do there -- trying to see if we have any SPNEGO-compatible creds in a cred_store. Basically, it means cred_store is most likely not accessible.

Now, in FreeIPA 4.5+ that should be correct -- a process where mod_auth_gssapi is running has no access to the keytab with HTTP/... service principal's key. The access is done by gssproxy, running in a separate process and triggered by an interposer plugin for GSSAPI in MIT Kerberos.

Few assumptions here:

• all required packages were installed (should be covered by whatever is used to create container image)
• gssproxy systemd service is running
• gssproxy socket is accessible to applications using GSSAPI (httpd process with mod_auth_gssapi in this case)
• /etc/gssproxy/10-ipa.conf configuration file exists
• gssproxy has read rights to /var/lib/ipa/gssproxy/http.keytab

You are saying there is no keytab in /var/.../gsproxy directory, so may be it is more generic issue with the container image?

[rshutt]

Firstly, I really appreciate your assist on this. You obviously know every last bit of this code intimately and you and the team ought to be quote proud. This is great stuff. It's too bad it got perfected when the cloud basically declared victory and nobody has accounts on anything anymore, but that's life :) . Thank you so much for your time with this and once I can get over whatever pain this is, making it work for others might not be so difficult :P

So the checklist...

1. Yeah the manifest is what it was and this normally works for people ;)
2. The gssproxy systemd service is definitely up, though even with increased debug the needle doesn't move on an httpd restart + kerberized ipa ping with admin@REALM [1]
3. The gssproxy socket exists, is being listened to by gssproxy and is 666 [2]
4. /etc/gssproxy/10-ipa.conf exists [3] and indicates that the resultant keytab should be owned by apache?
5. and DOES NOT HAVE read rights to /var/lib/ipa/gssproxy/http.keytab (though apache does?) [4]

[1]:

[root@ipa01 conf.d]# journalctl -u gssproxy
-- Logs begin at Tue 2019-08-06 05:27:12 UTC, end at Tue 2019-08-06 07:52:02 UTC. --
Aug 06 05:30:09 ipa01.randallman.net systemd[1]: Starting GSSAPI Proxy Daemon...
Aug 06 05:30:09 ipa01.randallman.net systemd[1]: Started GSSAPI Proxy Daemon.
Aug 06 07:40:39 ipa01.randallman.net systemd[1]: Reloading GSSAPI Proxy Daemon.
Aug 06 07:40:39 ipa01.randallman.net systemd[1]: Reloaded GSSAPI Proxy Daemon.
Aug 06 07:40:39 ipa01.randallman.net gssproxy[3378]: [2019/08/06 07:40:39]: Debug Enabled (level: 2)
Aug 06 07:40:39 ipa01.randallman.net gssproxy[3378]: [2019/08/06 07:40:39]: New config loaded successfully.
[root@ipa01 conf.d]# systemctl restart httpd
[root@ipa01 conf.d]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: admin@RANDALLMAN.NET

Valid starting       Expires              Service principal
08/06/2019 07:30:23  08/07/2019 07:30:23  krbtgt/RANDALLMAN.NET@RANDALLMAN.NET
08/06/2019 07:30:30  08/07/2019 07:30:23  HTTP/ipa01.randallman.net@RANDALLMAN.NET
[root@ipa01 conf.d]# ipa pingf
ipa: DEBUG: failed to find session_cookie in persistent storage for principal 'admin@RANDALLMAN.NET'
ipa: INFO: trying https://ipa01.randallman.net/ipa/json
ipa: DEBUG: Created connection context.rpcclient_139967430606224
ipa: INFO: [try 1]: Forwarding 'schema' to json server 'https://ipa01.randallman.net/ipa/json'
ipa: DEBUG: New HTTP connection (ipa01.randallman.net)
ipa: DEBUG: HTTP connection destroyed (ipa01.randallman.net)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 719, in single_request
    if not self._auth_complete(response):
  File "/usr/lib/python2.7/site-packages/ipalib/rpc.py", line 679, in _auth_complete
    message=u"No valid Negotiate header in server response")
KerberosError: No valid Negotiate header in server response
ipa: DEBUG: Destroyed connection context.rpcclient_139967430606224
ipa: ERROR: No valid Negotiate header in server response
[root@ipa01 conf.d]# journalctl -u gssproxy
-- Logs begin at Tue 2019-08-06 05:27:12 UTC, end at Tue 2019-08-06 08:51:57 UTC. --
Aug 06 05:30:09 ipa01.randallman.net systemd[1]: Starting GSSAPI Proxy Daemon...
Aug 06 05:30:09 ipa01.randallman.net systemd[1]: Started GSSAPI Proxy Daemon.
Aug 06 07:40:39 ipa01.randallman.net systemd[1]: Reloading GSSAPI Proxy Daemon.
Aug 06 07:40:39 ipa01.randallman.net systemd[1]: Reloaded GSSAPI Proxy Daemon.
Aug 06 07:40:39 ipa01.randallman.net gssproxy[3378]: [2019/08/06 07:40:39]: Debug Enabled (level: 2)
Aug 06 07:40:39 ipa01.randallman.net gssproxy[3378]: [2019/08/06 07:40:39]: New config loaded successfully.
[root@ipa01 conf.d]# 

[2]:

# ps -ef | egrep gssproxy
root      3379     1  0 05:30 ?        00:00:00 /usr/sbin/gssproxy -D
# lsof -p 3379
COMMAND   PID USER   FD      TYPE             DEVICE SIZE/OFF      NODE NAME
gssproxy 3379 root  cwd       DIR               0,58       74  68215581 /
gssproxy 3379 root  rtd       DIR               0,58       74  68215581 /
gssproxy 3379 root  txt       REG               0,58   133864 103300044 /usr/sbin/gssproxy
gssproxy 3379 root  mem       REG              253,2          103300044 /usr/sbin/gssproxy (path dev=0,58)
<SNIP>
gssproxy 3379 root    8u     unix 0xffff98f866204c00      0t0  34861964 /run/gssproxy.sock
gssproxy 3379 root    9w     FIFO                0,9      0t0  34861973 pipe
gssproxy 3379 root   10u     unix 0xffff98f866206c00      0t0  34861974 /run/gssproxy.sock
[root@ipa01 conf.d]# ls -ld /run/gssproxy.sock
srw-rw-rw-. 1 root root 0 Aug  6 05:30 /run/gssproxy.sock

[3]:

[root@ipa01 conf.d]# cat /etc/gssproxy/10-ipa.conf
#Installed and maintained by ipa update tools, please do not modify
[service/ipa-httpd]
  mechs = krb5
  cred_store = keytab:/var/lib/ipa/gssproxy/http.keytab
  cred_store = client_keytab:/var/lib/ipa/gssproxy/http.keytab
  allow_protocol_transition = true
  allow_client_ccache_sync = true
  cred_usage = both
  euid = apache

[service/ipa-api]
  mechs = krb5
  cred_store = keytab:/var/lib/ipa/gssproxy/http.keytab
  cred_store = client_keytab:/var/lib/ipa/gssproxy/http.keytab
  allow_constrained_delegation = true
  allow_client_ccache_sync = true
  cred_usage = initiate
  euid = ipaapi

[4]

[root@ipa01 conf.d]# ls -latr /var/lib/ipa/gssproxy/http.keytab
-rw-------. 1 apache apache 184 Aug  6 05:30 /var/lib/ipa/gssproxy/http.keytab

[root@ipa01 conf.d]# kinit -kt /var/lib/ipa/gssproxy/http.keytab HTTP/ipa01.randallman.net@RANDALLMAN.NET
[root@ipa01 conf.d]# echo $?
0
[root@ipa01 conf.d]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: HTTP/ipa01.randallman.net@RANDALLMAN.NET

Valid starting       Expires              Service principal
08/06/2019 09:01:22  08/07/2019 09:01:22  krbtgt/RANDALLMAN.NET@RANDALLMAN.NET

[abbra]

Ok, thanks for the confirmation. Gssproxy wouldn't be triggered by your kinit/ipa ping runs from root. It doesn't support interposing root processes.

It is also not triggered for any other access unless there is a variable GSS_USE_PROXY set to some value (yes) in the environment of that process. Could you please show output of

systemctl cat httpd

It should have Environment=GSS_USE_PROXY=yes

[rshutt]

It does, I checked httpd's environment: ``` [Service] Environment=KRB5CCNAME=/tmp/krb5cc-httpd Environment=GSS_USE_PROXY=yes Environment=KDCPROXY_CONFIG=/etc/ipa/kdcproxy/kdcproxy.conf ExecStartPre=/usr/libexec/ipa/ipa-httpd-kdcproxy ``` And double checked here: ``` [root@ipa01 /]# pgrep httpd 4536 <snip> [root@ipa01 /]# cat -v /proc/4536/environ LANG=C^@path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin^@NOTIFY_SOCKET=/run/systemd/notify^@KRB5CCNAME=/tmp/krb5cc-httpd^@GSS_USE_PROXY=yes^@KDCPROXY_CONFIG=/etc/ipa/kdcproxy/kdcproxy.conf^@ ```

…

On Tue, Aug 6, 2019 at 5:26 AM Alexander Bokovoy ***@***.***> wrote: Ok, thanks for the confirmation. Gssproxy wouldn't be triggered by your kinit/ipa ping runs from root. It doesn't support interposing root processes. It is also not triggered for any other access unless there is a variable GSS_USE_PROXY set to some value (yes) in the environment of that process. Could you please show output of systemctl cat httpd It should have Environment=GSS_USE_PROXY=yes — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#255?email_source=notifications&email_token=AMIHSGLLZUEUORC265JZMZDQDE7V3A5CNFSM4GFBL2T2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3UQVSI#issuecomment-518589129>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMIHSGJ3HYYQIVRAPDFYVS3QDE7V3ANCNFSM4GFBL2TQ> .

[fire]

I am having a hard time finding a repository with podman image building scripts and kubernetes yamls. Are I missing something?

Looking to see how I can contribute to Freeipa on a helm chart.

[adelton]

Building with podman is the same as building with docker -- podman build Dockerfile. Or just use the image from https://hub.docker.com/r/freeipa/freeipa-server/tags.

A pod-defining YAML file to get you started on Kubernetes could be for example

kind: Pod
metadata:
  name: freeipa-server
  labels:
    app: freeipa-server
  annotations:
    seccomp.security.alpha.kubernetes.io/pod: docker/default
spec:
  restartPolicy: Never
  containers:
  - name: freeipa-server
    image: docker.io/freeipa/freeipa-server:latest
    volumeMounts:
    - name: freeipa-server-data
      mountPath: /data
    ports:
    - containerPort: 80
      protocol: TCP
    - containerPort: 443
      protocol: TCP
    - containerPort: 53
      protocol: UDP
    - containerPort: 88
      protocol: TCP
    - containerPort: 88
      protocol: UDP
    env:
    - name: IPA_SERVER_HOSTNAME
      value: ipa.example.test
    - name: PASSWORD
      valueFrom:
        secretKeyRef:
          name: freeipa-server-password
          key: admin.password
    - name: IPA_SERVER_INSTALL_OPTS
      value: "-U -r EXAMPLE.TEST --setup-dns --no-forwarders --no-ntp"
    readinessProbe:
      exec:
        command: [ "/usr/bin/systemctl", "status", "ipa" ]
      initialDelaySeconds: 60
      timeoutSeconds: 10
      periodSeconds: 10
      successThreshold: 1
      failureThreshold: 3
  volumes:
  - name: freeipa-server-data
    hostPath:
      path: /opt/freeipa-server

[briantopping]

The whole point of diverging into an operator was the benefits that would come to managing FreeIPA clusters. It would be great if FreeIPA was simple and orthogonal enough that HPA was an option and things like that, but dependencies on legacies like dbus make that a difficult ask. They were good directions at the time that led to difficult containerization options today. I have no excuse for the delay on this except for aforementioned summer projects, the last one being installing a six person hot tub. Northern hemisphere summertime has come to a close. I’m happy to work intensively with others that have competency in Golang to get this finished.

[fire]

If I have your yaml configured into a Kubernetes stateful set, has there been any work in automatically joining them? Any designs?

I've seen the use of a kubernetes job to do this in cockroachdb (they have certificates).

[briantopping]

@fire I pushed the branch at https://github.com/briantopping/freeipa-operator/tree/WIP. This code is where I left it last time I worked on it, which was not compiling. Apologies for the state.

StatefulSets are just types managed by an internal operator, so to speak. So the goal here was to do the same thing, to get our operator code managing the lifecycle of FreeIPA pods. In turn, we'd be able to manage the storage and peering relationships more delicately, hopefully ensuring the process is flawless to the user and enabling higher level capabilities like backup and version upgrades more closely tied to the success of actual instance health.

[adelton]

dependencies on legacies like dbus make that a difficult ask

What is the problem with dbus specifically? I can imagine that the general fact that the FreeIPA container is multi-daemon systemd based is a problem but as for dbus, I feel we have ironed out the long-standing issue with certmonger startup in #283 and that was not really an issue caused by dbus, just manifested there.

[adelton]

If I have your yaml configured into a Kubernetes stateful set, has there been any work in automatically joining them? Any designs?

I'd like to see working and stable FreeIPA replica creation first, before attempting to do multiple ones with StatefulSets. I've added support for that via 506f523 but I've only tested that on OpenShift and we don't have any sort of CI around that -- we've recently lost even the FreeIPA master in OpenShift/Kubernetes automated testing that we had.

If someone is willing to setup testing environment where we could test FreeIPA servers and replicas in multiple version of OpenShift and Kubernetes with the existing code, it would greatly boost our ability to add and sustain support for StatefulSets.

[briantopping]

Why does Quay not have versions of FreeIPA yet? My k8s cluster just died with expired certificates. When I rebooted, it appears to have pulled a different version with different filesystem expectations or something.

I knew this was going to happen someday and have been noting it for at least a year.

Doesn't matter much what we do here if a well-intended change to the only docker image people can reference blows up user deployments. I simply don't want the latest version if I am not expecting it. Most production environments are the same.

[adelton]

The Quay plan is being discussed in #246.

In general though, if you want to use a specific version in a production, push (and tag) that specific version to your registry. Do not rely on external registries to be available or have the image next time you need it, no matter if it's Docker Hub or Quay.

[briantopping]

Do not rely on external registries to be available or have the image next time you need it, no matter if it's Docker Hub or Quay

There are many arguments against that position and they are relevant to this effort. If this operator is ever to be idempotent and the backing repository is not idempotent, then the operator will have to contain it's own repository since the operator can't depend on one in the environment. No operators do this, or it would be a part of the operator toolkits because it's an expectation.

• Docker itself has 38 pages of images on hub going back four years: https://hub.docker.com/_/docker?tab=tags&page=1&ordering=-last_updated
• Cilium is not even the most popular CNI for k8s and it has images going back three years: https://hub.docker.com/r/cilium/cilium/tags?page=1&ordering=-last_updated
• An extreme example is the Java commons-logging library. The original version from 2005 is still hosted at https://repo1.maven.org/maven2/commons-logging/commons-logging/.

The point is when a cache is not reliable, the dependency for reliability gets pushed out to the clients. This increases complexity by an enormous amount.

Does this operator now need to start including it's own registry? Can you find me other operators that have registries built into them?

[adelton]

Initially this issue was about Helm Chart, not Helm Operator.

I'm not sure why the Operator couldn't depend on the repository (registry) in the environment. In any case, this is not as much about where the registry lives but what's in there and how it's tagged. It should be just as possible to manually push (and tag) the "good" images into Docker Hub, to different namespace.

The experience with left-pad shows that relying on external sources not under one's control is not the best way to run production setups.

Red Hat is providing curated and tagged images registry.access.redhat.com/rhel7/ipa-server based loosely on this upstream effort. If stability is the primary concern (as it should, for production environment), that image might be much safer option. People showing interest in that image might also help Red Hat decide to upgrade it from its current Technology Preview status.

Working on freeipa-container outside of my work duties, I don't have the capacity to manually tag new things. I'm happy when failed Travis CI's cron job reminds me that new packages were pushed to yum repositories that broke the setup ... and even that does not always catch everything early enough and then users experience build issues, as yesterday's case #280 (comment) shows.

I'd expect members of the community to do the curating and tagging if they deem it important. I suggest to use #246 for discussions around that. It has been silent for the past half a year.

[briantopping]

Initially this issue was about Helm Chart, not Helm Operator.

As documented, a Helm chart is unfortunately not capable enough. I completely understand the problem around "spare time after work". :)

I didn't know that there was a production repository. The full link for the browser is https://access.redhat.com/containers/?tab=tags#/registry.access.redhat.com/rhel7/ipa-server. That should be sufficient for the needs of either a Helm chart or an operator to provide a reliable experience for users. I'll close #246 with that info.

[adelton]

Hello @briantopping, what is the story and status here? Does this issue need to stay open, or can we close it, potentially with a link somewhere where the work is being done?